How do I check if robots.txt blocks a page?
Find the group for the crawler's user agent (or the * group if there is none), then apply the longest matching Allow or Disallow rule; a tie goes to Allow. Paste the file here, type the bot and path, and the tool applies those rules for you.
- Paste the robots.txt content, or enter the site and press Fetch if the site allows cross-site reads.
- Type the user agent, for example
GooglebotorGPTBot. - Type a path or full URL to test.
- Read the verdict and the rule that decided it.
- Switch to the Sitemap tab to paste and check your sitemap XML.
Examples
Longest match wins
Input
User-agent: * Disallow: /admin/ Allow: /admin/public/ Path: /admin/public/help
Result
ALLOWED. The longest matching rule is Allow: /admin/public/
Why does Fetch sometimes fail?
This tool asks for the file straight from your browser and never sends your request through a server. Browsers only let one site read another site's files when that site allows it (CORS), and most do not for robots.txt. When that happens, open the file in a new tab, copy it, and paste it in.
Frequently asked questions
Does Disallow remove a page from Google?
No. Disallow stops crawling, but a blocked URL can still be indexed from links. To keep a page out of results, allow crawling and add a noindex tag.
How do I block AI crawlers?
Add a group for each crawler you want to block, for example User-agent: GPTBot followed by Disallow: /. Check each AI company's documentation for its crawler names.
What does a $ at the end of a rule mean?
It anchors the rule to the end of the URL. Disallow: /*.pdf$ blocks URLs ending in .pdf but not /file.pdf?download=1.
How big can a sitemap be?
The sitemaps.org protocol allows up to 50,000 URLs and 50 MB uncompressed per file. Larger sites split into several sitemaps listed in a sitemap index.