How to use this tool

Robots.txt Tester & Crawl Rule Validator

How to use the robots.txt tester

Back to the tool

One URL and one crawler go in; the answer is whether that crawler may fetch that address, and which line of robots.txt decides it.

  1. 01

    Enter the URL to test

    Paste the page itself, not the robots.txt file: the tool reads the origin from it and fetches /robots.txt there. Normalize URL still matters; set it to Domain only and you are asking about the home page, because the path is the half of the address that gets matched.

  2. 02

    Select the crawler to test

    Rules are per crawler, so the same URL can be allowed for one bot and blocked for another. The check stays locked until a crawler is chosen: there is no sensible default, and a wrong default would answer a question you never asked. All other crawlers is the * group.

  3. 03

    Run the test and read the deciding line

    One run is one request: the server only fetches the file, and the matching runs in your browser. The whole file comes back line-numbered, with the winning rule tinted and its number replaced by an icon. Copy and Download in the row above hand you the file itself: the download is a real robots.txt you can drop into a site.

  4. 04

    Read the verdict below the file

    Below the file: which robots.txt was read and what it answered, the path that was matched, the verdict with the winning line repeated next to it, and every Sitemap: line the file declares. The line is repeated on purpose: the file can be long enough that the tinted row is out of sight by the time you reach the verdict.

How to read the crawl rule verdict

The verdict is one word; everything else on the page exists to show you how that word was reached.

Verdict values

Allowed
No rule matches this path, or the rule that wins is an Allow. A host with no robots.txt at all reports the same way: no file means nothing is forbidden.
Disallowed
The winning rule is a Disallow, named right next to the verdict. Blocked from crawling is not removed from the index: a blocked URL can still be listed, which is why noindex belongs in a meta tag, not here.
Undetermined
The file could not be read, or it answered 200 with a page instead of rules. Google treats an unreadable robots.txt as a full block for about twelve hours, so answering Allowed here would be the one wrong answer.
URL path
The part of the address that is actually matched: path plus query string, without the fragment. It is case-sensitive, unlike the crawler name.
Sitemaps declared
Every Sitemap: line in the file, duplicates removed. These lines are global: they belong to the file, not to any crawler group, so they apply no matter which bot you picked.

Rule precedence

The longest pattern wins
Between two rules that both match, the longer pattern decides, counting * and $ as characters. That is why Disallow: /shop loses to Allow: /shop/public on /shop/public and wins on /shop.
A tie goes to Allow
Same length, Allow beats Disallow: the least restrictive reading, and what the reference parser does.
Only your crawler group
Every group naming your crawler is merged, even if the file splits them apart. The * group is used only when no group names it: a file with a Googlebot section does not also apply its * section to Googlebot.
* and $
* stands for any run of characters. $ anchors the pattern to the end of the URL, but only at the very end of the pattern; anywhere else it is an ordinary character.
Disallow with nothing after it
Forbids nothing. It is how a file says "no restrictions for this group", and it is not the same as Disallow: /.

When the file cannot be read

No robots.txt (404)
Nothing is disallowed, so the verdict is Allowed and there is no file to show. A chain of more than five redirects counts as no file too, which is where the reference crawler gives up.
Could not be read
A 5xx, a 429, a timeout, or a refused connection. The verdict is Undetermined, and no file is shown; the body of an error page is not your rules.
200 with HTML
A soft 404: the server answers as if the file existed but sends a page. Crawlers read that as junk, so the verdict is Undetermined; this one is worth fixing, because it is invisible from a browser.
Only the first 500 KiB
The file is cut at that size before matching, which is the limit the reference parser reads. Rules past that point do not exist for any crawler either.
The file exists but sets no rules
An empty file, or one with only comments and Sitemap: lines. Valid, and it forbids nothing.