AI Bot & LLM Crawler Access Checker

One page goes in; every AI crawler we track is measured against it twice, in two independent ways.

1. 01

   Enter the page you want AI crawlers to reach

   One page per run: the comparison here is between bots, not between URLs. Pick the exact page you care about, because robots.txt rules are matched against its path. A rule that blocks /blog/ will not show up if you test the homepage, and a CDN rule keyed on a path behaves the same way.

   URLexample.com/article

   Normalize URLFull URL
2. 02

   Run the check across all 22 bots

   There is no bot picker on purpose. The Content column measures each bot against the other 21 in the same run, so a partial run would take away the yardstick that makes the column mean anything. Every bot is fetched in turn, so a row that never got its turn can be retried on its own.

   Check AI bots

   22/22Resolved URL200example.com/article
3. 03

   Read both verdicts side by side

   robots.txt is what you declared; HTTP status is what your server actually did. Rows where the two agree are routine. The rows worth your attention are the ones where they disagree, and those carry a warning icon: allowed by the file but refused by the server, or blocked by the file yet served in full.

   Botrobots.txtHTTP status

   GPTBotAllowed200

   ClaudeBotAllowed403

   CCBotBlocked200

How to read the AI crawler access report

A row reads left to right: which bot, what robots.txt says about it, and what the server actually did. Every request is ours, sent with that bot’s User-Agent string; the vendor’s own crawler may still behave differently.

Result table columns

Resolved URL

Above the table, not in it. This is the address we measured, fetched once with our own identifying User-Agent: the control group every bot row is compared against.

Vendor

Who runs the crawler. Rows are grouped by vendor, so one company reads as one block.

Bot

The token you would write after User-agent: in robots.txt. A bot marked unverified UA has no published User-Agent string at all; one marked token as UA has only this token, which is what we send.

Purpose

What the crawler collects for: training a model, building a search index, fetching a page a person just asked about, or checking an ad landing page.

robots.txt

Layer one, computed from your file: does your robots.txt allow this token on this path? Hover the verdict to see which rule matched. The column disappears when the site has no robots.txt, and the line above the table says so.

HTTP status

Layer two, measured for real: what your server returned to a request carrying that bot User-Agent. This is the layer robots.txt cannot tell you about.

Content

How many visible characters that bot received. The yardstick is the other bots in the same run, so a number well under the rest means a stripped page, not a language difference.

Result

Whether the measurement itself finished. It is a badge, not a button: the retry for a failed row lives in the next column.

Actions

One button per row that needs one: a bot that timed out, or never got its turn, gets a retry that re-measures only that row. Rows that finished cleanly have nothing here.

Warning icon reference

Allowed, but a 4xx or 5xx

Your robots.txt invites the bot in and something in front of your server turns it away, usually a CDN or WAF bot rule you did not write. This is the finding the tool exists for.

Blocked, but a 2xx

The opposite gap: robots.txt says no and the page is served anyway. Nothing enforces robots.txt, so any crawler that ignores it walks off with the full page.

Far less content

The status looks healthy but the body is a fraction of what other bots got. That is usually a challenge page or a stripped variant, and a status column alone never shows it.

unverified UA

The vendor publishes no User-Agent string, so ours comes from access logs and may be out of date. Read that row with less confidence than the rest.

token as UA

The vendor publishes the robots.txt token but no User-Agent header, so the request we send carries the token itself. Real traffic from that crawler may look different, so read the row as indicative rather than exact.

Status code reference

200 OK

The bot got the page. Compare the Content column before calling it a win.

403 Forbidden

Refused. Almost always a bot rule rather than a broken page: the same URL usually opens fine in your browser.

404 Not Found

Nothing at that address for this visitor. If other bots got 200 for the same URL, the difference is being made on purpose.

429 Too Many Requests

Rate limited. Your site is throttling, not blocking: retry that row rather than reading it as a policy.