Vutral
How to use this tool

Bulk Meta Robots & X-Robots-Tag Checker

Data checkedMeta Robots

How to use the bulk meta robots tag checker

Back to the tool

Each URL is fetched once and both places an indexing rule can live are read back: the robots meta tag in the head and the X-Robots-Tag response header.

  1. 01

    Enter the URLs to check

    Single mode checks one address and runs on Enter. Switch to Bulk to paste a list, one URL per line, up to 30 per run. The pages worth pasting together are the ones that should differ: a product page that must be indexed next to a search page that must not, so one run shows both answers side by side.

  2. 02

    Choose the URL normalization level

    Normalize URL rewrites each address before fetching. Full URL keeps the path and query but drops tracking parameters and the fragment; No query removes the query string; Domain only keeps the domain. Indexing rules are set per template, and a query string is what tells a search page apart from its listing; drop it and you check the listing instead.

  3. 03

    Read the Robots column

    The column shows the directive strings exactly as declared, one per line, and judges nothing. A directive naming a single crawler is shown with that name after it, [googlebot] for example, since a word aimed at one crawler means less. An empty cell means nothing was declared, which is not a fault: such a page is indexable and followable.

  4. 04

    Inspect a row and export the results

    View sums the directives and marks each with its source: [html] for the meta tag, [header] for the header. The meta tag lives in a template, the header in server or CDN configuration. A second mark flags a directive naming one crawler; the verdict counts it only when User agent is set to that crawler. Copy table copies the rows tab-separated; Export CSV writes them to a file.

How to read the noindex and nofollow report

A row reads left to right: the address checked, where it landed, and the rule that page sets for crawlers.

Result table columns

URL
The address exactly as it was checked — after normalization, not always what you typed.
Resolved URL
Where the request ended and the status it ended on. Redirects are followed first, and the rule is read from that final page.
Robots
Every directive string the final page declares, one per line, unchanged, each with the crawler name after it when it addresses just one. — means none was declared.
Result
Whether the row finished, is still running, or stopped on an error. It says nothing about the rule itself.
Actions
View opens the detail panel for that row; Retry appears instead when the request never got through.

Detail panel verdicts

Indexing and following are allowed.
Something was declared, and none of it blocks anything: an explicit index, follow, or directives that only shape the snippet. The same outcome as declaring nothing, said out loud.
No robots directive — the default is index, follow.
Nothing was declared, so the default applies: indexable and followable. Correct for most pages.
nofollow — links on this page are not followed.
The page is indexable but its links pass nothing on. Rarely what a public page wants, and easy to inherit from a template built for something else.
noindex — this page will not be indexed.
The page is kept out of the index. Right for a search page or a thank-you page, and the most expensive mistake on a page you meant to rank. none counts here too: it is shorthand for noindex plus nofollow.
The two sources disagree — Google applies the stricter one.
One source blocks indexing and another does not. Google takes the stricter one, so the page is out; reading only the permissive source tells you the opposite.
Nothing here applies to every crawler — each directive above names one, and named directives are not checked.
Everything the page declares addresses a named crawler, so there is no general rule to judge. Read the listed directives yourself: a page that is open to everyone except Googlebot is still a page Google will not index.
Directives that name a crawler are listed but not checked — the verdict above covers only the ones that apply to every crawler.
The page declares both kinds. The verdict is true of the general rules only, and the named ones sit above it unjudged; worth reading when the crawler named is the one you care about.
Nothing here applies to Googlebot Smartphone — every directive above names a different crawler.
You ran as a named crawler, and every line the page declares names a different one. Nothing here restricts the crawler you picked; the listed lines are for others.
The verdict above is for Googlebot Smartphone: directives naming it are counted, directives naming other crawlers are not.
Shown when you ran as a named crawler and the page addresses some crawler by name. Lines naming the crawler you picked join the general rules in the verdict; lines naming any other crawler are listed and left out.
The directives counted for Googlebot Smartphone disagree — the verdict above follows the stricter one.
You ran as a named crawler and the lines that apply to it disagree: one blocks indexing, another does not. The verdict takes the stricter one. Google documents that it does the same; other crawlers publish their own rules.

Where a directive can come from

[html]
A <meta name="robots"> tag in the head of the page. Fix it in whatever template renders that head.
[header]
An X-Robots-Tag response header. The only way to set a rule on a file with no head, a PDF or an image, and easy to forget because the page source never shows it. Fix it in server or CDN configuration.
[googlebot] and other crawler names
A directive addressed to a single crawler: <meta name="googlebot">, or an X-Robots-Tag line prefixed with a crawler name. It is listed with that name, and judged only when User agent is set to that crawler. Read those lines yourself: a page blocked for Googlebot alone will not rank.