Robots.txt Validator & Tester
Which rule wins, and why — implementing the longest-match precedence Google actually uses.
Results
| URL | Verdict | Matching rule |
|---|
File validation
What the robots.txt Validator & Tester does
Reading a robots.txt and working out whether a particular URL is blocked is harder than it looks, because the rules do not apply in the order they are written. Google resolves conflicts by longest match wins: the rule whose path pattern matches the most characters takes precedence, and if an Allow and a Disallow tie, Allow wins. This tester implements that logic, so it tells you not just whether a URL is blocked but which rule decided it.
That matters because the intuitive reading is often wrong. Given Disallow: /search
followed by Allow: /search/help, the Allow wins for /search/help even
though it comes second and is more specific — because it is longer. Reverse the order in the
file and nothing changes. Group selection works differently again: a crawler uses the single most
specific matching user-agent group and ignores every other group entirely, including
*.
The tool also validates syntax. It flags directives that appear before any user-agent line (which
are silently ignored), paths that do not begin with a slash, non-standard directives such as
Host and Noindex that Google does not support, malformed sitemap URLs, and
lines that are not valid directives at all. It highlights a site-wide block prominently, because that
is the error with the largest consequences.
Wildcards are supported as Google implements them: * matches any sequence of
characters, and $ at the end of a pattern anchors the match to the end of the URL.
How to use it
- Paste your robots.txt into the left box.
- List the URLs or paths you want to test on the right — the ones you care about most, and the ones you fear are blocked.
- Choose the crawler to test as; different bots can match different groups.
- Check the matching rule column when a verdict surprises you. It shows exactly which line decided.
A worked example
Testing against the example file:
| URL | Verdict | Matching rule |
|---|---|---|
/cart/summary | Blocked | Disallow: /cart/ |
/search | Blocked | Disallow: /search |
/search/help | Allowed | Allow: /search/help |
/products?orderby=price | Blocked | Disallow: /*?orderby= |
/guides/how-we-test | Allowed | No rule matches |
Now switch the crawler dropdown to GPTBot and every row flips. GPTBot has its own
group in the file, so it obeys that group and ignores the * rules entirely
— it is blocked from the whole site by Disallow: /, including
/search/help, which the * group explicitly allowed. That is correct crawler
behaviour and it is the single most misunderstood part of robots.txt: creating a named group for a
crawler silently exempts it from every general rule you wrote.
Frequently asked questions
Which rule wins when Allow and Disallow both match?
The longer path pattern wins, regardless of order in the file. If both patterns are the same length, Allow wins. This is Google's documented behaviour and what the tester implements; some other crawlers use simpler first-match logic.
Why does one bot get a different answer from another?
Because a crawler obeys only the most specific user-agent group that matches it, and ignores all others including *. If you create a Googlebot group, Googlebot stops reading the * group entirely — so any rules you wanted to apply to it must be repeated inside its own group.
Are wildcards supported in robots.txt?
They are not in the original specification, but Google and Bing both support * for any sequence of characters and $ to anchor the end of a URL. Smaller crawlers may not, so avoid relying on them for anything critical.
Can I test a live site's robots.txt here?
Paste its contents in — visit yourdomain.com/robots.txt and copy what you see. The tool cannot fetch it for you, because browsers block cross-origin requests to other sites.