SEO Toolkit

Robots.txt Checker

All tools

Test a URL path against your robots.txt rules before a crawler finds the mistake for you.

How it works

Four simple steps

1

Paste robots.txt

Add the file contents, or drop the file onto the input.

2

Review the checklist

Groups, rules, sitemaps and unknown lines are summarized.

3

Test a path

Enter a user-agent and a path such as /admin/settings.

4

Read the verdict

See Allowed or Blocked and the rule that decided it.

Robots.txt Checker

About Robots.txt Checker

Someone added "Disallow: /p" to block a private folder and you suspect it also blocks /pricing and /products. Paste the file, type the user-agent and the path, and see straight away whether the page is allowed and which rule decided it.

The checker reads User-agent, Allow, Disallow, Sitemap and Crawl-delay lines and builds a checklist: the number of groups, allow and disallow totals, sitemap lines, a crawl-delay note (Google ignores it) and any unrecognized lines by number. The path tester picks the group whose user-agent matches the name you enter, falling back to the * group, then applies the longest matching rule and shows Allowed or Blocked with the rule responsible.

Know the simplifications before trusting a verdict. Rules are matched as plain prefixes, so wildcards such as * inside a path and the $ end anchor are not interpreted. Each User-agent line starts its own group, so two User-agent lines stacked above one set of rules are not combined the way real crawlers combine them. And when an Allow and a Disallow rule are the same length, this tool sides with Disallow, while Google picks the less restrictive Allow. For files using those features, confirm in Search Console.

To write a new file from scratch, try the robots.txt generator. For AI crawler rules, the llms.txt and AI bot policy studio drafts matching blocks. Everything is evaluated locally.

Keyboard shortcut

Ctrl Enter - Run tool
FAQ

Frequently asked questions

See all FAQs

Answers for this tool. For site-wide help, open the FAQ hub.

No. Rules are compared as simple prefixes, so * and $ are treated as literal characters. Check wildcard rules with a crawler-specific testing tool.

On a tie this tool applies the Disallow rule. Google resolves ties in favor of Allow, so the real result for Googlebot may differ.

Type Googlebot in the User-agent box. If the file has a Googlebot group, its rules are used; otherwise the tester falls back to the group for all crawlers. Names must match in full, though letter case is ignored.

View more FAQs

Related tools

View all

Keyword Density

Analyze keyword usage.

Open

SERP Preview

Preview search engine results.

Open

Meta Tag Analyzer

Analyze existing meta tags.

Open

Sitemap Validator

Validate XML sitemaps.

Open

Explore 321 free tools

Everything you need as a developer, marketer or creator - in one beautiful place.

Browse all tools