A11y Annotator
Audit HTML for alt text, labels, headings, and ARIA gaps with fix suggestions.
OpenWrite crawler rules as simple lists and download a correctly formatted robots.txt.
Keep * for all bots or enter one user-agent.
One path per line in the Allow box.
One path per line in the Disallow box.
Set the sitemap URL, then copy or download robots.txt.
A staging site got indexed last month because nobody blocked it, and now the production launch needs a robots.txt that keeps admin and cart pages out of crawlers' way without hiding the catalogue. Typing directives by hand works, but one misplaced slash changes what crawlers do.
The form produces a single rule group. Name the user-agent (the default * covers every crawler, or type a specific one such as Googlebot), list the paths to allow and the paths to disallow one per line, optionally set a crawl-delay in seconds, and give the absolute URL of your sitemap. The output follows a fixed order: the User-agent line, each Allow, each Disallow, Crawl-delay if set, and Sitemap last. It updates as you type and downloads as robots.txt.
Paths match as prefixes, so Disallow: /admin blocks /admin/, /admin-login and /administrator alike; add the trailing slash when you mean the folder only. Googlebot ignores Crawl-delay entirely, although some other crawlers respect it. Remember too that robots.txt controls crawling rather than indexing: a disallowed URL can still appear in results if other sites link to it, so use a noindex tag for pages that must stay out. After uploading, review the parsed directives with the robots.txt checker, and build the sitemap the file points to with the XML sitemap builder.
The file is composed in your browser from the fields above. Nothing goes live until you upload it to your site root.
Keyboard shortcut
Answers for this tool. For site-wide help, open the FAQ hub.
The form builds one group at a time. Generate the group for the first user-agent, copy it, change the user-agent and paths, then paste the second group below the first with a blank line between them.
Yes. The Sitemap directive expects an absolute URL including https://. A relative path such as /sitemap.xml is not valid there.
Not by itself. Blocking crawling also stops search engines from seeing a noindex tag on those pages. Let them be crawled with noindex until they drop out, then add the Disallow rule if you still need it.