Web Tools

Robots.txt Generator

All tools

Write crawler rules as simple lists and download a correctly formatted robots.txt.

How it works

Four simple steps

1

Name the crawler

Keep * for all bots or enter one user-agent.

2

List allowed paths

One path per line in the Allow box.

3

List blocked paths

One path per line in the Disallow box.

4

Add sitemap and save

Set the sitemap URL, then copy or download robots.txt.

Robots.txt Generator

About Robots.txt Generator

A staging site got indexed last month because nobody blocked it, and now the production launch needs a robots.txt that keeps admin and cart pages out of crawlers' way without hiding the catalogue. Typing directives by hand works, but one misplaced slash changes what crawlers do.

The form produces a single rule group. Name the user-agent (the default * covers every crawler, or type a specific one such as Googlebot), list the paths to allow and the paths to disallow one per line, optionally set a crawl-delay in seconds, and give the absolute URL of your sitemap. The output follows a fixed order: the User-agent line, each Allow, each Disallow, Crawl-delay if set, and Sitemap last. It updates as you type and downloads as robots.txt.

Paths match as prefixes, so Disallow: /admin blocks /admin/, /admin-login and /administrator alike; add the trailing slash when you mean the folder only. Googlebot ignores Crawl-delay entirely, although some other crawlers respect it. Remember too that robots.txt controls crawling rather than indexing: a disallowed URL can still appear in results if other sites link to it, so use a noindex tag for pages that must stay out. After uploading, review the parsed directives with the robots.txt checker, and build the sitemap the file points to with the XML sitemap builder.

The file is composed in your browser from the fields above. Nothing goes live until you upload it to your site root.

Keyboard shortcut

Ctrl Enter - Run tool
FAQ

Frequently asked questions

See all FAQs

Answers for this tool. For site-wide help, open the FAQ hub.

The form builds one group at a time. Generate the group for the first user-agent, copy it, change the user-agent and paths, then paste the second group below the first with a blank line between them.

Yes. The Sitemap directive expects an absolute URL including https://. A relative path such as /sitemap.xml is not valid there.

Not by itself. Blocking crawling also stops search engines from seeing a noindex tag on those pages. Let them be crawled with noindex until they drop out, then add the Disallow rule if you still need it.

View more FAQs

Related tools

View all

A11y Annotator

Audit HTML for alt text, labels, headings, and ARIA gaps with fix suggestions.

Open

Bundle Size Estimator

Compare approximate gzip sizes for popular npm packages offline.

Open

Import Cost Estimator

Estimate approximate import cost from pasted import lines and package names.

Open

URL Parser

Break down any URL into parts.

Open

Explore 321 free tools

Everything you need as a developer, marketer or creator - in one beautiful place.

Browse all tools