Keyword Density
Analyze keyword usage.
OpenWrite an llms.txt summary and the robots.txt lines for AI crawlers from one form.
Enter the name, canonical URL and a one-line summary.
Tick GPTBot, ClaudeBot, Google-Extended or CCBot and set allow or block.
Add docs to highlight and paths that crawlers should skip.
Tidy the placeholder lines, download llms.txt and merge the robots rules.
Your documentation site gets cited by AI assistants, and you want to point them at the right pages while keeping account areas out of training crawls. This studio drafts an llms.txt file listing your main docs and produces robots.txt rules for the AI user-agents you choose.
Enter a site name, canonical URL and one-line description, then list important doc paths and paths to keep out, one per line. The llms.txt output is Markdown: a title, your description as a quote, a Docs list with each path expanded to a full URL, an AI crawl policy paragraph, a Do not crawl list and an Optional section. Below it, a robots.txt block is written for each bot you tick: GPTBot, ClaudeBot (plus the older anthropic-ai token), Google-Extended and CCBot. If none are ticked, GPTBot is used.
The Allow AI training crawlers switch applies to every selected bot at once. When on, each gets Allow: / plus your disallowed paths; when off, each gets Disallow: /. To allow one bot and block another, generate twice and merge. Edit the draft before publishing: the second line is a "Generated locally" comment written as a Markdown heading, and the Optional section holds placeholder lines about a contact page and browser-based tools that may not fit your site. llms.txt is a proposed convention that crawlers may ignore, while robots.txt is what compliant bots follow.
Test the merged file with the robots.txt path checker, or start one with the Robots.txt Generator. Everything is generated in the browser.
Keyboard shortcut
Answers for this tool. For site-wide help, open the FAQ hub.
Not in one pass, because the allow switch applies to all ticked bots. Generate once with GPTBot allowed, again with CCBot blocked, and combine the two robots.txt blocks.
At the root of your site, next to robots.txt, so it is reachable at /llms.txt. The robots hints have to be merged into your existing robots.txt file, since they do nothing on their own.
No. Google-Extended controls whether content is used for Google's AI models. Regular indexing by Googlebot is governed by its own rules.