Robots.txt Generator
Build a valid robots.txt with presets for common setups, block AI crawlers in a click, add your sitemap and test any path against the rules.
Test a URL
How to create a robots.txt file
- Pick a preset or set the default access, then list the paths crawlers should not visit, with any exceptions to allow.
- Tick the crawlers you want to block completely, such as AI training bots, and add your sitemap URL.
- Test a path against the rules, then download robots.txt and upload it to the root of your site (example.com/robots.txt).
Features
AI crawler blocking
One-click rules for GPTBot, ClaudeBot, CCBot and more.
URL tester
See whether a path is allowed for a given crawler.
Presets & download
Standard, WordPress, block-all and AI-block presets.
Private
Everything runs in your browser; nothing you paste is uploaded.
Frequently asked questions
What is robots.txt?
A plain text file at the root of your site that tells crawlers which URLs they may request. It is a request, not a security barrier: well-behaved bots follow it, but it does not hide content from people or bad actors.
Does Disallow remove a page from Google?
Not necessarily. A disallowed URL can still appear in search results if other sites link to it. To keep a page out of the index, use a noindex meta tag or header and leave it crawlable so Google can see the tag.
How do I block AI training crawlers?
Add a User-agent group with Disallow: / for each bot, such as GPTBot, ClaudeBot, CCBot and Google-Extended. The Block AI crawlers preset does this for the main ones.
How do wildcards work?
In Google and Bing, * matches any sequence of characters and $ marks the end of the URL. Disallow: /*.pdf$ blocks all PDF files, and the longest matching rule wins, with Allow beating Disallow on ties.