Free tool
AI Crawler Robots.txt Generator
Allow or block GPTBot, ClaudeBot, PerplexityBot and other AI crawlers, and copy the robots.txt rules.
How it works
What is the AI Crawler Robots.txt Generator?
The AI Crawler Robots.txt Generator is a free tool that writes robots.txt rules for known AI crawlers. Toggle each bot between allowed and blocked, including GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot, grouped as training, search and retrieval, or user-triggered. Copy the rules into your robots.txt. They are a request to crawlers, not enforcement.
How to use the AI Crawler Robots.txt Generator
- Review the three groups: training, search and retrieval, and user-triggered.
- Set each crawler to Allowed or Blocked, or switch a whole group at once.
- Copy the generated rules.
- Add them to the robots.txt file at the root of your site and check the live file.
Which AI crawlers can I control with robots.txt?
The generator covers fourteen documented user-agent tokens. Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot and Meta-ExternalAgent. Search and retrieval: OAI-SearchBot, Claude-SearchBot and PerplexityBot. User-triggered: ChatGPT-User, Claude-User and Perplexity-User.
The list is limited to tokens their operators or long-standing public documentation describe. New crawlers appear often, so treat it as a solid base, not a complete census. The grouping is a simplification: some operators, such as Amazon and Meta, describe more than one use for a single crawler.
What is the difference between training, search and user-triggered crawlers?
Training crawlers collect pages that may be used to train models. Blocking them is a statement about your content being used as training data. Search and retrieval crawlers build the index an AI search product answers from; blocking them can remove your pages from those answers and their citations.
User-triggered fetchers request a page because a person asked the assistant about it or pasted its URL. Many sites block the first group and allow the other two, which is what the sample does. That choice is yours; the tool does not recommend one policy.
Are Google-Extended and Applebot-Extended real crawlers?
No. They are control tokens. Google-Extended tells Google whether content its existing crawlers fetched may be used for Gemini model training and grounding. Google states it does not affect inclusion or ranking in Google Search. Applebot-Extended does the same job for Apple's models, applied to pages Applebot crawled.
Because nothing crawls under those names, they never appear in server logs. You will only ever see them in robots.txt.
Does robots.txt actually stop AI crawlers?
It is a request, not enforcement. The major operators listed here say their crawlers follow robots.txt, but nothing technical makes a bot comply, and some vendors say their user-triggered fetchers may not follow it because a person requested the page.
To enforce a block you need the server or CDN to refuse the request, for example with a rule from the Cloudflare Rules Generator. To see what bots do in practice, paste your access log into the AI Crawler Log Analyzer.
Should I block AI crawlers?
It depends on what you want from AI products. If you want your brand and pages cited in AI answers, blocking search and retrieval crawlers works against you. If your concern is unlicensed use of your writing in model training, blocking the training group addresses that without touching AI search.
The rules this tool writes are meant to be added to an existing file. Build the rest with the Robots.txt Generator, and check which bots your live file already blocks with the AI Bot Crawler Checker.
Questions
- Does blocking GPTBot remove my site from ChatGPT search?
- OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the one for search. They are controlled separately, so blocking only GPTBot leaves search access in place.
- Does blocking Google-Extended hurt my Google rankings?
- Google says no. Google-Extended does not affect crawling, indexing or ranking in Google Search.
- Why does the output list allowed bots too?
- An explicit
Allow: /group makes your policy visible and stops a broader rule elsewhere in the file from applying to those bots by accident. - Will these rules affect Googlebot or Bingbot?
- No. Each group names specific AI tokens. Crawlers that are not named keep following your existing rules.
- Is the AI Crawler Robots.txt Generator free?
- Yes, with a free spyc.ai account. It runs in your browser, so nothing you type or paste is sent to a server.
Technical SEO Tools
More in Technical SEO Tools
Opengraph Viewer
Paste a URL and see the card it produces when someone shares it — title, description, image.
View OpengraphOpen Graph Generator
Write the Open Graph tags, preview the card, copy the markup.
Generate Open Graph tagsRobots.txt Checker
Fetch any site's robots.txt and test one URL against it: allowed, or blocked by which rule.
View robots.txtReady when you are
Before your next buyer asks AI about you, ask AI about yourself
One check, four engines, about twenty seconds. We show you the answers before we ask for anything.