spyc.ai

Free tool

Robots.txt Generator

Build a robots.txt with user-agent groups, allow and disallow rules, sitemaps and ready presets.

How it works

What is the Robots.txt Generator?

The Robots.txt Generator is a free tool that builds a robots.txt file from user-agent groups, each with its own Allow and Disallow rules, plus Sitemap lines and an optional Crawl-delay. Start from a preset, such as allow all, block all, WordPress or Shopify-style, edit the rows, and copy the finished file.

How to use the Robots.txt Generator

  1. Pick a preset or start with the default group for all crawlers.
  2. Edit each group: the user-agent it applies to and its Allow and Disallow paths.
  3. Add your sitemap URL, and a crawl-delay only if a specific crawler needs it.
  4. Copy the file and upload it as robots.txt at the root of your site.

What is a robots.txt file?

robots.txt is a plain text file at the root of a host that tells crawlers which paths they may request. It is made of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules. Sitemap lines can sit anywhere and apply to the whole file.

The file covers one host and protocol only. https://example.com/robots.txt says nothing about https://shop.example.com.

How do Allow and Disallow rules work?

A rule matches any URL whose path starts with the rule's value. Disallow: /private/ blocks everything in that folder. * matches any run of characters and $ marks the end of the URL. When an Allow and a Disallow both match, Google applies the most specific rule, meaning the longest path, and Allow wins a tie.

A crawler obeys only one group: the one with the most specific user-agent that matches it. A group for Googlebot replaces the * group for Googlebot; the two are not combined.

Does robots.txt stop a page from being indexed?

No. It stops crawling, not indexing. A blocked URL can still appear in search results if other pages link to it, usually with no description. To keep a page out of the index, let it be crawled and add a noindex robots meta tag or header.

robots.txt is also not access control. The file is public and compliance is voluntary, so never use it to hide private content.

Does Google respect crawl-delay?

No. Googlebot ignores the Crawl-delay line. Some other crawlers, including Bingbot, honour it. The generator still lets you add one per group and shows a note when you do, so you know which crawlers it reaches.

What should a robots.txt file contain?

For most sites, very little: a group for * that blocks admin, cart, checkout and internal search paths, and a Sitemap line. The WordPress preset blocks /wp-admin/ and allows admin-ajax.php. The Shopify-style preset is a short example of the kind of rules Shopify ships; Shopify generates its own file, which you change through its robots.txt theme template.

After uploading, test the live file with the Robots.txt Checker. To set rules for AI bots, use the AI Crawler Robots.txt Generator, and compare versions before a deploy with the Robots.txt Diff Tracker.

Questions

Where do I put robots.txt?
At the root of the host, so it loads at https://example.com/robots.txt. A file in a subfolder is ignored.
What does an empty Disallow mean?
Disallow: with no path blocks nothing, so the group allows everything. The generator writes it for any group that has no rules.
Can one group cover several crawlers?
Yes. Type the user-agents separated by commas and the generator writes one User-agent line for each above the shared rules.
Is there a size limit for robots.txt?
Google reads the first 500 kibibytes of the file and ignores anything after that. Few sites come close.
Is the Robots.txt Generator free?
Yes, with a free spyc.ai account. It runs in your browser, so nothing you type or paste is sent to a server.

Technical SEO Tools

More in Technical SEO Tools

Opengraph Viewer

Paste a URL and see the card it produces when someone shares it — title, description, image.

View Opengraph

Open Graph Generator

Write the Open Graph tags, preview the card, copy the markup.

Generate Open Graph tags

Robots.txt Checker

Fetch any site's robots.txt and test one URL against it: allowed, or blocked by which rule.

View robots.txt

Ready when you are

Before your next buyer asks AI about you, ask AI about yourself

One check, four engines, about twenty seconds. We show you the answers before we ask for anything.