Free tool
Robots.txt Diff Tracker
Compare two robots.txt versions and see which paths became blocked or unblocked for each user-agent.
How it works
What is the Robots.txt Diff Tracker?
The Robots.txt Diff Tracker is a free tool that compares two versions of a robots.txt file. Paste the old and the new file to see every added and removed line, then a summary for each user-agent group showing which paths became blocked, which were unblocked, and which Allow rules were added or removed.
How to use the Robots.txt Diff Tracker
- Paste the previous robots.txt into the old box.
- Paste the new or proposed robots.txt into the new box.
- Read the per user-agent summary to see what is newly blocked or unblocked.
- Scan the line-by-line view for sitemap, comment and crawl-delay changes.
Why compare two versions of robots.txt?
Because one line can take a site out of search. A stray Disallow: / left over from a staging environment, or a rule that blocks the folder holding your CSS and JavaScript, changes what crawlers can fetch the moment it goes live, and nothing on the site looks different to a visitor.
Comparing the file before a deploy, or after a platform or plugin update rewrote it, shows exactly what changed while it is still cheap to fix.
How does the robots.txt diff work?
It runs two comparisons. The first is a plain line diff: blank lines are dropped, the rest are compared in order, and lines that were added or removed are marked. That view catches everything, including comments, Sitemap lines and Crawl-delay.
The second parses both files into user-agent groups and compares the rules inside each. A Disallow that appears only in the new file is listed as now blocked; one that disappeared is listed as no longer blocked. Allow rules are tracked the same way.
What does it mean when a new user-agent group is added?
It can change more than it seems. A crawler follows only the most specific group that names it. If the old file had a single * group and the new one adds a group for GPTBot, that bot stops reading the * rules altogether and follows only its own group.
So a new group that contains one harmless rule can silently unblock every path the * group was protecting, for that crawler. The tool labels added and removed groups so you can check for this.
What are the limits of this comparison?
Paths are compared as text. The tool does not work out the combined effect of overlapping rules, such as a wildcard Disallow that covers the same URLs as a removed specific one. It also does not fetch anything, so it compares what you paste, not what is live.
To test whether a specific URL is blocked on the live site, use the Robots.txt Checker.
How do I keep track of robots.txt changes over time?
Keep a copy. Store robots.txt in version control if your site is deployed from a repository, or save a dated copy whenever you change it. When rankings or crawl stats shift, paste the last known good copy and the live file here to rule the file in or out within a minute.
If you are writing a new version, build it with the Robots.txt Generator or add AI bot rules with the AI Crawler Robots.txt Generator, then diff it against the current file before uploading.
Questions
- Does the tool fetch my live robots.txt?
- No. You paste both versions. Open
yoursite.com/robots.txtin a browser and copy the text to get the live one. - Are user-agent names case-sensitive?
- No. Crawlers match user-agent names without regard to case, and the tool compares them the same way. Paths are case-sensitive.
- Why does an empty Disallow not show as a rule?
Disallow:with no path blocks nothing, so it is not counted as a blocked path. It still shows in the line-by-line view.- What if the same user-agent appears in two groups?
- The rules are merged, which is how Google treats repeated groups for the same user-agent.
- Is the Robots.txt Diff Tracker free?
- Yes, with a free spyc.ai account. It runs in your browser, so nothing you type or paste is sent to a server.
Technical SEO Tools
More in Technical SEO Tools
Opengraph Viewer
Paste a URL and see the card it produces when someone shares it — title, description, image.
View OpengraphOpen Graph Generator
Write the Open Graph tags, preview the card, copy the markup.
Generate Open Graph tagsRobots.txt Checker
Fetch any site's robots.txt and test one URL against it: allowed, or blocked by which rule.
View robots.txtReady when you are
Before your next buyer asks AI about you, ask AI about yourself
One check, four engines, about twenty seconds. We show you the answers before we ask for anything.