Free tool
Robots.txt Sitemap Conflict Checker
Find sitemap URLs that robots.txt blocks for Googlebot, with the exact Disallow rule behind each conflict.
How it works
What is the Robots.txt Sitemap Conflict Checker?
The Robots.txt Sitemap Conflict Checker is a free tool that finds URLs listed in your sitemap that your robots.txt blocks. It reads robots.txt, loads the sitemaps it declares, and tests up to 2,000 sitemap URLs against the rules for Googlebot and for all crawlers, showing the exact Disallow line behind each conflict.
How to use the Robots.txt Sitemap Conflict Checker
- Enter any URL on the site you want to check.
- Run the check to load robots.txt and the sitemaps it declares.
- Read the rules causing conflicts, ordered by how many URLs each one blocks.
- Either remove the blocked URLs from the sitemap or relax the rule in robots.txt.
What is a robots.txt and sitemap conflict?
It is a URL that the sitemap submits for indexing while robots.txt forbids crawlers from fetching it. The two files give opposite instructions. Search Console reports these as blocked by robots.txt, or as indexed though blocked by robots.txt when Google indexes the bare URL from links without ever reading the page.
Conflicts are usually accidental: a broad rule such as Disallow: /search or Disallow: /*? that also matches pages someone later added to the sitemap.
How does the tool decide a URL is blocked?
It applies the matching rules Google documents. Every Allow and Disallow line in the relevant group is compared with the URL's path and query string. * matches any run of characters and $ anchors the end of the URL. The longest matching rule wins, and when an Allow and a Disallow are the same length, Allow wins.
Googlebot uses the group that names Googlebot when one exists and ignores the * group entirely in that case. The tool evaluates both and says whether a URL is blocked for Googlebot, for other crawlers, or for both.
Which sitemaps does it read?
The Sitemap: lines in robots.txt. If there are none, it tries /sitemap.xml. Sitemap indexes are expanded, up to 10 sitemap files and 2,000 URLs per run. Pages themselves are never requested, so the check is fast and puts no load on the site beyond those files.
Sitemap URLs on a different host are skipped and counted, because a robots.txt file only governs the host it is served from.
Does robots.txt stop a page from being indexed?
No. Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it, shown without a description because Google could not read the page. To keep a page out of the index, allow crawling and use noindex. A crawler that cannot fetch the page never sees the noindex.
That is why listing a blocked URL in the sitemap is a mistake in both directions: if you want it indexed, unblock it, and if you do not, take it out of the sitemap.
How do I fix sitemap URLs blocked by robots.txt?
Start from the rule, not the URL. The results group conflicts by the Disallow line responsible, so one edit often clears many. Narrow the rule, or add a more specific Allow line for the section that should be crawled.
To test individual rules and user agents, use the Robots.txt Checker. Once crawling is allowed, confirm the pages are otherwise indexable with the Sitemap Indexability Checker.
Questions
- How many URLs does it check?
- Up to 2,000 sitemap URLs from up to 10 sitemap files per run. Larger sitemaps are checked in part, and the result says so.
- Which user agents does it test?
- Googlebot and the wildcard group
*. If robots.txt has no Googlebot group, Googlebot follows the*rules and the two results are the same. - What if the site has no robots.txt?
- A missing robots.txt (a 404) means nothing is disallowed, so there can be no conflicts. The tool reports that and still reads the sitemap.
- Does it handle wildcards and Allow rules?
- Yes. It supports
*, the$end anchor, and longest-match precedence betweenAllowandDisallow. It does not normalise percent-encoding, so unusual encoded paths can differ from Google's result. - Is the Robots.txt Sitemap Conflict Checker free?
- Yes, with a free spyc.ai account. Each run is limited only by the caps described on this page.
Technical SEO Tools
More in Technical SEO Tools
Opengraph Viewer
Paste a URL and see the card it produces when someone shares it — title, description, image.
View OpengraphOpen Graph Generator
Write the Open Graph tags, preview the card, copy the markup.
Generate Open Graph tagsRobots.txt Checker
Fetch any site's robots.txt and test one URL against it: allowed, or blocked by which rule.
View robots.txtReady when you are
Before your next buyer asks AI about you, ask AI about yourself
One check, four engines, about twenty seconds. We show you the answers before we ask for anything.