Is llms.txt worth it?

An llms.txt file is worth publishing if you have documentation or a handful of pages that explain what you do, because it takes minutes with the LLMs.txt Generator and has no downside. Do not expect it to change anything measurable on its own: no major AI provider has confirmed reading it.

Does Google use llms.txt?

Google has not said it uses llms.txt, and its Search Advocate John Mueller has compared the file to the keywords meta tag. Google says pages need no special files or markup to appear in AI Overviews, and normal SEO practice applies.

What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers which URLs they may fetch, and crawlers that follow the standard respect it. llms.txt is a reading list of the pages a site wants language models to see. It allows or blocks nothing, and nothing enforces it.

What is the difference between llms.txt and llms-full.txt?

llms.txt is an index of links, while llms-full.txt contains the full text of those pages in one file. That makes llms-full.txt far larger. It suits sites whose documentation is the product, and most sites need only llms.txt.

Where does the llms.txt file go?

The llms.txt file goes at the root of the domain, so it loads at example.com/llms.txt. Nothing links to it and no header announces it, so a copy in a subdirectory will not be found.

Where do the file size and last-modified date come from?

The LLMs.txt Checker takes the last-modified date from the server's Last-Modified header, so it is blank when the server sends none. The size comes from the Content-Length header, or from the bytes actually received when that header is missing.

Can I check llms.txt on a subdomain?

Yes. The checker looks at the root of the host you enter, so docs.example.com is checked at docs.example.com/llms.txt, separately from example.com. Each subdomain needs its own file if you want one there.

Does the checker need the full llms.txt URL?

No. Enter the domain, such as example.com, and the checker requests /llms.txt and /llms-full.txt at its root. Any path you add is ignored, because both files only count at the root.

Free tool

LLMs.txt Checker

Whether your site has an llms.txt file, and a line-by-line read of what is in it.

How it works

What is the LLMs.txt Checker?

The LLMs.txt Checker is a free tool that looks for /llms.txt and /llms-full.txt at the root of a website and reports whether each file is found, with its size and last-modified date when the server sends one. It then runs a content check on the llms.txt file and lists any errors or warnings.

How to use the LLMs.txt Checker

Enter the website's domain, such as example.com. The tool checks the root of the domain, where both files live.
Press Check LLMs.txt. The tool requests /llms.txt and /llms-full.txt from that site.
Read the Domain Check tab: found or not found for each file, its URL, and its size and last-modified date when available.
Open the Content Validator tab to see the line counts, errors and warnings for the llms.txt file that was served.

What is llms.txt used for?

llms.txt is a markdown file at /llms.txt that points language models to the pages on a site worth reading. It starts with an # H1 naming the site, an optional > blockquote summary, then ## Section headings, each holding a list of links written as - [Name](https://example.com/page): what it covers.

It is a curated map, not a sitemap. The point is that a person chose the twenty pages worth reading instead of listing all twenty thousand. A companion file, llms-full.txt, puts the full text of those pages into one file so a model can read the content without fetching each link.

Is llms.txt actually used by AI search engines?

Not as far as anyone has confirmed. llms.txt is a proposal published by Jeremy Howard of Answer.AI in September 2024, not a standard. No major AI provider has publicly confirmed that its crawlers read it, and Google's John Mueller has compared it to the keywords meta tag.

Treat it accordingly: quick to publish, no downside, and no evidence it changes anything on its own. It also cannot make up for pages a crawler cannot reach. If your content is blocked in robots.txt or only renders with JavaScript, listing it in llms.txt changes nothing, because something still has to fetch those pages. The AI Crawler Checker tests whether 14 AI crawlers can reach them.

How do I check if a site has an llms.txt file?

Open the domain followed by /llms.txt in a browser, or run this checker, which requests both /llms.txt and /llms-full.txt at once and reports whether each one is found.

When you open the file yourself, you should see plain markdown, not a styled page or your 404 page. Hosts with catch-all routing can answer every path with an HTML page, so check the content as well as the address. The file should be served as text/plain or text/markdown. To test the format and every link in it, use the LLMs.txt Validator.

What does the llms.txt content check look for?

It checks that the served llms.txt file has content and a recognisable structure, and that any labelled URL or email in it is well formed. It counts total lines, content lines, empty lines and lines starting with #, which it lists as Comments, so markdown headings appear in that figure.

It reports an error when the file is empty or holds only headings, and when a url:, website: or repository: line has an invalid address or an email: line an invalid email. It warns when it finds no recognisable structure, such as headings, list items or links, and when the file is shorter than 50 characters. Lines under a markdown heading are accepted as written, so a tidy file passes easily. To test the H1, the summary, the sections and every link, use the LLMs.txt Validator.

Why does the checker say found when the file is wrong?

Because found means the server answered the request with a success status, and some servers answer every address that way. The checker follows redirects, so an llms.txt URL that redirects to your homepage, or a host whose catch-all routing serves an HTML page for any path, both come back as found.

The size and the content check give it away. An HTML page is usually far larger than a short markdown file, and its line counts look nothing like a list of links. Open the URL in a browser to confirm what is actually there.

Can robots.txt block access to llms.txt?

Yes. llms.txt is an ordinary URL, so a crawler that follows robots.txt will not fetch it if a rule disallows that path. A broad Disallow: / for an AI crawler blocks llms.txt along with everything else.

If you publish the file for AI tools to read, make sure the crawlers you want can reach it. The Robots.txt Checker shows the rules in your file, so you can confirm no group you care about disallows /llms.txt.

What is the difference between an llms.txt checker and a validator?

A checker tells you whether the files exist, and a validator tells you whether the file is right. This checker requests both llms.txt and llms-full.txt, reports found or not found with the size and date, and runs a light content check on llms.txt.

A validator goes further on one file: it checks the H1 title, the summary blockquote and the H2 sections against the llmstxt.org proposal, and tests whether every link responds. Run the checker first to confirm the files are live, then the validator on the file you care about.

Questions

Is llms.txt worth it?: An llms.txt file is worth publishing if you have documentation or a handful of pages that explain what you do, because it takes minutes with the LLMs.txt Generator and has no downside. Do not expect it to change anything measurable on its own: no major AI provider has confirmed reading it.
Does Google use llms.txt?: Google has not said it uses llms.txt, and its Search Advocate John Mueller has compared the file to the keywords meta tag. Google says pages need no special files or markup to appear in AI Overviews, and normal SEO practice applies.
What is the difference between llms.txt and robots.txt?: robots.txt tells crawlers which URLs they may fetch, and crawlers that follow the standard respect it. llms.txt is a reading list of the pages a site wants language models to see. It allows or blocks nothing, and nothing enforces it.
What is the difference between llms.txt and llms-full.txt?: llms.txt is an index of links, while llms-full.txt contains the full text of those pages in one file. That makes llms-full.txt far larger. It suits sites whose documentation is the product, and most sites need only llms.txt.
Where does the llms.txt file go?: The llms.txt file goes at the root of the domain, so it loads at example.com/llms.txt. Nothing links to it and no header announces it, so a copy in a subdirectory will not be found.
Where do the file size and last-modified date come from?: The LLMs.txt Checker takes the last-modified date from the server's Last-Modified header, so it is blank when the server sends none. The size comes from the Content-Length header, or from the bytes actually received when that header is missing.
Can I check llms.txt on a subdomain?: Yes. The checker looks at the root of the host you enter, so docs.example.com is checked at docs.example.com/llms.txt, separately from example.com. Each subdomain needs its own file if you want one there.
Does the checker need the full llms.txt URL?: No. Enter the domain, such as example.com, and the checker requests /llms.txt and /llms-full.txt at its root. Any path you add is ignored, because both files only count at the root.

AI Visibility Tools

Before your next buyer asks AI about you, ask AI about yourself

One check, four engines, about twenty seconds. We show you the answers before we ask for anything.

Check your brand free All free tools