What Is llms.txt and Do You Need One?

llms.txt is a Markdown file placed at the root of a website, at /llms.txt. It tells language models what the site is about and links to the pages that matter most, each with a one-line description. It was proposed by Jeremy Howard of Answer.AI in September 2024, and the format is documented at llmstxt.org.

Why it exists

Web pages are built for browsers: menus, scripts, cookie banners, footers. A model with a limited context window wastes much of it on that noise. A short Markdown summary written by the site owner is cheaper to read and more accurate than anything a model could guess from raw HTML.

The format

A valid file has a fixed order:

1. One H1 with the name of the site or project. This is the only required part.
2. A blockquote with a short summary.
3. Optional free text with details, without headings.
4. H2 sections that group links.
5. Links as Markdown list items, optionally followed by a colon and a note.

A section called Optional marks links that a model can skip when it is short on context.

# Acme Analytics

> Privacy-friendly web analytics for small teams. No cookies, GDPR-ready.

## Docs
- [Quick start](https://acme.com/docs/start): Add the script in 2 minutes
- [API](https://acme.com/docs/api): REST endpoints for stats and exports

## Optional
- [Changelog](https://acme.com/changelog): Release notes

Some sites also publish /llms-full.txt, the full text of their docs in one file. It is useful for coding tools, but check its size with a token counter: a file larger than a model’s context window gets cut.

llms.txt vs robots.txt vs sitemap.xml

File	Written for	Says
robots.txt	Crawlers	What may be crawled
sitemap.xml	Search engines	Every URL and when it changed
llms.txt	Language models and agents	Which pages matter and what they contain

llms.txt doesn’t grant or deny access. A bot blocked in robots.txt shouldn’t read llms.txt either.

Who actually reads it

This is where most articles overstate things:

• AI search engines. No major one has confirmed it uses llms.txt to rank or cite pages. Google has said it doesn’t use it. Server logs typically show little traffic to the file.
• Developer tools and agents. This is where it helps. Coding assistants, documentation platforms and MCP-based tools fetch llms.txt to understand a product or API quickly, and many docs platforms generate it automatically.

Should you add one?

Yes, if you run documentation, a SaaS product, an API or a developer tool. People ask coding assistants about your product, and llms.txt gives those assistants a clean map.

It’s optional for blogs, shops and local businesses. It costs ten minutes and does no harm, but don’t expect it to change your visibility in ChatGPT answers. Work on citable content first.

Create and check it

1. Generate a draft from your sitemap with the llms.txt generator.
2. Edit the descriptions. They should say what a page answers, not repeat its title.
3. Upload it to the root of your domain as llms.txt, served as plain text.
4. Run the llms.txt validator to check the structure and links.