llms.txt is a short markdown file at the root of a site that lists what the site considers its own best pages, written for AI systems rather than for humans in a browser. It went from one person's essay to many sites shipping a lint-checked file, and the tools around it — generators, validators, checkers — now show up in searches for 'llms.txt file generator' and 'llms.txt validator'.
What it is supposed to do
- Give an AI agent a curated map, so the model is not choosing pages by link popularity alone.
- Propose a machine-readable toc: your guides, your about, your standards, your changelog.
- Sit next to robots.txt without replacing it. robots.txt says what is allowed, llms.txt says what is worth reading.
Where it stands
OpenAI, Google and others are already indexing honest machine-readable signals from sites directly; the sitemap's recommendation to keep robots, sitemaps, dates and metadata accurate is the guidance that carries through. llms.txt's direct reading is trickier: the biggest AI crawlers document their robot rules and indexing of pages, while the proposal itself remains a community convention officially agreed with none of them. That is not a failure. It's a gacor documentation culture on this one — but you can't treat llms.txt the way you treat robots.txt, where noncompliance has consequences through specific servers.
This site publishes one anyway. It costs one small file, it documents your site honestly to anyone who arrives by script, and if a major crawler starts consuming it tomorrow, the file was already there.