What Is llms.txt, and Should Your Website Use It?

01

A new file appears at the root of the website

A business website may contain service pages, project evidence, articles, policies, documentation and location pages. A person can use the navigation and search. A software agent has to fetch pages, remove menus and scripts, follow links and decide which parts deserve its limited context.

The llms.txt proposal offers the agent a shorter route. A small Markdown file, normally published at /llms.txt, introduces the website and points towards selected resources with plain descriptions. It behaves more like a carefully written contents page than a complete copy of the site.

That idea is useful. The surrounding claims need more care. Publishing the file does not force an AI service to read it, does not grant or withdraw permission to crawl, and does not make Google rank the website more highly.

The sensible question is not whether every site needs the latest AI file. It is whether a maintained, machine-readable guide would help the agents and people trying to understand this particular site.

02

What llms.txt actually is

llms.txt is a proposed convention first published in 2024 and updated since. It is designed to help large language models and software agents use a website by giving them concise background, guidance and links to more detailed information.

The proposed format uses Markdown. An H1 names the site or project. A blockquote can provide a short summary. Further notes can explain important context, followed by H2 sections containing selected links and descriptions. An Optional section can hold useful but less important material.

The file can sit at the site root or within a path such as /docs/llms.txt. A file inside a path describes the content beneath that path. The proposal also recommends linking to clean Markdown versions of detailed pages where they are available.

This makes particular sense for technical documentation. A coding agent trying to find one API method does not need the pricing page, company history and every old announcement in the same context window. A concise map can point it straight towards the current reference and examples.

03

It is a curated map, not a copy of the website

A sitemap is expected to list many discoverable URLs. An llms.txt file should be selective. Its value comes from choosing the pages that best explain the organisation, product or documentation set and describing why each link matters.

For a business software consultancy, useful sections might cover identity, core services, selected project evidence, technical articles and contact details. For a software product, they might cover a quick start, API reference, authentication, common errors and release information.

Dumping every URL into the file removes that advantage. It gives the agent another long list to interpret and creates a maintenance problem whenever weak, duplicated or outdated pages remain in the index.

The useful mental model is a receptionist with a good directory. The directory does not contain every conversation held in the building. It helps a visitor find the right room and understand what happens there.

04

llms.txt is not robots.txt

The similar filename causes understandable confusion. robots.txt is used to tell compliant crawlers which URLs they may access. Google also warns that robots.txt is not a security control and is not the right way to guarantee that a web page stays out of search results.

llms.txt has a different job. It supplies context and selected links. Leaving a page out of llms.txt does not prevent an AI crawler from finding it through navigation, a sitemap or another website. Adding a page does not authorise access that has been blocked elsewhere.

Private information still needs real access control. Put it behind authentication and appropriate authorisation. Use the relevant crawler controls for public content, and check the documentation for each service because different crawlers do not all use the same identifiers or rules.

Treat llms.txt as publishing, not protection. Anything written in the file is public and may be copied, cached or quoted like other public website content.

05

It does not replace an XML sitemap or structured data

An XML sitemap helps search engines discover canonical pages, especially on large or frequently changing websites. Structured data describes recognised entities and page types in a machine-readable form. Internal links help people and crawlers understand how the site fits together.

llms.txt does not replace any of that work. The proposal is a separate, curated overview intended for agents that choose to use it. A website can sensibly provide all of these files because they answer different questions.

The comparison below is a useful way to stop the files being treated as interchangeable.

06

Google does not use it for Search visibility

Google has now addressed llms.txt directly in its guidance for generative AI features. It says Google Search does not use llms.txt or other special AI text files. Creating one for another system will neither help nor harm a site's visibility or rankings in Google Search.

That does not make the proposal pointless. It does remove one misleading sales pitch. A business should not buy an llms.txt generator because somebody promises a shortcut into AI Overviews or better Google positions.

Google's advice remains familiar: publish useful original content, keep important pages crawlable and indexable, use a clear technical structure, provide a good page experience and maintain accurate business information. Those jobs are harder than creating one text file because they improve the substance of the site rather than add another label to it.

Other agents may use llms.txt. The proposal lists growing adoption among documentation platforms and AI developer sites. Support is still service-specific, so the honest claim is that the file can help compatible tools, not that every model will fetch or obey it.

07

How to write a useful llms.txt file

Start with the questions an informed visitor would ask. Who is this organisation or project? What does it provide? Which pages contain the most reliable detail? Where are the boundaries, policies and current references?

Write a short introduction in ordinary language. Group links by purpose and give each one a description that explains what the agent will find. Prefer canonical, public and maintained pages. Link to clean Markdown versions when the site can provide them without creating a second manual publishing process.

Keep the file small enough to scan. A service site might need a few identity pages, core services, selected proof and a contact route. A documentation site may need more sections, but it should still guide the agent towards details rather than embed the whole manual.

Test the result from a cold start. Give an agent only the llms.txt URL and ask common questions about the site. Check whether it chooses the correct source and whether the descriptions remove ambiguity. The test does not prove every external service will use the file, but it reveals whether the map itself is clear.

08

Maintenance is where the idea earns or loses trust

A hand-written file can become stale. A service is renamed, a policy moves or an article is replaced, yet the old description remains at the root of the site. The file then gives a confident route to the wrong answer.

Generating the links from the same structured content used by the website can reduce that drift. Even then, curation still needs a human decision. Automation can list every page, but it cannot always tell which page is authoritative, commercially relevant or safe to recommend.

Give the file an owner and review it when navigation, services, documentation or canonical URLs change. Check for broken links during deployment. Keep claims consistent with the visible website rather than adding richer promises that people cannot verify on the linked page.

Do not fill it with keyword variations. A concise identity statement and useful descriptions are more valuable than repeating every town, service and marketing phrase in the hope that an AI system mistakes volume for authority.

09

Should your website add one?

Add llms.txt when the site has public information that an agent may genuinely need and somebody can maintain a useful route through it. Documentation sites, software products, professional profiles and businesses with several clear service or policy pages are reasonable candidates.

Leave it for later when the website is small, the content is thin or the important pages are already inaccurate. Fixing the visible information, navigation, crawlability and page quality will deliver more value to people and to search systems than publishing an immaculate index of weak pages.

The implementation cost can be small, especially when the site already stores its pages in structured data. The opportunity cost still matters. If the choice is between writing llms.txt and correcting an outdated service page, correct the page first.

Used properly, llms.txt is a helpful map for agents that support it. Used as an AI ranking promise, it becomes another distraction. Curate it, maintain it and keep the established web foundations underneath it.