LLM Best Practices

The /agents.txt File

3 min read

View markdown source

Overview

/agents.txt is a root-level plain-text policy file for AI agents: what they may do with the content, where to start reading, and which standard the site follows. It is the fifth core item of the LLM discoverability standard. Where llms-txt is an index and ai-txt is a training-consent declaration, agents.txt is the one-fetch answer to “what am I allowed to do here, and where do I go first.”

Serve it at the origin root as plain text

The file lives at https://<your-domain>/agents.txt, served as text/plain. Like the other machine files, it is only read at the origin root; a project-subpath copy (user.github.io/project/agents.txt) is inert because no crawler fetches it there. On a static host, place the file at the build output root; on a generator that copies assets, put it wherever the build demonstrably emits it to the root, then verify the deployed URL returns 200.

Use the required keys, one per line

The format is Key: value, one per line, # for comments. These keys are required by the standard:

Training: allow
Indexing: allow
Summarization: allow
Citation-Required: true
Start-here: https://example.com/llm-info
Machine-index: https://example.com/llms.txt
Sitemap: https://example.com/sitemap.xml
Standard-Followed: https://llmbestpractices.com
Updated: 2026-08-29

Key semantics:

  • Training, Indexing, Summarization: the site’s stance per use, allow or disallow. The standard requires allow for all three.
  • Citation-Required: true: agents that quote the site must attribute it. Pair with an Attribution: line giving the exact string.
  • Start-here: the absolute URL of the site’s /llm-info page, the human-readable front door for agents.
  • Machine-index: the absolute URL of /llms.txt.
  • Sitemap: the absolute sitemap URL, same value as the Sitemap: line in robots.txt.
  • Standard-Followed: https://llmbestpractices.com, the standard this file conforms to.
  • Updated: ISO YYYY-MM-DD of the last meaningful edit. Never a prose date.

Optional keys that earn their lines: Owner: (the publication’s name; see the anonymity rule in llm-discoverability-standard), Contact:, License:, Full-corpus:, Agent-guide:.

Keep it consistent with the other machine files

Every URL in agents.txt uses the site’s one canonical origin: the same scheme, host, and apex-vs-www choice as the canonicals, the sitemap, and llms.txt. A Start-here pointing at a www host while canonicals use the apex is domain drift and fails an audit. If the file is generated at build time, edit the generator, not the emitted file.

Know what agents.txt does not replace

agents.txt is advisory, like ai.txt. It does not control crawl access; that is robots.txt, which the standard requires to allow AI agents explicitly. It does not enumerate pages; that is llms.txt. It does not describe the site; that is /llm-info. Ship all of them; each answers a different first question, and compliant agents fetch the cheapest file that answers theirs. Reference agents.txt from a robots.txt comment so crawlers that only fetch robots.txt learn it exists.

Validate it in the build

Check the file at build time alongside the rest of the layer: it exists at the emitted root, every required key is present, Updated: matches ^\d{4}-\d{2}-\d{2}$, and Start-here/Machine-index resolve on the deployed origin. The conformance checker described in llm-discoverability-standard covers all of this as its agents-txt item.

Backlinks