LLM Best Practices

The /llms.txt Standard

6 min read

View markdown source

Overview

/llms.txt is a proposed standard from llmstxt.org for giving LLM agents a curated, human-readable index of a site. Treat it as the agent-facing counterpart to sitemap.xml. Ship one on any site you expect agents to browse.

What it is

/llms.txt is a single markdown file served at the site root. It opens with the site title and a blockquote summary, then groups page links under H2 category headings. Each link line has the format - [Title](url): One-line summary. The file is hand-curated or generated from a content index; it is not a crawl.

Where it lives

Serve it at https://<your-domain>/llms.txt with Content-Type: text/plain or text/markdown. Static hosts (GitHub Pages, Vercel, Netlify, Cloudflare Pages) serve the file directly when placed at the build root. Use Quartz’s static/ directory if the SSG does not pick it up from content/ automatically.

Format spec, with example

# Site Name

> One- or two-paragraph blockquote describing what this site is and who it is for.

## Category

- [Page Title](https://example.com/category/slug): One-sentence summary.
- [Another Page](https://example.com/category/other): One-sentence summary.

## Another Category

- [Page](https://example.com/other/page): Summary.

Rules that matter:

  • The H1 is the site name, not the file name.
  • The blockquote after the H1 is the site summary. Keep it tight; this is what an agent reads to decide whether to keep going.
  • Use H2 for category groupings. Avoid H3 unless the site is large enough to need sub-sections.
  • Each link line is one bullet, one link, one summary. Do not stack multiple links per bullet.
  • Use absolute URLs. Relative paths confuse agents that fetch the file in isolation.

Shape it to the house standard: Primary, Key pages, About

The LLM discoverability standard adds three required sections on top of the llmstxt.org shape.

## Primary
- [LLM Info](https://example.com/llm-info): Machine-readable site description for AI assistants; start here.

## Key pages
- [Page](https://example.com/page): The handful of load-bearing pages, one bullet each.

## About

Owner: Example Publication
Standard followed: https://llmbestpractices.com
Updated: 2026-08-29

Rules: ## Primary comes first and its first entry links /llm-info. ## Key pages lists the pages an agent should read when it will not read everything. ## About closes the file with Owner: (the publication’s own name where anonymity is policy), Standard followed:, and an ISO Updated: date as plain lines, not bullets. The once-per-file rule below applies to category sections; ## Primary and ## Key pages are routing tiers and may repeat a page’s canonical entry.

Difference from robots.txt and sitemap.xml

Different audiences, different jobs.

  • robots.txt tells crawlers what they may not fetch. It is a policy file.
  • sitemap.xml tells crawlers what exists. It is a machine index with no descriptions.
  • llms.txt tells LLM agents what is worth reading and why. It is a curated, descriptive index.

Ship all three. They do not overlap.

When to also include /llms-full.txt

/llms-full.txt is an optional companion file: the same index, but with the full markdown body of every page inlined. Ship it when the site is small enough that the entire corpus fits in a single LLM context window (rule of thumb: under ~500K tokens of content). For larger sites, omit it; agents should fetch pages individually.

Know what llms.txt does not do

llms.txt has no effect on Google ranking or AI Overviews inclusion. Google’s Search Central documentation, updated June 2026, states plainly that Search does not use llms.txt and that the file neither helps nor hurts a site’s ranking or generative-AI appearance. Adoption is also thin: independent 2026 studies put llms.txt on roughly 8 to 16 percent of reachable top-1,000 sites, and one large-scale crawl-log analysis found 97 percent of published files were never fetched, with no measurable citation effect.

  • Do not ship llms.txt expecting an SEO or AI Overview lift. Ship it for the audience that does read it: coding agents, agentic browsers, and a handful of AI vendors (Anthropic, Perplexity) that have said they consume it.
  • Stripe, Vercel, Cloudflare, and Anthropic all publish one; treat their adoption as a signal for agent tooling, not for search visibility.
  • Keep the file cheap to generate and maintain (see below) since the cost is low even where the payoff is narrow and audience-specific.

How to auto-generate from a content index

If pages already have YAML frontmatter with title, summary, and category, generate llms.txt at build time. Pseudocode:

groups = defaultdict(list)
for page in all_pages:
    if page.frontmatter.get("status") == "draft":
        continue
    groups[page.frontmatter["category"]].append(page)
 
lines = [f"# {site_title}", "", f"> {site_summary}", ""]
for category, pages in sorted(groups.items()):
    lines.append(f"## {category.title()}")
    for p in sorted(pages, key=lambda p: p.frontmatter["title"]):
        url = f"https://{base_url}/{p.path}"
        lines.append(f"- [{p.frontmatter['title']}]({url}): {p.frontmatter['summary']}")
    lines.append("")
write(lines, "static/llms.txt")

The generator should skip drafts and deprecated pages, sort categories alphabetically, and sort pages within each category by title. See ship-llms-txt for an end-to-end walkthrough.

Serve the markdown source and say so in the header

Ship a markdown twin of every page and announce it in the first lines of llms.txt. An agent that has to strip navigation, sidebars, and footers out of HTML spends tokens on markup and sometimes mangles code blocks. Serving the source removes that step, and it costs nothing at build time because the markdown already exists.

The convention that needs no lookup table is appending .md to the page URL.

https://example.com/coding/python      → rendered HTML, cite this to a human
https://example.com/coding/python.md   → raw markdown, parse this

State the rule in the llms.txt header, not only on a separate agent page, because the header is the first thing a crawler reads:

# Example Docs
 
> One-sentence description of the site.
 
Append `.md` to any URL below to get the raw markdown source instead of
rendered HTML. Every page is served both ways from this domain.

Advertise it in the page <head> as well, so the markdown form is discoverable from any entry point rather than only from llms.txt:

<link rel="alternate" type="text/markdown" href="/coding/python.md" title="Markdown source" />

Emit the tag only for URLs that actually have a markdown twin. Generated routes such as tag listings and the 404 page usually have no source file, and advertising a URL that 404s is worse than advertising nothing. On a static site generator this is a build step that copies each source file to <slug>.md in the output directory, alongside the emitter that already copies images and other non-markdown assets.

Common mistakes

  • Listing the same page twice across category sections. Pick one category home for each page; only ## Primary and ## Key pages may repeat it.
  • Letting summaries balloon past one sentence. The summary line is a routing hint, not the page itself.
  • Forgetting to regenerate the file on content changes. Add it to the build, not a manual step.
  • Linking to relative paths. Agents fetch the file in isolation; relative URLs break.
  • Mixing rendered HTML links and raw markdown links. Pick one form per file. Rendered HTML is the standard.

Validation

Lint the file at build time. The minimum check:

  • File exists at the build root.
  • File starts with # .
  • Every link line matches ^- \[.+\]\(https?://.+\): .+$.
  • No duplicate URLs.
  • Every URL returns 200 in a post-deploy smoke test.

For a richer check, parse the file with a markdown library and walk the link tree.

Backlinks