---
title: "The /agents.txt File"
slug: "agents-txt"
category: "seo"
tags: ["seo", "agents-txt", "ai-agents", "policy", "standards"]
status: "stable"
last_updated: 2026-08-29
summary: "The /agents.txt file: a root-level, key-value policy that tells AI agents what they may do, where to start reading, and which standard the site follows."
description: "What /agents.txt is, the required keys (Training, Indexing, Summarization, Citation-Required, Start-here, Machine-index, Sitemap, Standard-Followed, Updated), and how it differs from robots.txt, ai.txt, and llms.txt."
aliases: ["agents.txt", "agents.txt file", "agents.txt standard", "AI agent policy file"]
related: ["[[seo/llm-discoverability-standard]]", "[[seo/ai-txt]]", "[[seo/llms-txt]]", "[[seo/discoverability-files]]", "[[meta/for-ai-agents]]"]
---

## Overview

`/agents.txt` is a root-level plain-text policy file for AI agents: what they may do with the content, where to start reading, and which standard the site follows. It is the fifth core item of the [[seo/llm-discoverability-standard|LLM discoverability standard]]. Where [[seo/llms-txt]] is an index and [[seo/ai-txt]] is a training-consent declaration, `agents.txt` is the one-fetch answer to "what am I allowed to do here, and where do I go first."

## Serve it at the origin root as plain text

The file lives at `https://<your-domain>/agents.txt`, served as `text/plain`. Like the other machine files, it is only read at the origin root; a project-subpath copy (`user.github.io/project/agents.txt`) is inert because no crawler fetches it there. On a static host, place the file at the build output root; on a generator that copies assets, put it wherever the build demonstrably emits it to the root, then verify the deployed URL returns 200.

## Use the required keys, one per line

The format is `Key: value`, one per line, `#` for comments. These keys are required by the standard:

```
Training: allow
Indexing: allow
Summarization: allow
Citation-Required: true
Start-here: https://example.com/llm-info
Machine-index: https://example.com/llms.txt
Sitemap: https://example.com/sitemap.xml
Standard-Followed: https://llmbestpractices.com
Updated: 2026-08-29
```

Key semantics:

- `Training`, `Indexing`, `Summarization`: the site's stance per use, `allow` or `disallow`. The standard requires `allow` for all three.
- `Citation-Required: true`: agents that quote the site must attribute it. Pair with an `Attribution:` line giving the exact string.
- `Start-here`: the absolute URL of the site's `/llm-info` page, the human-readable front door for agents.
- `Machine-index`: the absolute URL of `/llms.txt`.
- `Sitemap`: the absolute sitemap URL, same value as the `Sitemap:` line in `robots.txt`.
- `Standard-Followed`: `https://llmbestpractices.com`, the standard this file conforms to.
- `Updated`: ISO `YYYY-MM-DD` of the last meaningful edit. Never a prose date.

Optional keys that earn their lines: `Owner:` (the publication's name; see the anonymity rule in [[seo/llm-discoverability-standard]]), `Contact:`, `License:`, `Full-corpus:`, `Agent-guide:`.

## Keep it consistent with the other machine files

Every URL in `agents.txt` uses the site's one canonical origin: the same scheme, host, and apex-vs-www choice as the canonicals, the sitemap, and `llms.txt`. A `Start-here` pointing at a www host while canonicals use the apex is domain drift and fails an audit. If the file is generated at build time, edit the generator, not the emitted file.

## Know what agents.txt does not replace

`agents.txt` is advisory, like `ai.txt`. It does not control crawl access; that is `robots.txt`, which the standard requires to allow AI agents explicitly. It does not enumerate pages; that is `llms.txt`. It does not describe the site; that is `/llm-info`. Ship all of them; each answers a different first question, and compliant agents fetch the cheapest file that answers theirs. Reference `agents.txt` from a `robots.txt` comment so crawlers that only fetch `robots.txt` learn it exists.

## Validate it in the build

Check the file at build time alongside the rest of the layer: it exists at the emitted root, every required key is present, `Updated:` matches `^\d{4}-\d{2}-\d{2}$`, and `Start-here`/`Machine-index` resolve on the deployed origin. The conformance checker described in [[seo/llm-discoverability-standard]] covers all of this as its `agents-txt` item.

## Related

- [[seo/llm-discoverability-standard]]
- [[seo/ai-txt]]
- [[seo/llms-txt]]
- [[seo/discoverability-files]]
- [[meta/for-ai-agents]]