Overview
This page is the atomic definition. The deep-dive lives at ai-txt.
Definition
ai.txt is an optional plain-text manifest served at /ai.txt that declares a site’s content-licensing preferences for AI systems. The format mirrors robots.txt: user-agent groups (Bytespider, GPTBot, ClaudeBot, and others) and Allow/Disallow directives, with extensions for training versus inference and for opt-in licensing terms. It is a soft signal; honoring it depends on the AI vendor. Spawning AI’s Source.Plus initiative and the IETF AI Preferences Working Group are converging on more formal standards.
When it applies
Ship ai.txt when the site has a deliberate policy on AI training (allow, deny, or license-required). Sites with no policy can skip it; the default for most crawlers is to fetch unless blocked.
Example
A publisher’s /ai.txt declares User-Agent: GPTBot followed by Disallow: / to opt out of OpenAI training, and a separate User-Agent: ClaudeBot block with Allow: / that permits Anthropic’s crawler.
Related concepts
- ai-txt - the deep-dive with format rules and vendor support.
- llms-txt - the parallel manifest for agent-readable site indexing.
- for-ai-agents - the agent-onboarding page on this site.
- llm-info-standard - the broader LLM-info standards survey.
- technical - the technical SEO checklist that touches crawler directives.
Citing this term
See ai.txt (llmbestpractices.com/glossary/ai-txt).