LLM Best Practices

AI Crawler User-Agents

3 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

Match the token to its job before you allow or block it: training, search indexing for an answer engine, or a fetch triggered by one user’s request. Blocking one token does not block the others. Every row below comes from the vendor’s own crawler documentation, checked 2026-10-01. Where the tokens sit next to llms.txt and ai.txt is in discoverability-files.

Tokens

Choose the token by job, not by vendor; the Job column says which control applies.

TokenOperatorJobrobots.txt
GPTBotOpenAITrainingHonored.
OAI-SearchBotOpenAISearch index for ChatGPTHonored.
ChatGPT-UserOpenAIUser-initiated fetchOpenAI: rules “may not apply”.
OAI-AdsBotOpenAISafety check of ad landing pagesNot stated.
ClaudeBotAnthropicTrainingHonored.
Claude-SearchBotAnthropicSearch qualityHonored.
Claude-UserAnthropicUser-initiated fetchHonored.
GooglebotGoogleSearch, including AI Overviews and AI ModeHonored; see below.
Google-ExtendedGoogleControl token only: use of crawled content for Gemini training and for grounding in Gemini Apps and Vertex AIToken, not a crawler.
Google-CloudVertexBotGoogleCrawls a site owner requests for Vertex AI AgentsHonored.
PerplexityBotPerplexitySearch indexHonored.
Perplexity-UserPerplexityUser-initiated fetch”Generally ignores” robots.txt.
ApplebotAppleSearch for Siri, Spotlight, SafariHonored.
Applebot-ExtendedAppleControl token only: Apple foundation model trainingToken, not a crawler.
Meta-ExternalAgentMetaTraining and indexingHonored.
Meta-WebIndexerMetaMeta AI searchHonored.
Meta-ExternalFetcherMetaUser-requested fetch”May bypass” robots.txt.
AmazonbotAmazonProduct improvement; may train Amazon AI modelsHonored.
Amzn-SearchBotAmazonAmazon search; no trainingHonored.
Amzn-UserAmazonUser-initiated fetchMay not follow all directives.
CCBotCommon CrawlPublic corpus used by many model trainersHonored.
MistralAI-TrainingMistralTrainingControllable.
MistralAI-IndexMistralSearch index; not trainingControllable.
MistralAI-UserMistralUser-initiated fetch; not trainingControllable.
DiffbotDiffbotGeneral crawling for its knowledge graph and search; not AI trainingHonored by default.

No English-language vendor crawler documentation exists for these, so treat any rule as best effort: Bytespider (ByteDance; its webmaster page is outside the English web and third parties report it ignores robots.txt), DeepSeek (no published crawler token), and cohere-ai (Cohere states it does not crawl to train models and lists no crawlers).

Allow or block

Pair every User-agent line with an explicit directive.

# Keep out of training, stay visible in ChatGPT search
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Blocking GPTBot leaves OAI-SearchBot and ChatGPT-User free to fetch and cite you. To stay out of an answer engine entirely, block its search and user-fetch tokens too.

Google specifics

Google-Extended has no user-agent string of its own: Google crawls with its normal agents and applies the token afterwards. It limits training and grounding in Gemini Apps and Vertex AI. It does not affect Search inclusion or ranking, and it does not remove you from AI Overviews or AI Mode. Those follow Googlebot, so limit them with nosnippet, data-nosnippet, max-snippet, or noindex.

Gotchas

Avoid relying on robots.txt alone for fetchers that user actions trigger.

GotchaFix
User-initiated fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User) may ignore robots.txt.Enforce at the WAF or CDN if you must block them.
User-agent strings are easy to spoof.Verify against the IP lists that OpenAI, Anthropic, Perplexity, and Common Crawl publish.
Vendors rename and split agents, so an old allowlist leaks.Re-check the operator docs each quarter.
A wildcard User-agent: * rule applies only to agents with no group of their own.Name each agent you care about.
Applebot follows your Googlebot rules when robots.txt does not name Applebot.Add an explicit Applebot group.

Backlinks