---
title: "AI Crawler User-Agents"
slug: "ai-crawlers"
category: "cheatsheets"
tags: ["cheatsheets", "reference", "seo", "ai-search", "discoverability"]
status: "stable"
last_updated: 2026-08-14
summary: "Reference table of current AI and LLM crawler user-agents, what each does, and the robots.txt token that controls it."
related: ["[[seo/discoverability-files]]", "[[seo/ai-search-optimization]]", "[[seo/llms-txt]]", "[[glossary/crawl-budget]]", "[[meta/for-ai-agents]]"]
---

## Overview

Match the bot to the behavior before you allow or block it. AI crawlers split into three jobs: training ingestion, search indexing for an AI answer engine, and live user-triggered fetches. Blocking one does not block the others, and the user-agent strings change as vendors split their fleets. This card lists the agents that matter as of 2026 and the exact robots.txt token for each. For where these tokens live and how they interact with `llms.txt` and `ai.txt`, see [[seo/discoverability-files]].

## Know the three crawler jobs

Treat the job, not the brand, as the unit of control.

- Training: ingests pages to train or fine-tune a model. Examples: `GPTBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`, `anthropic-ai`.
- Search indexing: builds the index an AI answer engine cites. Examples: `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`.
- User fetch: retrieves a single page because a user asked a chatbot about it. Examples: `ChatGPT-User`, `Claude-User`, `Perplexity-User`. These honor robots.txt but are not bulk crawlers.

## Reference table

| User-agent | Operator | Job | Block in robots.txt |
| --- | --- | --- | --- |
| `GPTBot` | OpenAI | Training | `User-agent: GPTBot` |
| `OAI-SearchBot` | OpenAI | Search index | `User-agent: OAI-SearchBot` |
| `ChatGPT-User` | OpenAI | User fetch | `User-agent: ChatGPT-User` |
| `ClaudeBot` | Anthropic | Training | `User-agent: ClaudeBot` |
| `Claude-SearchBot` | Anthropic | Search index | `User-agent: Claude-SearchBot` |
| `Claude-User` | Anthropic | User fetch | `User-agent: Claude-User` |
| `Google-Extended` | Google | Training (Gemini) | `User-agent: Google-Extended` |
| `Googlebot` | Google | Search index | `User-agent: Googlebot` |
| `PerplexityBot` | Perplexity | Search index | `User-agent: PerplexityBot` |
| `Perplexity-User` | Perplexity | User fetch | `User-agent: Perplexity-User` |
| `Applebot-Extended` | Apple | Training | `User-agent: Applebot-Extended` |
| `Bytespider` | ByteDance | Training | `User-agent: Bytespider` |
| `CCBot` | Common Crawl | Training corpus | `User-agent: CCBot` |
| `Amazonbot` | Amazon | Search and assistant | `User-agent: Amazonbot` |
| `Meta-ExternalAgent` | Meta | Training | `User-agent: Meta-ExternalAgent` |
| `Meta-ExternalFetcher` | Meta | User fetch | `User-agent: Meta-ExternalFetcher` |
| `cohere-ai` | Cohere | Training and inference | `User-agent: cohere-ai` |
| `MistralAI-User` | Mistral | User fetch | `User-agent: MistralAI-User` |
| `Diffbot` | Diffbot | Knowledge graph | `User-agent: Diffbot` |

## Allow or block with full rules, not a bare token

Set the policy per agent, then state the directive. A `User-agent` line with no `Disallow` or `Allow` is ambiguous; always pair them.

```
# Allow everything (this site's posture)
User-agent: GPTBot
Allow: /

# Block training but keep search-index access
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
```

Blocking `GPTBot` removes your pages from training but leaves `ChatGPT-User` and `OAI-SearchBot` free to fetch and cite you. To stay out of AI answers entirely, block the search-index and user-fetch agents too.

## Watch for fleet splits and stale tokens

Vendors rename and split agents; an allowlist written for last year leaks. Anthropic retired `Claude-Web` in favor of `Claude-User` and `Claude-SearchBot`; Google separated `Google-Extended` (training) from `Googlebot` (search); Meta split its single training agent into `Meta-ExternalAgent` (training) and `Meta-ExternalFetcher` (user fetch). Re-check operator docs each quarter, and prefer an explicit allow for the agents you want over a single wildcard. This site allows every agent because each page is built to be cited; see [[meta/for-ai-agents]] and the curated index at [[seo/llms-txt]]. For crawl-volume tradeoffs on large sites, see [[glossary/crawl-budget]] and [[seo/ai-search-optimization]].

## Related

- [[seo/discoverability-files]]
- [[seo/ai-search-optimization]]
- [[seo/llms-txt]]
- [[glossary/crawl-budget]]
- [[meta/for-ai-agents]]