---
title: "Token"
slug: "token"
category: "glossary"
tags: ["glossary", "ai-agents", "llm", "tokenization", "context-window", "cost"]
status: "stable"
last_updated: 2026-05-14
summary: "A token is the atomic text unit a language model processes; roughly 4 characters or 0.75 English words, though it varies by language and tokenizer."
related:
  [
    "[[glossary/context-window]]",
    "[[glossary/temperature]]",
    "[[glossary/top-p]]",
    "[[prompt-engineering/prompt-design]]",
    "[[glossary/semantic-cache]]",
    "[[ai-agents/embeddings]]",
  ]
---

## Overview

This page is the atomic definition. Prompt budgeting and cost management live at [[prompt-engineering/prompt-design]].

## Definition

A token is the atomic unit of text a language model encodes and processes. Tokenizers (OpenAI uses Byte Pair Encoding via tiktoken; Anthropic uses a similar BPE scheme) split text into subword units. Common English words are a single token; rare words, numbers, and code identifiers are often split into multiple tokens. Rough rule of thumb for English: 1 token per 4 characters, or 100 tokens per 75 words. Chinese, Japanese, and Korean are denser: 1 character may be 1-2 tokens but represents more information. Tokens matter for three reasons: (1) the [[glossary/context-window]] is measured in tokens; (2) API pricing is per-token (input and output priced separately); (3) the model generates one token at a time at inference, so longer outputs cost more and take longer. Count tokens before making API calls using the provider's tokenizer library to budget accurately.

## When it applies

Count tokens when designing prompts that approach the context limit, when estimating API costs, and when implementing streaming responses that should stop at a target length.

## Example

The phrase "ChatGPT is a large language model" tokenizes as:
`Chat`, `G`, `PT`, ` is`, ` a`, ` large`, ` language`, ` model` = 8 tokens.

The word "antidisestablishmentarianism" may tokenize into 6-8 tokens.

## Related concepts

- [[glossary/context-window]] - context windows are measured in tokens.
- [[glossary/temperature]] - temperature controls the probability distribution over the next token.
- [[glossary/top-p]] - top-p filters the token vocabulary before sampling.
- [[prompt-engineering/prompt-design]] - prompt budgeting starts with accurate token counts.
- [[glossary/semantic-cache]] - cache hit rates depend on prompt token stability.

## Citing this term

> See [[glossary/token|Token]] (llmbestpractices.com/glossary/token).

## Related

- [[glossary/context-window]]
- [[glossary/temperature]]
- [[glossary/top-p]]
- [[prompt-engineering/prompt-design]]
- [[glossary/semantic-cache]]