Tokens & Tokenization
|
This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production. This section’s bibliography lists the reference material consulted while preparing these pages. |
Examples on this page target Python 3.12, tiktoken 0.14.0, Hugging Face transformers 5.17.0 (with
tokenizers 0.23.2) and the Anthropic Python SDK against claude-sonnet-5, checked as current on 2026-09-28. A
transformer model never reads text directly: every input is first cut into tokens — sub-word chunks drawn
from a fixed vocabulary — and each token is looked up as an integer id before it reaches the model’s
embedding layer. This page covers how that split is produced, how to count tokens ahead of time, and why the
count matters beyond curiosity.
Why sub-word tokens
A tokenizer could in principle split text into whole words or into individual characters, but both extremes have problems. A word-level vocabulary needs an entry for every word form in every language the model will ever see, still fails on a word it has never encountered (an out-of-vocabulary word, typo, or brand-new product name), and wastes vocabulary slots on rare inflections. A character-level vocabulary never has that out-of-vocabulary problem, but it turns every sentence into a very long sequence of tiny units, which costs more compute per unit of meaning and makes it harder for the model to represent whole words as a single step.
Sub-word tokenization sits between the two: common words become a single token, rarer words split into a handful of frequent pieces, and a token sequence never fails outright on unseen text — in the worst case it falls back to individual bytes or characters. All of the tokenizer families below are different algorithms for choosing which sub-word pieces belong in the vocabulary.
BPE, WordPiece, SentencePiece and byte-level tokenizers
Byte Pair Encoding (BPE)
Byte Pair Encoding (Sennrich et al., 2016) is the tokenizer family behind GPT-style models. Training starts from a vocabulary of individual characters (or bytes) and repeatedly merges the most frequent adjacent pair of symbols into a new, single symbol, adding it to the vocabulary. After enough merges, common words end up as one token each, while rarer words remain split into two or more of the merged pieces. Encoding a new piece of text applies the same merge rules, in the order they were learned, until no more merges apply.
Byte-level BPE
A plain, character-level BPE vocabulary still needs an escape hatch for characters it never saw during
training (an emoji, an unfamiliar script). Byte-level BPE, used by GPT-2 and its successors, sidesteps this
by starting the merge process from the 256 possible byte values rather than characters: every possible input,
in any language or encoding, is representable from the start, because any string is a sequence of bytes.
tiktoken (below) is a byte-level BPE implementation.
WordPiece
WordPiece (used by BERT and its descendants) trains similarly to BPE — merging pairs of symbols iteratively — but scores each candidate merge by how much it increases the likelihood of the training corpus under a language-model objective, rather than by raw pair frequency alone. In practice, the resulting vocabularies and behaviour are close to BPE’s; the two are usually treated as siblings rather than as meaningfully different in outcome.
SentencePiece
SentencePiece is not a competing merge algorithm so much as a different starting point: it treats the input as a raw stream of Unicode characters, including spaces, instead of pre-splitting on whitespace first. That makes it well suited to languages that do not use spaces to separate words (Japanese, Chinese, Thai), where a whitespace pre-tokenization step would be meaningless. SentencePiece can run either a BPE-style or a unigram-language-model algorithm on top of that raw character stream, and it is the library several multilingual and non-English-first model families (for example Llama’s and T5’s tokenizers) build on.
| Family | Typical models |
|---|---|
BPE (byte-level) |
GPT-2/3/4-family, GPT-OSS (via |
WordPiece |
BERT and its descendants |
SentencePiece (BPE or unigram) |
Llama, T5, many multilingual models |
Vocabulary and special tokens
A tokenizer’s vocabulary is the fixed table mapping every token string to an integer id; its size is set at training time (commonly in the tens of thousands to around 200,000 entries for current frontier models) and never changes at inference time — a model cannot invent a new token id after training. Alongside ordinary sub-word entries, every vocabulary reserves a handful of special tokens that carry structural meaning rather than text content, for example a beginning-of-sequence marker, an end-of-sequence marker, a padding token used to line up batched sequences of different lengths, and an unknown-token fallback for whatever byte-level BPE and SentencePiece are specifically designed to make unnecessary. Chat-tuned models add further special tokens of their own to mark where a turn starts and which role is speaking, which is exactly what a chat template (next) is responsible for inserting correctly.
Chat templates
A chat-tuned model is trained on conversations formatted with a specific, model-specific arrangement of
special tokens and role markers — one model’s assistant turn might start with <|assistant|> and another’s
with [/INST], and using the wrong one measurably degrades output quality even though the underlying token
vocabulary is unaffected. Hugging Face’s apply_chat_template method reads a model’s chat template (a Jinja
template stored alongside the tokenizer) and applies it to a plain list of role/content dictionaries, so
callers never have to hard-code a specific model’s control tokens by hand:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta")
messages = [
{"role": "system", "content": "You are DocsAssistant, a documentation Q&A assistant."},
{"role": "user", "content": "What does ef_search control in an HNSW index?"},
]
# add_generation_prompt=True appends the tokens that open the model's own turn
prompt_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(prompt_text)
# tokenize=True returns a dict (input_ids, attention_mask) for model.generate(**inputs)
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")add_generation_prompt=True appends the tokens that signal "the assistant’s turn starts here"; without it, a
base or chat model can misread the prompt as still belonging to the user turn and continue it instead of
replying. See the official
Hugging Face chat templates guide for
apply_chat_template, add_generation_prompt and the related continue_final_message argument used for
response prefilling.
Hosted chat APIs (Anthropic’s Messages API, the OpenAI-compatible Chat Completions API Ollama also serves)
apply their own equivalent template server-side: the caller sends a list of role/content messages and the
provider inserts its own special tokens before running the model, so apply_chat_template is mainly relevant
when working directly with an open-weight model’s own tokenizer, not when calling a hosted API.
Tokens vs. words across languages and code
A token is not a word, and how many tokens a piece of text costs depends heavily on what the text is, not just how many words a human would count:
-
English prose tokenizes close to one token per word for common vocabulary, plus extra tokens for punctuation and for rarer or compound words that split into multiple pieces.
-
Non-English languages, especially those under-represented in a tokenizer’s training data, routinely cost more tokens per word than English for the same content, because their words are less likely to have earned a single dedicated merged token and instead split into several sub-word pieces (or, for byte-level BPE, multiple bytes per character for non-Latin scripts).
-
Code tokenizes differently again: indentation, punctuation-heavy syntax (
{,(,::,→) and identifiers insnake_caseorcamelCaseeach contribute their own tokens, so a short code snippet can cost noticeably more tokens than an English sentence of similar character length.
The following table counts the same idea expressed three ways — an English sentence, its Spanish translation,
and an equivalent Python snippet — using tiktoken 0.14.0’s o200k_base encoding (the encoding behind
current GPT-4o-family models), checked on 2026-09-28:
| Text | Content | Words | Tokens (o200k_base) |
|---|---|---|---|
English |
"How do I speed up similarity search in a vector database?" |
11 |
12 |
Spanish |
"¿Cómo acelero la búsqueda por similitud en una base de datos vectorial?" |
12 |
16 |
Code |
|
11 |
21 |
The Spanish sentence needs roughly a third more tokens than its English original despite having one more word, and the code snippet — similar in length to the English sentence — costs almost twice as many tokens, driven by identifiers, parentheses and operators that each consume their own token. This is why cost and context-window budgets (below) cannot be estimated from a word count alone, and why a multilingual or code-heavy workload should always be measured with the actual tokenizer rather than assumed to match English.
Reproducing the table:
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
english = "How do I speed up similarity search in a vector database?"
spanish = "¿Cómo acelero la búsqueda por similitud en una base de datos vectorial?"
code = "def cosine_similarity(a, b): return sum(x*y for x, y in zip(a, b))"
for label, text in [("english", english), ("spanish", spanish), ("code", code)]:
print(label, len(enc.encode(text)))Counting tokens with tiktoken
tiktoken is OpenAI’s own byte-level BPE tokenizer, usable standalone to count or inspect tokens without
loading a full model. encoding_for_model picks the right encoding for a given model name, and o200k_base is
the encoding current GPT-4o-family and GPT-5-family models use (older GPT-4/GPT-3.5-family models used
cl100k_base):
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
text = "DocsAssistant answers questions about HNSW indexes and pgvector."
tokens = enc.encode(text)
print(len(tokens), "tokens")
print(tokens[:5]) # token ids
print(enc.decode_single_token_bytes(tokens[0])) # bytes of the first tokenSee the official tiktoken repository and README for get_encoding,
encoding_for_model and the encode/decode API this example uses.
Counting tokens with Hugging Face AutoTokenizer
AutoTokenizer loads the tokenizer that shipped with a given Hugging Face model repository, so tokenize and
encode reflect that specific model’s own vocabulary rather than a generic approximation:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
text = "DocsAssistant answers questions about HNSW indexes and pgvector."
pieces = tokenizer.tokenize(text) # human-readable sub-word pieces
ids = tokenizer.encode(text) # integer token ids, including special tokens
print(len(pieces), pieces[:6])
print(len(ids))tokenize returns the sub-word strings themselves (useful for inspecting how a tokenizer split the text),
while encode returns the integer ids the model actually consumes, with special tokens (such as a
beginning-of-sequence marker) added according to that tokenizer’s configuration. See the official
Hugging Face AutoTokenizer /
PreTrainedTokenizer API reference for tokenize, encode and the rest of the tokenizer surface, and
Chat templates above for apply_chat_template, the same class’s chat-formatting method.
Counting tokens with the Anthropic API
Anthropic exposes a dedicated token-counting endpoint, client.messages.count_tokens, which accepts the same
message structure the Messages API itself takes (system prompt, messages, tools) and returns only the input
token count, at no cost and without generating a response — useful for checking a prompt against a context
window or a cost estimate before actually sending it:
import anthropic
client = anthropic.Anthropic()
response = client.messages.count_tokens(
model="claude-sonnet-5",
system="You are DocsAssistant, a documentation Q&A assistant.",
messages=[{"role": "user", "content": "What does ef_search control in an HNSW index?"}],
)
print(response.input_tokens)The count is model-specific: Anthropic’s newer model generations use a different tokenizer than older ones, so
the same text can produce a noticeably different count depending on the model passed, and a count taken
against one model should not be reused to estimate cost or context-window fit for another. See the official
Anthropic token counting guide and the
Messages token-counting API reference for the
exact request and response shape.
Tokens, cost and latency
Hosted chat APIs price a request by the number of input tokens (the prompt, including any system prompt and prior turns) and output tokens (the generated response) separately, usually at different per-token rates since generating a token costs more compute than reading one. For a request with input tokens billed at price per token and output tokens billed at price per token, the cost of that one request is:
Token count also drives latency, independently of price: a longer input takes longer to process before the
model can start generating, and a longer requested output takes longer to stream back token by token, since
decoder-only models generate one token at a time. Cost,
Latency & Throughput develops this into the full time-to-first-token / time-per-output-token model and a
worked cost-calculator example; this page’s point is narrower — that the token counts above (or a
count_tokens call, or a tiktoken/AutoTokenizer count) are the actual inputs to both formulas, not an
afterthought computed once a bill arrives.
From token ids to embeddings
Counting and encoding stop at integer token ids — the tokenizer’s job ends there. Inside the model, an embedding layer then looks up each token id in a learned table and turns it into a dense vector before any attention or feed-forward computation happens; Embeddings covers what that vector represents and how embedding models (a related but distinct use of the same underlying representation) are trained to place similar meanings close together in vector space.