Token
A token is the unit a language model reads and writes: a fragment of text, usually a word, part of a word or a punctuation mark, mapped to a number the model computes with. Prompts, fetched pages and answers are all measured, limited and priced in tokens.
How text is cut up
A tokenizer splits text against a fixed vocabulary learned from the training corpus. Common words become one token; uncommon words, names and technical terms are split into several pieces; a space or a capital letter can change the split. Anthropic’s glossary puts one of its tokens at roughly three and a half English characters, with the figure varying by language. The model never sees letters. It sees the sequence of token numbers, and it produces its answer one token at a time.
The page an assistant reads is a token budget
Everything an assistant holds at once, its instructions, the conversation and every page it retrieved, has to fit inside a context window counted in tokens. A long page is therefore cut, summarized or sampled before the model reasons over it, and whoever wrote the page does not choose where the cut falls. A homepage that opens with a slogan and reaches the address, the price and the service list only after several screens has spent its share of the budget on the slogan. The sentence you want quoted should not depend on the model having read everything before it.
A common question
Why does the position of a fact on a long page matter to a language model?
Everything an assistant holds at once, its instructions, the conversation and each page it fetched, has to fit a context window counted in tokens. A long page is therefore cut, summarized or sampled before the model reasons over it, and whoever wrote the page does not choose where the cut lands. A homepage that spends its opening screens on a slogan before reaching the address, the price and the service list has spent its budget on the slogan. The sentence you want quoted should not depend on everything above it being read.
Related terms
Sources
Tokenisation and the character estimate: Anthropic’s glossary of model terms. No audit figures on this page.