Token
A token is the smallest unit of text a language model actually reads, writes, and is billed for. It's usually a chunk smaller than a word — a common word, a word fragment, or a piece of punctuation. Models like ChatGPT, Claude, and Gemini don't process letters or sentences directly; they break everything down into tokens first, run their computation on those, then reassemble tokens back into readable text.
Why tokens matter for your visibility
Every question a user sends an AI assistant and every answer it returns is counted in tokens, and every model has a context window — a hard cap on how many tokens it can hold in a single exchange, covering the prompt, any retrieved pages, and the reply together. If your page is bloated with throat-clearing, repeated boilerplate, or nested qualifiers, it burns through that budget before it reaches the part that actually answers the question, and gets truncated or skipped. This isn't really a special "AI trick": it's the same reason a page cluttered with fluff underperforms with human readers too. Writing tightly, with the answer stated plainly near the top, means more of your actual content survives into whatever an assistant reads and cites.
How tokenization works
Before a model can process any text, a tokenizer splits it into pieces using a fixed vocabulary learned from training data. Common words often become a single token; rarer or longer words, and most non-English text, get split into multiple pieces. A compound German word like Suchmaschinenoptimierung can break into several tokens, while a short common word like und stays whole — which is one reason German and other morphologically dense languages tend to need noticeably more tokens than English to say the same thing. Each token is converted into a numeric vector that the model's neural network operates on, and the model then generates output one token at a time, each new token conditioned on everything before it. An answer is assembled piece by piece, not produced as a finished sentence in one step.
Common mistakes and misconceptions
The most common error is treating tokens as interchangeable with words — they're usually smaller, and the ratio varies by language. A second misconception is assuming a model can absorb text of unlimited length; in reality the context window is a fixed ceiling that covers input and output combined, and content that doesn't fit gets dropped or forces the model to lose track of earlier material. A third mistake, increasingly common in 2026, is assuming that adding AI-specific markup or a special file will help a model "read" your tokens more favorably. It won't: Google has stated plainly that no special schema, markup, or AI-facing file is required for its AI features, and warns against writing separate content "for AI." The fix for AI visibility is the same discipline that helps any careful reader: put the direct answer early, in plain language, and cut the padding that competes for the same limited token budget.
Relation to AI recommendations and GEO
In generative engine optimization, tokens are the actual unit an assistant is working with when it decides what to cite. A model (or the retrieval system feeding it) pulls in passages, converts them to tokens, and builds an answer from whatever fits. Short, fact-dense paragraphs that name concrete entities — a place, a service, a price — tokenize efficiently and are easier to lift cleanly into a response. Cost reinforces this: providers bill by the token, so systems have a built-in incentive to favor sources that deliver a complete answer economically. None of this replaces the more durable levers behind AI citation, like being mentioned across the web in the first place — but writing compactly means the tokens you do get counted for are the ones doing real work.
Example
Picture a small tax firm in Freiburg that advises startup founders. When someone asks an AI assistant for a tax advisor for founders in that city, the model tokenizes both the question and whatever website text it has pulled in. If the firm states its core facts — startup founding, Freiburg, initial consultation — compactly and near the top of the page, that handful of tokens fits easily into the answer's budget. A page that buries the same facts under paragraphs of scene-setting costs far more tokens to say the same thing, and is more likely to be trimmed or passed over entirely. Compact, direct writing wins the token budget.
Common questions
How many tokens is a typical sentence?
For English text, a rough rule of thumb is about 0.75 words per token, so a 15-word sentence runs roughly 20 tokens. For German and other languages with long compound words, the ratio is worse — expect noticeably more tokens for the same sentence, since compounds and rare technical terms get split into several pieces.
Why are AI models billed by the token instead of by the word or request?
Because the token, not the word, is the unit the model actually computes on. Every token processed as input and every token generated as output consumes computing power, so providers like OpenAI and Anthropic price their models per thousand tokens, typically with separate rates for input and output.