Tokens and Tokenization: Why AI Reads Text Differently Than You Do

A language model never sees the letters you typed. Before your prompt reaches the model, a separate piece of software chops it into chunks called tokens — sometimes a whole word, often a fragment, occasionally a single byte — and hands the model a list of ID numbers. The model does all its "reading" and "writing" in that alien vocabulary and only converts back to human text at the very end. Tokenization is the layer where your words stop being words.

This sounds like a pedantic implementation detail. It is not. Tokenization quietly decides what you pay, how much text fits in the context window, why the same idea costs more in Hindi than in English, and why an otherwise brilliant model insists there are two R's in "strawberry." Once you can see tokens, a whole category of AI weirdness stops being mysterious and starts being predictable. This post is about learning to see them.

Key takeaways
What a token actually is
Byte-pair encoding, the algorithm doing the chopping
The vocabulary and merge table: what actually gets stored
Bytes, characters, or words: why subwords won
The tokenizer family tree: BPE, WordPiece, Unigram, SentencePiece
Special tokens and the chat template
The four-characters rule of thumb
The non-English tax
Why tokens govern cost and context limits
Token healing and prompt boundaries
The "strawberry" problem, explained
Glitch tokens: the vocabulary's haunted houses
Common misconceptions
The future: byte-level and tokenizer-free models
How to actually look at tokens
Where tokenization touches everything else
FAQ

Key takeaways

A token is a chunk of text, not a word or a letter. Common words are usually one token; rare or long words get split into several pieces. As a rough English rule of thumb, one token is about four characters, or roughly 0.75 words.
Tokenization happens before the model and is fixed at training time. The model's entire vocabulary — often on the order of 100,000–200,000 tokens — is frozen. Everything you send gets encoded into that vocabulary, including emoji, code, and languages the model barely saw.
You are billed in tokens, and limited in tokens. Both API pricing and the context window are measured in tokens, not words or characters. Your prompt's token count, not its word count, is what actually matters.
The tokenizer is a separate program, not part of the model. It's a deterministic, reversible lookup — a vocabulary plus, for byte-pair encoding, an ordered merge table — that connects to the network only through the embedding table. That's why you can count tokens before ever calling the model.
The tokenizer explains real bugs. Character-counting tasks ("how many R's"), shaky arithmetic, non-English cost penalties, mangled rare words, and even "glitch tokens" all trace back to how text got split.
You can inspect it. Tokenizer playgrounds and counting libraries let you see the exact split, which is the fastest way to debug cost and context surprises.

What a token actually is

Start with the problem tokenization solves. A model needs a fixed-size vocabulary — a finite list of symbols it knows. Two obvious approaches both fail.

You could make every word a token. But then the vocabulary is effectively infinite (names, typos, URLs, variable_names, new slang) and the model is helpless the moment it meets a word it never saw. You could go the other way and make every character a token, giving a tiny vocabulary that handles anything. But then even a short paragraph becomes a very long sequence of symbols, and the model has to work far harder to learn that t-h-e means "the."

Tokenization is the compromise in between. It builds a vocabulary of subword pieces: whole common words stay whole, and rarer words get broken into fragments the model has seen before. So the is one token, tokenization might split into token + ization, and a genuinely novel string like flarnbxq falls apart into tiny pieces, possibly individual bytes. Nothing is ever truly out-of-vocabulary, because in the worst case the model can always fall back to raw bytes.

The result is a list of integers. The word "reading" might become token 25,481; "▁the" (with a leading-space marker) might be token 279. The model only ever manipulates those IDs. When people say a model "understands" text, what physically enters the network is a sequence of these numbers, each mapped to a learned vector. (What happens to those vectors — attention, prediction — is a separate story; see how AI chatbots work.)

It helps to be precise about the plumbing here, because the single most common misconception is that the tokenizer is part of the model. It is not. The tokenizer is a separate program with its own file, and it does no learning at inference time. Encoding is a deterministic, reversible lookup: text goes in, a fixed list of integers comes out, and decoding those same integers reproduces the original text byte-for-byte. There is no probability, no neural network, no cleverness in this step — just a table and a set of rules applied greedily. You could run the tokenizer with the model deleted and it would still turn your sentence into the same numbers. This separation is why you can count tokens before you ever call the model, and why two different models built on the same tokenizer produce identical token counts for the same string.

What connects those integers to the actual neural network is the embedding table — a large matrix with one row per token in the vocabulary. Token 279 doesn't enter the model as the number 279; the number is used as an index to pull out row 279, a learned vector of a few thousand values. That vector is the model's actual internal representation of the token, shaped over the whole of training. So the full journey is: text → (tokenizer) → integer IDs → (embedding lookup) → vectors → (the network) → a probability distribution over the next token → (sampling) → a new integer → (tokenizer, in reverse) → text. Every arrow that touches the tokenizer is mechanical bookkeeping; all of the intelligence lives between the embedding table and the output. Tokenization is the packaging, not the product — but the shape of the packaging constrains everything inside it, which is the whole reason this article exists.

Byte-pair encoding, the algorithm doing the chopping

Most modern models use a variant of byte-pair encoding (BPE) or a close cousin. The idea is refreshingly simple, and understanding it once demystifies the whole thing.

BPE learns its vocabulary from a big pile of training text using a greedy, bottom-up merge process:

Start from the smallest units. Begin with individual bytes or characters as the base vocabulary. Every possible input can be represented from these alone.
Count adjacent pairs. Scan the corpus and find the most frequently occurring pair of neighboring symbols. Early on, that might be t followed by h.
Merge it. Add th to the vocabulary as a new single token and replace every t+h in the corpus with it.
Repeat. Count again. Now maybe th+e is the most common pair, so the becomes a token. Keep merging — in, ing, ation, common whole words — until you hit a target vocabulary size.

The genius is that frequency drives granularity. Text that appears constantly gets compressed into large, efficient tokens. Text that's rare never earns a merge and stays fragmented. The model's vocabulary is therefore a compression scheme tuned to whatever data it was trained on — overwhelmingly English web text, code, and a long tail of everything else.

It's worth walking through a tiny worked example, because the mechanism is genuinely this concrete. Suppose your entire training corpus is the repeated word low low low lower lowest. Start with characters: l o w, l o w e r, l o w e s t. The most frequent adjacent pair is l+o, appearing five times, so you merge it into lo. Now the most frequent pair is lo+w, so you merge that into low. In two merges the token low exists, and it appears in all three words. Next the corpus contains low e r and low e s t; you might merge e+r into er, giving low + er. The vocabulary you end up with — l, o, w, e, r, s, t, lo, low, er — plus the ordered list of merges you performed, is the entire tokenizer. That ordered list is the crucial artifact: at encoding time you re-apply exactly those merges, in exactly that order, to any new text. Real tokenizers do this over hundreds of billions of characters and tens of thousands of merges, but the loop is identical.

One subtlety that trips people up: BPE merges never cross certain boundaries. Before merging begins, most tokenizers pre-tokenize the text with a regular expression that splits on whitespace, punctuation, and often the seams between letters, digits, and symbols. Merges then happen only within those pre-token chunks. This is why hello and world never fuse into a single helloworld token no matter how often they co-occur, and why a leading space usually gets glued to the front of the word that follows it rather than standing alone. The pre-tokenizer's regex is an underappreciated design choice — it is, for instance, why some tokenizers split runs of digits into groups of three and others split every digit, with real consequences for arithmetic.

That last point is the source of most tokenization surprises. The tokenizer is not neutral. It is biased toward the statistics of its training corpus and the hand-tuned rules of its pre-tokenizer, and that bias leaks into cost, speed, and behavior.

The vocabulary and merge table: what actually gets stored

If the tokenizer is a separate program, what does that program physically contain? Two things, and it's worth being able to picture them.

The first is the vocabulary: a flat dictionary mapping every known token to an integer ID. Token the → 279, token ▁straw → 496 (that ▁ is a visible stand-in for a leading space, a convention some tokenizers use), and so on, up to whatever the vocabulary size is — commonly somewhere in the range of 30,000 for older models to 100,000–200,000 or more for recent ones. This dictionary is finite and frozen. Every token the model will ever read or write is one of these entries; there is no entry number 200,001, and there never will be for that model.

The second, in a BPE tokenizer, is the merge table: the ordered list of pair-merges the training process discovered, from first to last. This is what makes encoding deterministic. To tokenize new text, the tokenizer splits it into characters (or bytes), then walks the merge list top to bottom, applying each merge everywhere it can before moving to the next. Because the list is ordered by the order merges were learned — which tracks frequency — the highest-priority merges fire first, and the greedy process converges on the same split every time. Encoding is essentially "replay the training merges against this string."

A few consequences fall out of this design that matter in practice:

The vocabulary is a sunk decision. Vocabulary and merges are fixed before the model trains a single step, because the embedding table has to have exactly one row per token. You cannot add a word to a deployed model's vocabulary the way you'd add a contact to your phone. New jargon, a client's product name, a fresh emoji — all of it gets encoded using the existing pieces, splitting into whatever fragments the frozen table allows.
Bigger vocabulary, fewer tokens, but heavier model. A larger vocabulary compresses text into fewer tokens (cheaper, roomier context) but inflates the embedding and output layers, and spreads training signal more thinly across rare tokens that each appear less often. This is a genuine engineering trade-off, not a free lunch, and it's a big reason different labs land on different vocabulary sizes.
The tokenizer ships with the model. When you download an open-weights model, the tokenizer files (tokenizer.json, vocab, merges) come alongside the weights. Use the wrong tokenizer with a set of weights and you get fluent-looking garbage, because the integer IDs no longer point at the rows the model was trained to expect. The pairing is exact.

Bytes, characters, or words: why subwords won

We said word-level and character-level tokenization both fail, and subwords are the compromise. It's worth being concrete about the failure modes, because they explain design choices in every modern tokenizer.

Word-level tokenization — one token per whitespace-separated word — was common in older natural-language systems. Its fatal flaw is the open vocabulary problem: language generates new "words" endlessly (covfefe, n=42, x_train_final_v2, a misspelling, a brand new hashtag), and a word-level model meets these as a single "unknown" token, throwing away all information. It also can't share anything between run, runs, running, and runner, which any human sees as related.

Character-level tokenization solves coverage perfectly — every string is representable — but at a punishing cost: sequences become very long (roughly 4× longer than subword sequences for English), and since the compute of attention grows with sequence length, you pay dearly for it. The model also has to spend capacity relearning that t-h-e is a unit, essentially rediscovering subwords from scratch inside its weights.

Subword tokenization threads the needle: frequent whole words stay as single tokens (short sequences, like word-level), while anything unusual decomposes into smaller known pieces (full coverage, like character-level). The remaining question is what the smallest unit should be — the floor you can never fall through.

Here byte-level BPE is the elegant answer used by most current models. Instead of Unicode characters, the base vocabulary is the 256 possible bytes. Because any text in any language, plus emoji, plus arbitrary binary-ish content, is ultimately a sequence of bytes, this guarantees there is no such thing as a truly out-of-vocabulary input — the worst case is that an exotic string falls back to several single-byte tokens. This is why you never see a modern chat model choke with an "unknown token" error on a weird symbol: it just spends more tokens. A single emoji or an uncommon CJK character often costs three or four byte-tokens, since it occupies multiple bytes in UTF-8 and the tokenizer never learned to merge them. Elegant coverage, uneven cost — a theme we'll keep hitting.

The tokenizer family tree: BPE, WordPiece, Unigram, SentencePiece

"Tokenization" is not one algorithm. There are three main training schemes and one influential toolkit, and knowing the differences saves you from thinking every model chops text the same way.

BPE (byte-pair encoding). The greedy merge process described above. Build a vocabulary by repeatedly fusing the most frequent adjacent pair. Encoding replays the merges. This is the workhorse behind the GPT family and many open models (usually in its byte-level form). Its defining trait is that the merge order is the tokenizer — encoding is rule-replay, not search.

WordPiece. Introduced with BERT, WordPiece looks almost like BPE but chooses each merge by a slightly different criterion: instead of picking the most frequent pair, it picks the pair that most increases the likelihood of the training data — roughly, the merge whose combined token is more informative than its parts sitting separately. At encoding time WordPiece is typically greedy longest-match: starting from the front of a word, take the longest vocabulary entry that fits, mark continuation pieces with a ## prefix (token, ##ization), and repeat. Different objective, similar spirit.

Unigram language model. The odd one out, and the default behind many multilingual and non-English models. Unigram works top-down: it starts with a large candidate vocabulary and iteratively removes tokens that contribute least, keeping a probability for each surviving token. Crucially, for any given string there are many possible segmentations, and Unigram scores them — at encoding time it picks the most probable segmentation rather than a fixed greedy one. This probabilistic framing makes it easy to sample alternative tokenizations during training (a regularization trick called subword regularization) and tends to produce cleaner splits for morphologically rich languages.

SentencePiece is not a fourth algorithm but a toolkit — it can train either BPE or Unigram — and its important contribution is treating the input as a raw stream of Unicode with no assumptions about whitespace. It encodes the space itself as a visible symbol (the ▁ you keep seeing) so that detokenization is perfectly reversible and language-agnostic, which matters enormously for languages like Chinese and Japanese that don't put spaces between words. When you see ▁ in a token dump, you're almost certainly looking at a SentencePiece-trained tokenizer.

The practical upshot: token counts, and the exact way a word splits, are not portable across model families. A prompt that measures 3,000 tokens on a BPE tokenizer may be 3,300 on a Unigram one, and a word that stays whole in one may fragment in another. Never assume; measure against the specific model you're using.

Special tokens and the chat template

Not every token corresponds to text you typed. Tokenizers reserve a set of special tokens — control symbols the model uses to structure a conversation — and these are as load-bearing as any word.

The classic ones are sequence markers: a beginning-of-sequence token, an end-of-sequence token, and a padding token used to make batches uniform. Modern chat models add role and turn markers — special tokens that mean "the system instruction starts here," "the user is speaking," "the assistant's turn begins," "this turn is over." When you use a chat API and pass a tidy list of system, user, and assistant messages, the provider silently renders them into one long string studded with these special tokens, following a fixed chat template. The model was trained on exactly that format, which is how it knows where your instruction ends and where it's supposed to start generating.

Three things follow that are easy to miss:

Special tokens count against your budget and your bill. Every turn marker, every role header, every wrapper the template adds is real tokens. This is part of why a multi-turn conversation costs more than the visible text suggests, and why very short back-and-forth messages have surprisingly high per-message overhead.
The end-of-turn token is how generation stops. When a model "decides" it's finished answering, what physically happens is it emits the end-of-turn special token, and the serving software stops. Get the template wrong — wrong markers, missing end token — and models ramble, impersonate the user, or refuse to stop. A large share of "my local model won't shut up" bug reports are chat-template mismatches, not model problems.
Special tokens are a security surface. Because these tokens carry structural authority, letting raw user text inject the literal string of a role marker is one flavor of prompt injection. Good tokenizers refuse to encode the special strings from ordinary user input, encoding the characters as plain text instead — but it's a real boundary worth knowing exists.

The four-characters rule of thumb

You don't need to run the tokenizer to estimate size. For ordinary English prose, these approximations hold well enough for planning:

Unit	Rough token count
1 token	~4 characters of English
1 token	~0.75 words
100 tokens	~75 words
1,000 words	~1,300–1,400 tokens
A dense page of text	~500–800 tokens

These are English averages, and the operative word is rough. Whitespace, punctuation, and capitalization all consume tokens. Code, with its brackets, indentation, and snake_case identifiers, tends to run token-heavy. Numbers are notorious: depending on the tokenizer, a long number like 31415926 may split into several chunks in ways that have nothing to do with mathematical place value — one reason arithmetic can be shaky. And any non-English text can blow these estimates apart, which brings us to the most consequential quirk.

The non-English tax

Because BPE learns its merges from a mostly-English corpus, English gets the most efficient encoding. Common English words are single tokens. Other languages are not so lucky.

A language written in a non-Latin script — or simply one that appeared less often in training — gets encoded into more, smaller tokens for the same meaning. A sentence that is 10 tokens in English might be 20, 30, or more tokens translated into a lower-resource language, even when the human-visible length is similar. Scripts like Chinese, Japanese, Korean, Arabic, and many Indic and African languages routinely pay this penalty, sometimes falling all the way down to multiple tokens per character.

This has three concrete consequences, and none of them are cosmetic:

Cost. You pay per token. The same conversation can cost several times more in one language than another. Speakers of lower-resource languages effectively subsidize the efficiency English speakers enjoy.
Context. More tokens per sentence means fewer sentences fit in a fixed context window. A document that fits comfortably in English may overflow when translated.
Latency. Models generate one token at a time, so more tokens per response means slower answers.

There's a mechanical reason the penalty is so steep for non-Latin scripts specifically, and it comes straight from byte-level BPE. Latin letters are one byte each in UTF-8, so English had a low floor to begin with, and then earned thousands of merges on top. A character like 한 or 日 or न occupies three bytes in UTF-8, and if the tokenizer never learned merges to fuse those byte-sequences back into whole characters (because the script was underrepresented in training), a single visually-simple character can cost three tokens before you even reach the word or morpheme level. You are, in effect, paying to spell out each character byte by byte. So the tax compounds: a low base rate for Latin, plus rich merges for English words, versus a high base rate for multi-byte scripts and few merges to offset it.

None of this reflects the model being "worse" at the language in some deep cognitive sense — it's the encoding that's inefficient. But the effect is real money and real limits, and it's the clearest example of tokenization being a hidden tax rather than a neutral pipe. It also has a subtle downstream effect worth naming: because rare-language text fragments into more, smaller pieces, the model has to spread its "attention" over a longer, noisier sequence to represent the same idea, which can make it genuinely harder to reason over — so the encoding penalty and a mild capability penalty sometimes travel together, even though they're different problems.

Why tokens govern cost and context limits

Two of the most practical numbers in working with any model are both denominated in tokens.

Pricing. API providers charge per token, almost always with separate rates for input (the tokens you send) and output (the tokens the model generates). Output is typically several times more expensive than input, because generation is the sequential, compute-heavy part. This is why a chatty system prompt you resend on every request quietly dominates your bill, and why "summarize this into three bullets" is cheap on output but potentially expensive on input if the thing being summarized is huge. If you're trying to reason about spend, the unit that matters is tokens, and the deeper economics — batching, caching, why output costs more — are worth understanding on their own; see AI inference cost economics.

The context window. A model's advertised context length — whether it's tens of thousands or millions — is a token budget, not a character or word budget. Everything counts against it: the system prompt, the conversation history, retrieved documents, tool definitions, the user's message, and the space reserved for the reply. When people say a long chat "forgot" something from earlier, what often happened is the token budget filled up and the oldest content got dropped or summarized away. The word "context window" makes it sound like memory; it's really a token ledger with a hard ceiling.

It's worth doing the arithmetic once, because it reframes how you think about a chat product. Say a system prompt is 1,500 tokens and you send it on every one of 50 turns in a conversation. That's 75,000 input tokens spent on the same instructions, before counting a single word the user or model actually said. Meanwhile, the conversation history grows with every turn: turn 20 re-sends turns 1 through 19 as input, so cost per turn climbs roughly linearly and the total cost of a long chat grows closer to quadratically than linearly. This is the real economics behind three things you may have noticed — why providers push prompt caching so hard (cache the fixed prefix and stop paying to re-read it), why long conversations feel like they get more expensive as they go, and why trimming a bloated system prompt pays off on every future request. The unit that makes all of this legible is the token.

A useful mental habit: whenever you think "how long is this," retrain yourself to ask "how many tokens is this." Word count and character count are proxies that break exactly when it matters most — with code, numbers, and non-English text.

Token healing and prompt boundaries

Here's a failure mode almost nobody warns you about, and it comes from the seam where your prompt ends and generation begins.

Recall that tokenization is greedy and that spaces are usually glued to the front of the following word. Now imagine you prompt a model to complete a URL and your prompt ends in https: — or you end a prompt with a trailing space, or mid-word. The tokenizer encodes your prompt as one fixed sequence of tokens, and the model then predicts the next token from there. But the token your prompt happened to end on may not be the token the model would naturally have chosen as the start of the continuation. If the natural next chunk was ://www, but your prompt already committed to the token boundary right after :, the model is now forced onto an awkward split it rarely saw in training, and completions get subtly worse — stray characters, doubled punctuation, degraded quality right at the boundary.

The fix, implemented by some libraries, is called token healing: before generating, back up over the last token or two of the prompt and let the model re-choose them together with the continuation, so the boundary lands where the tokenizer would naturally have put it. You don't usually control this directly in a hosted chat API, but the lesson generalizes and is very actionable: end your prompts at natural boundaries. Don't leave a trailing space, don't stop mid-word, and be wary of building prompts by naive string concatenation that leaves a token straddling the seam. A prompt that ends cleanly on a word or a newline gives the model the boundary it was trained to expect. It's a small habit that quietly removes a class of "why is the output slightly mangled at the start" bugs.

The "strawberry" problem, explained

The most famous demonstration of the tokenizer's fingerprints is asking a model to count the letters in a word. "How many R's are in strawberry?" and watching a capable model confidently answer wrong.

The reason is almost entirely tokenization. To the model, "strawberry" is not the ten letters s-t-r-a-w-b-e-r-r-y. It's a small handful of tokens — something like straw + berry, or str + aw + berry, depending on the tokenizer. The individual letters have been fused into chunks before the model ever sees them. Asking it to count R's is like asking you to count the pen strokes in a word you're reading at a glance — the information has been abstracted away at the input layer.

The model can sometimes recover by reasoning character-by-character if prompted to spell the word out first, and newer reasoning-heavy models often get it right precisely because they've been trained to decompose the word deliberately. But the underlying difficulty is structural: character-level tasks are hard for a system that fundamentally perceives text in subword chunks. This same root cause explains trouble with reversing strings, counting characters, some rhyming and wordplay tasks, and certain kinds of arithmetic. When a model fails at something a child could do with a pencil, "check the tokenizer" is a shockingly good first guess. It's a different failure mode from a factual hallucination — the model isn't inventing anything, it genuinely can't see the letters.

Arithmetic deserves its own paragraph, because it's the same disease wearing a different coat. Humans do math positionally: we know the 7 in 70 means seventy because of where it sits. But if a tokenizer chops the number 1234567 into chunks like 123 + 4567, or worse, into pieces that don't align to place value at all, the model receives digits pre-grouped in a way that has nothing to do with tens, hundreds, and thousands. It then has to reconstruct place value from tokens that fight the concept. This is why long multiplication is shaky while short sums are fine, and why the specific way a model's tokenizer handles digits — every-digit-separate versus grouped-by-three — measurably changes its arithmetic reliability. It's also why "show your work" or "add these one column at a time" helps: you're forcing a character/digit-level process on top of a subword substrate. None of this means the model can't do math; it means the input representation is working against it, and prompting can partly compensate.

Glitch tokens: the vocabulary's haunted houses

There's a stranger consequence of freezing a vocabulary before training, and it's one of the most revealing bugs in all of language models. Because the vocabulary is built from raw scraped text, it can acquire tokens for strings that appear in the raw corpus statistics but almost never appear in the cleaned training data the model actually learns from. A username from a scraped forum, a fragment of a spam log, an artifact of some data dump — frequent enough during vocabulary construction to earn their own token, then filtered out or vanishingly rare during model training.

The result is a glitch token (or "undertrained token"): a token that exists, has a row in the embedding table, but was seen so little during training that its embedding is essentially random noise. Feed the model one of these and it behaves bizarrely — refusing to repeat the token, spelling something entirely different, spewing unrelated text, or breaking character. The most famous examples were a cluster of tokens (the internet settled on SolidGoldMagikarp as the mascot) that turned out to be scraped usernames; asking the model to repeat them produced surreal, off-the-rails responses. There was nothing wrong with the network's reasoning — it had simply never learned what that particular vector was supposed to mean, because the token pointed at a row nobody ever trained.

Glitch tokens are mostly a curiosity now, since labs actively hunt for and prune them, but they're a perfect teaching case. They prove, concretely, that the tokenizer and the model are separate systems trained on different data, that the vocabulary is frozen and can contain "dead" entries, and that a token is just an index into a table that may or may not have received any learning signal. If you ever see a model melt down on one specific weird string, you've probably found an undertrained token.

Common misconceptions

A few beliefs are common enough, and wrong enough, to call out directly.

"A token is a word." No. A token is a chunk that may be a whole word, a word fragment, a single character, a byte, punctuation, or a space-plus-word. Common words are often one token, but "unbelievable" or "antidisestablishmentarianism" fragment, and a space is frequently part of the token, not a separator between tokens.
"The tokenizer understands language." No. It's a deterministic compression table with zero comprehension. All the understanding is in the model's weights downstream. The tokenizer would happily and identically chop a string of pure gibberish.
"More tokens means the model thought harder." Not for input. Input tokens are just how much text you sent; a fragmented, token-heavy prompt isn't "richer," it's just more expensive. (Output tokens during a reasoning process are a different matter — there, more tokens really can mean more deliberation.)
"Token count is the same across models." No. Different tokenizers give different counts for identical text. Budget and cost estimates must be tied to the specific model.
"I can just add a word to the vocabulary." Not for a trained model. The vocabulary and the embedding table are fixed together; new terms get encoded from existing pieces. Genuinely extending a vocabulary means adding untrained embedding rows and retraining, which is exactly how glitch-token-like problems are born.
"Bigger vocabulary is strictly better." No — it's a trade-off between fewer tokens per text and a heavier model with thinner training signal per rare token. The right size depends on the target languages and domains.
"Tokens are how the model measures meaning." No. Tokens measure text, mechanically. Meaning lives in the vectors the tokens index into.

The future: byte-level and tokenizer-free models

Given that so many rough edges — the strawberry problem, wobbly arithmetic, the non-English tax, glitch tokens — trace back to the tokenizer, an obvious question is: why not get rid of it? Feed the model raw bytes or characters and let it learn its own units. This is an active research direction, and it's worth knowing where it stands so you can read the trend rather than the hype.

The appeal is real. A model that operates directly on bytes has no fixed vocabulary to freeze, no merge table biased toward English, no undertrained tokens, and full character-level vision — it would, in principle, see the three R's in strawberry. It would also treat every language and script on equal footing at the input, dissolving the encoding tax.

The obstacle is just as real: raw bytes make sequences much longer, and the cost of the core attention mechanism grows with sequence length, so naive byte-level models are far more expensive to train and run for the same amount of text. The interesting recent work therefore doesn't remove chunking so much as make it learned and dynamic — architectures that group bytes into patches on the fly, spending more granularity where the content is dense and less where it's predictable, instead of committing to a fixed merge table up front. The bet is that a model can decide its own boundaries better than a frozen table decided during preprocessing.

For now, the honest status is: subword tokenization is still the overwhelming default in production, because it's cheap, simple, deterministic, and good enough almost everywhere. Byte-level and dynamic-chunking approaches are promising and improving, but they haven't displaced BPE at scale. The practical takeaway isn't "tokenization is going away tomorrow" — it's that the specific failures you can trace to tokenization today are understood well enough that people are engineering the layer away. Until they succeed, the layer is worth seeing clearly, which is the entire point of this article.

How to actually look at tokens

The fastest way to build intuition is to stop guessing and start looking. A few durable practices:

Use a tokenizer playground. Model providers and open-source projects publish interactive tools where you paste text and see it split into colored token chunks with a live count. Paste in an English sentence, then the same sentence in another language, then a block of code, then a long number. The visual difference in how they fragment teaches more than any explanation.
Count tokens in code. Tokenizer libraries let you get an exact count before sending a request. This is how you budget context, estimate cost, and avoid the surprise of a request rejected for being over the limit.
Match the tokenizer to the model. Different model families use different tokenizers, so token counts don't transfer perfectly across providers. If you switch models, re-measure; a prompt that was 3,000 tokens on one may be 3,300 on another.
Watch for the leading space. Most tokenizers treat "the" at the start of text and " the" mid-sentence as different tokens, because the space is bundled in. This is why token counts can seem slightly off from naive expectations — spaces live inside tokens, not between them.

Making a habit of measuring is the whole game. Almost every "why did this cost so much" or "why did it truncate" mystery resolves in seconds once you can see the split.

Where tokenization touches everything else

Tokenization isn't an isolated curiosity; it's upstream of most of how you use models. It shapes how you write prompts, because a bloated system prompt is a per-request tax you pay forever and a drain on your context budget. It shapes retrieval pipelines, because the documents you pull in have to fit the same token ledger as everything else, forcing hard choices about chunk size and how much to include. It even shapes model comparisons: a model with a more efficient tokenizer for your language or domain can be effectively cheaper and roomier than a rival with a bigger advertised context window but a wasteful split.

The reason it stays invisible is that it works well enough, most of the time, in English. It's the edges — other languages, code, numbers, character-level tasks, tight budgets — where the seams show. And those edges are exactly where careful practitioners spend their time. Learning to see tokens is one of the highest-leverage bits of AI literacy precisely because the mechanism is simple, fixed, and touches everything downstream of it.

FAQ

What is tokenization in AI? Tokenization is the process of breaking text into small units called tokens — whole words, word fragments, or individual bytes — that a language model can process. Each token is mapped to an ID number, and the model does all of its computation on those numbers rather than on the original letters. It happens before the model runs and uses a fixed vocabulary learned during training, typically via an algorithm like byte-pair encoding.

How many tokens is a word? For typical English, one word is roughly 0.75 to 1 token on average, or put the other way, one token is about 0.75 words and roughly four characters. Common short words are usually a single token; long, rare, or technical words get split into multiple tokens. These are English averages — code, numbers, and non-English languages can use far more tokens for the same amount of visible text.

Why do I get charged per token instead of per word? Because tokens, not words, are the actual unit the model processes. Providers price per token — usually with a lower rate for input and a higher rate for output — because compute cost scales with the number of tokens processed and generated, not with human word counts. It also makes billing consistent across languages, code, and content where "word" isn't well defined.

Why can't AI reliably count the letters in a word? Because it never sees the individual letters. A word like "strawberry" is fused into a couple of subword tokens before the model processes it, so the letter-level detail is abstracted away. Counting characters, reversing strings, and similar tasks are hard for the same reason. Models can sometimes recover by spelling the word out step by step, but the difficulty is built into how text is tokenized.

Does tokenization treat all languages equally? No. Byte-pair encoding vocabularies are learned mostly from English-heavy training data, so English encodes very efficiently while many other languages — especially non-Latin scripts — need more tokens for the same meaning. That means higher cost, slower responses, and less content fitting in the context window for those languages. It's an artifact of the encoding, not necessarily of the model's underlying ability.

Is a bigger vocabulary always better? Not automatically. A larger vocabulary can encode text into fewer tokens, which saves cost and context space, but it also makes the model's input and output layers larger and can spread training signal more thinly across rare tokens. Tokenizer design is a trade-off between compression efficiency and model size, and the best choice depends on the target languages and domains — which is why different model families make different calls.

Is the tokenizer part of the model? No. The tokenizer is a separate program with its own files (a vocabulary and, for BPE, a merge table). It does no learning at inference time — encoding is a deterministic, reversible lookup. It connects to the model only through the embedding table, where each token ID indexes a learned vector. That separation is why you can count tokens before calling the model, and why using the wrong tokenizer with a set of weights produces garbage: the ID numbers stop pointing at the rows the model was trained to expect.

Do BPE, WordPiece, and SentencePiece produce the same tokens? No. BPE fuses the most frequent adjacent pair and replays those merges to encode; WordPiece (used by BERT-style models) picks merges by a likelihood criterion and encodes by greedy longest-match with ## continuation markers; the Unigram scheme prunes a large vocabulary down and chooses the most probable segmentation per string. SentencePiece is a toolkit that can train BPE or Unigram while treating whitespace as a visible symbol. The upshot is that token counts and exact splits aren't portable across model families — always measure against the specific model.

What is a glitch token? A glitch (or undertrained) token is a token that earned a place in the vocabulary because its string was frequent in the raw scraped corpus, but was then rare or absent in the cleaned training data — so its embedding never received a real learning signal. Feeding one to the model can produce bizarre behavior: refusing to repeat it, spelling something unrelated, or breaking down entirely. The SolidGoldMagikarp family (scraped usernames) is the famous example. Labs now prune these, but they're a vivid demonstration that the tokenizer and the model are separate systems.

The one-sentence version

A model reads in tokens, is billed in tokens, and is limited by tokens — so the moment you can see how your text gets chopped, you can predict its cost, its context fit, and a surprising share of its failures. Everything else about tokenization is detail on top of that.

Table of contents