How AI Reads Text — Tokenizer Explainer

01 · Splitting

AI never sees your words, only pieces

Your text is cut into small pieces called tokens, and each piece becomes a number. Pick a way to split, then step through a whole sentence.

Your sentence — as you typed it

Characters 0 Tokens 0 Words 0

This is the sentence as you see it — whole words, spaces, punctuation. The model never sees it this way. Press Split it into tokens to watch it get chopped up.

What the model thinks comes next longer bar = more likely

Token IDs here are illustrative. Every model has its own list, so the same piece gets a different number elsewhere.

Text becomes numbered pieces first. Everything else the model does starts from those numbers.

Why not just use whole words?

You could split on spaces, but the model would then need a separate number for every word it has ever seen, including misspellings, slang and every language. That is millions of entries. A smaller, smarter list of pieces works better.

Split	How it cuts	“procurement”	Trade-off
Word	One token per word	`procurement`	Simple, but a huge list, and new words break it
Subword	Meaningful pieces	`procure` + `ment`	Best balance, and used by most modern models
Character	One token per letter	`p` `r` `o` …	Tiny list, but very long sequences
Byte	Raw computer codes	`112` `114` `111` …	Copes with any language, but sequences get long

Why subwords win. “Procure”, “procurement” and “procuring” share a root, so a subword list can reuse the same piece for all three.

02 · Learning the pieces

Pieces are learned by merging common pairs

The tokenizer starts with single letters, finds the pair that appears together most often, and glues it into one piece. Press Play and watch it repeat.

Step 0 / 0

Press Play to watch BPE build its list of pieces step by step, or use the Next button to go one step at a time.

Tokens 0 tokens

initial split

Speed

Nobody wrote these pieces by hand. They come from counting what appears together most often.

Show the workings

Pair frequencies (top 15)

Pieces so far (0)

Merge history

No merges yet. Press Play to start.

A worked example, and training versus everyday use

The textbook example. Start from low lower newest widest, with ▁ marking the end of each word. One possible run: e + s becomes es, then es + t becomes est, then l + o becomes lo, then lo + w becomes low. Several pairs tie on count, so the order you see in the playground may differ.

Nothing here knows English. “low” and “est” emerge purely because those letters keep appearing together.

Learning happens once. The merge rules are learned from a very large body of text before the model is trained, then saved and fixed. The playground above shows that learning step on a tiny sample.

Everyday use is a lookup. When you send a prompt, your text is split into letters and the saved rules are applied in order. No new rules are discovered. A real tokenizer has tens of thousands of entries, so common words like “vendor” and “contract” are usually single tokens, unlike in this small demo.

03 · Length

Every token takes up room

A model can only read so much at once, and tokens are how that room is counted. The same facts take more or fewer tokens depending on how you write them.

Try your own text

0characters

0words

~0estimated tokens

Tokens are the unit of length. Plain, tight prompts leave more room for your documents.

Rules of thumb, rough sizes and tips

Rule of thumb for English: one token is about 4 characters, or about three quarters of a word. Estimates here use that rule. Real counts depend on the model’s own tokenizer.

Content	Rough size
A short question (“Review this proposal”)	about 5 tokens
A paragraph of vendor data (10 lines)	about 150 tokens
A prompt template with instructions	about 400 tokens
A full eight-section assessment	about 800 tokens

Numbers and dates add up. Each digit group, comma and dot can be its own token, so a table full of figures is bigger than it looks.
Cut repeated instructions. Say each rule once.
Keep examples short. One clear example beats three long ones.
Ask for the length you want. “In five bullets” keeps the answer compact.

About prices. What a token costs depends on how a model is accessed and changes over time, so this page does not quote prices. For questions about limits or usage in Amazon Quick, ask your DNA / IT team.

04 · The big picture

Tokenizing is stop one of six

Every message follows this path. The tokenizer comes first, so everything after it depends on how your text was split.

Stop 1 · Your text

Pieces, IDs and numbers shown here are illustrative.

Three ways to build the list of pieces

How your text is split shapes every stop that follows.

Compare the three methods

Method	Direction	How it chooses	Used by
BPE (Byte Pair Encoding)	Bottom-up: merge	The most frequent pair	GPT-family and LLaMA models
WordPiece	Bottom-up: merge	The most likely pair (probability)	BERT
Unigram	Top-down: prune	Removes the least useful pieces	T5

The same text can give different token counts on different models, which changes how much fits in one request.

Special tokens: the model’s control codes

Every list also holds a few reserved tokens with special jobs.

Token	Purpose
`[PAD]`	Fills short inputs so a batch has equal lengths
`[UNK]`	Stands in for a piece not found in the list
`[CLS]`	Marks the start of an input (BERT-specific)
`[SEP]`	Separates two segments of text
`[MASK]`	Hides a token for the model to predict during BERT training

What happens at the later stops, and how we got here

Embeddings: each token ID becomes a long list of numbers (often hundreds to thousands) that captures meaning. “Procurement” and “purchasing” end up close together, “procurement” and “airplane” far apart.
Position: the model also needs to know where each token sits, so “The contract is approved” and “Is the contract approved?” are not treated as identical.
Attention (the transformer): every token can look at every other token and weigh which matter. In “The vendor’s SLA performance is below target”, “below target” leans on “SLA performance” and “vendor”.
Output: the model scores every possible next token, picks one, adds it, and repeats. A setting called temperature controls how adventurous that pick is: low is more predictable, high is more varied.

How text representation evolved. Each step fixed a limit of the one before.

One-hot encoding: no meaning. “Cute” and “adorable” look as different as “cute” and “airplane”.
Word2vec / GloVe: meaningful, but one fixed vector per word whatever the context.
RNN / LSTM / GRU: reads in order, but long inputs squeeze through a bottleneck.
ELMo: meaning that depends on context. “Bank” differs in “river bank” and “bank account”.
Transformers (attention): look at all tokens in parallel with no bottleneck. This is what powers today’s large language models.