Tokenization in NLP: how tokenizers let machines read

When this post was first published in May 2020, NLP had just had a huge year. Optimized libraries and efficient hardware made neural networks faster to deploy, and one of the remaining bottlenecks was tokenization, in particular complete, framework-independent implementations.

The two most prominent trends of that time were the Transformer architecture (2017) and the BERT language model (2018), the best-known model built on it. Since then, transformers have become the basis of large language models such as GPT, Gemini, Claude and Llama, and every one of them still begins and ends with a tokenizer.

Why is reading difficult for machines?

A person can understand language before learning to read. Machines don’t have that phonetic beginning. Knowing nothing about language, they need systems that let them process text without being able, like humans, to associate sounds with the meaning of words.

How can machines start processing text if they know nothing about grammar, sounds, words or sentences? You can write rules that make the machine look words up in a dictionary, but then it learns nothing, and you would need a static data set for every combination of words and all their grammatical forms.

Instead, you need to teach the machine to “read” text in a way it can learn from: the more it reads, the more it learns. People rely on the sounds they learned earlier. Machines have no such knowledge, so they need to be told how to divide text into standard units. They do this with a process called “tokenization,” in which text is broken down into smaller chunks or “tokens” that are then fed as input into a deep learning model such as BERT. But do we need tokenization at all?

Do we need Tokenizers?

To teach a deep learning model such as BERT or GPT-2 to perform well on natural language tasks, we need to give it a lot of text. The model learns some level of syntactic and semantic understanding, and the depth of the semantic understanding of these models is still an area of active research. They are thought to learn syntax in the lower layers of the network and meaning in the higher ones.

The architecture decides which tasks a model handles well. GPT-2, for example, uses a decoder architecture because its job is to predict the next word in a sequence. BERT uses an encoder architecture because it is built to understand text, for tasks such as question answering and classification; one of its training tasks was next-sentence prediction. However they are designed, they all need to receive text through their input layer, and that layer accepts numbers, not letters.

That is the role of tokenization in NLP: it turns text into a sequence of token IDs from a fixed vocabulary, and turns the model’s output IDs back into text. A spam filter, a translation app and a chatbot all start this way. Open-source libraries make the step fast and reusable: Hugging Face Tokenizers, written in Rust, tokenizes a gigabyte of text in under 20 seconds on a server CPU. Such a tokenizer runs a pipeline of steps:

• Normalization: cleanup such as removing extra whitespace, lowercasing or stripping accents.
• Pre-tokenization: splitting the raw text into words on whitespace and punctuation.
• The model: the subword algorithm, such as BPE, WordPiece or Unigram, that splits the words into tokens.
• Post-processing: adding the special tokens the model expects, such as the [CLS] token at the start of every BERT input.

Types of tokenization

There are four basic ways to cut text into tokens, each a trade-off between vocabulary size and how much meaning one token carries:

• Word tokenization splits text on spaces and punctuation, so “The cats sat.” becomes “The”, “cats”, “sat” and “.”. It is simple, but the vocabulary grows to hundreds of thousands of entries, and any word not seen in training becomes an unknown token.
• Character tokenization uses single letters, digits and symbols. Nothing is ever unknown, but sequences get long and each token carries little meaning.
• Subword tokenization sits in between: frequent words stay whole and rare words are built from pieces, as in “token” plus “ization”. This is the standard for today’s language models.
• Sentence tokenization splits a text into sentences. It is a first step in tasks such as summarization, rather than the input to a model.

Subword tokenization

Classical word representation does not handle rare words well, and characters split information too finely. The subword sits between the two: not too fine-grained, yet able to handle unseen and rare words.

Byte pair encoding (BPE) is the most popular subword algorithm. Philip Gage described it in 1994 as a data compression method that replaces the most frequent pair of bytes with a new symbol. In 2015 Rico Sennrich, Barry Haddow and Alexandra Birch adapted it for neural machine translation, to encode rare and unknown words as sequences of subword units. Training a BPE tokenizer works like this:

• Start from characters. Take a training text with the words low (5 times), lower (2), newest (6) and widest (3), and split every word into letters.
• Count every pair of neighboring symbols. Here “e s” and “s t” are the most frequent, with 9 occurrences each, from newest and widest.
• Merge the most frequent pair into a new token: “e” and “s” become “es”, then “es” and “t” become “est”, then “l” and “o” become “lo”, and “lo” and “w” become “low”.
• Repeat until the vocabulary reaches the size you chose. GPT-2’s vocabulary has 50,257 tokens; Meta’s Llama 3, released in April 2024, uses 128,000.

The result generalizes. After these four merges, a word the tokenizer has never seen, such as “lowest”, splits into the known pieces “low” and “est”. To tokenize new text, the tokenizer applies the learned merges in the same order.

GPT-2 added an important variant in 2019, byte-level BPE. Instead of over 130,000 Unicode characters, it starts from the 256 possible byte values, so any text in any script can be encoded without unknown tokens. OpenAI’s tiktoken, the tokenizer for its models, is also a BPE tokenizer, and on average each of its tokens corresponds to about 4 bytes of text.

The Unigram approach, proposed by Taku Kudo in 2018, works in the opposite direction. It starts from a large set of candidate subwords, such as the most frequent substrings, and repeatedly removes the ones whose loss hurts the model least, until it reaches the target size. When it tokenizes a word, it picks the most probable segmentation, and during training it can sample different segmentations of the same word, which helps on small and out-of-domain data sets.

WordPiece

When BERT was released in 2018, it used WordPiece with a vocabulary of 30,000 tokens. The algorithm itself was older: Mike Schuster and Kaisuke Nakajima described it at Google in 2012, for Japanese and Korean voice search. WordPiece marks pieces that continue a word with a prefix, “##” in BERT, so a word can come out as “token” and “##ization”.

WordPiece looks very similar to BPE; think of it as a bridge between the BPE and Unigram approaches. BPE merges the pair of tokens with the highest count, nothing more. WordPiece chooses merges with a likelihood score that divides the frequency of a pair by the frequencies of its parts, so it prefers pairs whose parts rarely appear apart.

SentencePiece

SentencePiece, released by Taku Kudo and John Richardson in 2018, tries to put the subword tools and techniques under one banner. It’s a bit like a Swiss Army knife for subword tokenization. These are the problems it addresses:

• Other models assume the input is already split into words. BPE and Unigram are great models, but both need pre-tokenized text. SentencePiece takes raw text and does everything itself.
• Language dependence. Pre-tokenized input means splitting rules for each language, and languages written without spaces, such as Chinese, Japanese or Thai, make that messy very quickly.
• Decoding is hard. When a tokenizer throws spaces away, you cannot always tell how to put the original text back together, which makes results hard to reproduce or verify.
• No complete solution. BPE and Unigram are parts of a solution: you cannot plug in raw text and get a result. SentencePiece brings everything together in one package.

How does SentencePiece solve all these problems?

SentencePiece uses several features, described in its paper and its GitHub repository, that address the issues above:

• It trains directly on raw sentences, with no language-specific pre-tokenizer.
• It treats the space as an ordinary symbol, written as “▁”, so tokenization is lossless: joining the pieces and turning the marker back into a space restores the original text exactly.
• It implements both BPE and the Unigram model, including the sampling of different segmentations during training.
• It is self-contained: one model file holds the normalization rules, the vocabulary and the segmentation model, so the same text gets the same tokens in any environment.

SentencePiece is the tokenizer behind models such as T5, ALBERT, XLNet and mBART.

Why tokens matter for large language models

For large language models, tokens are the unit of almost everything: prices are quoted per million tokens, and the context window, the amount of text a model takes in at once, is measured in tokens. Some practical consequences:

• The same text costs different amounts in different languages. A NeurIPS 2023 study found that the same text can need up to 15 times more tokens in one language than in another.
• Spelling and counting are harder than they look. Asked how many r’s are in “strawberry”, models have often answered two: they see a few tokens, not ten letters.
• Some researchers want to drop fixed tokenizers altogether. Meta’s Byte Latent Transformer, published in December 2024, reads raw bytes grouped into patches of varying size and matched the performance of tokenizer-based models at up to 8 billion parameters.

With the rapid development of deep learning, it is easy to look only at the core of models such as BERT, GPT or Llama. However, the format that allows these models to process text is critical to their learning. Understanding subword tokenizers helps you follow new work in the field, build these networks yourself, or deal with multilingual issues in traditional NLP tasks.