Tokens: how a model reads text

A model never sees letters or words. A tokenizer cuts text into tokens, pieces from a fixed vocabulary, each with a number. The numbers are all it reads.

ids = tokenizer.encode("My parcel has not arrived")
print(ids)
print([tokenizer.decode([i]) for i in ids])
print(len(ids), "tokens")

encode gives the numbers. Decoding them one at a time shows the pieces. Common words are one token each, and most tokens carry the space in front of the word with them.

Words that are not common

for text in ["refund", "unsubscribing", "Hello, my parcel has not arrived", "नमस्ते, मेरा पार्सल नहीं आया"]:
    ids = tokenizer.encode(text)
    print(len(ids), [tokenizer.decode([i]) for i in ids])

A less common word is split into pieces. The last two lines say the same thing, and the Hindi one takes several times as many tokens: this tokenizer has fewer pieces for Hindi, so it falls back to fragments. Pieces that show as strange characters are parts of a single letter, readable only when joined.

This matters for money and space. Model APIs charge per token, and every model has a limit on how many tokens it can read at once. The same message can cost several times more in one language than another.

And back to text

ids = tokenizer.encode("The parcel arrived but the box was crushed")
print(tokenizer.decode(ids))