All language subtitles for 014基于单词的标记器

Let's take a look at word-based tokenization.
Word-based tokenization is the idea of splitting

the raw text into words, by splitting on spaces
or other specific rules like punctuation. In this

algorithm, each word has a specific number, an
"ID", attributed to it. In this example, "Let's"

has the ID 250, do has ID 861, and tokenization
followed by an exclamation point has the ID 345.

This approach is interesting, as the model has
representations that are based on entire words.

The information held in a single number is high

as a word contains a lot of contextual
and semantic information in a sentence.

However, this approach does have its limits.
For example, the word dog and the word

dogs are very similar, and their meaning is
close. However, the word-based tokenization

will attribute entirely different IDs to these
two words, and the model will therefore learn

different meanings for these two words. This
is unfortunate, as we would like the model

to understand that these words are indeed related
and that dogs is the plural form of the word dog.

Another issue with this approach is that there
are a lot of different words in a language.

If we want our model to understand all
possible sentences in that language,

then we will need to have an ID for each
different word, and the total number of words,

which is also known as the vocabulary
size, can quickly become very large.

This is an issue because each ID is mapped to a
large vector that represents the word's meaning,

and keeping track of these mappings requires an
enormous number of weights when the vocabulary

size is large. If we want our models to stay
lean, we can opt for our tokenizer to ignore

certain words that we don't necessarily need. For
example, when training our tokenizer on a text,

we might want to take the 10,000
most frequent words in that text

to create our basic vocabulary, instead
of taking all of that language's words.

The tokenizer will know how to convert
those 10,000 words into numbers,

but any other word will be converted to the
out-of-vocabulary word, or the "unknown" word.

This can rapidly become an issue: the model
will have the exact same representation

for all words that it doesn't know, which
will result in a lot of lost information.