All language subtitles for 016基于子词的标记器

Let's take a look at subword-based tokenization.
Understanding why subword-based tokenization

is interesting requires understanding the flaws
of word-based and character-based tokenization.

If you haven't seen the first videos on
word-based and character-based tokenization,

we recommend you check them out
before looking at this video.

Subword-tokenization lies in between
character-based and word-based tokenization

algorithms. The idea is to find a middle ground
between very large vocabularies, large quantity of

out-of-vocabulary tokens, loss of meaning across
very similar words, for word-based tokenizers,

and very long sequences, less meaningful
individual tokens for character-based tokenizers.

These algorithms rely on the following
principle: frequently used words should not

be split into smaller subwords, but rare words
should be decomposed into meaningful subwords.

An example is the word dog: we would like to have
our tokenizer to have a single ID for the word

dog, rather than splitting it into characters:
d, o, and g. However, when encountering the word

dogs, we would like our tokenizer to understand
that at the root, this is still the word dog,

with an added s while slightly changes the meaning
while keeping the original idea. Another example

is a complex word like tokenization, which can
be split into meaningful subwords. The root of

the word is token, and ization completes the
root to give it a slightly different meaning.

It makes sense to split the word into two: token,
as the root of the word (labeled as the "start" of

the word). ization as additional information
(labeled as a "completion" of the word).

In turn, the model will now be able to make
sense of token in different situations.

It will understand that the words token, tokens,
tokenizing, and tokenization are linked and have

a similar meaning. It will also understand that
tokenization, modernization, and immunization,

which all have the same suffixes, are probably
used in the same syntactic situations.

Subword-based tokenizers generally have a way
to identify which tokens are start of words, and

which tokens complete start of words: token as the
start of a word. ##ization as completing a word.

Here the ## prefix indicates that ization is
part of a word rather than the beginning of it.

The ## comes from the BERT tokenizer, based on the
WordPiece algorithm. Other tokenizers use other

prefixes, which can be placed to indicate part of
words like seen here, or start of words instead!

There are a lot of different algorithms that can
be used for subword tokenization, and most models

obtaining state-of-the-art results in English
today use some kind of subword-tokenization

algorithm. These approaches help in reducing
the vocabulary sizes by sharing information

across different words, having the ability to
have prefixes and suffixes understood as such.

They keep meaning across very similar words,
by recognizing similar tokens making them up.