All language subtitles for 007Transformer解码器模型

In this video, we'll study the decoder architecture.
An example of a popular decoder-only architecture

is GPT-2. In order to understand how decoders
work, we recommend taking a look at the video

regarding encoders: they're extremely similar
to decoders. One can use a decoder for most

of the same tasks as an encoder, albeit with,
generally, a little loss of performance. Let's

take the same approach we have taken with
the encoder to try and understand the architectural

differences between an encoder and a decoder.
We'll use a small example, using three words.

We pass them through the decoder. We retrieve
a numerical representation of each word. Here,

for example, the decoder converts the three
words “Welcome to NYC” in these three

sequences of numbers. The decoder outputs
exactly one sequence of numbers per input

word. This numerical representation can also
be called a "Feature vector", or "Feature

tensor". Let's dive in this representation.
It contains one vector per word that was passed

through the decoder. Each of these vector
is a numerical representation of the word

in question. The dimension of that vector
is defined by the architecture of the model.

Where the decoder differs from the encoder
is principally with its self-attention mechanism.

It's using what is called "masked self-attention".
Here for example, if we focus on the word

"to", we'll see that its vector is absolutely
unmodified by the "NYC" word. That's because

all the words on the right (also known as
the right context) of the word is masked.

Rather than benefitting from all the words
on the left and right, I.e., the bidirectional

context, decoders only have access to the
words on their left. The masked self-attention

mechanism differs from the self-attention
mechanism by using an additional mask to hide

the context on either side of the word: the
word's numerical representation will not be

affected by the words in the hidden context.
So when should one use a decoder? Decoders,

like encoders, can be used as standalone models.
As they generate a numerical representation,

they can also be used in a wide variety of
tasks. However, the strength of a decoder

lies in the way a word has access to its left
context. The decoders, having only access

to their left context, are inherently good
at text generation: the ability to generate

a word, or a sequence of words, given a known
sequence of words. In NLP, this is known as

Causal Language Modeling. Let's look at an
example. Here's an example of how causal language

modeling works: we start with an initial word,
which is "My". We use this as input for the

decoder. The model outputs a vectors of dimension
768. This vector contains information about

the sequence, which is here a single word,
or word. We apply a small transformation to

that vector so that it maps to all the words
known by the model (mapping which we'll see

later, called a language modeling head). We
identify that the model believes the most

probable following word is "name". We then
take that new word, and add it to the initial

sequence. From "My", we are now at "My name".
This is where the "autoregressive" aspect

comes in. Auto-regressive models re-use their
past outputs as inputs in the following steps.

Once again, we do that the exact same operation:
we cast that sequence through the decoder,

and retrieve the most probable following word.
In this case, it is the word "is". We repeat

the operation until we're satisfied. Starting
from a single word, we've now generated a

full sentence. We decide to stop there, but
we could continue for a while; GPT-2, for

example, has a maximum context size of 1024.
We could eventually generate up to 1024 words,

and the decoder would still have some memory
of the first words of the sequence! If we

go back several levels higher, back to the
full transformer model, we can see what we

learned about the decoder part of the full
transformer model. It is what we call, auto-regressive:

it outputs values that are then used as its
input values. We repeat this operations as

we like. It is based off of the masked self-attention
layer, which allows to have word embeddings

which have access to the context on the left
side of the word. If you look at the diagram

however, you'll see that we haven't seen one
of the aspects of the decoder. That is: cross-attention.

There is a second aspect we haven't seen,
which is it's ability to convert features

to words; heavily linked to the cross attention
mechanism. However, these only apply in the

"encoder-decoder" transformer, or the "sequence-to-sequence"
transformer (which can generally be used interchangeably).

We recommend you check out the video on encoder-decoders
to get an idea of how the decoder can be used

as a component of a larger architecture!