All language subtitles for 008Transformer 模型：编码器-解码器

In this video, we'll study the
encoder-decoder architecture.

An example of a popular encoder-decoder model is
T5. In order to understand how the encoder-decoder

works, we recommend you check out the videos
on encoders and decoders as standalone models.

Understanding how they behave individually will
help understanding how an encoder-decoder behaves.

Let's start from what we've seen about the
encoder. The encoder takes words as inputs,

casts them through the encoder, and
retrieves a numerical representation

for each word cast through it. We now know that
the numerical representation holds information

about the meaning of the sequence. Let's put
this aside and add the decoder to the diagram.

In this scenario, we're using the decoder
in a manner that we haven't seen before.

We're passing the outputs of the encoder directly
to it! Additionally to the encoder outputs,

we also give the decoder a sequence. When
prompting the decoder for an output with no

initial sequence, we can give it the value
that indicates the start of a sequence.

And that's where the encoder-decoder magic
happens. The encoder accepts a sequence as input.

It computes a prediction, and outputs a
numerical representation. Then, it sends

that over to the decoder. It has, in a sense,
encoded the sequence. And the decoder, in turn,

using this input alongside its usual sequence
input, will take a stab at decoding the sequence.

The decoder decodes the sequence, and outputs a
word. As of now, we don't need to make sense of

that word, but we can understand that the decoder
is essentially decoding what the encoder has

output. The "start of sequence word" indicates
that it should start decoding the sequence.

Now that we have both the feature vector and
an initial generated word, we don't need the

encoder anymore. As we have seen before with the
decoder, it can act in an auto-regressive manner;

the word it has just output can now be used
as an input. This, in combination with the

numerical representation output by the encoder,
can now be used to generate a second word.

Please note that the first word is still here; as
the model still outputs it. However, it is greyed

out as we have no need for it anymore. We can
continue on and on; for example until the decoder

outputs a value that we consider a "stopping
value", like a dot, meaning the end of a sequence.

Here, we've seen the full mechanism of the
encoder-decoder transformer: let's go over it one

more time. We have an initial sequence, that is
sent to the encoder. That encoder output is then

sent to the decoder, for it to be decoded. While
we can now discard the encoder after a single use,

the decoder will be used several times: until
we have generated every word that we need.

Let's see a concrete example; with Translation
Language Modeling; also called transduction;

the act of translating a sequence. Here, we would
like to translate this English sequence "Welcome

to NYC" in French. We're using a transformer model
that is trained for that task explicitly. We use

the encoder to create a representation
of the English sentence. We cast this

to the decoder and, with the use of the start of
sequence word, we ask it to output the first word.

It outputs Bienvenue, which means "Welcome".

We then use "Bienvenue" as the input sequence for
the decoder. This, alongside the feature vector,

allows the decoder to predict the second
word, "à", which is "to" in English.

Finally, we ask the decoder to predict
a third word; it predicts "NYC",

which is, once again, correct. We've translated
the sentence! Where the encoder-decoder really

shines, is that we have an encoder and a
decoder; which often do not share weights.

We, therefore, have an entire block (the encoder)
that can be trained to understand the sequence,

and extract the relevant information. For the
translation scenario we've seen earlier, for

example, this would mean parsing and understanding
what was said in the English language; extracting

information from that language, and putting
all of that in a vector dense in information.

On the other hand, we have the decoder, whose
sole purpose is to decode the feature output by

the encoder. This decoder can be specialized in
a completely different language, or even modality

like images or speech. Encoders-decoders
are special for several reasons. Firstly,

they're able to manage sequence to sequence
tasks, like translation that we have just seen.

Secondly, the weights between the encoder and the
decoder parts are not necessarily shared. Let's

take another example of translation. Here we're
translating "Transformers are powerful" in French.

Firstly, this means that from a sequence
of three words, we're able to generate

a sequence of four words. One could argue
that this could be handled with a decoder;

that would generate the translation in an
auto-regressive manner; and they would be right!

Another example of where sequence to sequence
transformers shine is in summarization.

Here we have a very long
sequence, generally a full text,

and we want to summarize it. Since the
encoder and decoders are separated,

we can have different context lengths (for
example a very long context for the encoder which

handles the text, and a smaller context for the
decoder which handles the summarized sequence).

There are a lot of sequence to sequence
models. This contains a few examples of

popular encoder-decoder models
available in the transformers library.

Additionally, you can load an encoder
and a decoder inside an encoder-decoder

model! Therefore, according to the
specific task you are targeting,

you may choose to use specific encoders
and decoders, which have proven their worth

on these specific tasks. This wraps things up
for the encoder-decoders. Thanks for watching!