All language subtitles for 010管道函数内部发生了什么？（TensorFlow）

What happens inside the pipeline function?

In this video, we will look at what actually
happens when we use the pipeline function of

the Transformers library. More specifically, we
will look at the sentiment analysis pipeline, and

how it went from the two following sentences to
the positive labels with their respective scores.

As we have seen in the pipeline presentation,
there are three stages in the pipeline. First,

we convert the raw texts to numbers the
model can make sense of, using a tokenizer.

Then, those numbers go through
the model, which outputs logits.

Finally, the post-processing steps transforms
those logits into labels and scores.

Let's look in detail at those three steps, and how
to replicate them using the Transformers library,

beginning with the first stage, tokenization. The
tokenization process has several steps. First,

the text is split into small chunks called tokens.
They can be words, parts of words or punctuation

symbols. Then the tokenizer will had some special
tokens (if the model expect them). Here the model

uses expects a CLS token at the beginning and a
SEP token at the end of the sentence to classify.

Lastly, the tokenizer matches each token to its
unique ID in the vocabulary of the pretrained

model. To load such a tokenizer, the Transformers
library provides the AutoTokenizer API.

The most important method of this class is
from_pretrained, which will download and cache

the configuration and the vocabulary associated
to a given checkpoint. Here, the checkpoint used

by default for the sentiment analysis pipeline is
distilbert base uncased finetuned sst2 english.

We instantiate a tokenizer associated with that
checkpoint, then feed it the two sentences.

Since those two sentences are not of the same
size, we will need to pad the shortest one to

be able to build an array. This is done by
the tokenizer with the option padding=True.

With truncation=True, we ensure that any sentence
longer than the maximum the model can handle

is truncated. Lastly, the return_tensors option
tells the tokenizer to return a TensorFlow tensor.

Looking at the result, we see we
have a dictionary with two keys.

Input IDs contains the IDs of both sentences, with
0s where the padding is applied. The second key,

attention mask, indicates where padding has been
applied, so the model does not pay attention to

it. This is all what is inside the tokenization
step. Now let's have a look at the second step,

the model. As for the tokenizer, there is an
TFAutoModel API, with a from_pretrained method.

It will download and cache the configuration
of the model as well as the pretrained

weights. However, the TFAutoModel API will
only instantiate the body of the model,

that is, the part of the model that is
left once the pretraining head is removed.

It will output a high-dimensional tensor that
is a representation of the sentences passed,

but which is not directly useful
for our classification problem.

Here the tensor has two sentences, each of sixteen
tokens and the last dimension is the hidden size

of our model 768. To get an output linked
to our classification problem, we need to

use the TFAutoModelForSequenceClassification
class. It works exactly as the AutoModel class,

except that it will build a model with a
classification head. There is one auto class for

each common NLP task in the Transformers library.
Here, after giving our model the two sentences,

we get a tensor of size two by two: one result for
each sentence and for each possible label. Those

outputs are not probabilities yet (we can see they
don't sum to 1). This is because each model of the

Transformers library returns logits. To make sense
of those logits, we need to dig into the third and

last step of the pipeline: post-processing. To
convert logits into probabilities, we need to

apply a SoftMax layer to them. As we can see,
this transforms them into positive numbers that

sum up to 1. The last step is to know which of
those corresponds to the positive or the negative

label. This is given by the id2label field
of the model config. The first probabilities

(index 0) correspond to the negative label, and
the seconds (index 1) correspond to the positive

label. This is how our classifier built with the
pipeline function picked those labels and computed

those scores. Now that you know how each steps
works, you can easily tweak them to your needs.