All language subtitles for 009管道函数内部发生了什么？（PyTorch）

What happens inside the pipeline
function? In this video,

we will look at what actually happens when we use
the pipeline function of the Transformers library.

More specifically, we will look at the
sentiment analysis pipeline, and how it

went from the two following sentences to the
positive labels with their respective scores.

As we have seen in the pipeline presentation,
there are three stages in the pipeline.

First, we convert the raw texts to
numbers the model can make sense of,

using a tokenizer. Then, those numbers go
through the model, which outputs logits.

Finally, the post-processing steps transforms
those logits into labels and scores.

Let's look in detail at those three steps, and how
to replicate them using the Transformers library,

beginning with the first stage, tokenization. The
tokenization process has several steps. First,

the text is split into small chunks called tokens.
They can be words, parts of words or punctuation

symbols. Then the tokenizer will had some special
tokens (if the model expect them). Here the model

uses expects a CLS token at the beginning and a
SEP token at the end of the sentence to classify.

Lastly, the tokenizer matches each token to its
unique ID in the vocabulary of the pretrained

model. To load such a tokenizer, the Transformers
library provides the AutoTokenizer API.

The most important method of this
class is from_pretrained, which will

download and cache the configuration and the
vocabulary associated to a given checkpoint.

Here, the checkpoint used by default for the
sentiment analysis pipeline is distilbert base

uncased finetuned sst2 english. We instantiate
a tokenizer associated with that checkpoint,

then feed it the two sentences. Since those
two sentences are not of the same size,

we will need to pad the shortest
one to be able to build an array.

This is done by the tokenizer
with the option padding=True.

With truncation=True, we ensure that any sentence
longer than the maximum the model can handle

is truncated. Lastly, the return_tensors option
tells the tokenizer to return a PyTorch tensor.

Looking at the result, we see we have a dictionary
with two keys. Input IDs contains the IDs of both

sentences, with 0s where the padding is
applied. The second key, attention mask,

indicates where padding has been applied,
so the model does not pay attention to it.

This is all what is inside the tokenization
step. Now let's have a look at the second step,

the model. As for the tokenizer, there is an
AutoModel API, with a from_pretrained method.

It will download and cache the configuration
of the model as well as the pretrained weights.

However, the AutoModel API will only
instantiate the body of the model,

that is, the part of the model that is
left once the pretraining head is removed.

It will output a high-dimensional tensor that is a
representation of the sentences passed, but which

is not directly useful for our classification
problem. Here the tensor has two sentences,

each of sixteen tokens and the last dimension is
the hidden size of our model 768. To get an output

linked to our classification problem, we need to
use the AutoModelForSequenceClassification class.

It works exactly as the AutoModel class,
except that it will build a model with a

classification head. There is one auto class for
each common NLP task in the Transformers library.

Here, after giving our model the two
sentences, we get a tensor of size two by two:

one result for each sentence and for each possible
label. Those outputs are not probabilities yet

(we can see they don't sum to 1). This is because
each model of the Transformers library returns

logits. To make sense of those logits, we need to
dig into the third and last step of the pipeline:

post-processing. To convert logits into
probabilities, we need to apply a SoftMax

layer to them. As we can see, this transforms
them into positive numbers that sum up to 1.

The last step is to know which of those
corresponds to the positive or the negative label.

This is given by the id2label field of the
model config. The first probabilities (index 0)

correspond to the negative label, and the seconds
(index 1) correspond to the positive label.

This is how our classifier built with the
pipeline function picked those labels and computed

those scores. Now that you know how each steps
works, you can easily tweak them to your needs.