All language subtitles for 018将输入批量处理在一起（PyTorch

How to batch inputs together? In this video, we
will see how to batch input sequences together.

In general, the sentences we want to pass through
our model won't all have the same lengths.

Here we are using the model we saw
in the sentiment analysis pipeline

and want to classify two sentences.
When tokenizing them and mapping each

token to its corresponding input IDs,
we get two lists of different lengths.

Trying to create a tensor or a NumPy array from
those two lists will result in an error, because

all arrays and tensors should be rectangular.
One way to overcome this limit is to make the

second sentence the same length as the first by
adding a special token as many times as necessary.

Another way would be to truncate the first
sequence to the length of the second, but we

would them lose a lot of information that might
be necessary to properly classify the sentence.

In general, we only truncate sentences when
they are longer than the maximum length the

model can handle. The value used to pad the second
sentence should not be picked randomly: the model

has been pretrained with a certain padding ID,
which you can find in tokenizer.pad_token_id.

Now that we have padded our sentences,
we can make a batch with them. If

we pass the two sentences to the model
separately and batched together however,

we notice that we don't get the same results for
the sentence that is padded (here the second one).

If you remember that Transformer models make
heavy use of attention layers, this should

not come as a total surprise: when computing
the contextual representation of each token,

the attention layers look at all the other words
in the sentence. If we have just the sentence or

the sentence with several padding tokens added,
it's logical we don't get the same values.

To get the same results with or without padding,
we need to indicate to the attention layers

that they should ignore those padding tokens.
This is done by creating an attention mask,

a tensor with the same shape as the input
IDs, with zeros and ones. Ones indicate the

tokens the attention layers should consider in the
context and zeros the tokens they should ignore.

Now passing this attention
mask along with the input ids

will give us the same results as when we sent
the two sentences individually to the model!

This is all done behind the scenes by the
tokenizer when you apply it to several sentences

with the flag padding=True. It will
apply the padding with the proper value

to the smaller sentences and create
the appropriate attention mask.