All language subtitles for 011实例化 Transformers模型 (PyTorch)

How to instantiate a Transformers model?

In this video we will look at how we can create
and use a model from the Transformers library.

As we've seen before, the AutoModel class allows
you to instantiate a pretrained model from any

checkpoint on the Hugging Face Hub. It will
pick the right model class from the library to

instantiate the proper architecture and load the
weights of the pretrained model inside it. As we

can see, when given a BERT checkpoint, we end up
with a BertModel, and similarly for GPT-2 or BART.

Behind the scenes, this API can take
the name of a checkpoint on the Hub,

in which case it will download and cache the
configuration file as well as the model weights

file. You can also specify the path to a local
folder that contains a valid configuration file

and a model weights file. To instantiate the
pretrained model, the AutoModel API will first

open the configuration file to look at the
configuration class that should be used.

The configuration class depends on the type of
the model (BERT, GPT-2 or BART for instance).

Once it has the proper configuration class,
it can instantiate that configuration,

which is a blueprint to know how to create the
model. It also uses this configuration class

to find the proper model class, which is combined
with the loaded configuration, to load the model.

This model is not yet our pretrained model as it
has just been initialized with random weights.

The last step is to load the weights from the
model file inside this model. To easily load

the configuration of a model from any checkpoint
or a folder containing the configuration folder,

we can use the AutoConfig class.

Like the AutoModel class, it will pick the
right configuration class from the library.

We can also use the specific class corresponding
to a checkpoint, but we will need to change the

code each time we want to try a different model.
As we said before, the configuration of a model is

a blueprint that contains all the information
necessary to create the model architecture.

For instance the BERT model associated with
the bert-base-cased checkpoint has 12 layers,

a hidden size of 768, and a vocabulary size
of 28,996. Once we have the configuration,

we can create a model that has the same
architecture as our checkpoint but is

randomly initialized. We can then train it from
scratch like any PyTorch module/TensorFlow model.

We can also change any part of the
configuration by using keyword arguments.

The second snippet of code instantiates a
randomly initialized BERT model with ten layers

instead of 12. Saving a model once it's trained
or fine-tuned is very easy: we just have to use

the save_pretrained method. Here the model will
be saved in a folder named my-bert-model inside

the current working directory. Such a model can
then be reloaded using the from_pretrained method.