All language subtitles for 012实例化 Transformers模型 (TensorFlow)

How to instantiate a Transformers model?

In this video we will look at how we can create
and use a model from the Transformers library.

As we've seen before, the TFAutoModel class
allows you to instantiate a pretrained model

from any checkpoint on the Hugging Face Hub.

It will pick the right model class from the
library to instantiate the proper architecture

and load the weights of the pretrained model
inside it.

As we can see, when given a BERT checkpoint,
we end up with a TFBertModel, and similarly

for GPT-2 or BART.

Behind the scenes, this API can take the name
of a checkpoint on the Hub, in which case

it will download and cache the configuration
file as well as the model weights file.

You can also specify the path to a local folder
that contains a valid configuration file and

a model weights file.

To instantiate the pretrained model, the AutoModel
API will first open the configuration file

to look at the configuration class that should
be used.

The configuration class depends on the type
of the model (BERT, GPT-2 or BART for instance).

Once it has the proper configuration class,
it can instantiate that configuration, which

is a blueprint to know how to create the model.

It also uses this configuration class to find
the proper model class, which is combined

with the loaded configuration, to load the
model.

This model is not yet our pretrained model
as it has just been initialized with random

weights.

The last step is to load the weights from
the model file inside this model.

To easily load the configuration of a model
from any checkpoint or a folder containing

the configuration folder, we can use the AutoConfig
class.

Like the TFAutoModel class, it will pick the
right configuration class from the library.

We can also use the specific class corresponding
to a checkpoint, but we will need to change

the code each time we want to try a different
model.

As we said before, the configuration of a
model is a blueprint that contains all the

information necessary to create the model
architecture.

For instance the BERT model associated with
the bert-base-cased checkpoint has 12 layers,

a hidden size of 768, and a vocabulary size
of 28,996.

Once we have the configuration, we can create
a model that has the same architecture as

our checkpoint but is randomly initialized.

We can then train it from scratch like any
PyTorch module/TensorFlow model.

We can also change any part of the configuration
by using keyword arguments.

The second snippet of code instantiates a
randomly initialized BERT model with ten layers

instead of 12.

Saving a model once it's trained or fine-tuned
is very easy: we just have to use the save_pretrained

method.

Here the model will be saved in a folder named
my-bert-model inside the current working directory.

Such a model can then be reloaded using the
from_pretrained method.