All language subtitles for [English (United States)] Stanford CS229 I Machine Learning I Building Large Language Models (LLMs)

So, let's get started.

So I'll be talking about
building LLMs today.

So I think a lot of you have
heard of LLMs before, but just

as a quick recap.

LLMs standing for
large language models

are basically all the
chat bots that you've

been hearing about recently.

So, ChatGPT, from OpenAI,
Claude, from Anthropic, Gemini

and Llama, and other
types of models like this.

And today we'll be talking
about how do they actually work.

So it's going to be an overview
because it's only one lecture

and it's hard to
compress everything.

But hopefully, I'll
touch a little bit

about all the components
that are needed

to train some of these LLMs.

Also, if you have questions,
please interrupt me

and ask if you have a question.

Most likely other people in
the room or on Zoom have other.

Have the same questions.

So, please ask.

Great.

So what matters
when training LLMs.

So there are a few key
components that matter.

One is the architecture.

So as you probably all
LLMs are neural networks,

and when you think
about neural networks,

you have to think about what
architecture you're using.

And another component,
which is really important

is the training loss and
the training algorithm.

So, how you actually train
these models, then it's data.

So, what do you train
these models on.

The evaluation,
which is how do you

know whether you're
actually making progress

towards the goal of LLMs and
then, the system component.

So that is like
how do you actually

make these models run on
modern hardware, which

is really important because
these models are really large.

So now more than ever,
systems are actually

really an important
topic for LLMs.

So those five components, you
probably all know that LLMs.

And if you don't
know LLMs are all

based on transformers
or at least some version

of transformers.

I'm actually not going to talk
about the architecture today.

One, because I gave a lecture
on transformers a few weeks ago

and two, because you can find
so much information online

on transformers.

There's much less information
about the other four topics.

So, I really want
to talk about those.

And another thing to say
is that most of academia

actually focuses on
architecture and training

algorithm and
losses as academics

and I've done that for
a big part of my career,

is simply we like thinking
that this is like we make

new architectures,
new models, and it

seems like it's very important.

But in reality, honestly, what
matters in practice is mostly

the three other topics.

So, data, evaluation and
systems, which is what most

of industry actually focuses on.

So, that's also
one of the reasons

why I don't want to talk too
much about the architecture,

because really the rest
is super important.

Great.

So, overview of
the lecture, I'll

be talking about pretraining.

So, pretraining, you
probably heard that word.

This is the general word.

This is kind of the classical
language modeling paradigm where

you basically train your
language model to essentially

model all of internet.

And then, there's
a post training,

which is a more
recent paradigm which

is taking these
large language models

and making them
essentially AI assistants.

So, this is more of a
recent trend since ChatGPT.

So, if you ever heard
of GPT3 or GPT2,

that's really pretraining land.

If you heard of ChatGPT,
which you probably have,

this is really
post training land,

so I'll be talking about both,
but I'll start with pretraining

and specifically
I'll talk about what

is the task of pretraining LLMs
and what is the loss that people

actually use.

So, language modeling,
this is a quick recap.

Language models at a
high level are simply

models of probability
distribution over sequences

of tokens or of words.

So it's basically
some model of p of x1

to XL, where x1
is basically what

one and XL is the last one in
the sequence or in the sentence.

So, very concretely, if you
have a sentence like the mouse

ate the cheese, what
the language model gives

you is simply a probability
of this sentence being uttered

by a human or
being found online.

So, if you have another sentence
like "The the mouse ate cheese."

Here, there's
grammatical mistakes.

So, the model should
know that this should

have some syntactic knowledge.

So, it should know that
this has less likelihood

of appearing online.

If you have another sentence
like the cheese ate the mouse,

then the model should
hopefully know about the fact

that usually cheese
don't eat mouse.

So, there's some
semantic knowledge

and this is less likely
that the first sentence.

So, this is basically at a high
level what language models are.

One word that you probably have
been hearing a lot in the news

are generative models.

So, this is just something
that can generate.

Models that can
generate sentences

or can generate some data.

The reason why we say language
models are generative models

is that once you have a
model of a distribution,

you can simply sample
from this model.

And now we can generate data.

So we can generate sentences
using a language model.

So the type of models that
people are all currently using

are what we call
autoregressive language models.

And the key idea of
autoregressive language models

is that you take this
distribution over words

and you basically decompose
it into the distribution

of the first word, multiply
by the distribution of

or the likelihood of the
distribution of the second word

given the first
word, and multiply it

by P of the third word
given the first two words.

So, there's no
approximation here.

This is just the chain rule
of probability, which you

hopefully you all know about.

Really no approximation.

This is just one way of
modeling a distribution.

So, slightly more
concisely, you can write it

as a product of P's of the next
word, given everything which

happened in the past.

So, of the context.

So, this is what we call
autoregressive language models.

Again, this is really
not the only way

of modeling distribution.

This is just one way.

It has some benefits
and some downsides.

One downside of
autoregressive language models

is that when you actually
sample from this autoregressive

language model,
you basically have

a for loop, which generates
the next word, then conditions

on that next word.

And then we generate
in other words.

So, basically if you
have a longer sentence

that you want to generate, it
takes more time to generate it.

So, there are some downsides
of this current paradigm,

but that's what
we currently have.

So, I'm going to
talk about this one.

Great.

So, autoregressive
language models.

At a high level, what a task of
autoregressive language model

is simply predicting the
next word, as I just said.

So, if we have a sentence
like she likely prefers,

one potential, next
word might be dogs.

And the way we do it is
that we first tokenize.

So, you take these words or
subwords you tokenize them

and then you give an
ID for each token.

So here you have
one, two, three.

Then, you pass it
through this black box.

As I already said,
we're not going

to talk about the architecture.

You just pass it through,
pass it through a model,

and you then get a distribution,
a probability distribution

over the next word or
over the next token.

And then you sample
from this distribution,

you get a new token and
then you detokenize.

So, you get a new
ID, you detokenize

and that's how you basically
sample from a language model.

One thing which is
important to note

is that the last two
steps are actually

only needed during inference.

When you do training,
you just need

to predict the most likely
token and you can just

compare to the real token
which happened in practice,

and then, you basically
change the weights

of your model to increase
the probability of generating

that token.

Great.

So, autoregressive
neural language models.

So to be slightly
more specific, still,

without talking about
the architecture,

the first thing we do is
that we have all of these.

Sorry, yes.

On the previous slide.

Predicting the probability
of the next token,

does this mean that your
final output vector has

to be the same dimensionality
as the number of tokens

that you have?

Yes.

How do you deal with
if you have more token.

Adding more token
to your [INAUDIBLE]?

Yeah so we're going to
talk about tokenization

actually later so you will
get some sense of this.

You basically can deal
with adding new tokens.

I'm kind of exaggerating.

There are methods for doing
it, but essentially people

don't do it.

So it's really
important to think about

how you tokenize your
text, and that's why

we'll talk about that later.

But it's a very
good point to note

is that you basically--
the vocabulary size, so

the number of tokens that
you have is essentially

the output of your
language model.

So it's actually pretty large.

So autoregressive
neural language models.

First thing you do is that you
take every word or every token.

You embed them so you get
some vector representation

for each of these tokens.

You pass them through some
neural network, as we said,

it's a transformer.

Then you get a representation
for all the word

and all the words
in the context.

So it's basically
a representation

of the entire sentence.

You pass it through
a linear layer,

as you just said, to
basically map it to the number

so that the output--
the number of outputs

is the number of tokens.

You then pass it
through some softmax

and you basically get a
probability distribution

over the next words given
every word in the context.

And the last that you
use is basically--

it's essentially a task of
classifying the next token.

So it's a very simple, kind
of, machine learning task.

So you use the
cross-entropy loss.

Where you basically look at the
actual target that happened,

which is the target
distribution, which

is a one hot encoding,
which in this case says,

I saw the real word
that happened is cat.

So that's a one hot
distribution over cat.

And here this is the actual--

do you see my mouse?

Oh, yeah.

This is the distribution
that you generated.

And basically you
do cross entropy,

which really just increases the
probability of generating cat

and decreases all the
probability of generating

all the other tokens.

One thing to notice is
that, as you all know again,

this is just equivalent
to maximizing the text log

likelihood because
you can just rewrite

the max over the probability
of this autoregressive language

modeling task as just being
this minimum of I just

added the log here
and minus, which

is just the minimum of the loss,
which is the cross entropy loss.

So basically
minimizing the loss is

the same thing as maximizing
the likelihood of your text.

Any question?

Questions?

OK, tokenizer.

So this is one thing
that people usually

don't talk that much about.

Tokenizers are
extremely important.

So it's really important that
you understand at least what

they do at a high level.

So why do we need tokenizers
in the first place?

First, it's more
general than words.

So one simple thing
that you might think

is we're just going to take
every word that we will have.

You just say every word
is a token in its own.

But then what happens is if
there's a typo in your word?

Then you might not have
any token associated

with this word with a typo.

And then you don't know
how to actually pass

this word with a typo into
a large language model.

So what do you do next?

And also, even if you think
about words, words is a very--

words are fine with
Latin-based languages.

But if you think about
a language like Thai,

you won't have a simple
way of tokenizing

by spaces because there are
no spaces between words.

So really, tokens are much
more general than words.

It's the first thing.

Second thing that
you might think

is that you might tokenize
every sentence, character

by character.

You might say A is one
token, B is another token.

That would actually work
and probably very well.

The issue is that then your
sequence becomes super long.

And as you probably
remember from the lecture

on transformers, the
complexity grows quadratically

with the length of sequences.

So you really don't want to
have a super-long sequence.

So tokenizers basically try to
deal with those two problems

and give common subsequences
a certain token.

And usually how you should be
thinking about it is around

an average of every token
is around 3-4 letters.

And there are many
algorithms for tokenization.

I'll just talk about one of them
to give you a high level, which

is what we call Byte Pair
Encoding, which is actually

a pretty common.

One of the two most
common tokenizers.

And the way that you
train a tokenizer

is that first you start with
a very large corpus of text.

And here, I'm really not talking
about training a large language

model yet, this is purely
for the tokenization step.

So this is my large corpus of
text with these five words.

And then you associate
every character

in this corpus of text
a different token.

So here, I just split
it up every character

with a different
token, and I just

color coded all of those tokens.

And then what you do is that
you go through your text,

and every time you see pairs
of tokens that are very common,

the most common pair of
token, you just merge them.

So here you see three
times the tokens t and o

next to each other.

So you're just going to
say this is a new token.

And then you continue,
you repeat that.

So now you have tok, tok
which happens three times.

Toke with an E that
happens 2 times and token,

which happens twice, and then
ex which also happens twice.

So this is the-- if you were to
train a tokenizer on this corpus

of text, which is
very small, that's

how you would finish
with a token--

with like trained tokenizer.

In reality, you do it on
much larger corpus of text.

And this is the
real tokenizer of--

actually, I think this
is GPT3 or ChatGPT.

And here you see how it would
actually separate these words.

So basically you
see the same thing

as what we gave in
the previous example.

Token becomes its own token.

So tokenizer is
actually split it up

into two tokens token and -izer.

So yeah, that's all
about tokenizers.

Any questions on that?

Yeah.

How do you deal with
spaces, and how do you

deal with [INAUDIBLE].

Yeah so actually there's
a step before tokenizers,

which is what we call
pre-tokenizers, which

is exactly what you just said.

So this is mostly--

in theory, there's no reason to
deal with spaces and punctuation

separately.

You could just say every
space gets its own token,

every punctuation
gets its own token,

and you can just
do all the merging.

The problem is that-- so
there's an efficiency question.

Actually, training these
tokenizers takes a long time.

So you better-- because you have
to consider every pair of token.

So what you end up doing is
saying if there's a space,

this is very--
like pre-tokenizers

are very English specific.

You say if there's
a space, we're

not going to start looking
at the token that came before

and the token that
came afterwards.

So you're not merging
in between spaces.

But this is just like a
computational optimization.

You could theoretically
just deal with it

the same way as you deal
with any other character.

And--

Yeah.

When you merge tokens to delete
the tokens that you merged away

or do you keep the smaller
tokens that emerge?

You actually keep
the smaller tokens.

I mean, in reality, it doesn't
matter much because usually

on a large corpus of text, you
will have actually everything.

But you usually
keep the small ones.

And the reason why
you want to do that

is because if-- in case there's,
as we said before, you have

some grammatical
mistakes or some typos,

you still want to
be able to represent

these words by character.

So, yeah.

Yes.

Are the tokens unique?

So I mean, say in this case
T-O-K-E-N is there only one

occurrence or could--

do you need to leave multiple
occurrence so they could have--

take on different
meanings or something?

Oh I see what you say.

No, it's every token
has its own unique ID.

So a usual-- this
is a great question.

For example, if you
think about a bank, which

could be bank for like
money or bank like water,

it will have the same token.

But the model will
learn, the transformer

will learn that based on the
words that are around it,

it should associate that--

I'm saying-- I'm being
very handwavy here,

but associate that with
a representation that

is either more like the bank
money side or the bank water

side.

But that's a transformer
that does that.

It's not a tokenizer.

Yes.

Yes.

So you mentioned
during tokenization,

keep the smaller tokens
you started with, right.

Like if you start with
a T you keep the T

and then you build
your tokenize out to

[INAUDIBLE] allow input token.

So let's say maybe you didn't
train on token, but in your data

you are trying to encode token.

So how does the tokenizer know
to encode it with token or to

[INAUDIBLE]?

Yeah.

The great question.

You basically when you--
so when you tokenize,

so that's after training
of the tokenizer

when you actually
apply the tokenizer

you basically always
choose the largest token

that you can apply.

So if you can do token,
you will never do T,

you will always do token.

But there's actually--
so people don't usually

talk that much about
tokenizers, but there's

a lot of computational benefits
or computational tricks

that you can do for making
these things faster.

So I really don't think
we-- and honestly, I

think a lot of people think
that we should just get away

from tokenizers and just
kind of tokenize character

by character or bytes by bytes.

But as I said, right now
there's this issue of length,

but maybe one day, like
in five or 10 years,

we will have different
architectures

that don't scale quadratically
with the length of the sequence.

And maybe we'll move
away from tokenizers.

So can you share
with us the drawback?

Why do people want to move
away from the tokenizer?

Yeah.

So I think one good
example is math.

If you think about math,
actually numbers right now

are not tokenized.

So for example, 327 might
have its own token, which

means that models,
when they see numbers,

they don't see them
the same way as we do.

And this is very
annoying because I mean,

the reason why we can
generalize with math

is because we can deal with
every letter separately

and we can then do composition.

Where you know that
basically if you add stuff,

it's the same thing as
adding every one separately

plus like whatever
the unit that you add.

So they can't do that.

So then you have to do
special tokenization.

And, like, one of the
big changes that GPT4 did

is changing the way
that they tokenize code.

So for example, if you have
code, you know you have often,

in Python, these four
spaces at the beginning.

Those were dealt with
strangely before.

And as a result, like,
the model couldn't really

understand how to
deal with code.

So tokenize actually
matter a lot.

OK, so I'll move on right now,
but we can come back later

on tokenizers.

Great.

So we talked about a task
the loss the tokenizer,

let's talk a little
bit about evaluation.

So the way that LLMs
are usually evaluated

is what we call-- is using
what we call perplexity.

At a high level it's basically
just your validation loss.

The slight difference
with perplexity

is that we use something that
is slightly more interpretable,

which is that we use the
average per token loss,

and then you exponentiate it.

And the reason why
you exponentiate it

is because you want--

I mean, the loss has
a log inside and you--

like one humans
are actually pretty

bad at thinking in log space.

But two logs depend
on the base of the log

while when you exponentiate
you basically have everything

in the vocabulary size unit.

And the average per
token is just so

that your perplexity is
independent of the length

of your sequence.

So perplexity is just
two to the power average

of the loss of the sequence.

So perplexity is between one
and the length of the vocabulary

of your tokenizer.

One it's simply well,
if you predict perfectly

the thing which every
word, then every word

will have basically
products of ones.

So the best perplexity
you can have is one.

If you really have no
idea, you basically

predict with one divided
by size of vocabulary

and then you do simple
math and you basically

get perplexity of
size of vocabulary.

So the intuition
of perplexity is

that it's basically
the number of tokens

that your model is, kind
of, hesitating between.

So if your model is perfect,
it doesn't hesitate.

It know exactly the word.

If it really has
no idea, then it

hesitates between all
of the vocabulary.

So perplexity really improved.

That's perplexity on a standard
data set between 2017 and 2023.

It went from a kind of 70
tokens to less than 10 tokens

over these five, six years.

So that means that the
models were previously

stated between 70 words every
time it was generating a word,

and now it's hesitating
between less than 10 words.

So that's much better.

Perplexity is actually
not used anymore

in academic benchmarking,
mostly because it depends

on the tokenizer that you use.

It depends on the actual data
that people are evaluating on.

But it's still very important
for development of LLMs.

So when you actually
train your own LLM people

will still really look
at the perplexity.

One common other way and
now more common in academia

of evaluating these LLMs is just
by taking all the classical NLP

benchmarks, and I'll give you
a few examples later and just,

kind of, aggregating everything.

So collect as many automatically
evaluatable benchmarks

and just evaluate
across all of them.

So one such-- or
actually two such

benchmarks are what we call
HELM, which is from Stanford.

And another one is the
Hugging Face open leaderboard,

which are probably the two
most common ones right now.

So just to give you
an idea, in HELM,

all of these type
of tasks, which

are mostly things that
can be easily evaluated

like question answering.

So think about many different
question answering tasks.

And the benefit with
question answering

is that you usually know
what is the real answer.

So you can-- the way that
you evaluate these models

and I'll give you a concrete
example in one second,

is that you can just look at
how likely the language model is

to generate the real answer
compared to some other answers.

And that's essentially,
at a high level,

how you evaluate these models.

So to give you a
specific example,

MMLU is probably the most common
academic benchmark for LLMs.

And this is just a
collection of many question

and answers in all
of those domains.

For example, college
medicine, college physics,

astronomy and these
type of topics.

And the questions are things
like, so this is in astronomy.

What is true for
type-1a supernova?

Then you give four
different potential answers

and you just ask the model
which one is more likely.

So there are many
different ways of doing it.

Either you can look at the
likelihood of generating

all these answers, or
you can ask the model

which one is the most likely.

So there are different ways
that you can prompt the model,

but at a high level, you
know which one is correct.

And there are three
other mistakes.

Yes.

Creating unconstrained
text as an output.

Yeah.

How do you evaluate
a model if it

gives something that's
semantically completely

identical, but is not the
exact tokens that you expect?

Yeah.

So that's a great question.

I'll talk more about that later.

Here, in this case, we
don't do unconstrained.

So the way you would evaluate
MMLU is basically either

you ask the first
question, and then you

look at the likelihood of
the model generating A,

the likelihood of the model
generating B, C, and D

and you look at which
one is the most likely.

Or you can ask the
model out of A, B, C, D,

which one is the most likely.

And you look at whether the
most likely next token is A, B,

C, or D. So you
constrain the model

to say it can only
answer these four things.

You say you constraint--

Yeah.

You constrain the
prompt or do you

mean of its whole
probability distribution

that it outputs
you only comparing

the outputs of like-- you're
only comparing the A token the

[INAUDIBLE].

Yeah.

So in the second case I gave
you, you would do exactly the--

actually would do both.

You would prompt the
model saying A, B, C, or D

plus you would constrain to
only look at these four tokens.

In the first case, you don't
even need to generate anything.

So in the first case,
you literally just

look, given it's
a language model,

it can give a distribution
over sentences.

You just look at what is
the likelihood of generating

all of these words?

What is the likelihood of
generating the second choice?

And you just look at whether the
most likely sentence is actually

the real answer.

So you don't actually
sample from it,

you really just
use P of X1 to XL.

Does that make sense?

That being said, evaluation
of open-ended questions

is something we're going
to talk about later,

and it's actually
really important

and really challenging.

Yes.

Earlier you mentioned
[INAUDIBLE] metrics

like perplexity
are not I usually

use because it
depends on how you do

your tokenization,
some design choices.

I was wondering if you
could speak more to that.

Yeah.

So think about perplexity.

I told you perplexity is
between 1 and vocabulary size.

So now imagine that ChatGPT
uses a tokenizer that has 10,000

tokens but Gemini from Google
uses a tokenizer that had

100,000 potential tokens.

Then actually the Gemini one
will have the upper bound

of the perplexity that you can
get is actually worse for Gemini

than for ChatGPT.

Does that make sense?

So that's just an idea.

It's actually a little bit
more complicated than that,

but that's just one
festival with a bit

of where you can see that the
tokenizer actually matters.

Great.

OK, so evaluation challenges.

There are many.

I'll just talk about
two really briefly.

One, as I told you, there are
two ways of doing evaluation

for these MMLUs.

Actually, there are
many more than two

but I gave you two examples.

And it happens that
for a long time,

even though that was a
very classical benchmark

that everyone uses actually
different companies

and different
organizations were actually

using different ways
of evaluating MMLU.

And as a result, you get
completely different results.

For example, Llama-65b, which
was the first model of meta

in the llama series, had
on HELM 63.7 accuracy

but on this other
benchmark had like 48.8.

So really the way that you
evaluate, and this is not even

talking about prompting
this is really just the way

that you evaluate the models.

Prompting is another issue.

So really, there are a
lot of inconsistencies.

It's not as easy as it looks.

First thing.

Yeah, sorry.

How can we make sure
that all these models

are trained on the benchmark?

Second thing.

This is a great question.

Train test contamination.

This is something
which I would say

is really important
in academia in--

given that the talk is mostly
about training large language

models, for companies, it's
maybe not that important

because they know
what they trained on.

For us, we have no idea.

So, for us, it's a real problem.

So there are many
different ways of trying

to test whether the test set--

or sorry, whether the
test set was actually

in the training set.

One, kind of, cute trick
that people in the lab,

in [? Tatsuo's ?] lab have
found, is that what you can do

is that given that most
of the data set online

are not randomized,
you can just look at--

and that language models,
what they do is just

predict the next word.

You can just look at
the entire test set.

What if you generate
all the examples

in order versus all the
examples in a different order.

And if it's more likely to
generate a thing in order, given

that there's no
real order there,

then it means that probably
it was in the training set.

Does that make sense?

So there are many--
that's like one of them.

There are many other
ways of doing it.

Train test
contamination, again, not

that important for development,
really important for

academic benchmarking.

Great.

So there are many
other challenges,

but I'll move on for now.

Great.

Data.

So data is another
really big topic.

At a high level people
just say you basically

train large language
models on all of internet.

What does that even mean?

So people sometimes say,
well, of clean internet,

which is even less defined.

So internet is very dirty
and really not representative

of what we want in practice.

If I download a random
website right now,

you would be shocked
at what is in there.

It's definitely
not your Wikipedia.

So I'll go really briefly
on what people do.

I can answer some
questions, but I mean,

data is on its own
it's a huge topic.

Basically, first what you do
is download all of internet.

What that means is that
you use web crawlers that

will go on every web page, on
internet or every web page that

is on Google.

And that is around 250
billion pages right now.

And that's around
1 petabyte of data.

So this is actually a Common
Crawl is one web crawler.

So people don't usually
write their own web crawlers

what they do is that they
use standard web crawlers,

and Common Crawl is one of them
that basically every month adds

all the new websites that were
added on internet that are found

by Google, and they put it in
a big basically a big data set.

So that's-- on Common Crawl, you
have around 250 billion pages

right now.

So 1E6 gigabytes of data.

Once you have this--

so this is a random web page.

Like literally random
from this Common Crawl.

And what you see is
that one, it really

doesn't look at type of things
that you would usually see,

but actually-- so
this is an HTML page.

It's hard to see, but
if you look through

will see some content.

For example, here,
Test King World

is your ultimate source for
the system x high performance

server.

And then you have three dots.

So you don't even-- the
sentence is not even finished.

That's how random
internet looks like.

So, of course, it's
not that useful

if you just train a
large language model

to generate things like this.

So what are some of the
steps that are needed?

First one, you extract
the text from the HTML.

So that's what I just
tried to do by looking

at basically the correct tags.

There are a lot of
challenges through this.

For example, extracting
math is actually

very complicated, but pretty
important for training

large language models.

Or for example, boilerplates.

A lot of your forums will
have the same type of headers,

the same type of footers.

You don't want to repeat
all of this in your data,

and then you will filter
undesirable content.

So not safe for work,
harmful content, PII.

So usually every
company has basically

a blacklist of websites
that they don't

want to train their models on.

That blacklist is very
long and you basically

say if it comes from there,
we don't train on this.

There are other ways
of doing these things.

Is that you can train a small
model for classifying what

is PII, removing these things.

It's hard.

Every point here that
I'm going to show you

is a hard amount of
work, but I'm just

going to go quickly through it.

So filter undesirable content.

Second or fourth
is de-duplication.

As I said, you might have
things like headers and footers

in forums that are
always the same.

You want to remove that.

Another thing that
you might have

is a lot of URLs that are
different, but actually show

the same website.

And you might also have a lot of
paragraphs that come from common

books that are basically
de-duplicated 1,000 times

or 10,000 times on internet.

So you have to de-duplicated.

Also very challenging because
you have to do that at scale.

Once you do the
de-duplication, you

will do some
heuristic filtering.

You will try to remove
low-quality documents.

The way you do that are things
like rules-based filtering.

For example, if you see that
there are some outlier tokens.

If the distribution of
tokens in the website

is very different than the
usual distribution of tokens,

then it's probably some outlier.

If you see that the length
of the words in this website

is super long, there's something
strange going on that website.

If you see that the website
has only three words,

maybe, is it worth
training on it.

Maybe not.

If it has 10 million words,
maybe there's something also

wrong going on that page.

So a lot of rules like this.

Yes.

Why do we filter out
undesirable content

from our data set instead
of putting it in as,

like, a supervised loss?

Can we not just say, here's
this like, hate speech website,

let's actively try to--

let's actively penalize
the model for getting it.

We'll do exactly that,
but not at this step.

That's why the post-training
will come from.

Pretraining the
idea is just to say

I want to model, kind of, how
humans speak, essentially.

And I want to remove all
these headers, footers

and menus and things like this.

But it's a very good
idea that you just had.

And that's exactly
what we'll do later.

Next step,
model-based filtering.

So once you filter a lot
of data, what you will do--

that's actually a
very cute trick.

You will take all
of Wikipedia and you

will look at all
the links that are

linked through Wikipedia pages.

Because probably if something
is referenced by Wikipedia,

it's probably some
high-quality website.

And you will train a classifier
to predict whether something

comes from-- whether a
document comes from one

of these references
from Wikipedia

or whether it's
from the random web.

And you will try
to basically say,

I want more of the things that
come from Wikipedia references.

Does that make sense?

So yeah.

So you will train a
machine learning model.

Usually also very simple
models because you

need to do that really at scale.

I mean, just think about
the 250 billion pages.

Next one, you will try
to classify your data

into different domains.

You will say, OK, this is
entertainment, this is books,

this is code, this is like
these type of domains.

And then you will try to
either up or down weight

some of the domains.

For example, you might say--

you might see that actually if
you train more on code, then

actually your model becomes
better on reasoning.

So that's something that
people usually say in

a very hand-wavy way.

If you train your
model more on code,

actually it helps reasoning.

So you want to update
the coding distribution

because that helps for general
language modeling skills.

Books is usually also another
one that people usually update.

Entertainment, they
usually down weight.

So things like this.

Of course, you want to do it--
so people used to do it, maybe

kind of heuristically.

Now there's entire
pipelines that we'll

talk about of how to do
these things slightly

more automatically.

And then at the end of
training, you usually train--

after training on all
of this data that we saw

you usually train on
very high quality data

at the end of training your
large language model where you

decrease your learning rate.

And that basically
means that you're,

kind of, overfitting your model
on a very high quality data.

So usually what you
do there is Wikipedia.

You basically
overfit on Wikipedia

and you overfit on, like,
human data that was collected.

The other thing is like
continual pretraining

for getting longer context.

I'm going to skip over
all of these things.

But that's just to give
you a sense of how hard it

is when people just say I'm
going to train on internet,

that's a lot of work.

And, really, we haven't
figured it out yet.

So collecting well
data is a huge part

of practical, large
language model.

Some might say that
it's actually the key.

Yes.

[INAUDIBLE] about data.

So basic question.

So usually when you start
with like a petabyte of data,

after you go through
all the steps,

what's the typical amount
of data you have remaining.

And then how large a
team does it typically

take to go through all the
data steps you talked about?

Sorry how la-- is your
question how large

is the data after you filter?

Yeah.

After you filter and then
you go through all the steps.

How large a team do you
need to go through, like,

all the filtration
steps you mentioned.

How slow is it or--

How many people
would you need to be

able to do this [INAUDIBLE]?

OK that's a great question.

I'm going to somewhat
answer about the data.

How large is the data set
at the end of this slide.

For number of people that work
on it, that's a good question.

I'm actually not quite
sure, but I would say, yeah,

I actually don't
quite know but I

would say it's probably even
bigger than the number of people

that work on the tuning of
the pretraining of the model.

So the data is bigger
than the modeling aspect.

Yeah, I don't think
I have a good sense.

I would say probably in LLAMA's
team, which have 70-ish people,

I would say maybe
15 work on data.

Yeah.

All these things, you don't
need that many people,

you need a lot of compute also.

Because for data you
need a lot of CPUs.

So, yeah.

And I'll answer
the second question

at the end of this slide.

So as I just, kind
of, alluded to really,

we haven't solved data
at all for pretraining.

So there's a lot of research
that has to be done.

First, how do you process
these things super efficiently?

Second, how do you
balance kind of all

of these different domains?

Can you do synthetic
data generation?

That's actually a
big one right now.

And because we don't have--

we'll talk about that
later, but we don't have

enough data on the internet.

Can you use multimodal data
instead of just text data?

And how does that improve
even your text performance?

There's a lot of secrecy
because, really, this

is the key of most of the
pretraining large language

models.

So for competitive dynamics,
usually these companies

don't talk about how they
do the data collection.

And also there's a
copyright liability issue.

They definitely don't
want to tell you

that they've trained on
books even though they did

because if not can sue them.

Common academic benchmarks.

So that will, kind of,
answer what you asked.

It started-- so those
are the smaller ones.

The names are not
that important,

but it started from around
$150 billion tokens, which are

around 800 gigabytes of data.

And now it's around
15 trillion--

15 trillion tokens,
which is also

the size of the models that
are-- right now the best models

are probably trained
on that amount of data.

So 15 trillion tokens,
which is probably,

I guess, two orders of
magnitude bigger than that.

So 80E3 gigabyte.

So that would be around 100
to 1,000 times filtering

of the Common Crawl,
if I'm not mistaken.

So, yeah.

One very famous one is the Pile.

So this is an academic
benchmark, the Pile.

And we can just look at what
distribution of data they have.

It's things like
archive, PubMed Central,

which is all the biology stuff.

Here it's Wikipedia, you see
Stack Exchange, some GitHub

and some books and
things like this.

Again, this is on
the smaller side.

So this is-- if we look at here,
this is on 280B so, in reality,

it's like 100 times bigger
so you cannot have that much

of GitHub and of Wikipedia.

In terms of closed
source models.

Just to give you
an idea, Llama 2

it was trained on
2 trillion tokens,

Llama 3 15 trillion
tokens, which is currently

the best model that we know
on how much it was trained on,

which is the same thing as is
the best academic or the biggest

academic benchmark, which
is 15 trillion tokens.

GPT4 we don't really
but it's probably

in the same order of magnitude
or it's probably around that.

Actually, it's probably
around 13 from leaks.

If the leaks are true.

Great.

So scaling laws.

Any other questions on data
before we go to scaling laws?

Sorry I know I'm giving
you a lot of information,

but there's a lot into
training, large language models.

Great scaling laws.

So the idea is that what people
saw around 2020, or at least

from a long time, but they've
been able to theoretically show

it or empirically
show it since 2020,

is that the more data
you train your models on

and the larger the models,
the better the performance.

This is actually pretty
different than what

you've seen in this class.

In this class we teach
you about overfitting.

Overfitting doesn't happen
with large language models.

Larger models,
better performance.

It's something that
really took a long time

for the community who took
this type of class to realize.

But for the exam,
overfitting exists.

So, OK, the idea of scaling loss
is that if-- given that more

data and larger
models will always

give you better
performance, can we

predict how much better
your performance will

be if you increase the amount of
data and the size of your model?

And surprisingly, it works.

So here you see three plots
from a very famous paper called

Scaling Laws from OpenAI.

Here you see on
the x-axis compute.

So how much did you train--

like, how much compute did
you spend for training?

And here you see test loss.

So this is essentially,
I mean, perplexity,

but it's your validation loss.

So it's a log of the perplexity.

And if you put these
two on log scale,

then you see that the
performance or the--

sorry, the scaling
law is linear.

That means that if you
increase your compute

by a certain amount, you can say
by how much your test loss will

actually decrease.

Same thing with data and
same thing for parameters.

If you increase
the data set size,

your loss will
decrease by an amount

that is somewhat predictable.

If you increase the
number of parameters,

the loss will
decrease by an amount,

which is somewhat predictable.

This is really amazing.

Very surprising.

I mean, it looks innocuous when
you look at these type of plots,

but that's crazy because it
means that you can predict

how well we're going to
perform in two or three years,

depending on how much
compute we will add,

assuming that these
things will hold.

There's nothing
theoretical about it.

Yes.

Two things.

One, what is the loss
that they're using here.

Is this perplexity?

So it's-- I said perplexity was
like 2 to the power of the loss.

So this is the power
of the perplexity.

And then the second
thing is, when

you increase the
number of parameters

or you increase the data
set size [INAUDIBLE] data

[INAUDIBLE] times, doesn't
that just inherently

increase your compute?

Like does all of this
[INAUDIBLE] come to just how

[INAUDIBLE] you [INAUDIBLE]?

Yes.

--or something
specific [INAUDIBLE]?

No, this is a great question.

So the compute here is actually
a factor of two things, the data

and the parameter.

What I'm showing here
is that you can--

well, actually, we're going
to talk about that in details.

But basically, if you increase
the number of parameters,

you should increase the
number of data that you have.

So you actually don't
go multiple times

to the same data set.

No one does epochs
in at least not yet

because we haven't still
kind of enough data.

So yeah, this is
all the same trend,

which is increase
compute decrease loss.

Yes.

Have we seen the numbers for
the last two years or this

is still holding?

It is still holding.

I don't have good
numbers to show you,

but it is still
holding, surprisingly.

Yes.

Is there no evidence that
control quality density

will ever plateau?

In theory, we would expect
it plateau, [INAUDIBLE]?

No empirical evidence of
plateauing anytime soon.

Why?

We don't know.

Will it happen?

Probably.

I mean, it doesn't need
to because it's actually

in log scale.

So it's not like
as if it had to go.

It had to plateau.

Like mathematically, it could
continue decreasing like this.

I mean, most people think
that it will probably

plateau at some point.

We don't know when.

So that's-- I'll talk more
about scaling laws now.

So why are scaling
laws really cool?

Imagine that I gave you--

you're very fortunate I gave
you 10,000 GPUs for this month.

What model will you train?

How do you even go about
answering that question?

And I mean, this
is a hypothetical,

but that's exactly what these
companies are faced with.

The old pipeline,
which was basically

tune hyperparameters
on the big models.

So let's say I have
30 days, I will train

30 models for one day each.

I will pick the best one and
that will be the final model

that I will use in production.

That means that the model
that I actually used

was only trained for one day.

The new pipeline is that you
first find a scaling recipe.

So you find something that
tells you, for example,

like one common thing
is that if you increase

the size of your model, you
should decrease your learning

rate.

So you find a
scaling recipe such

that you know if I increase
the size of my model,

here's what I should do
with some hyperparameters.

Then you tune your
hyperparameters

on smaller models
of different sizes.

Let's say I will say for
three days, of my 30 days,

I will train many
different models.

And I will do
hyperparameter tuning

on these small models,
each of different sizes.

Then I will fit a
scaling law and try

to extrapolate from these
smaller models, which

one will be the best if I
train it for much longer--

or sorry if I train
it for a larger model.

And then I will train
the final huge model

for 27 days instead
of just one day.

So the new pipeline
is not train things

or do hyperparameter tuning
on the real scale of the model

that you're going
to use in practice,

but do things on smaller
ones at different scales.

Try to predict how
well they will perform

once you make them bigger.

I will give-- I will give you a
very concrete example right now.

Let's say transformers
versus LSTMs.

Let's say you have
these 10,000 GPUs,

you are not sure which
one you should be using.

Should I be using a
transformer-based model

or LSTM-based model.

What I will do is I
will train transformers

at different scales.

So here you see different
parameters on the x-axis,

y-axis is my test source.

I will then train different
LSTMs at different scales.

Once I have these points,
I will see oh it, kind of,

fits a scaling law.

I will fit my
scaling law and then

I will be able to predict if
I had 10 times more compute,

here's how well I would
perform for the LSTM.

It's actually slightly
less linear for the LSTM,

but you can probably try to
predict where you would end up.

And clearly from this
plot, you would see

that transformers are better.

One thing to notice when you
read these type of scaling laws

is that there are two
things that are important.

One is really your
scaling rate, which

is the slope of the-- the
slope of the scaling law.

The other thing
is your intercept,

you could start
worse, but actually

become better over time.

It just happens that
LSTMs are worse for both.

But I could show you
another one where things--

you can predict that actually
after a certain scale

you're better off using that
type of model than others.

So that's why scaling laws
are actually really useful.

Any questions on that?

Yeah.

So these are all,
kind of, very--

how sensitive are these to small
differences in the architecture.

Like one like
transformer architecture

versus another
transformer architecture.

Do you think we have
to fit your own curve

and, basically, say like oh
scaling laws tell me this should

be some logarithmic function.

Like, let me
extrapolate that for

my own specific architecture.

Yeah, so usually, for
example, if you're an academic

and you want to-- now at
least that's pretty recent

and you want to propose
a new activation.

That's exactly what you will do.

You will fit a scaling law,
show another scaling law

with the standard
like, I don't GELU

and you will say
that it's better.

In reality, once you start
thinking about it in scaling

laws terms, you really
realize that actually

all the architecture
differences that we

can make, like the small,
minor ones, all they do

is maybe change a little
bit the intercept.

But really that doesn't
matter because just

train it for 10 hours longer or
like wait for the next computer

GPUs and these things
are really secondary.

Which is exactly why I was
telling you originally,

people spend too much time on
the architecture and losses.

In reality, these things
don't matter as much.

Data though.

If you use good data, you will
have much better scaling laws

than if you use bad data.

So that really matters.

Another really cool thing
you can do with scaling laws

is that you can ask yourself,
how to optimally allocate

training resources.

Should I train larger models.

Because we saw that it's better
when you train larger models,

but we saw that it's also
better when you use more data.

So which one should I do?

Should I just train on
more data, a smaller model,

or should I train a
larger model on less data?

So Chinchilla is a very famous
paper that first showed this.

The way they did it,
I want to give you

a little bit of a sense
of what these plots are.

Here you see training
loss again on the x-axis,

you see parameter differences,
sorry, parameter size--

number of parameters.

So the size of the model.

And here all these
curves are what

we call ISO flops, which is that
all the models on this curve

have been trained with the
same amount of compute.

The way that you do
that is that you train--

you change.

Sorry, you vary the number of
tokens that were trained on

and the size of the models,
but you vary in such a way

that the total compute
is constant, OK.

So all these curves that you
see with different colors

have different amount of
compute that were trained on.

Then you take the best one
for each of those curves.

Once you have the best one
for each of those curves,

you can ask-- you can
plot how much flops it was

and which curve were you
on and how much parameters

did you actually use for
training that specific point.

You put that on the log
log scale again and now

you fit a scaling law again.

So now I have something
which tells me

if I want to train a model of 10
to the power 23 flops, here is

exactly the number of parameters
that I should be using.

100 B.

And you can do the same
thing with flops and tokens.

So now you can predict--

if I tell you exactly I
have one month of compute,

what size of model
should I be training?

Fit the scaling
law, and I tell you.

Of course that all
looks beautiful.

In reality like there's a
lot of small things of like,

should you be counting,
like, embedding parameters,

there's a lot of complexities.

But if you do things well,
these things actually do hold.

So the optimal number of
parameters that Chinchilla paper

have found is to use 20
tokens for every parameter

that you train.

So if you add one
more parameter,

you should train your thing on--
your model on 20 more tokens.

So one caveat here is that this
is optimal training resources.

So that is telling me if you
have 10 to the power, 23 flops

or if you have 100, I don't know
how much that is, $100 million

or 10-- no, that's
much less, actually.

Let's say I have
$5 million to train

my best model that
gets the lowest

loss what would I train on?

In reality, these companies need
to think about inference also.

If you have a smaller model,
they will spend less over time.

So actually, if you
consider the inference cost,

you have other papers that
try to show that, it's

around 150 parameters, sorry--

tokens per parameters, because
you prefer having a smaller

model because over
time you're going

to actually spend less money
on inference of these models.

So 150 to 1, that's around what
the best models are trained

on right now, at least
the ones that are

used in practice in production.

Great.

Any questions on Chinchilla?

Great.

Oh sorry.

In practice, how expensive
is inference for these models

relative to training?

Actually, very expensive.

I will not talk about
inference because that would

be another entire lecture.

But just think
about ChatGPT where

they have I don't know
how much it is now,

like 600 million
people that use it.

Like, that's a lot.

Yeah.

So it's actually very expensive.

There's a lot of optimization
you can do for inference though.

And that's an entire
other lecture.

I'm going to skip that this
time, but it's very interesting.

OK tunings.

As I said, there are
many things that you

can answer with scaling laws.

I just try to give
you two examples,

but really there
are many things.

What data do you use.

What mixture-- what data
mixing weighting you use.

The mixtures, that's what
we talked about before.

What architecture you use,
whether you should make

your models wider or deeper?

Should you be
paying for more GPUs

or actually
collecting more data?

All these things are
things you can try

to answer with scaling laws.

One thing I want to say
is the bitter lesson.

If you ever heard
of Richard Sutton,

very famous blog post in
2019, what he realized,

which I think not
enough people realize,

I didn't-- definitely did
not realize at that time,

is that once you see these type
of scaling laws you know that

the more compute you have, the
better models you will get.

So with scale, you
will get better model.

And you also know by
Moore's law or these type

of variants of Moore's
law that you will always

have better compute.

Then the only thing
that matters is just

to have architectures that
can leverage computation.

So what matters is basically
systems data and less

so the architecture, like
the small architecture

differences like, your
activation and things like this.

So I think that's one of the
reasons why most of research

focuses on some things that
for industry matters less.

And I was one of
those researchers

for a large part of my career.

So don't spend time
over complicating.

Do the simple
things, do it well.

See all them.

That's really what OpenAI taught
us with ChatGPT and with all

the GPTs before.

OK, I want to give you some back
of the envelope computation.

So I might be off by
a few factors here,

but I just want to give you
a sense of how costly it is

to train some of these models.

I'll give us an example.

llama3 400b which is currently
the best open source model that

you can get.

It was trained on 15.6 tokens.

It has 405 billion parameters.

So just now that
you know what is

like this optimal tokens per
parameter, that's around 40.

So that's a little bit
more than Chinchilla,

but less than this like
inference optimal model.

So they went for
training optimallity

Flops for this model.

So one simple way
to compute flops

is 6 times the
number of parameters,

times the number of
data that you train on.

So if you do the simple
calculation here,

it's 3.8 e25 flops.

The reason why this
is important is

that if you follow it
a little bit, the news,

there's an executive order
from Biden that basically

says that once you have one e26
parameters, sorry, flops, then

you have special
scrutiny on your models.

So they went to
2X less than that.

So they really went
right below this

to not have special scrutiny.

So 3.8.

I might be off by a little
bit, but it's definitely

under the 1 e26

So parameter p is parameters
n is data, number of tokens.

This is just an approximation.

Yeah.

OK.

Compute and we know that they
trained on 16,000 h100s and we

know the throughput
they set it to.

So if you do the computation,
it takes around 70 days

or 26 million GPU hours.

At least that's what my back
of the envelope computation.

They actually said that
they use 30 million

instead of 26 million GPU hours.

So maybe they had
some challenges.

I don't really know.

But if you follow the
simple computation,

it's around 70 days.

Cost.

I mean this it's
hard to approximate,

but I'm just going to say
it's, kind of, the rent.

Like, what if I wanted to
rent H100, that many H 100

for that many days,
how much will I pay?

H100 a lower bound on
the renting costs of H100

is around two hours--

$2 per hour.

So if you multiply this
by 26,000,000 hours,

you get $52 million.

So they probably
pay less than that,

but not actually much less
because all these services

that actually rent GPUs, they
don't make that much money.

So it's probably slightly
less, but not that much less.

Now salary I said 50
employees, 500k per year.

Yeah it's probably
the right ballpark.

$25 million.

So if you put altogether
around $75 million

for training this llama model.

I'm probably off
by like 10 million,

but that's kind
of right ballpark.

Carbon emitted.

A lot of people might ask
like also the cost is not

the only thing
that is important.

So I did the computation.

It's around 4000 tons
of CO2 equivalent.

That is actually only
2000 return tickets

from JFK to London.

So right now carbon
emitted is actually not--

I mean, it's huge, but
it's not meaningful yet.

I think in maybe GPT6,
GPT7, once you multiply this

by 100, that might
become a real issue.

Right now it's
still not, I think,

an issue in the grand
scheme of things.

Next model the way you should be
thinking about these models is

that every new generation, the
number of flops essentially

multiplies 10x, or at least
that's what they try if they

have enough energy.

And if they can buy enough GPUs.

Great.

Any question on these
back of the envelope math.

No.

OK.

So now we talked
about pretraining,

I wanted to also
chat about systems

because now we know compute
is really important so there's

a question of how do
you optimize the--

how do you optimize the compute?

I will leave that for
the end because I'm not

sure how much time we will have.

I think it's important,
but hopefully I'll

be able to talk about it later.

It's slightly different
than what we've

been talking about right now.

So I'll move on to
post-training for now.

So the task of
post-training, the reason why

we need to do post
training is, as I told you

before, it's to
make AI assistants.

So language modeling
is not really the thing

that you want when you
have an AI assistant.

For example, if you
ask to GPT3, which

is a purely language model--

a pure language model,
not a non-aligned one.

If you ask a question
explain the moon landing

to a six-year-old, the
completion that you would get

is something explain the theory
of gravity to a six-year-old.

Because what it learned
is that on internet,

if you have one
question, you usually

have maybe another bullet point
of other similar questions

you don't usually have
question and then answer later.

This is not what you want
from an AI assistant.

So how do we do this
alignment, which

is this post training and
making these models assistants?

So the goal of this
alignment is to basically get

LLMs follow the
instructions that

are given by users and
maybe some designers,

kind of, desires.

So think about motivation.

You don't want the
model-- like OpenAI

doesn't want the model to
say stuff that is very toxic.

So here you see on
the left-hand side

that when you ask a question, it
actually provides a real answer.

So it's not like before the LLM.

And on the right-hand side,
you see that it would--

if you ask to write a tweet
describing how a certain part

of the population are evil, it
will say that it cannot do that.

So that's kind of
this alignment.

The background here is
that basically the data

that you want for training
some of these models is--

like, we know what we want.

Which is just asking
humans, this is a question,

this is the answer
that you want.

But the thing is that it's very
expensive to collect that data,

and it's hard to find it online.

In contrast, pretraining
data is not what you want,

but there's a lot of it.

So what we will do, or
the main idea is simply

take a pretrained
large language model

pretrained on all of internet
and then just fine tune.

So you just change a little bit
the weights on the type of data

that you actually want.

And hopefully given
it, you already

pretrained it on
all of internet,

it basically learns or knows
how to speak in English

and knows standard
language syntax

then you can really fine tune
it with very little data.

OK, SFT.

So Supervised Fine Tuning is
really exactly what I just said.

Which is the idea of
fine-tuning the large language

model on basically the
desired answers that

are collected from humans.

So why is it called
supervised fine tuning?

Because you basically want to
do language modeling on the real

answers.

So language modeling is this
like next word prediction,

and that's the fine tuning part.

And then you want to do it on
desired answers given by humans

so that's why we
call it supervised.

So how do we collect this data?

Well, I just said it.

You just ask humans
to tell you this

is a question this is
the answer that you would

want from some of these models.

So this is an example.

I can't read very
well on my computer,

but my kid needs
to do a science--

no let's read this one.

Can you write a
short introduction

about the relevance
of the term monopsony?

And then it says monopsony
refers to a market

structure, blah blah, blah.

And that's a human
network there.

So, actually, this
is Open Assistant,

which was a way to collect
data online by humans.

So this type of supervised
fine tuning or alignment

is really the key of ChatGPT.

This is what made the big jump
from GPT 3, which was mostly

something that was
known by AI researchers

to ChatGPT, which became
known by basically everyone.

So the problem
with human data is

that it's very slow to
collect and very expensive.

So one possible
simple idea is to use

LLMs to scale data collection.

So that's exactly what we
did with Alpaca one year ago.

What we did is that
we asked humans,

so we use a data set of
human question answers.

So there were 175
question answers here,

and we asked the best
model at the time,

so text-davinci 003 to basically
generate many more of these

question and answers.

So all we did is, this is
what humans would write now,

write similar answers
and similar questions.

And we collected 52,000
LLM-generated question answers.

And then what we did is
simply we took llama 7B,

which was the best
pre-trained model at the time.

And we just fine tuned this
with supervised fine tuning,

as I told you.

And that's how we got
the Alpaca 7B model.

And this is the type of
data that we collected.

So things like what
does algorithm mean?

And algorithm is a step by
step set of instructions

you use to solve a problem or
achieve a goal, blah, blah,

blah, blah.

So the data is not actually--
it's actually pretty good,

given that it was LLM generated
by LLMs from essentially two

generations ago.

So that really started
at least for us

as an academic
replication of ChatGPT.

Now it really--
there's a big field

of synthetic data
generation of how

to use LLMs to basically make
development of LLMs faster.

And basically by decreasing
the amount of human hours that

you need.

Quantity of data.

So we talked about what type
of data and how we collect it.

One thing which is
surprising with SFT

is that you don't
need that much data.

So what this paper showed
this is called LIMA,

is that if you scale the amount
of data that you use from

supervised fine tuning
from 2000 to 32,000,

it really doesn't help much.

So here scaling laws
definitely don't help.

And so the intuition here
is that all you learn

is you learn how to format
your desired answers.

Another way of saying it is that
your pre-trained models, they

essentially model the
distribution of every user

on internet, one that
might write bullet points,

another one that might
answer question-- answer

question with an answer.

So all you tell your
model is like, wait,

you should actually
be optimizing

more for this type of
user than another one.

So you're not
actually teaching it--

you're not teaching anything
through this SFT, so

supervised fine
tuning, all you do

is you tell the model to
optimize for one type of user

that it saw already in
a pretrained data set.

So the knowledge is already
in the pretrained LLM

and you basically just
specialize to one type of user.

Great.

Any question on SFT?

Yes.

So I know it's a big
issue with synthetic data

where if you keep generating
data from the same distribution,

eventually you're not
learning a new distribution,

you're essentially
playing with it.

Just bootstrapping that.

Yeah.

Surely you can't scale
that forever, right.

You can't keep going
on and generating

from the same distribution.

You hope to learned
something new.

Yeah.

So are there-- it's an
active area of research

but any thoughts
that you have around

how people are maybe thinking
around this and better ways

to bootstrap?

Or to give up on this idea and
realize that the chart shows

you don't need that many so
just get humans to generate

2000 really good prompts.

Yeah.

So that's a very good question.

So for the data
stuff, so I'm saying

it's not that important
for SFT, but there

will be another thing we'll talk
about right after where actually

data does matter.

My intuition based on not
that much empirical results

is that you can still get,
even though you use your LLMs,

if you use purely
LLM generated text

and you do that for like three
or four generations of LLMs,

I agree with you that probably
you won't improve much.

But for me what is important is
how do you use human in the loop

with LLMs?

Not purely LLMs,
not purely humans,

but maybe what
you can do is just

have the model
regenerate some new text

and just humans
write a few edits.

Edits are much faster than
writing the entire text.

And I think that if you have
that type of collaboration,

then from an information
theoretical point of view,

you still get
additional information,

but you're still much faster
than if you use humans.

And I think that
as a field we'll

probably move towards these
type of things, which is really

just finding the examples that
are important and asking humans.

It's kind of active
learning, just

asking humans exactly when
you need to get their inputs.

Yes.

Do we train with the
same loss function

and the same general
training algorithm

for the supervised
fine tuning bit

as we do for the pretraining?

Because the examples
you showed, I

think the important thing
of the good examples

is like super
factually accurate.

Like there's these
more complex things

and it's still just
like [INAUDIBLE].

Same loss.

So that's why here--

yeah, I didn't-- maybe
didn't emphasize enough.

This is just language modeling.

Fine tune the LLM with language
model and the desired answers.

So this is literally
the same loss.

It will be different
in two seconds,

but the first step
of SFT is literally

the same loss where
you just say, OK, I

want to actually specialize
on that type of data.

So there's even a question
of what is pretraining,

what is post-training?

Because, in reality, it's
just like a different data

that you use.

The reason why we usually call
it post-training is that the way

we collect that data
is very different.

Great, great questions.

Yes.

Maybe it's the same
question, but why would

these 2000 examples have
such a overweighted influence

on fine tuning?

So that's why we--

also that's another reason
why we call it post-training

is that we use different
type of hyperparameters.

So, I told you
basically at the end

of pretraining you
essentially end up

with a learning rate of 0.

Here, you're going to
increase your learning rate.

So like 1e minus
5, 1e minus-- yeah.

And so the way that you give
to them is actually different.

OK.

Second step or second
part of this post training

is what we call
reinforcement learning

from human feedback or RLHF.

Some of you might
have heard of that.

The idea is that SFT has
a problem, namely that you

do behavioral cloning, which
means that you just try to clone

what the humans would say.

And that has many issues.

One of them is that you're
bound by human abilities.

So if-- humans actually humans
won't generate the things

that they think is actually
the best thing to generate.

So if you ask me
to write a book,

I mean, I can definitely
enjoy your book.

I can probably say one book
is better than another,

but I'm definitely not going to
be as good as writing the book

that I want to read.

So you're going to be
bound by the human ability

to generate things, even though
the humans might be better

at distinguishing
between things.

That's one issue.

Issue number two, I find that
actually pretty interesting

is that it--

if you ever heard of the
word hallucination. so this

is LLMs generating fake--
like false information.

Hallucination might--
at least people

have hypothesized that can come
from the supervised fine tuning

even if you do supervised fine
tuning on data that is correct.

And the reason why
that is is that if--

given I told you that basically
SFT is with very little data.

And it's with data
that the model

doesn't learn anything new.

So what if the human gives an
answer that the model didn't

know was true.

From the model perspective,
the human basically

is telling the model generate
this thing that seems plausible

but actually have no
idea if it's true or not.

So just to give you a
very concrete example,

if we go back to this
monopsony example,

can you write blah blah
blah about monopsony?

Imagine that the human wrote a
reference on this type of book.

And that book might exist.

That might be a
correct reference,

but what if the LLM
never saw this reference

during pretraining.

Then it doesn't know that
it's a correct reference.

So really what
you tell the model

is to generate or make up some
plausible sounding reference

rather than actually
tell the real reference

that it saw during pretraining.

So hallucination might
be caused by this SFT.

So that's problem number two.

Does that all make sense?

Great.

Problem number 3, price.

Generating the ideal
answers is very pricey.

And that comes back
to your question

of humans writing the
entire answer is actually

pretty expensive.

So that's why RLHF comes in.

The idea is that instead of
cloning the behaviors of humans,

we're going to maximize
human preference.

And the way we're going to
do that, so the pipeline,

is that for a certain--
for every instruction,

you're going to ask a model
to generate two answers

and usually use a
pretty good model.

So you usually don't use an LLM
here, you use a SFT fine tune,

you use a fine tuned LLM already
to give pretty good answers.

And then you ask labelers which
of these two answers was better?

So select the preferred one.

And then with different
types of algorithms,

we're going to talk about
the algorithms, you just fine

tune the model to generate
more of the green thing

than the red thing.

So more of the good stuff.

So now the question
is how and we're

going to talk about
that right now.

So there are two ways that
we're going to talk about

and two that are mainly
use in the community.

The first one is simply the idea
of using reinforcement learning.

So hopefully you all know what
reinforcement learning is now.

So when you think about
using reinforcement learning,

one important question is
like, what is the reward

that we're optimizing.

So in this case, there
are really two options

that I could think about.

The first one, you
could just say,

I'm going to compare the output
generated by some baseline,

the output generated
by my model.

And I'm just going to ask the
human to say which one is better

and I'm going to use
this as a reward.

So if I'm better
than the baseline,

this is a plus 1, if
not, it's a minus 1.

So now it's binary reward.

The problem with binary reward
is that it's very sparse

and you don't get much
information out of it.

Like maybe your answer
was slightly better,

maybe it was like way
better and you don't really

know from this how
much better it was.

So option 2 is
that you can train

what we call a reward model,
which is simply a classifier.

So you use machine
learning to classify

how much better two outputs
are from the preference--

from the perspective
of the human.

So this is a little bit
meta, but what you basically

do is that you train--

you take a reward model, which
is just a large la-- also

a large classifier, and you
basically ask this reward model,

you give it the input
and the actual output

that you have, one
of the two outputs.

And you just exponentiate that
so that's the softmax loss

that you all know about.

And now you divide by
the exponentiated reward

on the first example--

I'm sorry, on the
first output and this

is on the second output.

And you basically train--

so the reason why you do that
is that you train your model,

you train this
reward model to be

able to classify how much better
one output is to another one.

So another slightly less
convoluted way of saying it

is that your reward
model will output

some reward that will be used
as the logits of your softmax.

So now if you have high
logits in your softmax,

it means that you highly
likely this output is better.

So that's what we call
Bradley-Terry model.

Yes.

Will this reward
model [INAUDIBLE]

lower the entire output, or
is it going to [INAUDIBLE]?

So this takes the entire--

yeah, this takes the
entire output at once.

So it takes all the
input and all the output

and it gives one number.

Yes.

So [INAUDIBLE] reward model,
where would the human be then?

Sorry.

With the reward model,
where would the human be?

Like--

I see.

OK sorry.

Maybe I wasn't clear.

You train this reward model
to fit this green and red

preference from humans.

So basically you
train a classifier

to say whether the humans
prefer red or green.

But instead of using
the binary reward, which

is what the human would
tell you you basically use

the logits of the softmax.

And the thing with the logits
is that logits are continuous.

So now you know that if
your reward model said

it has high logits,
then, in some ways,

the human highly preferred this
answer to some other answer.

Great.

So as I just said, continuous
information is better.

So that's what people use
in practice or at least

used to use in practice.

I'll tell you about the
other algorithm later.

So what do you do at the
end is that you basically

try to just use reinforcement
learning that you know about.

Now we know we have a reward.

What you sample through
is the generation

from your large language model.

And then you just use
some regularization term.

So the reason why we do
this regularization term

is for avoiding what we
call overoptimization.

So this reward
model might not be

really represent--
might not perfectly

model human preferences.

So you don't want to
maximize this thing

to essentially infinity.

And you do it using a PPO,
which is a common reinforcement

learning algorithm.

One thing to note here, because
it will be important for later,

is that when we use
maximum likelihood--

sorry, now the large
language models

are actually a policy for
your reinforcement learning.

It's not maximizing
maximum likelihood anymore.

Which means that you're not
modeling any distribution

anymore.

And the reason why
this is important

is that models that went
through this type of PPO

actually don't give
you likelihoods

of text that are meaningful.

Because what you
optimize them to do

is basically just
optimize for generating

the most likely thing,
not optimize for modeling,

all the answers that
humans might say.

Another way of saying
that is that there's

nothing that incentivizes
here the model to not give

a single possible generation.

Nothing here says it's good
if you have some distribution

with some entropy.

If you haven't followed, it's
not that important but just good

to know.

Great.

So PPO is exactly what
ChatGPT did originally.

So here is on their
blog post on what

they have is step one do
supervised fine tuning, which

now you all know about.

Step two, train a reward
model on human preferences.

Step three, do PPO
multiple steps,

which is where you
see this blue arrow.

So you continue-- you train
the model once with the PPO,

you collect new
data, you continue.

And that's why-- and that's
exactly what ChatGPT did.

And that was the
big breakthrough

between GPT 3 and ChatGPT.

One thing to note is that
PPO has many challenges.

Reinforcement learning
is something that

is super nice theoretically.

In practice, anyone
who ever worked

with reinforcement learning
knows it's such a mess.

There's a lot of things
like rollouts, outer loops,

clipping so many complications.

So it's messy.

This is the idealized PPO
used for LLM settings,

so that's already
much more complicated

than this expectation
we saw before.

And in practice it's actually
much more complicated.

So we have one implementation
of it that we had to do,

and I'm not going
to go through it.

But basically have so
much stuff that you

have to think about
when you implement

that type of PPO algorithm.

So you have clipping everywhere,
you have a lot of complexities

and things are not
well documented.

All this to say that we're going
to there was a new method that

was proposed also from
Stanford one year ago

called DPO, which is essentially
a simplification of PPO.

And the way-- what they did
or the idea that they have

is that instead of using
reinforcement learning,

you can just maximize the
probability of generating

the stuff that you
like and minimizing

the probability of the
stuff that you don't like.

So if you think about the human
preference, the red and green,

maximize green, minimize red.

So the loss is actually
this one where what you see

this is simply some
log of the model.

So this is the likelihood of
a model generating the things

that the human preferred,
given the inputs.

And what you try
to do is basically

maximize the likelihood of
generating the things that you

like, minimize the likelihood of
the things that you don't like.

All the rest of the terms
here it's not too important.

It's actually really not that
complicated to understand.

But at a high level, it's really
just maximizing the things

you like, minimizing the rest.

And one thing to note, which
I was going to say just here,

is that actually all
the rest is chosen such

that the global minima of
PPO and the global minima

of like this DPO,
under some assumptions,

are essentially equivalent.

So this is the right thing
to do mathematically.

I'm not going to go
through the derivations,

but that's the
right thing to do.

It's pretty different with
PPO in the sense that now--

with PPO, what you had to do is
collect the human preferences,

then train a reward model
with maximum likelihood,

then use reinforcement learning.

Now all you do is basically
maximum likelihood.

Much simpler.

Yes.

I mean, yeah.

So it seems like this is A,
much simpler and B, like,

what you would just intuitively
do with [INAUDIBLE]?

Why did they start
with this reward model.

Like what led them doing that?

I think it's a great question.

I don't really know.

What I can tell you is that.

At ChatGPT the people
who did basically

this PP-- sorry, who
did ChatGPT initially

are the ones who
actually wrote PPO.

And I think they
were just-- like,

there are a lot of
reinforcement learning people.

And I think that for them
it was very intuitive.

So there's also some
additional potential benefits.

For example, I don't want to--

yeah, for example, if
you use the reward model,

the cool thing here with
reinforcement learning

is that you can use unlabeled
data with the reward model.

So here you can only use the
labeled data for doing DPO--

For PPO-- for PPO, you first
train your reward model

and then you can
use unlabeled data

where the reward
model will basically

label this unlabeled data.

So this additional,
kind of, potential--

there could be
potential improvements.

In practice it happens
that there are none.

And I think just that a
lot of people in this team

were reinforcement
learning experts, including

the main author of
PPO, John Schulman.

So much simpler than PPO, and
it's basically performs as well.

So now this is the standard
thing that people use.

At least in the open
source community,

I believe it's actually the
standard also in industry.

So that's called DPO.

Gains so those are all
the papers on the left.

Here this is on the
summarization task.

You see, all I
want to show you is

that basically the
pretrained models were OK

and they improve of scale.

If you do supervised
fine tuning,

you improve them
a little bit more,

if you do PPO or something
with RLHF human feedback,

you get performance
that are, oftentimes

depending on a benchmark,
even better than humans.

So this is the human
reference summaries.

Same thing.

This is on a paper that
we have Alpaca farm where

we see the evaluation
here is not too important

but basically see
pretrained model.

You jump to SFT and then you
jump to PPO, DPO and PPO,

DPO have the exact
same performance.

So basically RLHF helps.

That's, kind of, the
conclusion and DPO is simple.

Data.

The way that you collect
that type of data.

First idea is just use humans
as we already talked about.

Guidelines are very
complicated for what

humans should be labeling,
and it's really not that easy.

And actually, if you ever
do some of the labeling,

you will see that it's
extremely complicated.

Like if I Zoom in to this.

Here, I have a question tell
me about self-driving cars.

And you read both
self-driving cars

are vehicles that are
capable of detecting

the surroundings,
blah, blah blah, blah.

Self driving cars are
cars that are equipped

with sensors, blah
blah, blah to navigate

without the need for a driver.

I mean, both seem OK.

Which one is better?

It's actually hard
to say at a glance.

And as a result, the
problem with humans

is that you will
start optimizing

a lot of high-level features.

For example, the
second one is longer.

I can guarantee you that
most humans will choose

the second one,
even though I mean,

maybe the first one is better.

I don't know.

I haven't read it carefully.

So challenges of humans.

First, slow and expensive.

Second, as I just mentioned,
it's hard to focus on things

that matter, like correctness.

And people usually
look at things

that don't matter as much
like the form, like length.

And as a result,
so what I show here

is that when you do RLHF,
the more you do RLHF,

the longer the output
of the models become.

So if you've ever been
annoyed at ChatGPT

answering you super
long sentences,

this is because of RLHF.

Annotator distribution shift.

Like the distribution
of annotators

that you use matters a
lot, and you have to think,

like, what is even the
humans that we want

to represent in these models?

Another question is
crowdsourcing ethics.

Like usually these--
basically a lot

of the labeling that is
done, the people who do them

are not paid well
and they have to go

through a lot of toxic
data because you basically

want the model to avoid
saying the toxic data.

So crowdsourcing ethics too.

So many challenges
with human data.

So what we did, also
last year, is again,

the same thing as Alpaca, just
the idea of like oh well, there

are challenges
with humans, maybe

we can just replace
them with LLMs.

So what we did is
simply replace--

I see that.

I'm just realizing that the
slides are not centered.

Anyways you replace a human
preference with preferences.

So here, on this figure, you
see on the x-axis, the price

that we paid for
collecting human data.

It's around $300
for 1,000 examples.

And this is on mechanical
Turkers which are usually

like cheaper than maybe
some of the other companies

that you could go through.

And on the y-axis,
it's basically

the agreement with other humans,
with the mode of other humans.

And what you see is that
actually, as I told you before,

labeling is really complicated.

Humans agree with
themselves only around 66%

of the time on a binary task.

And it's not that the
humans are not good

here because we were five
main authors on this paper.

We tried to label
this data ourselves,

and we only had, like, 67 or
68% accuracy, even though we

talked-- like we talked
for like three hours of how

we should be doing labeling.

But really, it's complicated.

It's not an easy task.

And here I just showed
many different models.

And, basically, you see that
models are much cheaper,

and they can actually
get higher agreement

with the mode of humans
than humans themselves.

And the reason why is because
humans have a lot of variance,

models have no variance.

So there might be a
little bit more biased

but have less variance.

So it works surprisingly well.

And now it's, kind
of, the standard

in open source community.

I think even in
industry a lot of people

use both humans and
LLMs for improving

the collection of RLHF data.

And this is like-- this is
the paper from last year,

but honestly, now it's more like
the LLMs would be around this

agreement, and
this costs around,

I would say 50 50x than humans
and better agreement with human

than humans themselves.

OK.

So that gets us to
evaluation of post training.

That goes back to
your initial question

at the beginning of the lecture.

How do you evaluate
something like ChatGPT?

The answers that GPT could
give are basically unbounded.

And it's not that
there's one right answer,

there are many answers
that are just as good.

So there are many challenges.

One, you can't use
validation loss

because one method
might use PPO,

the other one might use DPO.

Validation loss
is not comparable.

Second, you can't use--

sorry, perplexity.

That's the thing
I told you before.

These models are not calibrated.

They don't give distributions.

They just optimize
for one thing.

So you can't use perplexity for
actually evaluating these type

of models once they aligned--

sorry, once they're aligned.

Third, there's a large
diversity of questions

that humans might
ask to these models.

Generation open QA some question
answering some summarization

and all of these things.

So there's so many
things you have to cover.

Then the tasks are
really open ended,

so it's very hard to automate.

So that's what you were
alluding to before.

So the idea is that
instead of trying

to come up with really
easily automated benchmarks,

it's just we're going to ask
questions that users actually

ask to these models in practice.

And we're just going
to ask annotators

to say between these two
models, which one is better.

What's the better output.

So basically the
exact same thing

as basically the data
from RLHF but you

use it now for evaluation.

Yes I'm not sure
I understand what

you mean by can't use
perplexity not calibrated.

Like RLHF still doing like
next token prediction.

So--

Why can't perplexity
be used then?

So think about the
optimal solution

after doing PPL is
basically one model that

gives you essentially a delta.

Like basically it says that
there's only one sentence

that is--

that could be generated
for that question.

So now if you use
it on something

that is slightly semantically
differently different,

it would actually give a
likelihood of 0 for that answer.

So in reality, it's not that
extreme because as you say,

it's still a
distribution, but it just

shows you that there's
a fundamental issue

with perplexity.

Once these models
are not LLMs anymore,

they were not trained,
at least with PPO

they're not trained to do
maximum likelihood anymore,

they were trained
to be policies.

So probably the most
common or the most--

yeah, the most common benchmark
or the most trusted one

is what we call ChatBotArena,
which is basically

go on internet, have random
users on the internet,

blindly talk with two chatbots,
just ask many questions,

see the two answers and
rate, which one is better.

And you do that over hundreds
of thousands of users and then

you get the actual preferences
and you get rankings of models.

So you can go right
now on ChatBotArena

and actually interact
with these models.

One potential issue
just to highlight

is that while people who want
to do these type of things

are usually more like
tech-driven or like tech savvy.

So a lot of the questions
that you will ask

are more like tech
stuff discussing

software errors,
inquiries about AI tools

and all of these things.

So another issue
is cost and speed.

If you really want
to use something

like this for
development process,

it will be too costly because
you will need to basically pay

a lot of humans to do that.

So one simple idea is,
again, as we said many times,

just use LLM instead of humans.

You probably know the
drill at this point.

Steps for every instruction
generate outputs

by some baseline and the model
that you want to evaluate.

So here you imagine that
I'm comparing an answer

from ChatGPT and from Misrule.

I'm just asking a model, another
model, which one is better.

And I just basically
average that out.

Yeah.

I asked ChatGPT 4,
which one is better.

I averaged that out over
my entire distribution,

over my entire
benchmark or data set,

and that gives me a win rate.

So a win probability for one
model compared to another one.

And now you can rank models.

And this is the
AlpacaEval leaderboard.

So the benefits of this
is that actually we

show-- we get 98% correlation
with ChatBotArena.

So very high
correlation with humans.

So this is yeah,
comparison with correlation

with other benchmarks.

And it takes less than three
minutes and less than $10

to run.

So it's pretty cheap.

And there are downsides though.

One of them is poor correlation.

So as we already saw
before, LLMs prefer,

this is one spurious
correlation, not many.

I'll just talk about one.

LLMs prefer longer outputs.

Actually humans also
prefer longer outputs.

But the problem or the
issue once you use LLMs

is that once there is bias, you
will continue optimizing that.

Humans at some point,
I can guarantee you

if I ask a simple
question, and you give me

five pages of
answers, I'll be like,

no, I don't like that answer.

But LLMs if they have this bias
and they were trained for that,

they will continue
preferring longer outputs.

So here we see the
preference just showing

that humans and models
prefer longer outputs.

And here is another view of
the initial AlpacaEval data set

benchmark, where when we asked--

when we rank GPT4, when we
look at the win rate of GPT4

versus actually GPT4 itself,
if we use the standard GPT4,

it gets 50%, kind of, by
definition because we're

comparing GPT4 versus GPT4.

But if we ask a GPT4 to
be slightly more verbose,

so we just say in the prompt,
be verbose in your answers,

then it gets a
win rate of 64.4%.

So really there's
a huge variance.

And if we ask it
to be concise, it

gets 20% so there's
a huge variance

depending on whether you ask
it to be concise or verbose.

That's very annoying.

So one possible solution,
which is what we did,

is just use some
regression analysis.

I'm not going to
go into details,

but basically use
causal inference

tools to control for length.

And right now actually
length matters much less.

So if you ask it to be verbose,
you still get some gains,

but much less.

Great.

So that's all about
post training.

And now for the
next eight minutes,

I might talk about systems
or just answer questions.

Yes.

Can you go back to your
post training, internal post

training.

How did we tune those
parameters using

the small body of
fine-tuning data

and have such big
effect on the model?

You mentioned earlier that
there's a different set

of hyperparameters.

Are we changing just some of
the weights, the later weights

or other weights.

What's actually happening?

Yeah.

Yeah, I, kind of, skimmed
through all of this.

You change all the weights.

Actually, industry will
change all the weights.

In open source
land, you might have

heard of Laura, which is
going to change basically only

some of the weights or it
actually, to be more specific,

it's going to add
some differences

to the output of every layer.

But in industry, you're going to
just fine tune all the weights.

And also to say something
else about the data, actually,

this last step, RLHF
you usually going

to collect a lot more
data than with SFT.

So if FSFT is like 5,000,
10,000, maybe 50,000 with,

RLHF I think you're going to be
more around like the one million

order of magnitude.

It's still much less
than pretraining though.

Yeah.

Because pretraining
is 15 trillion tokens.

I mean, this is like--
that's not even a drop

and yet you influence
the weight a lot.

So because you do it--

I mean, you have to think that
how you do it is you use--

I mean, as I said, the learning
rate that you're going to use

is going to be different,
but also you only do that.

So just imagine if I trained--

even if I trained
on one sentence,

but over and over
again at some point

my model will only
generate that sentence

even if it was just
one sentence instead of

the 15 trillion tokens.

So if you use a
large enough learning

rate and for enough
time, you will basically

overfit that sentence.

So the key thing to remember
is that the data is not--

it's not as if you mix
some post-training data

and some pretraining data.

You do pretraining, and then
you just start fine-tuning only

on the post-training.

So another way, maybe
another perspective

is that the pretraining
is just the initialization

of your model.

And once you view it that
way, that this is just

initialization of weights,
then there's nothing special.

Like you don't need to remember
that you train on a lot of data

before.

The only thing that matters is
that you had an initialization

and now I actually
train the model.

So maybe you think
about it that way.

Like this is a Markov
property in some ways.

It's just like you
had your weights.

This is my initialization.

Now I'm training that one.

Does that answer your question?

Kind of but you said
something just now about it's

almost the equivalent of just
rerunning the fine tuning

data many times.

Is it actually-- is that what
actually happens in order

to give so much more preference?

You might-- I actually don't
know right now how they do it

in industry.

When we did our packet,
we had to do three epochs.

So you did run it
three times through it.

But I mean, even
the number of times

that you run it through,
it's actually not important.

The only thing-- the only thing
is the effective learning rate

that what matters.

So yeah.

Great.

So I think I have five minutes.

OK I might try to give a
high-level overview at least

from one of the systems trick.

Systems, as we said, for
everyone bottleneck is--

sorry compute is
the huge bottleneck.

One question you might ask
is, why not buy more GPUs?

GPUs are expensive,
but also are scarce.

Even if you have $10
million right now,

you cannot buy the best GPUs.

[INAUDIBLE]

There's also some
physical limitations.

When you have multiple
GPUs, you have

to communicate between them.

That takes time.

So just buying more
GPUs is not that easy.

So it's really
important to think about

how do you allocate resources
and how do you optimize

your pipeline, so system?

101 on GPUs, I'm sorry,
I'm going slightly faster.

I hope that some of you
at least can follow.

GPUs are basically
optimized for throughput.

CPUs are optimized for latency.

So GPUs, the way you
have to think about it

is that there's one--

there's one command that
is run on many, many cores

at the same time on
different type of data.

So this is how you see a GPU.

You see there are
many different codes.

We call them streaming
multiprocessors,

which is very different than
the usual CPU architecture.

So just think high throughput
parallelization for GPUs.

GPUs are optimized for
fast matrix multiplication.

So every time you will do--
you will do something on GPU.

If you can do it with a
matrix multiplication,

it's going to be 10 times
faster than with anything else.

That is a little bit
annoying because it

means that we are,
kind of, bottlenecked

to doing anything with
matrix multiplications.

Another thing to
note with GPUs is

that compute has
been improving faster

than memory and communication.

So right now GPUs usually
are hard to keep--

Like the data that
you sent to GPUs

is actually hard to keep
up with the processes.

So most of your
GPUs are actually

going to be idle if you
just run normal code,

if you don't optimize your code.

So communication-- and this
will continue over time.

Another thing to know
about GPUs is that there's

a memory hierarchy.

This is the same thing
actually with CPUs,

but basically the closer
you are to your cores,

the less memory there is,
but the faster things run.

If you are further,
more memory slower.

Oh yeah I'm going to skip that.

OK actually, I'm
going to say it.

I told you about this--

the fact of communication.

The metric that
people usually look at

is model FLOP utilization.

So what is the theoretical
maximum that GPU could run at,

number of flops that you
could use per second--

divide-- sorry, the number
of observed throughput

divided by this
theoretical maximum.

And in general, if you
reach 50% you're very happy.

Like Facebook I looked
at llama was at 45

or something like this.

So that means that data
doesn't come fast enough

even for these big companies.

So one simple trick,
and that might

be the only one I'm
going to tell you about,

is low precision.

One simple idea is
that well, if I'm

going to put my floats
in low precision,

then there's going
to be fewer bits

that I have to send to my GPUs.

If there's fewer bits,
it's faster communication,

lower memory consumption.

Things are going to go faster.

And for deep learning
it just happens

that decimal is
not that important.

So when you do matrix
multiplication, when

you do like for example, SGD,
there's already so much noise

that if you update something
by 0.01 or 0.015, who cares.

So basically instead of using
32 bits per float, which

is what people used to use,
or 64 for example, which

is what you would
use in other domains,

you use 16 bits for
matrix multiplication.

So for every float
you use 16 bits.

And for training
you have this type

of what we call automatic
mixed precision.

Which is that some of the
things are in 32 bits,

others are in 60 bit--

on 16 bits.

Generally, the way you
should be thinking about

it is that your weights
are stored-- of your model,

are stored in 32 bits.

But just before the computation
you put everything in 16 bits.

Like this you do
computation super fast.

And at the end you update
your weights in 32 bits.

And the reason why you do all
the updates in 32 bits is just

think that if your
learning rate, for example,

is very small, you still
want to be able to make

a difference in your weights.

So all the computation
is done in 16 bits,

but the weights are
actually stored in 32 bits.

So that's like the standard
way that people are doing it.

OK, I'll actually
talk just about this,

and then I'll skip all the rest,
operator fusion, because I think

this is actually pretty cool.

As I just said,
communication is very slow

and actually every time
you use a PyTorch line,

it basically moves variable
to global memory of your GPU.

So when you have something like
this x dot cosine equal x1,

and then you do x1 dot cosine.

What is happening
behind the scenes

is that you take the
x, which is data.

You ship it to your actual
processors of your GPUs.

You apply the cosine.

You ship it back to the
main memory of your GPU

and then you see the next line.

You ship it back to the
computer-- to the GPU processor,

you apply another cosine
and you ship it back again.

So another way to
see that is that you

go from your DRAM, which is
your global memory and your GPU

and you ship it to compute.

You ship it back for every line.

This is a naive way of doing it.

This seems very wasteful.

So the idea, simple
idea of operator fusion

is just communicate, do all the
computation, ship it back once.

And this is exactly
what fused kernels are.

So if you ever want to make
your compute-- your computations

in PyTorch much faster,
just apply torch dot

compile on your model.

This is going to make your
model around 2 times faster.

And what it does is simply
that it rewrites your code--

your PyTorch code basically
in C++ in CUDA to do

the communication only once
then do all the operations,

then ship it back.

OK I'm not going to have
time to talk about tiling.

Tiling is important.

Parallelization.

Parallelization is important.

And mixture of experts.

Mixture of experts is important.

Outlook.

There are many things
we haven't talked about.

We haven't talked about
architectures we definitely

haven't talked about inference.

There are many other things
that are important with LLMs.

What is the UI that you use?

I mean, arguably ChatGPT,
the big novelty was just

have a simple UI to use it.

Multi-modality.

What are all the
misuses you could have.

The fact that there might not
be enough data on the internet

to train all these models.

Legality of data collection,
so many other things.

If you are interested
in all these topics,

I would suggest three classes.

CS224N is probably the one
that touches the least on LLMs,

but it gives some background
and historical context

of all the LLMs and gives
some adjacent material.

CS324 I think it's called--

I think it's just called
Large Language Models, more

in depth reading and lectures
on everything I talked about.

CS336 which is large
language model from scratch,

you actually build your own LLM.

It's an amazing class also
given by my two supervisors.

Very heavy workload,
so be careful.

Great.