All language subtitles for 03_cost-function-with-regularization.en

In the last video we saw that
regularization tries to make

the parental values W1 through
WN small to reduce overfitting.

In this video, we'll build on that
intuition and developed a modified cost

function for your learning algorithm that
can use to actually apply regularization.

Let's jump in, recall this example from
the previous video in which we saw that if

you fit a quadratic function to this data,
it gives a pretty good fit.

But if you fit a very
high order polynomial,

you end up with a curve
that over fits the data.

But now consider the following,
suppose that you had a way to

make the parameters W3 and
W4 really, really small.

Say close to 0.

Here's what I mean.

Let's say instead of minimizing
this objective function,

this is a cost function for
linear regression.

Let's say you were to modify
the cost function and

add to it 1000 times W3 squared
plus 1000 times W4 squared.

And here I'm just choosing 1000
because it's a big number but

any other really large
number would be okay.

So with this modified cost function,

you could in fact be penalizing
the model if W3 and W4 are large.

Because if you want to minimize this
function, the only way to make this

new cost function small is if W3 and
W4 are both small, right?

Because otherwise this
1000 times W3 squared and

1000 times W4 square terms
are going to be really, really big.

So when you minimize this function,

you're going to end up with W3
close to 0 and W4 close to 0.

So we're effectively nearly canceling out
the effects of the features execute and

extra power of 4 and
getting rid of these two terms over here.

And if we do that, then we end up with
a fit to the data that's much closer to

the quadratic function,

including maybe just tiny contributions
from the features x cubed and extra 4.

And this is good because it's a much
better fit to the data compared to if all

the parameters could be large and you end
up with this weekly quadratic function

more generally,
here's the idea behind regularization.

The idea is that if there are smaller
values for the parameters,

then that's a bit like
having a simpler model.

Maybe one with fewer features, which
is therefore less prone to overfitting.

On the last slide we penalize or

we say we regularized only W3 and W4.

But more generally, the way that
regularization tends to be implemented is

if you have a lot of features,
say a 100 features, you may not know which

are the most important features and
which ones to penalize.

So the way regularization is typically
implemented is to penalize all of

the features or more precisely,
you penalize all the WJ parameters and

it's possible to show that this will
usually result in fitting a smoother

simpler, less weekly function
that's less prone to overfitting.

So for this example, if you have data with
100 features for each house, it may be

hard to pick an advance which features
to include and which ones to exclude.

So let's build a model that
uses all 100 features.

So you have these 100
parameters W1 through W100,

as well as 100 and first parameter B.

Because we don't know which of
these parameters are going to be

the important ones.

Let's penalize all of them a bit and
shrink all of them by adding

this new term lambda times the sum from
J equals 1 through n where n is 100.

The number of features of wj squared.

This value lambda here is
the Greek alphabet lambda and

it's also called
a regularization parameter.

So similar to picking
a learning rate alpha,

you now also have to choose a number for
lambda.

A couple of things I would like
to point out by convention,

instead of using lambda
times the sum of wj squared.

We also divide lambda by 2m so
that both the 1st and

2nd terms here are scaled by 1 over 2m.

It turns out that by scaling
both terms the same way

it becomes a little bit easier to
choose a good value for lambda.

And in particular you find that even
if your training set size growth,

say you find more training examples.

So m the training set size is now bigger.

The same value of lambda that you've
picked previously is now also

more likely to continue to work if
you have this extra scaling by 2m.

Also by the way,

by convention we're not going to penalize
the parameter b for being large.

In practice, it makes very little
difference whether you do or not.

And some machine learning engineers and
actually some learning algorithm

implementations will also include lambda
over 2m times the b squared term.

But this makes very little
difference in practice and

the more common convention which was
used in this course is to regularize

only the parameters w rather
than the parameter b.

So to summarize in this modified
cost function, we want to minimize

the original cost, which is the mean
squared error cost plus additionally,

the second term which is called
the regularization term.

And so this new cost function trades
off two goals that you might have.

Trying to minimize this first term
encourages the algorithm to fit

the training data well by minimizing the
squared differences of the predictions and

the actual values.

And try to minimize the second term.

The algorithm also tries to
keep the parameters wj small,

which will tend to reduce overfitting.

The value of lambda that you choose,
specifies the relative importance or

the relative trade off or
how you balance between these two goals.

Let's take a look at what different
values of lambda will cause you're

learning algorithm to do.

Let's use the housing price prediction
example using linear regression.

So F of X is the linear regression model.

If lambda was set to be 0,
then you're not using the regularization

term at all because the regularization
term is multiplied by 0.

And so if lambda was 0,
you end up fitting this overly wiggly,

overly complex curve and it over fits.

So that was one extreme
of if lambda was 0.

Let's now look at the other extreme.

If you said lambda to be a really, really,

really large number,
say lambda equals 10 to the power of 10,

then you're placing a very heavy weight
on this regularization term on the right.

And the only way to minimize
this is to be sure that all

the values of w are pretty
much very close to 0.

So if lambda is very, very large,

the learning algorithm will choose W1,
W2, W3 and

W4 to be extremely close to 0 and
thus F of X is basically equal to b and

so the learning algorithm fits
a horizontal straight line and under fits.

To recap if lambda is 0
this model will over fit If

lambda is enormous like
10 to the power of 10.

This model will under fit.

And so what you want is some value of
lambda that is in between that more

appropriately balances these first and
second terms of trading off,

minimizing the mean squared error and
keeping the parameters small.

And when the value of lambda is not
too small and not too large, but

just right, then hopefully you end up
able to fit a 4th order polynomial,

keeping all of these features, but
with a function that looks like this.

So that's how regularization works.

When we talk about model selection,
later into specialization will

also see a variety of ways to
choose good values for lambda.

In the next two videos will flesh out
how to apply regularization to linear

regression and logistic regression, and
how to train these models with great in

dissent with that, you'll be able to avoid
overfitting with both of these algorithms.