All language subtitles for 01_cost-function-for-logistic-regression.en

Remember that the cost
function gives you a way to

measure how well
a specific set of

parameters fits
the training data.

Thereby gives you a way to

try to choose better parameters.

In this video, we'll look at how

the squared error
cost function is

not an ideal cost function
for logistic regression.

We'll take a look at a
different cost function that

can help us choose

better parameters for
logistic regression.

Here's what the training set for

our logistic regression
model might look like.

Where here each row might
correspond to patients

that was paying a visit to

the doctor and one dealt
with some diagnosis.

As before, we'll use

m to denote the number
of training examples.

Each training example has
one or more features,

such as the tumor size,
the patient's age,

and so on for a
total of n features.

Let's call the features
X_1 through X_n.

Since this is a binary
classification task,

the target label y takes
on only two values,

either 0 or 1.

Finally, the logistic
regression model

is defined by this equation.

The question you
want to answer is,

given this training set,

how can you choose
parameters w and b?

Recall for linear regression,

this is the squared
error cost function.

The only thing I've
changed is that I put

the one half inside

the summation instead of
outside the summation.

You might remember that in the
case of linear regression,

where f of x is the
linear function,

w dot x plus b.

The cost function
looks like this,

is a convex function or a
bowl shape or hammer shape.

Gradient descent
will look like this,

where you take one step,

one step, and so on to converge
at the global minimum.

Now you could try to use

the same cost function
for logistic regression.

But it turns out that
if I were to write f of

x equals 1 over 1 plus e to

the negative wx plus b and

plot the cost function
using this value of f of x,

then the cost will
look like this.

This becomes what's called a

non-convex cost
function is not convex.

What this means is that if

you were to try to
use gradient descent.

There are lots of local minima
that you can get sucking.

It turns out that for
logistic regression,

this squared error cost
function is not a good choice.

Instead, there will be

a different cost
function that can

make the cost function
convex again.

The gradient descent can be

guaranteed to converge
to the global minimum.

The only thing I've changed
is that I put the one half

inside the summation instead
of outside the summation.

This will make the math you

see later on this slide
a little bit simpler.

In order to build a
new cost function,

one that we'll use for
logistic regression.

I'm going to change a little bit

the definition of the cost
function J of w and b.

In particular, if you look
inside this summation,

let's call this term inside

the loss on a single
training example.

I'm going to denote the
loss via this capital

L and as a function

of the prediction of
the learning algorithm,

f of x as well as of
the true label y.

The loss given the predictor
f of x and the true label

y is equal in this case to 1.5
of the squared difference.

We'll see shortly
that by choosing

a different form for
this loss function,

will be able to keep the
overall cost function,

which is 1 over n
times the sum of

these loss functions to
be a convex function.

Now, the loss function
inputs f of x and

the true label y and tells

us how well we're
doing on that example.

I'm going to just
write down here at

the definition of
the loss function

we'll use for
logistic regression.

If the label y is equal to 1,

then the loss is
negative log of f of

x and if the label
y is equal to 0,

then the loss is negative
log of 1 minus f of x.

Let's take a look at why

this loss function
hopefully makes sense.

Let's first consider the
case of y equals 1 and plot

what this function
looks like to gain

some intuition about what
this loss function is doing.

Remember, the loss function

measures how well
you're doing on

one training example
and is by summing

up the losses on all of

the training examples
that you then get,

the cost function,
which measures how

well you're doing on the
entire training set.

If you plot log of f,

it looks like this curve here,

where f here is on
the horizontal axis.

A plot of a negative of the
log of f looks like this,

where we just flip the curve
along the horizontal axis.

Notice that it intersects
the horizontal axis at

f equals 1 and continues
downward from there.

Now, f is the output of
logistic regression.

Thus, f is always between
zero and one because

the output of
logistic regression

is always between zero and one.

The only part of the
function that's relevant is

therefore this part over here,

corresponding to f
between 0 and 1.

Let's zoom in and take

a closer look at this
part of the graph.

If the algorithm predicts
a probability close to

1 and the true label is 1,

then the loss is very small.

It's pretty much 0

because you're very close
to the right answer.

Now continue with the example

of the true label y being 1,

say everything is
a malignant tumor.

If the algorithm predicts 0.5,

then the loss is at
this point here,

which is a bit higher
but not that high.

Whereas in contrast,
if the algorithm were

to have outputs at 0.1 if it

thinks that there is

only a 10 percent chance
of the tumor being

malignant but y really is 1.

If really is malignant,

then the loss is this much
higher value over here.

When y is equal to 1,

the loss function
incentivizes or nurtures,

or helps push the
algorithm to make

more accurate predictions
because the loss is lowest,

when it predicts
values close to 1.

Now on this slide,

we'll be looking at
what the loss is

when y is equal to 1.

On this slide, let's look
at the second part of

the loss function corresponding
to when y is equal to 0.

In this case, the loss is
negative log of 1 minus f of x.

When this function is plotted,

it actually looks like this.

The range of f is
limited to 0 to 1

because logistic regression only

outputs values between 0 and 1.

If we zoom in,

this is what it looks like.

In this plot, corresponding
to y equals 0,

the vertical axis
shows the value of

the loss for different
values of f of x.

When f is 0 or very close to 0,

the loss is also going to be
very small which means that

if the true label is

0 and the model's prediction
is very close to 0,

well, you nearly got it right so

the loss is appropriately
very close to 0.

The larger the value
of f of x gets,

the bigger the loss
because the prediction

is further from
the true label 0.

In fact, as that
prediction approaches 1,

the loss actually
approaches infinity.

Going back to the tumor
prediction example just says if

the model predicts that
the patient's tumor is

almost certain to
be malignant, say,

99.9 percent chance
of malignancy,

that turns out to actually
not be malignant,

so y equals 0 then we

penalize the model
with a very high loss.

In this case of y equals 0,

so this is in the case of

y equals 1 on the
previous slide,

the further the
prediction f of x is

away from the true value of y,

the higher the loss.

In fact, if f of x approaches 0,

the loss here actually goes

really large and in fact
approaches infinity.

When the true label is 1,

the algorithm is
strongly incentivized

not to predict something
too close to 0.

In this video, you saw why

the squared error cost function

doesn't work well for
logistic regression.

We also defined the loss for

a single training example and

came up with a new
definition for

the loss function for
logistic regression.

It turns out that with this
choice of loss function,

the overall cost function will
be convex and thus you can

reliably use gradient descent

to take you to the
global minimum.

Proving that this
function is convex,

it's beyond the
scope of this cost.

You may remember that
the cost function is

a function of the entire
training set and is,

therefore, the average or 1 over

m times the sum of

the loss function on the
individual training examples.

The cost on a certain set
of parameters, w and b,

is equal to 1 over

m times the sum of

all the training examples of

the loss on the
training examples.

If you can find the value
of the parameters, w and b,

that minimizes this,
then you'd have

a pretty good set of
values for the parameters

w and b for logistic regression.

In the upcoming optional lab,

you'll get to take a look at how

the squared error cost function

doesn't work very well
for classification,

because you see that the
surface plot results in

a very wiggly costs surface
with many local minima.

Then you'll take a look at

the new logistic loss function.

As you can see here,

this produces a nice and
smooth convex surface plot

that does not have all
those local minima.

Please take a look
at the cost and

the plots after this video.

We've seen a lot in this video.

In the next video,

let's go back and take

the loss function for
a single train example

and use that to define

the overall cost function
for the entire training set.

We'll also figure out

a simpler way to write
out the cost function,

which will then later
allow us to run

gradient descent to find

good parameters for
logistic regression.

Let's go on to the next video.