Original subtitles

Now let's dive more deeply in gradient descent to gain

better intuition about what

it's doing and why it might make sense.

Here's the gradient descent algorithm

that you saw in the previous video.

As a reminder, this variable,

this Greek symbol Alpha, is the learning rate.

The learning rate controls how big of a step you take

when updating the model's parameters, w and b.

This term here, this d over dw,

this is a derivative term.

By convention in math,

this d is written with this funny font here.

In case anyone watching this has PhD in

math or is an expert in multivariate calculus,

they may be wondering, that's not the derivative,

that's the partial derivative. Yes, they be right.

But for the purposes of

implementing a machine learning algorithm,

I'm just going to call it derivative.

Don't worry about these little distinctions.

What we're going to focus on now

is get more intuition about

what this learning rate and what this derivative

are doing and why when multiplied together like this,

it results in updates to parameters w and b.

That makes sense. In order to do this let's use

a slightly simpler example where we

work on minimizing just one parameter.

Let's say that you have a cost function J of

just one parameter w with w is a number.

This means the gradient descent now looks like this.

W is updated to w minus the learning rate Alpha

times d over dw of J of w. You're

trying to minimize the cost by adjusting the

parameter w. This is like

our previous example where we had temporarily set b

equal to 0 with one parameter w instead of two,

you can look at two-dimensional graphs

of the cost function j,

instead of three dimensional graphs.

Let's look at what

gradient descent does on just function J of

w. Here on the horizontal axis is parameter w,

and on the vertical axis is the cost j of w. Now less

initialized gradient descent with some starting value for

w. Let's initialize it at this location.

Imagine that you start off at

this point right here on the function J,

what gradient descent will do is it will update

w to be w minus learning rate

Alpha times d over dw of J of

w. Let's look at what this derivative term here means.

A way to think about the derivative at this point on

the line is to draw a tangent line,

which is a straight line that

touches this curve at that point.

Enough, the slope of this line is

the derivative of the function j at this point.

To get the slope, you can

draw a little triangle like this.

If you compute the height divided by

the width of this triangle, that is the slope.

For example, this slope might be 2 over 1,

for instance and when

the tangent line is pointing up and to the right,

the slope is positive,

which means that this derivative is a positive number,

so is greater than 0.

The updated w is going to be

w minus the learning rate times some positive number.

The learning rate is always a positive number.

If you take w minus a positive number,

you end up with a new value for w, that's smaller.

On the graph, you're moving to the left,

you're decreasing the value of w. You may notice

that this is the right thing to do if your goal

is to decrease the cost J,

because when we move towards the left on this curve,

the cost j decreases,

and you're getting closer to the minimum

for J, which is over here.

So far, gradient descent,

seems to be doing the right thing.

Now, let's look at another example.

Let's take the same function j of w as above,

and now let's say that you initialized

gradient descent at a different location.

Say by choosing a starting value for

w that's over here on the left.

That's this point of the function j.

Now, the derivative term,

remember is d over dw of J of w,

and when we look at the tangent line

at this point over here,

the slope of this line is

a derivative of J at this point.

But this tangent line is sloping down into the right.

This lines sloping down into

the right has a negative slope.

In other words, the derivative of J at

this point is a negative number.

For instance, if you draw a triangle,

then the height like this is

negative 2 and the width is 1,

the slope is negative 2 divided by 1,

which is negative 2,

which is a negative number.

When you update w,

you get w minus the learning rate times

a negative number.

This means you subtract from w, a negative number.

But subtracting a negative number

means adding a positive number,

and so you end up increasing

w. Because subtracting a negative number is the

same as adding a positive number to

w. This step of gradient descent causes w to increase,

which means you're moving to the right of the graph and

your cost J has decrease down to here.

Again, it looks like

gradient descent is doing something reasonable,

is getting you closer to the minimum.

Hopefully, these last two examples show

some of the intuition behind what a derivative term

is doing and why this host gradient descent change

w to get you closer to the minimum.

I hope this video gave you some sense for why

the derivative term in gradient descent makes sense.

One other key quantity in

the gradient descent algorithm

is the learning rate Alpha.

How do you choose Alpha?

What happens if it's too

small or what happens when it's too big?

In the next video,

let's take a deeper look at

the parameter Alpha to help

build intuitions about what it does,

as well as how to make a good choice for a good value

of Alpha for your implementation of gradient descent.