Original subtitles

Welcome back everyone to this lecture on back propagation the last theory topic we're going to cover

is back propagation and we're going to start by trying to build an intuition behind back propagation

and then we'll dive into the calculus and notation of back propagation.

I want to point out that back propagation is probably the hardest part of the entire theoretical deep

learning process because of the calculus and the notation involved that calculus especially when we

start talking about back propagation in dealing with a matrix of weights and another matrix of biases.

So keep that in mind this is gonna be pretty difficult especially if you're rusty on your calculus notation

with that in mind though if you understand the basic intuition that you basically just move backwards

through a network to update the weights and biases then that's actually enough to continue on the rest

of the course.

So if you're fuzzy on the calculus portion of this lecture don't worry too much about that because it's

not like you're gonna be needing to compute the gradient herself.

The actual code is going to do that for us so fundamentally we want to know how the cost function results

change with respect to the weights in the network.

That way we can update the weights to minimize the cost function and we really talked a little bit about

that when talking about things like gradient descent and how it approaches the cost function so let's

go to begin with a very simple network in order to understand it back propagation.

So this is a super simple network essentially each layer only has one neuron.

So we'll see how back propagation works with just a network of a couple of neurons and then we can easily

expand this to networks with multiple neurons per layer so as we already know each input basically receives

a weight and a bias.

So there's an incoming weight attached that edge and then each node or that is to say each neuron has

its own bias.

So we get the sort of formula weight 1 plus bias one way to plus bias to weight three plus bias 3 and

so on so this means that we have some sort of cost function that's dependent on those weights and biases

and we've already seen how this process propagates forward.

So let's go ahead and start at the end to learn about back propagation so we already noted that the

way we notate layers is by having the very last layer be called.

So then our notation becomes that the neuron all the way on the right is in layer l the neuron to the

left of it is L minus 1 L minus 2 and so on for L minus n layers.

Now let's go ahead and just focus on the very last two layers of our network because back propagation

starts all the way at the end of the network layer l once we've gone through our feed forward process

and when focusing on these two layers I want to remind ourselves of the notation we've been using so

far we've been defining Z as W. times x plus B and X recall X that notation of X that's really only

valid at the very first layer because X stands for the actual raw feature inputs as you keep going from

neuron to neuron further into the layer X technically becomes the output of the previous neuron which

is defined as a because remember after we apply an activation function to Z such as sigmoid Zee we label

that as a.

So as you go further and further along into these layers z would actually be Z equal to w times a plus

b because X is technically only valid at that very first layer as the raw feature input.

Once you actually pass that into a neuron you technically not dealing with the raw features anymore.

Instead you're dealing with the output of the previous neurons layer which is better stated as a the

sigmoid of Z or whatever activation function you choose.

OK so what does that actually mean when we take it into account for that very last layer.

Well that means that z at the last layer is going to be equal to those weights at the last layer times

a of L minus one.

So what is a L minus 1 will AFL minus 1 is simply the output of the previous layers neuron so a L minus

one plus B level the biases at that very last layer so again a of L..

So the activation function.

Output at the very last layer is equal to sigmoid or the activation function of zero.

So notice Isaiah L is defined by the weights and biases at that layer l.

And then it's defined by the output of the previous zero.

So hopefully can kind of make these connections and that means that then the cost function is going

to be equal to a L minus Y Y is the actual true output squared.

So what we actually want to understand is how sensitive is the cost function to changes in W and this

is where partial derivatives come into play because we want to figure out the relationship between that

final cost function and the weights.

In this case at the layer L so we're going to say take the partial derivative of that cost function

with respect to weights and layer L and if you know some calculus then you know that there's a chain

rule.

And so if you were to take the formulas we just saw here and apply the chain rule for them in order

to solve for this partial derivative we mentioned here because we want to understand the relationship

between that cost function and the weights in the network then you end up calculating this formula.

So this is just the chain rule that basically allows you to take the derivative of a function within

a function.

So here easy some calculus for the chain rule.

We can determine that the parts of the relative of that cost function with respect to that weights is

equal to the partial derivative of the Z with respect to the weights times the partial derivative of

the A.

With respect to z times the past third of the cost function with respect to a such as chain rule allows

us to pull apart these functions within functions because we saw from these previous three formulas

that the cost function is defined by AFL a value then defined by Z of Alan and Z as defined by W of

Allenby of L now recall that the cost function is not just a function of the weights but it's also a

function of the biases.

So we want to be able to understand the relationship of the cost function changing not only the weights

with the bias along the network as well so we can then calculate the same partial derivative so the

partial derivative of the cost function with respect to those biased terms in the same way.

Essentially just kind of swapping out that weight for the bias.

Now the main idea here is that we can use the gradient to go back through the network and adjust our

weights and biases to minimize the output of the error vector on the last output layer.

And recall that the gradient is essentially that derivative when you're dealing with n dimensions so

using some calculus notation we can expand this idea to networks with multiple neurons per layer and

there's gonna be some notation you'll see in just a little bit which again if you're a little rusty

on linear algebra or calculus it's called the Hatem art product and it's actually a product that you're

already familiar with because it's kind of the default with NUM pi and these different deep learning

libraries where you're actually performing elements by elements multiplication.

So again the head smart product that little dot notation that kind of looks a little bit like a hydrogen

symbol.

Basically what it means is just do an element by element multiplication and that means that the two

matrices should be the exact same size which makes sense because that's gonna match up for things like

weights and biases so given this notation and back propagation we just have a few main steps to train

you're on networks now No again you don't need to fully understand these intricate details on calculus

or the notation to continue on the coding portions of this course let's review now the actual learning

process for a network we start off with just a very basic feed forward process that we're already familiar

with.

And step 1 using the input x that is the original features we set the activation function a for the

input layer so that very first input layer then means that we have z is equal to w x plus B and then

a that output essentially coming out of the input layer is gonna be equal to your activation function

of Z.

In this case represented as sigmoid of Z.

So this resulting a then feeds into the next layer.

So then you have the next layer taking an A.

Which means it's Z is going to be w times a plus b of the previous layer.

So then you go into the next layer and then you'd take the output of that a stick it into C then so

on and so on so we think about this for each layer.

All we're doing is for computing those zis and those A's since A's based top Z.

So if I'm at some layer L then what I'm going to be doing is setting Z of my current layer l equal to

the weights at all times the output from the previous layer that is a L minus 1 plus the biases of my

current layer bevel.

Once I have my sea avail I'm going to pass that through my activation function.

In this case sigmoid and then I get the a of my current layer L and then I can pass that to the next

layer of L plus 1 and so on and so on.

And then we come to step 3.

And here we've written it out in the full calculus notation of computing our error vector.

But essentially what we want to do is we take a look and focus on just that very first term alter is

doing is essentially expressing the rate of change of that cost function with respect to the output

activations and in the case of the quadratic cost function then that's essentially the same thing as

saying the activation of the last output layer minus Y which was the true value.

And essentially what we want to do is be able to calculate this error vector and back propagate it essentially

calculate the error back through every other single error that way we can adjust the weights and biases

for that error so replacing that first term with a capital L minus Y we get the following formula.

And again reason that had a smart product here.

So what I want to do is I want to write a generalized error vector formula and I'm gonna write it out

in terms of the error in the next layer which makes a lot of sense because all we're doing is we're

moving backwards and a real quick note here is it's a little tricky to find a font where a lower case

l. looks different than the number one.

So I have two little bullet points there to show you what I'm talking about.

So a lower case l. essentially looks like a straight line the number one has a little dash on top.

So keep that in mind as you see what's happening next.

The reason I'm not gonna be using a capital L is because in general we should be using capital L to

denote the very last output layer.

And when I want to do is show you the formula for the error vector and those calculations for any layer

l lowercase L that is inside of the network.

So what I'm going to do is for this back propagation step for every single layer starting at the very

last layer.

Capital L the moving to capital L minus one capital L minus two etc. all the way for all these layers.

The generalized than error term so that error term that Delta the little L or lowercase L is going to

be equal to the weight matrix L plus 1.

That's the lowercase L then transpose that so that t that term right there that's a transpose of the

weight matrix in the next layer on the right hand side of the L plus 1 that's the lowercase L and then

we have their multiplied by the error term at that.

Also next layer and then we take the head of our product again with the Z of L pass into the activation

function.

So again all we're doing there is back propagating the error.

And here we have the generalized error for any layer lowercase L so when we apply that actual transpose

weight matrix that weight matrix of L plus 1 transposed we can think intuitively of this as moving the

air backward through the network giving us some sort of measure of the air at the output of that elf

layer we then take the Hatem out product of that times they had a marked Product there of the Z at that

layer pass into the activation function and what does those is then this moves the air backward through

the activation function in layer l giving us the error an l in the weighted input to layer l.

So again that's a generalized term which is why you see me using that lowercase L and then we can understand

that the gradient of the cost function is given by these two formulas send for each layer L minus 1

L minus 2 and so on.

All we're really doing is computing that partial derivative of the cost function with respect to the

weights and the biases there and Jane K. That's just notation for the actual neurons themselves.

This then allows us to adjust the weights and biases to help minimize that cost function.

So I know this was quite difficult to understand in terms of the notation of the calculus and don't

worry if you didn't get it right away.

This usually takes at least for me it took definitely a couple of hours to understand.

Just trying to write it out with paper and pencil.

So what I've done is I've linked to you in this lecture some external links you should see it as a little

folder pop up as you're kind of watching this lecture or right next to the lecture title you should

see a little dropdown folder you can click there's some external links there that actually go and essentially

derive step by step.

All these equations.

So if this overview wasn't enough for you and you want to see every step and the whole derivation of

this process plus the proof of those kind of four step fundamental back propagation equations check

out the external links for lots of details on that but if you have the general understanding that you're

basically calculating the error term at the very last layer and then going backwards through the network

to calculate all those errors and then adjusting the weights and biases accordingly to minimize that

cost function.

If you understand that general intuition you know enough to go ahead and continue with the course.

All right.

Thanks and I'll see you at the next lecture.