Original subtitles

1

Hello and welcome back to the course on deep learning this is an additional tutorial to talk about the

2

soft and cross entropy functions.

3

It is not 100 percent necessary in order for you to go through all of the parts that we've been through

4

in the main part of this section where we're talking about the convolutional neural networks but at

5

the same time I thought it would be a good addition to your bag of knowledge and skill set.

6

So let's go ahead and dig into these functions.

7

So to start off with what we have here is the conclusion of a neural network that we built in the main

8

part of the section and then at the end it pops out some probabilities for zero point ninety five for

9

a dog 0.05 five or 5 percent for a cat.

10

Given that photo on the left as an input This is after the train has been conducted this is actually

11

it's running and it's classifying a certain image.

12

And so the question here is how come these two values add up to one.

13

Because as far as we know from everything I learned about artificial neural networks there is nothing

14

to say that these two final neurons are connected between each other.

15

So how would they know what the value of the hold each one of them know what the value of the other

16

one is.

17

And how would they know to add their values up to one.

18

Well the answer is they wouldn't.

19

In the classic version of our artificial neural network and the only way that they do is because we

20

introduce a special function called the soft max function in order to help us out of the situation.

21

So normally what would happen is the dog and the cat neurons would have any kind of real values that

22

they don't have to be they don't have to add up to one.

23

But then we would apply the soft max function which is written up over there at the top and that would

24

bring these values to be between 0 and 1 and it would make them add up to 1 and 3 PPTA.

25

The soft max function or the normalized exponential function is a generalization of the logistic function

26

that quote unquote squash has a k dimensional vector of arbitrary real values to a k dimensional vector

27

of real values in the range of zero to one that add up to 1.

28

So basically it does exactly what we want.

29

It brings these values to be between 0 and 1 and make sure that they add up to 1.

30

And the way it works is that the way that this is possible is that because at the bottom we're here

31

you can see that there is a summation.

32

So it takes the exponent and puts it in the power of Zed and adds it up so that one's a two across all

33

of your classes.

34

All of these values.

35

And so there that's your normalization happening right there.

36

So that's how the Saucebox function works and it makes sense to introduce the soft next function into

37

convolutional neural networks because how strange would it be if you had a possible classes of a dog

38

and a cat and for the dog class you had possibility of 80 percent.

39

And for the cat claws you had a good 45 percent right.

40

It just doesn't make sense like that and therefore it's much better when you introduce the soft next

41

function and that's what you will find happening most of the time in convolutional and neural networks.

42

Now the other thing is that the soft max function comes hand-in-hand with something called the Cross

43

entropy function and it's a very handy thing for us.

44

So let's first look at the formula.

45

This is what the cross entry function looks like.

46

We're actually going to be using a different calculation going to be using this representation of the

47

century but the results are basically the same.

48

This is just easier to calculate.

49

And what I know this might sound very unrelated to anything right now just formulas on your screen but

50

there will be some additional re recommended reading at the end of this section so don't worry if you're

51

not picking up on the math.

52

Like even if we haven't explained the math right now.

53

But the point here is that what is across entropy well across entropy function.

54

Remember how we previously in artificial neural networks we had a function called the mean squared arrow

55

function which we used as the cost function for assessing our natural performance.

56

And our goal was to minimize the MSE in order to optimize our network performance.

57

Well that was our cost function then there and in convolutional neural networks we can still use MSE

58

but a better option in convolutional neural networks after you apply the soft max function turns out

59

to be the cross entropy function.

60

And in convolutional neural networks when you apply the cross entry functions not cost called the cost

61

function anymore is called the last function and they are very similar.

62

Theyre just a little terminological differences and like little bit different and on what they mean.

63

But for all purposes its pretty much the same thing.

64

And what happens is the last function is again something that we want to minimize in order to maximize

65

the performance of our network.

66

So lets have a look at a quick example on how of how this function can be applied.

67

So lets say we put an image of a dog into our network.

68

The predicted value for dog is 0.9 and this is doing the training so we know that we know the label

69

that is a dog.

70

So the predictive value 0.9 the prigged value for cat is 0.1 then here we have the label so we know

71

its a dog because this is training 0 1 for dogs or for cat.

72

And so in this case you need to use you need to plug these numbers into your formula for the cross entropy.

73

So how you do it is the values on the left going to the verbal cue.

74

The one that is under the logarithm in the on the right side and the values from the right would go

75

into P and so it's important to remember which one goes there because if you get them wrong you don't

76

want to be taking a logarithm for all me from zero value and or going from 1.

77

So you just want to plug them in.

78

Make sure you plug them into the correct places.

79

And then you basically add that up.

80

So that's how the cross entry works and we'll look at a actually right now we're going to look at a

81

specific step by step example of applying this function in real life and Ill kind of make make more

82

sense what Cross entropy is and it'll be less like that.

83

My goal in this toil is to make you more comfortable of cross century because it can sound very convoluted

84

and no pun intended it can.

85

Like convolutional neural networks it can sound very complex and scary but it's not.

86

That's that's the point.

87

So let's go ahead and apply it just so we know that it's not scary.

88

So here's your all that.

89

And also this will explain why we're doing this why we're looking into different cause functions.

90

So neural network one neural network let's say we have two neural networks and then we pass an image

91

of a dog and we know that this is a dog and not a cat.

92

And then we have another image our cat this time an animal and it's a cat not a dog and here we have

93

a we are looking at a hole which is in fact a dog not a cat.

94

If you look very closely.

95

So we want to see what our neural networks were will predict in the first case.

96

Neural network 1 90 percent dog 10 percent cat correct no network number to 60 percent dog 40 percent

97

cat still correct worse.

98

But correct.

99

Second option first neural network 10 percent cat dog 90 percent cat.

100

Correct.

101

You know that number to 30 percent dog 70 percent cat worse but still correct.

102

And then finally neural network in in image year old network won 40 percent dog 60 percent cat incorrect

103

neural network number to 10 percent dog and 90 percent cat incorrect and worse.

104

So the key here is that even though both net folks got it wrong in the last one through all three images

105

neural network one was outperforming neural network.

106

So even in the last case it was very it had it gave dog like a 40 percent chance as opposed to neural

107

network to only give dog a 10 percent chance or neural network one is outperforming across the board

108

when compared to neural network 2.

109

And so now we're going to look at the functions that they can measure performance that we've kind of

110

talked about the rating.

111

So let's put these into a table so there's neural network 1 you have the wrong number.

112

So that's the image number.

113

And then for image one you have.

114

What's it predicted 90 percent dog chimps and cat.

115

So there's the hat Marable's and then you have the actual value so dog correct cat incorrect.

116

Same thing for image number two and same thing for a minimum of three and same for neural network number

117

two.

118

So Dog 60 percent kept 40 percent in the first image.

119

That's what it predicted crotons was dog not a cat.

120

And so on.

121

And so now let's see what errors we can actually get.

122

So what errors we can calculate to estimate the performance and monitor the performance of our networks.

123

So one type of error is called the classification error.

124

And that is basically just asking it did you get it right or not.

125

Regardless of the probabilities is just DID YOU GET IT RIGHT.

126

Or did you get it right.

127

So in both cases for both neural networks each of them they got one.

128

So this is how you they go wrong.

129

So they got one out of three wrong.

130

So 33 percent error rate for your network 1 and 30 percent error rate for neural network.

131

As a baseline from this standpoint both neural networks perform at the same level but we know that's

132

not true.

133

We know that neural network Ikhwan is outperforming neural network.

134

That's why a classification error is not a good measure especially for the purposes of back propagation

135

mean square error different and by the way I did these calculations in Excel I just didn't want to bore

136

you with them but you can Tony just sit down and do them on a paper or in Excel.

137

These are very straightforward calculations just basically take the sum of squared errors and then just

138

take the average across your observations and that's pretty much it.

139

So for the for neural network one gets 25 percent for neural network 2 you get 71 percent error rates

140

so as you can see this one is more accurate.

141

It's telling us that nearly one has a much lower error rate than your own network.

142

And then cross entropy again.

143

We've seen the formula you can also calculate this is actually even easier to calculate than the mean

144

square error Cross area across entropy gives you 38 percent for neural network 1 and 1.0 6 for neural

145

network 2.

146

So you can see the results are a bit different.

147

When you look at them like that when you look at you know the miniskirt area and cross entropy and the

148

question of why would you use cross entropy over means squared error isn't just about the kind of like

149

the numbers that they say but all these calculations were just to show you that this is all it's all

150

doable you can just do it on a paper it's it's not.

151

It is not very intense mathematics.

152

These are pretty pretty simple straightforward things.

153

But the question of why would you use means cause entropy over means there is a very very good question

154

to ask.

155

I'm glad you asked that the answer to that is like there's several advantages of cross entropy over

156

mean squared error which are not obvious.

157

And so I'll I'll mention a couple but other then I'll I'll let you know where you can find out more.

158

So one of them is that if you for instance your at the very start of your back propagation your output

159

value is very very very very tiny very tiny.

160

So it's much smaller than the actual value that you want.

161

Then at the very start the gradient in your great and decent world will be very very low and you won't

162

be enough.

163

It be very hard for the neural network to actually start doing something and start moving around and

164

start adjusting those weights and start Movistar actually moving in the right direction.

165

Whereas when you use something like the cross entropy because it's got that logarithm in it it actually

166

helps the network assess even a small area like that and do something about it.

167

Here's how to think about it.

168

So let's say in again this is very in and in very intuitive approach.

169

There's going to be a link to the mathematics and you can derive these things through the mathematics

170

in more detail but a very intuitive approach.

171

Let's say your like your outcome that you want.

172

Is is one and right now you are at one one millionth of one.

173

Right.

174

$0.00 or is there one and then you improve next time you improve your outcome from from one millionth

175

to one thousandth.

176

And in terms of if you calculate the squared error you just subtracting one from the other.

177

Or basically in each case you're Kalka in a square and you'll see that the squared errors when you compare

178

one case versus other it didn't change that much.

179

You didn't improve your network that much when you looking at the mean square there.

180

But if you're looking at the cross entropy because you're taking a logarithm and then you're comparing

181

that to dividing one to the other.

182

You will see that you have actually improved your network significantly so that that jump from one million

183

to 1000 in mean squared error terms will be very low.

184

It will be insignificant and it won't.

185

It won't guide your gradient boosting process or your back propagation in the right direction.

186

It all it will guided in the right direction but it'll be like a very slow guidance it won't have enough

187

power whereas if you do recross entropy across entropy will understand that even though these are very

188

small adjustments that are just you know making a tiny change in absolute terms in relative terms it's

189

a huge improvement.

190

And we are definitely going in the right direction.

191

Let's keep going that way so cross entropy will help your neural network get to the right gets to the

192

optimal state is a better way for the neural network to get to get it to an optimal state.

193

But bear in mind that this only works when it across entropy is only the preferred method only for classification.

194

So if you're talking about things like regression like which we had in artificial neural networks then

195

you would rather go with me and squared error whereas cross entropy is better for classification and

196

again it has to do with the fact that we're using soft next function so that's a kind of intuitive explanation

197

of that a good place to learn a bit more about that if you're really interested in you know why are

198

we using cross versus mean square error.

199

Google a video by Geoffrey Hinton called the soft max output function and he explains it very well and

200

you know being the godfather of deep learning who can explain it better anyway.

201

And by the way any video by Geoffrey Hinton is golden.

202

He's just got a huge talent for explaining things anyway.

203

So that's that soft nice versus cross and I hope that gives you kind of like an intuitive understanding

204

of what's going on here.

205

But more importantly that you're not put off by the term cross entropy because headline will mention

206

it in the practical stories and I wanted to make sure that you're prepared for that.

207

And it's just another way of calculating your last function.

208

And another way of optimizing your network which is specifically tailored to classification problems

209

and therefore convolutional neural networks and comes in hand hand-in-hand with the soft max function.

210

So additional reading if you'd like a light introduction into cross entropy if you're interested in

211

the concentrate a bit more of course.

212

A good article to check out is called a friendly introduction to cross entropy loss by Rob DePietro

213

2016.

214

Here's the link below.

215

Very very nice very soft and nothing no super complex math.

216

Good analogies good examples using analogies of cars and you look at cars and talks about information

217

and bits and restrictions and you know how would you decode this whole Unico that it's so it's a good

218

article to have a look at and we'll give you a good overview of a cross entry like from an introductory

219

standpoint.

220

If you want to dig into the heavy math like what you see here then check out an article by or a blog

221

by how to implement a neural network Intermezzo too so in terms of use is like an intermediary thing

222

like a.

223

Intermittency in.

224

You know like when you go to a theater and you have like a break between the first part and the second

225

part.

226

So because he's like going through all these steps and then he's like and then he says I got to explain

227

this first.

228

And yes so that's why it's called intermezzo.

229

No other reason as far as I understand the articles by Peter Rolands 2016 as well so both are quite

230

recent.

231

And you know check out this if you'd like to dig into the mathematics behind Kross entropy behind the

232

soft Max and cross entropy in this article actually.

233

So there we go.

234

That's all there is to these two.

235

Hopefully I was able to add some additional clarity and good luck with that.

236

It's going to be fun and enjoy the practical tutorials.

237

I'll see you next time.

238

Until then enjoy learning.