Original subtitles

1

Hello and welcome back to the course on artificial intelligence.

2

Today we are finally talking about Kule learning.

3

All right so we've already got this equation the bellmen equation which we've added lots of components

4

to.

5

We've got the reward here which can be not just at the very end but it can be at any given step.

6

We've got the discount factor.

7

We've got the probability because now we're looking at mark of a decision processes.

8

And here we've got the possibility of ending up in a different states regardless of what action we take

9

or actually given the action we take.

10

There can be multiple states that we can end up in and then we got the value of the next states because

11

he kind of like a recursive function and so on but you probably still have one question.

12

The question is where in all of this is no letter Q Why is it all called q.

13

Learning.

14

So where's the cue.

15

And that's the question that we're going to be answering today.

16

So far we've been dealing with values the value of being in a certain state and now we're going to look

17

at how Q fits into all of that as well.

18

So here we've got two examples on the left is what we would be do so far our agent has been analyzing.

19

Ok I'm over here.

20

This is a mark of decision process so doesn't matter how I got here.

21

The rest of the environment doesn't care of the steps that it took me to get here from now on.

22

I have to make the optimal decision where to go here here or here.

23

Based on the current state and all the future states that come from here but not from the past.

24

And so he can see that there's three options there's state one state to state three.

25

And based on his experience he has calculated the values in these states and now he's going to using

26

the bellmen equation.

27

So even though this is a classic Proceso he knows that he'll go here but there's a chance that he will

28

go left right and so on.

29

So based on these values going to make a decision that's what we do so far and that is totally legitimate

30

approach here.

31

But now we're get modified a little bit.

32

We're going to take the same exact concept same exact problem but here instead of looking at the values

33

of each state that he can end up in we're going to look at the values or the value of each action.

34

So we're we're not going to use the letter V anymore because for the value of the state we're going

35

to use a Q and the you might have a question why the letter Q Well.

36

Q Some people speculate that.

37

Q Will I read this.

38

I think on Quora.

39

Somebody mentioned that Q is because of quality but at the same time I couldn't find any other references

40

to that so it might not be because that might just because that's the letter that was used at the time

41

and now it became super popular because it's all called key learning because of that.

42

So no exact reason was hold.

43

Q But nevertheless at least it helps us distinguish between V and Q So Q here.

44

There were presents rather than the value of the state it represents lets go of quality.

45

It represents the quality of the action that represents.

46

OK so I've got four actions.

47

What are the different qualities of these action.

48

What is the value of the action or the quality of the action which action is more lucrative so I need

49

a metric telling me all right how do I quantify this action and then I can compare them and that is

50

exactly what Q is.

51

And so he's got four possible actions.

52

As always go up right left or down.

53

And based on the action theres going to be a formula which tells us the quantifiable value of that action

54

which we're calling the Q q value of that action.

55

So let's have a look at how we going to derive this formula.

56

Q What how does it actually relate to these.

57

Because as you can imagine because actions lead to states there has to be some sort of link between

58

the two.

59

Right we've got we've already determined how to calculate this and we're pretty good at it.

60

We know how to use the Belman equation in very different environments with lots of different complications.

61

Well let's leverage that knowledge to understand how we can now calculate.

62

Q In order to make the same predictions because as you can imagine the environment doesn't change depending

63

depending on what approach we use the environment is going to be the same regardless.

64

So therefore this approach and this approach should always give the same result and therefore that's

65

another reason why these two should be linked.

66

So let's have a look.

67

So here is our view approach where we just get to look at the value of any given state this state or

68

any other state.

69

And here we go into we're just using the lead here because that's the current state.

70

And so therefore the terminology will be the same in both equations and here we using q as a Q Is the

71

of the state s and the action.

72

A because action is up but in which state we perform that action we perform that action in the State.

73

OK so now we're going to ride out the Belman equation for the first approach as you can see here we've

74

got the of s or the value of any given state s is the maximum of the reward that you get a maximum bet

75

based on the actions you have three.

76

In this case you actually have four actions so maxim out of all the possible actions of this part which

77

we've heard discussed many times so this is our reward that we get from performing that action in that

78

state plaza discount in fact multiplied by the expected value of the new state that we're going to be

79

in an expected value because it is a stochastic process.

80

We don't know exactly for sure that we're going to end up over here we might end up on the left or the

81

right sort of probability.

82

That's why these probabilities are in you.

83

All right so that's our value.

84

And now let's look at.

85

Q So Q is going to be defined.

86

We're going to use this to define Q So let's say the agent from this location from this state perform

87

the action up.

88

What is the q value going to be called to.

89

Well first of all let's see what he will get in return for performing this action up.

90

First thing that you'll get is a reward right.

91

Knows no doubt about it.

92

There's going to be some sort of rule or might be zero but we know that the whole is the way this reinforcement

93

learning process works is that some towns are performing certain actions from a given state or two.

94

So I'm going to add that in here.

95

And then we're going to add what are we going to add.

96

Well let's think about it.

97

What is the next thing that happens after he's going there.

98

Well next thing that happens is that now the agent is in a certain state he could end up here with a

99

80 percent probability or some probability.

100

But actually up here right here.

101

But wherever he ends up now there's we already have a quantified metric for that state he's in.

102

And that is actually the value of that state.

103

But because he came up in many different states and three of the possible different states we have to

104

look at the expected value of the state that he'll be in.

105

And so we're going to add that in we're going to add.

106

Of course the discounted factor as we previously had because that is somewhere in the future.

107

And then we're going to add the some of across all possible states across all possible states that he

108

could end up by taking this action.

109

Terms of probability.

110

So what we're saying here is that OK so by performing an action you're going to get a reward Plus which

111

is a quantified metric Plus you're going to get.

112

You end up in a state we don't know which one it could be here.

113

Could be here it could be here.

114

But here is the expected value of the state that you're going to end up in.

115

And now we're going to multiply by discounting factor because that is one move away.

116

So that is our Q value for this for performance section and what you will notice here right away is

117

that.

118

Q The Q value is actually exactly identical to what's inside these brackets over here.

119

And why is that.

120

Well if you think about it here we're taking the maximum of the results will get the maximum across

121

all possible actions so we got for action taking the maximum across all possible actions of the result

122

that we'll get by taking each of those actions and enqueue we're defining.

123

Interesting.

124

What will we get by taking a certain action.

125

So if you think about it it makes sense that the value of a state.

126

So for instance this state is the maximum of all of the possible Q values.

127

Right so here in the States by being in the state the agent has one key value to keep the 3Q value for

128

q value.

129

So yes positive for possible Q values while the value of the stay it makes sense that the value of the

130

state is the maximum of all of those four key values.

131

That is exactly what we can see here.

132

That's a good confirmation of this new formula that we derive.

133

If that wasn't the case if that if that didn't match up then we would have questions would be like.

134

So why why doesn't it match.

135

Why doesn't it match up if.

136

Q value is a quantified metric of performing an action and V depends on the floor.

137

Is like is the maximum of the possible results of the four actions that he can perform over that makes

138

sense.

139

And that confirms the formula that we've just derived and now we're going to make it even more interesting.

140

We're going to get rid of the Wii entirely because you can see here you've got Wii is a recursive function.

141

So and then you've got me and then B and then B and then B and so on.

142

So you can express this view through all of the following Vee's the most optimal these will come up

143

here.

144

We're expecting Q As a funk a recursive function of the OR as a function of the next V and then you'd

145

have to plug in this V and then we get back to the B.

146

So what are we going to do is we're actually going to take this V and we're going to we're going to

147

replace it with Q Right so let's have a look at that.

148

We're going to take this V of the next state and we're going to plug this into that formula over here.

149

And as you can see now so this part doesn't change this probability doesn't change.

150

But as we just discussed the of s is the maximum by all actions of q of S and a right over here.

151

So that's what we're going to replace in here.

152

So we're going to say a maximum of of course is the new action the action that we're going to take because

153

here we've got the Wii of as prime.

154

So here now we've got the maximal console at a prime.

155

So the actions that we're going to take from this state are from wherever whichever other state we end

156

up in but the action we're going to take from there and Maxima across all those and the maximum is of

157

all the cube values that will that are available to us in that new state as prime comma a prime.

158

And that's action.

159

So that's the.

160

So there's going to be another four.

161

Q values there.

162

So now as you can see let's go through again.

163

So as from what we derive this word would be just cause just through logic and intuition so that we

164

can see that VNS are actually view of AS and of and a are linked.

165

The of S is the maximum across all actions of Cuba S and you can see right here so this this part is

166

identical to this part.

167

And then we're going to leverage that and we're going to replace this bit with VNS from here but not

168

this exact funnel we're going to take this internal part and replace it with kill the innocent.

169

So we're going to plug that in here and this part is going to be q of s prime a prime maximum of cube

170

by Crucell a Priam's of Q As Prime a prime.

171

And now we have our formula.

172

So now we have a recursive formula for the q value so now the agent can think what's the value of the

173

section what's the quality of this section was the new value of this action.

174

Well it depends on the reward I get in the immediate step after that plus it depends on the discounted

175

factor times the maximum of all the possible Q actions in that state.

176

But I don't know if I'm going to get their side need to also look at that state in that state and that's

177

why we have this expected value over here so we have some probability times the maximum that's expected

178

value.

179

So a very similar formula as you can see but this time we're expressing things through the q values

180

and that's why this whole algorithm is called Kill learning because this is what is looked at this is

181

what the agents actually use they don't look at the states look at their possible actions and then based

182

on the actions on the q value of the actions they will decide which action to take.

183

So they'll just look at the maximum Q value in this given state it has four actions.

184

What is the best action to take so it can compare sort of comparing the different states that can end

185

up end up in is going to compare the possible actions that it currently has then by finding the optimal

186

one is going to take that action and then engage is going to repeat that process repeat that process

187

and so on.

188

So now you can see how all this comes together how the reward the discounting facts or the stochastic

189

market decision processes and the values and the q values all come together in order to cueist this

190

one super powerful Belman equation for q values which we can now apply and let our agents learn how

191

to beat the environment.

192

And so that is a intuitive explanation of what's going on.

193

I know we went through the formulas but it is necessary because this is like our formula that's we've

194

been going through this whole chapter and I think it's a good transition from the To.

195

Q And it illustrates how there are links between Yishun And if you'd like to get a bit more of a rigorous

196

approach mathematical approach and like you see the mathematics behind it and learn a bit more about

197

q values and how they work.

198

Then we've got some additional reading for you.

199

This paper is called Markov decision processes concepts and algorithms by martn von Autor low 2009.

200

So you cut the link here as always and here you can read in a bit more detail to understand all the

201

nitty gritty behind Hugh values and so on and now that we've discussed all of these things relating

202

to the Belman equation now we are ready to look at something more complex such as this paper in order

203

if if we want to get some additional information on this in order to kind of get a deeper understanding.

204

But even if you don't read the newspaper or radio you should have a good working knowledge of what learning

205

is all about and how agents come up with the actions that they need to take in a certain environment.

206

So I hope you enjoy today Statoil and I look forward to your next them.

207

Until then enjoy.

208

I.