Original subtitles

1 1

Hello and Welcome back to the course 2

2

on Deep Learning. 3

3

Today we're talking about the neuron, 4

4

which is the basic building block 5

5

of artificial neural networks. 6

6

So let's get started. 7

7

Previously we saw an image which looked like this. 8

8

And these are actual, real life neurons 9

9

which have smeared on to glass, colored a little bit, 10

10

and they are observed through a microscope. 11

11

So this is what they look like as you can see, 12

12

quite an interesting structure. 13

13

A body, and a lot of different tails, 14

14

kind of branches coming out of them. 15

15

And this is very interesting but the question is 16

16

how can we recreate that in a machine? 17

17

Because we really need to recreate that in a machine 18

18

since the whole purpose of Deep Learning is to 19

19

mimic how the human brain works. 20

20

In the hopes that by doing so, 21

21

we are going to create something amazing. 22

22

We are going to create an amazing infrastructure 23

23

for machines to be able to learn. 24

24

And why do we hope for that? 25

25

Well because the human brain is, 26

26

well just happens to be one of the most powerful 27

27

learning tools on the planet, 28

28

or like learning mechanisms on the planet. 29

29

And we just hope that if we recreate that 30

30

we'll have something as awesome as that. 31

31

So our challenge right now, 32

32

our very first step to creating 33

33

artificial neural networks, 34

34

is to recreate a neuron. 35

35

So how do we do that? 36

36

Well, first let's take a closer look 37

37

at what it actually is. 38

38

This image was first created by 39

39

a Spanish neural scientist, 40

40

Santiago Ramón y Cajal, in 1899. 41

41

And what he did was he dyed neurons in 42

42

actual brain tissue and looked 43

43

at them under a microscope. 44

44

And while he was looking at them 45

45

he actually drew what he saw. 46

46

And this is what he saw. 47

47

He saw two neurons or two large neurons 48

48

over there at the top, 49

49

which had all these branches coming out of them 50

50

towards their top parts and then each had a 51

51

rod or thread coming out towards the bottom, very long one. 52

52

And that's what he saw. 53

53

And now, you know, technology has advanced 54

54

quite a lot and we have seen neurons much closer 55

55

and more detailed and now we can actually draw 56

56

what it looks like diagrammatically. 57

57

So let's have a look at that. 58

58

Here's a neuron, this is what it looks like. 59

59

Very similar to what Santiago Ramón drew over here. 60

60

Here in this neuron what we can see is that 61

61

its got a body, that's the main part of the neuron. 62

62

And then its got some branches at the top, 63

63

which are called dendrites. 64

64

And its also got an axon, 65

65

which is that long tail of the neuron. 66

66

So what are these dendrites for and what's the axon for. 67

67

Well, the key point to understand here is that 68

68

neurons by themselves are pretty much useless. 69

69

It's like an ant. 70

70

An ant on its own can't do much, 71

71

like 5 ants together maybe they can pick something up. 72

72

But again, they can't build an ant hill, 73

73

they can't establish a colony, 74

74

they can't work together as a huge organism. 75

75

But at the same time, when you have lots and lots of ants, 76

76

like you have a million ants, they can build a whole colony, 77

77

they can build an ant hill. 78

78

Same thing with neurons. 79

79

By itself it's not that strong, 80

80

but when you have lots of neurons together, 81

81

they work together to do magic. 82

82

And how do they work together? That's a question. 83

83

Well, that's what the dendrites and axon are for. 84

84

So the dendrites are kind of like the 85

85

receivers of the signal for the neuron, 86

86

and axon is the transmitter of the signal for the neuron. 87

87

And here's an image of how it all works conceptually. 88

88

So at the top you got a neuron, 89

89

and you can see that its dendrites are connected 90

90

to axons of other neurons that are like 91

91

even further away above it. 92

92

And then the signal from this neuron travels down 93

93

its axon and connects or passes onto 94

94

the dendrites of the other neuron. 95

95

And that's how they're connected. 96

96

And in that small image over there, 97

97

you can see that 98

98

the axon doesn't actually touch the dendrite. 99

99

(laughs) A lot of machine learning, 100

100

or a few machine learning scientists 101

101

are very adamant about the fact 102

102

that it doesn't touch. 103

103

It doesn't touch, it has been proven that 104

104

there is no physical connection there. 105

105

But the point that we are interested in 106

106

is that that connection between them, 107

107

that the whole concept of the signal being passed, 108

108

that's called the synapse. 109

109

You can see over there, in that little image, 110

110

that figure bracket is synapse. 111

111

That's the term we're going to be using. 112

112

Instead of calling our artificial neurons, 113

113

the lines we're gonna have, 114

114

or the connectors for artificial neurons 115

115

we're not be calling them axons or dendrites, 116

116

because then the question is 117

117

whose connection is this? 118

118

Is it that neuron's or is it this neuron's? 119

119

We're just going to call them synapses. 120

120

And that kind of just answers all the questions. 121

121

I mean it's basically just where the signal is passed. 122

122

Doesn't matter who that element belongs to. 123

123

That's just a representation of the signal 124

124

being passed and we see that just now. 125

125

So basically that's how a neuron works. 126

126

Let's move on to how we're going to represent 127

127

neurons or how we're going to create neurons in machines. 128

128

So now we're moving away from neural science 129

129

and moving into technology. 130

130

And here we go. 131

131

So, here's our neuron, 132

132

also sometimes called the node. 133

133

The neuron gets some input signals. 134

134

And it has an output signal. 135

135

So dendrites and axons, remember? 136

136

But again, we're gonna call these synopses. 137

137

These input signals, 138

138

we're going to represent them with 139

139

other neurons as well. 140

140

So, in this specific case, 141

141

you can see that 142

142

this neuron, this green neuron, 143

143

is getting signals from yellow neurons. 144

144

And in this course, we are going to try 145

145

to stick to a certain color coding regime, 146

146

where yellow means an input layer. 147

147

So basically all the neurons that are 148

148

on the outer layer, on the first front of 149

149

where the signals coming in. 150

150

By signal, it might be a bit of an overkill 151

151

to call this a signal. 152

152

It's just basically input value. 153

153

So you know how even like in a simple 154

154

linear regression you have input values, 155

155

and then you have a predicted value. 156

156

Same thing here. 157

157

So you have input values, 158

158

and there they are, the yellow ones. 159

159

And on the right to you we see just now 160

160

it'll be red, it'll be the output value. 161

161

The thing that I wanted to point out here is that 162

162

in this specific example we are looking at 163

163

a neuron which is getting its signals from 164

164

the input layer neurons. 165

165

So they are also neurons but 166

166

they are input layer neurons. 167

167

Sometimes you'll have neurons which 168

168

get their signal from other hidden layer neurons, 169

169

so from other green neurons. 170

170

And the concept is gonna be exactly the same. 171

171

Just in this case, for simplicity's sake, 172

172

we're portraying this example. 173

173

And in terms of the input layer, 174

174

the way to think about it is 175

175

in the analogy of the human brain, 176

176

the input layer is your senses, right. 177

177

So whatever you can see, hear, 178

178

feel, touch or smell. 179

179

And of course, 180

180

there's a lot of things you can see, 181

181

there's a lot of information coming in. 182

182

But those are your... 183

183

that's what your brain is limited to, 184

184

it's pretty much a (laughs) 185

185

it's pretty much lives in a box 186

186

made out of bones and it's only... 187

187

It's a mind blowing fact to think about. 188

188

Your brain is just locked in a black box, 189

189

and the only thing... 190

190

and it can't see, it can't hear, 191

191

the only thing it's getting 192

192

is electrical impulses coming from 193

193

these organs that you have, 194

194

which are called your ears, nose, eyes, 195

195

your sense of touch and whatever... and your taste. 196

196

It's just getting signals but 197

197

it basically lives in this dark black box 198

198

and it's making sense of the world through your senses. 199

199

It's phenomenal. 200

200

So you have these inputs that are coming in, 201

201

and in terms of human brain those are your five senses, 202

202

in terms of machine learning or deep learning, 203

203

that is basically your input values, 204

204

so your independent variables, 205

205

and we will get to that in a second. 206

206

So your input values, 207

207

the signal is passed through synapses to your neuron, 208

208

and then your neuron has an output value, 209

209

that it passes further on down the chain. 210

210

In this specific case, in terms of color coding, 211

211

again yellow means input layer. 212

212

So we kind of simplifying everything here. 213

213

We're saying we're only gonna have like the input layer, 214

214

then we're gonna have one hidden layer, 215

215

with the green, which is a hidden layer, 216

216

and then we're gonna have our output layer right away. 217

217

So just so that we can get used to those colors for now. 218

218

So there we go, that's the basic structure. 219

219

So now let's look at a bit more detail 220

220

at these different elements that we have. 221

221

So we got the input layer. 222

222

And what do we have here? 223

223

Well, we have these inputs which are 224

224

in fact independent variables. 225

225

So independent variable one, 226

226

independent variable two, 227

227

and independent variable m. 228

228

The important thing to remember here, 229

229

is that these independent variables 230

230

are all for one single observation. 231

231

So think of it as one row in your data base. 232

232

One observation. 233

233

You just take all of the independent variables, 234

234

maybe it's the age of the person, 235

235

the amount of money in their bank account, 236

236

how do they drive or walk to work, 237

237

what method of transportation do they use. 238

238

But that's all descriptions of one specific person, 239

239

that you are, either you're training your model on, 240

240

or you're performing some prediction on. 241

241

And the other thing you need to know 242

242

about these variables is that 243

243

you need to standardize them. 244

244

You need to either standardize them which means 245

245

make sure they have a mean of zero and variance one, 246

246

or you can also sometimes and 247

247

Hadelin will point out these tricks 248

248

in a bit more detail, 249

249

perhaps in the practical tutorials 250

250

you might come across these, 251

251

sometimes you might want to not standardize 252

252

you might wanna normalize them. 253

253

Meaning that instead of making sure that 254

254

mean is zero and variance is one, 255

255

you just subtract the minimum value and 256

256

then you divide it by maximum minus minimum, 257

257

so by the range of your values and 258

258

therefore you get values between zero and one. 259

259

Depend on the scenario you might wanna do one 260

260

or the other but basically you want 261

261

all of these variables to be quite similar, 262

262

in about the same range of values. 263

263

Why's that? 264

264

Well all of these values are going to 265

265

go into a neural network where as 266

266

we all see just now they will be added up and 267

267

multiplied by weights added up and so on. 268

268

It's just going to be easier for 269

269

the neural network to process them 270

270

if they are all about the same. 271

271

And that's just how it is going to 272

272

be able to work properly. 273

273

And if you want to read more about 274

274

standardization, normalization and other things 275

275

you can do with your input variables, 276

276

a good additional reading paper is called 277

277

Efficient BackProp by Yan LeCun 1998, 278

278

the link's over there. 279

279

So Yan LeCun, we're actually going to 280

280

talk about this phenomenal person 281

281

in the place of Deep Learning 282

282

in the part of the course where 283

283

we're talking about illusional neural networks. 284

284

You'll see that this is definitely a person 285

285

who knows what he's talking about. 286

286

He's a close friend of Geoffrey Hinton, 287

287

who we already seen, who we've already mentioned. 288

288

So in this paper you will learn more about 289

289

standardization and normalization. 290

290

But you can pick up lots of 291

291

other different tips and tricks and 292

292

be a good source of additional reading 293

293

as you go through this course. 294

294

So check it out if you're interested 295

295

in some additional reading. 296

296

There we go, so that's what we need to do 297

297

with the variables. 298

298

And here we've got the output value. 299

299

So what can our output value be? 300

300

Well we've got a couple of options. 301

301

Output value can be, 302

302

it can be continuous, for instance, price; 303

303

it can be binary, for instance, 304

304

a person will exit or stay; 305

305

or it can be a categorical variable. 306

306

If it's a categorical variable, 307

307

the important thing to remember here is that 308

308

in that case, your output value won't be just one, 309

309

it'll be several output values, 310

310

because these will be your dummy variables, 311

311

which will be representing your categories. 312

312

And that's just how it works. 313

313

Just important to remember that, 314

314

in that case that's how you're going to be getting 315

315

your categories out of the artificial neural network. 316

316

But let's go back to our simple case 317

317

of one output value. 318

318

And now one more point, a point I've already made, 319

319

I just want to reiterate this point. 320

320

On the left you've got a single observation, 321

321

so one row from your data set, 322

322

and on the right you have a single observation as well. 323

323

That is the same observation. 324

324

So important to remember that whatever inputs 325

325

you're putting in, that's for one row, 326

326

and then the output you get back is 327

327

for that exact same row. 328

328

Or if you're training your neural network then 329

329

you're putting the inputs in for that one row, 330

330

you're putting the output in for that one row. 331

331

So if you wanna simplify the complexity, 332

332

think of it as like a simple linear regression, 333

333

or a multi-variant linear regression. 334

334

So you're putting in your values, 335

335

you have your output. 336

336

There's no question about it 337

337

when we are talking about things like regression, 338

338

because we're so used to it. 339

339

Same thing here. It's nothing too complex. 340

340

We're just putting in values, 341

341

we're getting an output. 342

342

But just remember that every time 343

343

it's one row that you're dealing with. 344

344

So you don't get confused and start putting in 345

345

like thinking these are different rows that 346

346

you're putting into your artificial 347

347

neural network or something. 348

348

This is all just values in that one row. 349

349

So different observation, 350

350

different characteristics of, 351

351

or attributes relating to that one observation. 352

352

Every single time. 353

353

Okay so next thing that we wanna 354

354

talk about here is the synapses. 355

355

Here we've got synapses and 356

356

they all actually get assigned weights. 357

357

We're gonna talk more about weights further down, 358

358

but in short, weights are crucial to 359

359

artificial neural networks functioning. 360

360

Because weights are how neural networks learn. 361

361

By adjusting the weights, 362

362

the neural network decides in every single case, 363

363

what signal is important and what signal 364

364

is not important to a certain neuron, 365

365

what signal gets passed along and 366

366

what signal doesn't get passed along, 367

367

or to what strength, to what extent 368

368

signals get passed along. 369

369

So weights are crucial, 370

370

they are the things that get adjusted 371

371

through the process of learning. 372

372

When you're training your artificial neural network, 373

373

you're basically adjusting all of the weights 374

374

in all of the synapses across this 375

375

whole neural network and 376

376

that's where gradient descent and 377

377

back propagation come into play and 378

378

those are concepts that we'll also discuss. 379

379

So basically those are the weights. 380

380

That's all you need to know for now. 381

381

Here we've got the neuron. 382

382

So signals go into the neuron and 383

383

what happens in the neuron? 384

384

So this is the interesting part. 385

385

We're talking about the neuron today, 386

386

what happens inside the neuron? 387

387

So, a few things happen. 388

388

First thing, and the first step is that 389

389

all of these values that it's getting, get added up. 390

390

So it takes the added, so the weighted sum 391

391

of all of the input values that it's getting. 392

392

Very simple, right? 393

393

It's very very straight forward. 394

394

Just add up, multiply by the weight, add them up. 395

395

And then, it applies an activation function. 396

396

Now we're gonna talk more about activation function 397

397

further down but it's basically a function 398

398

that is assigned to this neuron or to this olier, 399

399

and it is applied to this weighted sum, 400

400

and then from that the neuron understands 401

401

if it needs to pass on a signal. 402

402

That's the signal it passes on, 403

403

the function applied to, the weighted sum. 404

404

But basically depending on the function, 405

405

the neuron will either pass on the signal or 406

406

it won't pass the signal on. 407

407

And that's exactly what happen here in step three. 408

408

The neuron passes on that signal 409

409

to the next neuron down the line. 410

410

And that's what we're going to talk about 411

411

in the next tutorial because it is 412

412

quite an important topic. 413

413

We want to delve deeper into 414

414

the activation function. 415

415

But hopefully for now, 416

416

everything is, should be pretty clear, 417

417

how you've got input values, 418

418

you've got weights, 419

419

you've got these synapses, 420

420

you've got something that happens in the neuron, 421

421

you've got weighted sum 422

422

and then the activation function applied to them 423

423

that is passed on then that is repeated 424

424

throughout the whole neural network, 425

425

on and on and on and on. 426

426

Thousands hundreds of thousands of times 427

427

depending on how big, how many neurons you have, 428

428

how many synapses you have in your neural network. 429

429

So there we go! 430

430

Hope you enjoyed today's tutorial, 431

431

can't wait to see you next time. 432

432

And until then, enjoy Deep Learning!