All language subtitles for Machine-Learning-for-Everybody-Full-Course_en

Kylie Ying has worked at many interesting places such as MIT,

She's a physicist, engineer, and basically a genius. And now she's

about machine learning in a way that is accessible to absolute

What's up you guys? So welcome to Machine Learning for Everyone.

is interested in machine learning and you think you are considered

is for you. In this video, we'll talk about supervised and

we'll go through maybe a little bit of the logic or math behind

we can program it on Google CoLab. If there are certain things

you're somebody with more experience than me, please feel free to

and we can all as a community learn from this together. So with

Without wasting any time, let's just dive straight into the code

concepts as we go. So this here is the UCI machine learning

they just have a ton of data sets that we can access. And I found

the magic gamma telescope data set. So in this data set, if you

to summarize what I what I think is going on, is there's this

these high energy particles hitting the telescope. Now there's a

actually records certain patterns of you know, how this light hits

properties of those patterns in order to predict what type of

whether it was a gamma particle, or some other head, like hadron.

the attributes of those patterns that we collect in the camera. So

know, some length, width, size, asymmetry, etc. Now we're going to

help us discriminate the patterns and whether or not they came

So in order to do this, we're going to come up here, go to the

to click this magic zero for data, and we're going to download

notebook open. So you go to colab dot research dot google.com, you

I'm just going to call this the magic data set. So actually, I'm

magic example. Okay. So with that, I'm going to first start with

you know, I always import NumPy, I always import pandas. And I

And then we'll import other things as we go. So yeah

we run that in order to run the cell, you can either click this

on my computer, it's just shift enter and that that will run the

to order I'm just going to, you know, let you guys know, okay,

So I've copied and pasted this actually, but this is just where I

And in order to import that downloaded file that we we got from

over here to this folder thing. And I am literally just going to

Okay. So in order to take a look at, you know, what does this file

do we have the labels? Do we not? I mean, we could open it on our

pandas read CSV. And we can pass in the name of this file.

And let's see what it returns. So it doesn't seem like we have the

I'm just going to make the columns, the column labels, all of

So I'm just going to take these values and make that the column

All right, how do I do that? So basically, I will come back here,

calls. And I will type in all of those things. With f size, f

We have f symmetry, f m three long, f m three trans, f alpha.

Okay, great. Now in order to label those as these columns down

So basically, this command here just reads some CSV file that you

separated values, and turns that into a pandas data frame object.

then it basically assigns these labels to the columns of this data

this data frame equal to DF. And then if we call the head is just

give me the first five things. Now you'll see that we have labels

All right, great. So one thing that you might notice is that over

we have G and H. So if I actually go down here, and I do data

you'll see that I have either G's or H's, and these stand for

And our computer is not so good at understanding letters, right?

understanding numbers. So what we're going to do is we're going to

one for H. So here, I'm going to set this equal to this, whether

I'm just going to say as type int. So what this should do is

if it equals G, then this is true. So I guess that would be one.

be false. So that would be zero, but I'm just converting G and H

really matter. Like, if G is one and H is zero or vice versa. Let

now and talk about this data set. So here I have some data frame,

values for each entry. Now this is a you know, each of these is

it's one item in our data set, it's one data point, all of these

thing when I mentioned, oh, this is one example, or this is one

these samples, they have, you know, one quality for each or one

up here, and then it has the class. Now what we're going to do in

predict for future, you know, samples, whether the class is G for

that is something known as classification. Now, all of these up

and features are just things that we're going to pass into our

the label, which in this case is the class column. So for you

10 different features. So I have 10 different values that I can

And I can spit out, you know, the class the label, and I know the

this is actually supervised learning. All right. So before I move

little crash course on what I just said. This is machine learning

question is, what is machine learning? Well, machine learning is a

that focuses on certain algorithms, which might help a computer

programmer being there telling the computer exactly what to do.

programming. So you might have heard of AI and ML and data

all of these. So AI is artificial intelligence. And that's an area

goal is to enable computers and machines to perform human like

Now machine learning is a subset of AI that tries to solve one

using certain data. And data science is a field that attempts to

from data. And that might mean we're using machine learning. So

and all of them might use machine learning. So there are a few

The first one is supervised learning. And in supervised learning,

So this means whatever input we get, we have a corresponding

models and to learn outputs of different new inputs that we might

I might have these pictures, okay, to a computer, all these

with a certain color. Now in supervised learning, all of these

them, this is the output that we might want the computer to be

over here, this picture is a cat, this picture is a dog, and this

Now there's also unsupervised learning. And in unsupervised

to learn about patterns in the data. So here are here are my input

images, they're just pixels. Well, okay, let's say I have a bunch

And what I can do is I can feed all these to my computer. And I

my computer is not going to be able to say, Oh, this is a cat, dog

you know, the output. But it might be able to cluster all these

Hey, all of these have something in common. All of these have

down here have something in common, that's finding some sort of

And finally, we have reinforcement learning. And reinforcement

there's an agent that is learning in some sort of interactive

penalties. So let's think of a dog, we can train our dog, but

any wrong or right output at any given moment, right? Well, let's

Essentially, what we're doing is we're giving rewards to our

Hey, this is probably something good that you want to keep doing.

But in this class today, we'll be focusing on supervised learning

and learning different models for each of those. Alright, so let's

first. So this is kind of what a machine learning model looks like

that are going into some model. And then the model is spitting out

So all these inputs, this is what we call the feature vector. Now

of features that we can have, we might have qualitative features.

categorical data, there's either a finite number of categories or

qualitative feature might be gender. And in this case, there's

the example, I know this might be a little bit outdated. Here we

two genders, there are two different categories. That's a piece of

example might be okay, we have, you know, a bunch of different

a nation or a location, that might also be an example of

these, there's no inherent order. It's not like, you know, we can

three, etc. Right? There's not really any inherent order built

data sets. That's why we call this nominal data. Now, for nominal

to feed it into our computer is using something called one hot

know, I have a data set, some of the items in our data, some of

some might be from India, then Canada, then France. Now, how do we

we have to do something called one hot encoding. And basically,

well, if it matches some category, make that a one. And if it

So for example, if your input were from the US, you would you

0100. Canada, okay, well, the item representing Canada is one and

France is one. And then you can see that the rest are zeros,

Now, there are also a different type of qualitative feature. So

there are different age groups, there's babies, toddlers,

adults, and so on, right. And on the right hand side, we might

bad, not so good, mediocre, good, and then like, great. Now, these

data, because they have some sort of inherent order, right? Like,

being a baby than being an elderly person, right? Or good is

bad. So these have some sort of inherent ordering system. And so

we can actually just mark them from, you know, one to five, or we

let's give it a number. And this makes sense. Because, like, for

just said, how good is closer to great, then good is close to not

to five, then four is close to one. So this actually kind of makes

computer as well. Alright, there are also quantitative pieces of

pieces of data are numerical valued pieces of data. So this could

you know, they might be integers, or it could be continuous, which

So for example, the length of something is a quantitative piece of

feature, the temperature of something is a quantitative feature.

Easter eggs I collected in my basket, this Easter egg hunt, that

feature. Okay, so these are continuous. And this over here is the

that go into our feature vector, those are our features that we're

our computers are really, really good at understanding math, right

they're not so good at understanding things that humans might be

Well, what are the types of predictions that our model can output?

there are some different tasks, there's one classification, and

just saying, okay, predict discrete classes. And that might mean,

this is a pizza, and this is ice cream. Okay, so there are three

pictures of hot dogs, pizza or ice cream, I can put under these

Hot dog, pizza, ice cream. This is something known as multi class

binary classification. And binary classification, you might have

only two categories that you're working with something that is

isn't binary classification. Okay, so yeah, other examples. So if

sentiment, that's binary classification. Maybe you're predicting

dogs. That's binary classification. Maybe, you know, you are

trying to figure out if an email spam or not spam. So that's also

Now for multi class classification, you might have, you know, cat,

rabbit, etc. We might have different types of fruits like orange,

maybe different plant species. But multi class classification just

and binary means we're predicting between two things. There's also

when we talk about supervised learning. And this just means we're

values. So instead of just trying to predict different categories,

with a number that you know, is on some sort of scale. So some

be the price of aetherium tomorrow, or it might be okay, what is

Or it might be what is the price of this house? Right? So these

discrete classes. We're trying to predict a number that's as close

using different features of our data set. So that's exactly what

supervised learning. Now let's talk about the model itself. How do

Or how can we tell whether or not it's even learning? So before we

let's talk about how can we actually like evaluate these models?

whether something is a good model or bad model? So let's take a

set has this is from a diabetes, a Pima Indian diabetes data set.

number of pregnancies, different glucose levels, blood pressure,

age, and then the outcome whether or not they have diabetes one

So here, all of these are quantitative features, right, because

So each row is a different sample in the data. So it's a different

and each row represents one person in this data set. Now this

different feature. So this one here is some measure of blood

over here, as we mentioned is the output label. So this one is

And as I mentioned, this is what we would call a feature vector,

features in one sample. And this is what's known as the target, or

vector. That's what we're trying to predict. And all of these

And over here, this is our labels or targets vector y. So I've

bar to kind of talk about some of the other concepts in machine

we have our x, our features matrix, and over here, this is our

will be fed into our model, right. And our model will make some

is we compare that prediction to the actual value of y that we

that's the whole point of supervised learning is we can compare

oh, what is the truth, actually, and then we can go back and we

iteration, we get closer to what the true value is. So that whole

okay, what's the difference? Where did we go wrong? That's what's

Alright, so take this whole, you know, chunk right here, do we

chocolate bar into the model to train our model? Not really,

how do we know that our model can do well on new data that we

create a model to predict whether or not someone has diabetes,

data, and I see that all my training data does well, I go to some

model. I think you can use this to predict if somebody has

be effective or not? Probably not, right? Because we haven't

generalize. Okay, it might do well after you know, our model has

over again. But what about new data? Can our model handle new

model to assess that? So we actually break up our whole data set

types of data sets, we call it the training data set, the

set. And you know, you might have 60% here 20% and 20% or 80 10

many statistics you have, I think either of those would be

the training data set into our model, we come up with, you know,

corresponding with each sample that we put into our model, we

between our prediction and the true values, this is something

what's the difference here, in some numerical quantity, of course.

and that's what we call training. Okay. So then, once you know,

we can put our validation set through this model. And the

check during or after training to ensure that the model can handle

single time after we train one iteration, we might stick the

the loss there. And then after our training is over, we can assess

hey, what's the loss there. But one key difference here is that we

this loss never gets fed back into the model, right, that feedback

Alright, so let's talk about loss really quickly. So here, I have

I have some sort of data that's being fed into the model, and then

here is pretty far from you know, this truth that we want. And so

model B, again, this is pretty far from what we want. So this loss

let's give it 1.5. Now this one here, it's pretty close, I mean,

to this one. So that might have a loss of 0.5. And then this one

but still better than these two. So that loss might be 0.9. Okay,

performs the best? Well, model C has a smallest loss, so it's

take model C. After you know, we've come up with these, all these

C is probably the best model. We take model C, and we run our test

test set is used as a final check to see how generalizable that

you know, finish training my diabetes data set, then I could run

data and I can say, oh, like, this is how we perform on data that

any point during the training process. Okay. And that loss, that's

of my test set, or this would be the final reported performance of

So let's talk about this thing called loss, because I think I kind

right? So loss is the difference between your prediction and the

So this would give a slightly higher loss than this. And this

because it's even more off. In computer science, we like formulas,

of describing things. So here are some examples of loss functions

up with numbers. This here is known as L one loss. And basically,

absolute value of whatever your you know, real value is, whatever

subtracts the predicted value, and takes the absolute value of

value is a function that looks something like this. So the further

right in either direction. So if your real value is off from your

then your loss for that point would be 10. And then this sum here

we're taking all the points in our data set. And we're trying to

everything is. Now, we also have something called L two loss. So

which means that if it's close, the penalty is very minimal. And

then the penalty is much, much higher. Okay. And this instead of

the the difference between the two. Now, there's also something

It looks something like this. And this is for binary

loss that we use. So this loss, you know, I'm not going to really

But you just need to know that loss decreases as the performance

other measures of accurate or performance as well. So for example,

So let's say that these are pictures that I'm feeding my model,

might be apple, orange, orange, apple, okay, but the actual is

three of them were correct. And one of them was incorrect. So the

three quarters or 75%. Alright, coming back to our colab notebook,

bit. Again, we've imported stuff up here. And we've already

this is this is all of our data. This is what we're going to use

again, if we now take a look at our data set, you'll see that our

So now this is all numerical, which is good, because our computer

Okay. And you know, it would probably be a good idea to maybe kind

have anything to do with the class. So here, I'm going to go

in the columns of this data frame. So this just gets me the list.

right? It's called so let's just use that might be less confusing

thing, which is the class. So I'm going to take all these 10

to plot them as a histogram. So and now I'm going to plot them as

take that data frame, and I say, okay, for everything where the

of our gammas, remember, now, for that portion of the data frame,

these, okay, what this part here is saying is, inside the data

the class is equal to one. So that's all all of these would fit

And now let's just look at the label column. So the first label

be this column. So this command here is getting me all the

for this specific label. And that's exactly what I'm going to put

just going to tell you know, matplotlib make the color blue, make

set alpha, why do I keep doing that, alpha equal to 0.7. So that's

And then I'm going to set density equal to true, so that when we

the hadrons here, we'll have a baseline for comparing them. Okay,

just basically normalizes these distributions. So you know, if you

and then 50 of another type, well, if you drew the histograms, it

one of them would be a lot bigger than the other, right. But by

distributing them over how many samples there are. Alright, and

on here and make that the label, the y label. So because it's

And the x label is just going to be the label.

What is going on. And I'm going to include a legend and PLT dot

the plot. So if I run that, just be up to the last item. So we

item. And now we can see that we're plotting all of these. So here

made this gamma. So this should be hadron. Okay, so the gammas in

here we can already see that, you know, maybe if the length is

to be gamma, right. And we can kind of you know, these all look

clearly, if there's more asymmetry, or if you know, this asymmetry

probably hadron. Okay, oh, this one's a good one. So f alpha seems

distributed. Whereas if this is smaller, it looks like there's

Okay, so this is kind of what the data that we're working with, we

Okay, so the next thing that we're going to do here is we are

our validation, and our test data sets. I'm going to set train

this. So NumPy dot split, I'm just splitting up the data frame.

where I'm sampling everything, this will basically shuffle my

exactly I'm splitting my data set, so the first split is going to

to say 0.6 times the length of this data frame. So and then cast

to be the first place where you know, I cut it off, and that'll be

then go to 0.8, this basically means everything between 60% and

set will go towards validation. And then, like everything from 80

my test data. So I can run that. And now, if we go up here, and we

these columns seem to have values in like the 100s, whereas this

all these numbers is way off. And sometimes that will affect our

is way off. And sometimes that will affect our results. So one

is scale these so that they are, you know, so that it's now

standard deviation of that specific column. I'm going to create a

And I'm going to pass in the data frame. And that's what I'll do

going to be, you know, I take the data frame. And let's assume

you know, that the label will always be the last thing in the data

data frame, dot columns all the way up to the last item, and get

well, it's the last column. So I can just do this, I can just

and then get those values. Now, in, so I'm actually going to

the standard scalar from sk learn. So if I come up here, I can go

And I'm going to import standard scalar, I have to run that cell,

And now I'm going to create a scalar and use that skip or so

And with the scalar, what I can do is actually just fit and

is equal to scalar dot fit, fit, transform x. So what that's doing

fit the standard scalar to x, and then transform all those values.

going to be our new x. Alright. And then I'm also going to just

one huge 2d NumPy array. And in order to do that, I'm going to

okay, take an array, and another array and horizontally stack them

the H stands for. So by horizontally stacked them together, just

okay, not on top of each other. So what am I stacking? Well, I

so that it can stack x and y. And now, okay, so NumPy is very

right? So in this specific case, our x is a two dimensional

thing, it's only a vector of values. So in order to now reshape it

NumPy dot reshape. And we can pass in the dimensions of its

one comma one, that just means okay, make this a 2d array, where

what what this dimension value would be, which ends up being the

same as literally doing this. But the negative one is easier

do the hard work. So if I stack that, I'm going to then return the

thing is that if we go into our training data set, okay, again,

And we get the length of the training data set. But where the

so remember that this is the gammas. And then if we print that,

we'll see that, you know, there's around 7000 of the gammas, but

So that might actually become an issue. And instead, what we want

our our training data set. So that means that we want to increase

so that these kind of match better. And surprise, surprise, there

that will help us do that. It's so I'm going to go to from in the

going to import this random oversampler, run that cell, and come

add in this parameter called oversample, and set that to false for

oversample, then what I'm going to do, and by oversample, so if I

then I'm going to create this ROS and set it equal to this random

I'm just going to say, okay, just fit and resample x and y. And

take more of the less class. So take take the less class and keep

the size of our data set of that smaller class so that they now

data set, and I pass in the training data set where oversample is

is train and then x train, y train. Oops, what's going on? These

what I'm doing now is I'm just saying, okay, what is the length of

14,800, whatever. And now let's take a look at how many of these

we can just sum that up. And then we'll also see that if we

many of them are the other type, it's the same value. So now these

rebalanced. Okay, well, okay. So here, I'm just going to make this

then the next one, I'm going to make this the test data set.

switch oversample here to false. Now, the reason why I'm switching

validation and my test sets are for the purpose of you know, if I

how does my sample perform on those? And I don't want to

I don't care about balancing those I'm, I want to know if I have a

unlabeled, can I trust my model, right? So that's why I'm not

what is going on? Oh, it's because we already have this train. So

that data frame again. And now let's run these. Okay. So now we

And we're going to move on to different models now. And I'm going

about each of these models. And then I'm going to show you how we

first model that we're going to learn about is KNN or K nearest

already drawn a plot on the y axis, I have the number of kids that

on the x axis, I have their income in terms of 1000s per year. So,

making 40,000 a year, that's where this would be. And if somebody

would be somebody has zero kids, it'd be somewhere along this

somewhere over here. Okay. And now I have these plus signs and

I'm going to represent here is the plus sign means that they own a

to represent no car. Okay. So your initial thought should be okay,

classification because all of our points all of our samples have

the plus label. And this here is another sample with the minus

width that I'll use. Alright, so we have this entire data set. And

own a car and maybe around half the people don't own a car. Okay,

point, let me use choose a different color, I'll use this nice

point over here? So let's say that somebody makes 40,000 a year

that would be? Well, just logically looking at this plot, you

they wouldn't have a car, right? Because that kind of matches the

them. So that's a whole concept of this nearest neighbors is you

And then you're basically like, okay, I'm going to take the label

So the first thing that we have to do is we have to define a

in, you know, 2d plots like this, our distance function is

And Euclidean distance is basically just this straight line

would be the Euclidean distance, it seems like there's this point,

that point, etc. So the length of this line, this green line that

as Euclidean distance. If we want to get technical with that, this

let me zoom in. The distance is equal to the square root of one

squared plus extend that square root, the same thing for y. So y

other squared. Okay, so we're basically trying to find the length,

between x and y, and then square each of those sum it up and take

going to erase this so it doesn't clutter my drawing. But anyways,

so here in the nearest neighbor algorithm, we see that there is a

telling us, okay, how many neighbors do we use in order to judge

we use a K of maybe, you know, three or five, depends on how big

I would say, maybe a logical number would be three or five. So

to three. Okay, well, of this data point that I drew over here,

Okay, so of this data point that I drew over here, it looks like

this one, this one. And then this one has a length of four. And

bit further than four. So actually, this would be these would be

points are blue. So chances are, my prediction for this point is

probably don't have a car. All right, now what if my point is

somewhere over here, let's say that a couple has four kids, and

well, now my closest points are this one, probably a little bit

right? Okay, still all pluses. Well, this one is more than likely

let me get rid of some of these just so that it looks a little bit

let's go through one more. What about a point that might be right

definitely this is the closest, right? This one's also closest.

the two of these. But if we actually do the mathematics, it seems

this one is right here. And this one is in between these two. So

than this one. And that means that that top one is the one that

what is the majority of the points that are close by? Well, we

here, and we have one minus here, which means that the pluses are

that this label is probably somebody with a car. Okay. So this is

work. It's that simple. And this can be extrapolated to further

know, if you have here, we have two different features, we have

the number of kids. But let's say we have 10 different features,

function so that it includes all 10 of those dimensions, we take

and then we figure out which one is the closest to the point that

that's K nearest neighbors. So now we've learned about K nearest

be able to do that within our code. So here, I'm going to label

And we're actually going to use a package from SK learn. So the

packages and so that we don't have to manually code all these

be really difficult. And chances are the way that we would code

or it'd be really slow, or I don't know a whole bunch of issues.

hand it off to the pros. From here, I can say, okay, from SK

neighbors, I'm going to import K neighbors classifier, because

so I run that. And our KNN model is going to be this K neighbors

a parameter of how many neighbors, you know, we want to use. So

we just use one. So now if I do K, and then model dot fit, I can

weight y train data. Okay. So that effectively fits this model.

why can and I guess yeah, let's do y predictions. And my y

dot predict. So let's use the test set x test. Okay. Alright, so

that we have those. But if I get my truth values for that test

we actually do. So just looking at this, we got five out of six of

actually take a look at something called the classification report

So if I go to from SK learn dot metrics, import classification

say, hey, print out this classification report for me. And let's

y test and the y prediction. We run this and we see we get this

to tell you guys a few things on this chart. Alright, this

pretty good. That's just saying, hey, if we just look at, you

what it's closest to, then we actually get an 82% accuracy, which

versus how many total are there. Now, precision is saying, okay,

for class one, or class zero and class one. What precision is

diagram over here, because I actually kind of like this diagram.

And on the left over here, we have everything that we know is

actually truly positive, that we've labeled positive in our

this is everything that's truly negative. Now in the circle, we

were labeled positive by our model. On the left here, we have

because you know, this side is the positive side and the side is

truly positive. Whereas all these ones out here, well, they should

are labeled as negative. And in here, these are the ones that

actually negative. And out here, these are truly negative. So

the ones we've labeled as positive, how many of them are true

okay, out of all the ones that we know are truly positive, how

so going back to this over here, our precision score, so again,

that we've labeled as the specific class, how many of them are

recall how out of all the ones that are actually this class, how

is 68% and 89%. Alright, so not too shabby, we can clearly see

like this, the class zero is worse than class one. Right? So that

hadrons and for our gammas. This f1 score over here is kind of a

recall score. So we're actually going to mostly look at this one

test data set. So here we have a measure of 72 and 87 or point

which is not too shabby. All right. Well, what if we, you know,

that, okay, so what was it originally with one? We see that our f1

point seven two and then point eight seven. And then our accuracy

three. Alright, so we've kind of increased zero at the cost of one

is 81. So let's actually just make this five. Alright, so you

we have 82% accuracy, which is pretty decent for a model that's

the next type of model that we're going to talk about is something

in order to understand the concepts behind naive Bayes, we have to

conditional probability and Bayes rule. So let's say I have some

this table right here. People who have COVID are over here in this

have COVID are down here in this green row. Now, what about the

tested positive are over here in this column. And people who have

this column. Okay. Yeah, so basically, our categories are people

people who don't have COVID, but test positive, so a false false

and test negative, which is a false negative, and people who don't

which good means you don't have COVID. Okay, so let's make this

in the margins, I've written down the sums of whatever it's

sum of this entire row. And this here might be the sum of this

question that I have is, what is the probability of having COVID

test? And in probability, we write that out like this. So the

line, that vertical line means given that, you know, some

okay, so what is the probability of having COVID given a positive

saying, okay, let's go into this condition. So the condition of

slice of the data, right? That means if you're in this slice of

given that we have a positive test, given in this condition, in

test. So what's the probability that we have COVID? Well, if we're

of people that have COVID is 531. So I'm gonna say that there's

now we divide that by the total number of people that have a

so that's the probability and doing a quick division, we get that

96.4%. So according to this data set, which is data that I made up

not actually real COVID data. But according to this data, the

that you tested positive is 96.4%. Alright, now with that, let's

this section here. Let's ignore this bottom part for now. So Bayes

the probability of some event A happening, given that B happened.

happened. This is our condition, right? Well, what if we don't

if we don't know what the probability of A given B is? Well, Bayes

can actually go and calculate it, as long as you have a

of A and the probability of B. Okay. And this is just a

so here we have Bayes rule. And let's actually see Bayes rule in

So here, let's say that we have some disease statistics, okay. So

And we know that the probability of obtaining a false positive is

false negative is 0.01. And the probability of the disease is 0.1.

the disease given that we got a positive test? Hmm, how do we even

what what do I mean by false positive? What's a different way to

is when you test positive, but you don't actually have the

that you have a positive test given no disease, right? And

it's a probability that you test negative given that you actually

that into a chart, for example, and this might be my positive and

be my diseases, disease and no disease. Well, the probability that

have no disease, okay, that's 0.05 over here. And then the false

testing negative, but I don't actually have the disease. This so

positive, and you don't have the disease, plus a probability that

don't have the disease, that should sum up to one. Okay, because

then you should have some probability that you're testing positive

testing negative. But that probability, in total should be one. So

negative and no disease, this should be the reciprocal, this

should be 0.95 because it's one minus whatever this probability

up here, this should be 0.99 because the probability that we, you

test negative and have the disease plus the probability that we

disease should equal one. So this is our probability chart. And

being point 0.1 just means I have 10% probability of actually of

in the general population, the probability that I have the disease

probability that I have the disease given that I got a positive

can write this out in terms of Bayes rule, right? So if I use this

probability of a positive test given that I have the disease times

divided by the probability of the evidence, which is my positive

Alright, now let's plug in some numbers for that. The probability

that I have the disease is 0.99. And then the probability that I

over here 0.1. Okay. And then the probability that I have a

what is the probability that I have a positive test given that I

and then having having the disease. And then the other case, where

negative test given or sorry, positive test giving no disease

having a disease. Okay, so I can expand that probability of having

these two different cases, I have a disease, and then I don't. And

having positive tests in either one of those cases. So that

plus 0.05. So that's the probability that I'm testing positive,

And the times the probability that I don't actually have the

0.1 probability that the population doesn't have the disease is

multiplication. And I get an answer of 0.6875 or 68.75%. Okay. All

that we can expand Bayes rule and apply it to classification. And

base. So first, a little terminology. So the posterior is this

Hey, what is the probability of some class CK? So by CK, I just

categories, so C for category or class or whatever. So category

dogs, category three, lizards, all the way, we have k categories,

So what is the probability of having of this specific sample x, so

of this one sample. What is the probability of x fitting into

so that that's what this is asking, what is the probability that,

this class, given all this evidence that we see the x's. So the

here, it's saying, Okay, well, given that, you know, assume,

class is class CK, okay, assume that this is a category. Well,

actually seeing x, all these different features from that

prior. So like in the entire population of things, what are the

probability of this class in general? Like if I have, you know, in

percentage? What is the chance that this image is a cat? How many

down here is called the evidence because what we're trying to do

we're creating this new posterior probability built upon the prior

right? And that evidence is a probability of x. So that's some

is a rule for naive Bayes. Whoa, okay, let's digest that a little

let me use a different color. What is this side of the equation

what is the probability that we are in some class K, CK, given

input, this is my second input, this is, you know, my third,

say that our classification is, do we play soccer today or not?

okay, is it how much wind is there? How much rain is there? And

So let's say that it's raining, it's not windy, but it's

So let's use Bayes rule on this. So this here

is equal to the probability of x one, x two, all these joint

times the probability of that class, all over the probability of

Okay. So what is this fancy symbol over here, this means

so how our equal sign means it's equal to this like little

proportional to okay, and this denominator over here, you might

the class like this, that number doesn't depend on the class,

for all of our different classes. So what I'm going to do is make

going to say that this probability x one, x two, all the way to x

to the numerator, I don't care about the denominator, because it's

single class. So this is proportional to x one, x two, x n given

that class. Okay. All right. So in naive Bayes, the point of it

this joint probability, we're just assuming that all of these

are all independent. So in my soccer example, you know, the

or the probability that, you know, it's windy, and it's rainy,

things are independent, we're assuming that they're independent.

actually write this part of the equation here as this. So each

all of them together. So the probability of the first feature,

times the probability of the second feature and given this

all the way up until, you know, the nth feature of given that it's

all of this. All right, which means that this here is now

expanded times this. So I'm going to write that out. So the

And I'm actually going to use this symbol. So what this means is

it means multiply everything to the right of this. So this

but do it for all the i's. So I, what is I, okay, we're going to

the first x i all the way to the nth. So that means for every

these probabilities together. And that's where this up here comes

oops, this should be a line to wrap this up in plain English.

is a probability that you know, we're in some category, given that

features is proportional to the probability of that class in

each of those features, given that we're in this one class that

of it, you know, of us playing soccer today, given that it's

Wednesday, is proportional to Okay, well, what is what is the

anyways, and then times the probability that it's rainy, given

times the probability that it's not windy, given that we're

we playing soccer when it's windy, how you know, and then how many

that's Wednesday, given that we're playing soccer. Okay. So how do

classification. So that's where this comes in our y hat, our

something called the arg max. And then this expression over here,

the arg max. Well, we want. So okay, if I write out this, again,

being in some class CK given all of our evidence. Well, we're

this expression on the right. That's what arc max means. So if K

one through K, so this is how many categories are, we're going to

to solve this expression over here and find the K that makes that

that instead of writing this, we have now a formula, thanks to

approximate that right in something that maybe we can we maybe we

we have the answers for that based on our training set. So this

these and finding whatever class whatever category maximizes this

this is something known as MAP for short, or maximum a

Pick the hypothesis. So pick the K that is the most probable so

of misclassification. Right. So that is MAP. That is naive Bayes.

just like how I imported k nearest neighbor, k neighbors

I can go to SK learn naive Bayes. And I can import Gaussian naive

Right. And here I'm going to say my naive Bayes model is equal.

had above. And I'm just going to say with this model, we are going

All right, just like above. So this, I might actually, so I'm

exactly, just like above, I'm going to make my prediction. So

naive Bayes model. And of course, I'm going to run the

just going to put these in the same cell. But here we have the y

is still our original test data set. So if I run this, you'll see

we get worse scores, right? Our precision, for all of them, they

you know, for our precision, our recall, our f1 score, they look

categories. And our total accuracy, I mean, it's still 72%, which

72%. Okay. Which, you know, is not not that great. Okay, so let's

Here, I've drawn a plot, I have y. So this is my label on one

my features. So let's just say I only have one feature in this

we see that, you know, I have a few of one class type down here.

because it's zero. And then we have our other class type one up

y. Okay. So many of you guys are familiar with regression. So

draw a regression line through this, it might look something like

doesn't seem to be a very good model. Like, why would we use this

Right? It's, it's iffy. Okay. For example, we might say, okay,

everything from here downwards would be one class type in here,

But when you look at this, you're just you, you visually can tell,

make sense. Things are not those dots are not along that line. And

are doing classification, not regression. Okay. Well, first of

this model, if we just use this line, it equals m x. So whatever

which is the y intercept, right? And m is the slope. But when we

is it actually y hat? No, it's not right. So when we're working

what we're actually estimating in our model is a probability,

and one, that is class zero or class one. So here, let's rewrite

Okay, well, m x plus b, that can range, you know, from negative

right? For any for any value of x, it goes from negative infinity

But probability, we know probably one of the rules of probability

between zero and one. So how do we fix this? Well, maybe instead

equal to that, we can set the odds equal to this. So by that, I

divided by one minus the probability. Okay, so now becomes this

take on infinite values. But there's still one issue here. Let me

The one issue here is that m x plus b, that can still be negative,

I have a negative slope, if I have a negative b, if I have some

but that can be that's allowed to be negative. So how do we fix

the log of the odds. Okay. So now I have the log of you know, some

the probability. And now that is on a range of negative infinity

because the range of log should be negative infinity to infinity.

the probability? Well, the first thing I can do is take, you know,

the not the e to the whatever is on both sides. So that gives me

over the one minus the probability is now equal to e to the m x

that out. So the probability is equal to one minus probability e

e to the m x plus b minus P times e to the m x plus b. And now we

one side. So if I do P, so basically, I'm moving this over, so I'm

to the m x plus b is equal to e to the m x plus b and let me

little bigger. So now my probability can be e to the m x plus b

Okay, well, let me just rewrite this really quickly, I want a

Okay, so what I'm going to do is I'm going to multiply this by

and then also the bottom by negative m x plus b, and I'm allowed

this over this is one. So now my probability is equal to one over

one plus e to the negative m x plus b. And now why did I rewrite

It's because this is actually a form of a special function, which

function. And for the sigmoid function, it looks something like

that some x is equal to one over one plus e to the negative x. So

is rewrite this in some sigmoid function, where the x value is

So maybe I'll change this to y just to make that a bit more clear,

the variable name is. But this is our sigmoid function. And

looks like is it goes from zero. So this here is zero to one. And

curved s, which I didn't draw too well. Let me try that again.

something if I can draw this right. Like that. Okay, so it goes in

And you might notice that this form fits our shape up here.

Oops, let's draw it sharper. But if it's our shape up there a lot

Alright, so that is what we call logistic regression, we're

to the sigmoid function. Okay. And when we only have, you know,

one feature x, and that's what we call simple logistic regression.

so that's only x zero, but then if we have x zero, x one, all the

multiple logistic regression, because there are multiple features

when we're building our model, logistic regression. So I'm going

And again, from SK learn this linear model, we can import logistic

And just like how we did above, we can repeat all of this. So

this log model, or LG logistic regression. I'm going to change

So I'm just going to use the default logistic regression. But

you see that you can use different penalties. So right now we're

an L2 penalty. But L2 is our quadratic formula. Okay, so that

you know, outliers, it would really penalize that. For all these

you can toggle these different parameters, and you might get

If I were building a production level logistic regression model,

would want to figure out how to do that. So I'm going to go ahead

I would want to figure out, you know, what are the best parameters

based on my validation data. But for now, we'll just we'll just

So again, I'm going to fit the X train and the Y train. And I'm

so I can just call this again. And instead of LG, NB, I'm going to

precision 65% recall 71, f 168, or 82 total accuracy of 77. Okay,

better than I base, but it's still not as good as K and N.

classification that I wanted to talk about is something called

or SVMs for short. So what exactly is an SVM model, I have two

x one on the axes. And then I've told you if it's you know, class

blue and red labels, my goal is to find some sort of line between

the data. Alright, so this line is our SVM model. So I call it a

line, but in 3d, it would be a plane and then you can also have

proper term is actually I want to find the hyperplane that best

classes. Let's see a few examples. Okay, so first, between these

and C, which one is the best divider of the data, which one has

or the other, or at least if it doesn't, which one divides it the

is has the most defined boundary between the two different groups.

pretty straightforward. It should be a right because a has a clear

know, everything on this side of a is one label, it's negative and

is the other label, it's positive. So what if I have a but then

like this, and my C, maybe like this, sorry, they're kind of the

But now which one is the best? So I would argue that it's still a,

Right? And why is it still a? Because in these other two, look at

to these points. Right? So if I had some new point that I wanted

say I didn't have A or B. So let's say we're just working with C.

that's right here. Or maybe a new point that's right there. Well,

looking at this. I mean, without the boundary, that would probably

right? I mean, it's pretty close to that other positive. So one

is something known as the margin. Okay, so not only do we want to

well, we also care about the boundary in between where the points

are, and the line that we're drawing. So in a line like this, the

might be like here. And I'm trying to draw these perpendicular.

if I switch over to these dotted lines, if I can draw this right.

are what's known as the margins. Okay, so these both here, these

And our goal is to maximize those margins. So not only do we want

two different classes, we want the line that has the largest

on the margin lines, the data. So basically, these are the data

divider. These are what we call support vectors. Hence the name

so the issue with SVM sometimes is that they're not so robust to

if I had one outlier, like this up here, that would totally change

vector to be, even though that might be my only outlier. Okay. So

in mind. As you know, when you're working with SVM is, it might

are outliers in your data set. Okay, so another example of SVMs

data like this, I'm just going to use a one dimensional data set

say we have a data set that looks like this. Well, our, you know,

perpendicular to this line. But it should be somewhere along this

anywhere like this. You might argue, okay, well, there's one here.

draw another one over here, right? And then maybe you can have two

SVMs work. But one thing that we can do is we can create some sort

that one thing I forgot to do was to label where zero was. So

Now, what I'm going to do is I'm going to say, okay, I'm going to

have x, sorry, x zero and x one. So x zero is just going to be my

x one equal to let's say, x squared. So whatever is this squared,

you know, maybe somewhere here, here, just pretend that it's

Right. And now my pluses might be something like

that. And I'm going to run out of space over here. So I'm just

use your imagination. But once I draw it like this, well, it's a

right? Now our SVM could be maybe something like this, this. And

our data set. Now it's separable where one class is this way. And

Okay, so that's known as SVMs. I do highly suggest that, you know,

mentioned, if you're interested in them, do go more in depth

do we how do we find this hyperplane? Right? I'm not going to go

because you're just learning what an SVM is. But it's a good idea

technique behind finding, you know, what exactly are the are the

that we're going to use. So anyways, this transformation that we

as the kernel trick. So when we go from x to some coordinate x,

what we're doing is we are applying a kernel. So that's why it's

So SVMs are actually really powerful. And you'll see that here. So

to import SVC. And SVC is our support vector classifier. So with

we are going to, you know, create SVC model. And we are going to,

could have just copied and pasted this, I should be able to do

again, fit this to X train, I could have just copied and pasted

done that. Okay, taking a bit longer. All right. Let's predict

let's see if I can hover over this. Right. So again, you see a lot

parameters here that you can go back and change if you were

but in this specific case, we'll just use it out of the box again.

you'll note that Wow, the accuracy actually jumps to 87% with the

there's nothing less than, you know, point eight, which is great.

I mean, everything's at 0.9, which is higher than anything that we

So so far, we've gone over four different classification models,

logistic regression, naive Bayes and cannon. And these are just

them. Each of these they have different, you know, they have

go and you can toggle. And you can try to see if that helps later

they perform, they give us around 70 to 80% accuracy. Okay, with

let's see if we can actually beat that using a neural net. Now the

I wanted to talk about is known as a neural net or neural network.

like this. So you have an input layer, this is where all your

all these arrows pointing to some sort of hidden layer. And then

sort of output layer. So what is what is all this mean? Each of

something known as a neuron. Okay, so that's a neuron. In a neural

features that we're inputting into the neural net. So that might

x n. Right. And these are the features that we talked about there,

the pregnancy, the BMI, the age, etc. Now all of these get

are multiplied by some w number that applies to that one specific

feature. So these two get multiplied. And the sum of all of these

so basically, I'm taking w zero times x zero. And then I'm adding

I'm adding you know, x two times w two, etc, all the way to x n

input into the neuron. Now I'm also adding this bias term, which

to shift this by a little bit. So I might add five or I might add

I don't know. But we're going to add this bias term. And the

the sum of this, this, this and this, go into something known as

okay. And then after applying this activation function, we get an

neuron would look like. Now a whole network of them would look

So I kind of gloss over this activation function. What exactly is

looks like if we have all our inputs here. And let's say all of

of addition, right? Then what's going on is we're just adding a

the some sort of weight times these input layer a bunch of times.

and factor that all out, then this entire neural net is just a

layers, which I don't know about you, but that just seems kind of

literally just write that out in a formula, why would we need to

we wouldn't. So the activation function is introduced, right? So

function, this just becomes a linear model. An activation function

this. And as you can tell, these are not linear. And the reason

our entire model doesn't collapse on itself and become a linear

something known as a sigmoid function, it runs between zero and

one all the way to one. And this is ReLU, which anything less than

greater than zero is linear. So with these activation functions,

is no longer just the linear combination of these, it's some sort

that the input into the next neuron is, you know, it doesn't it

become linear, because we've introduced all these nonlinearities.

model, the loss, right? And then we do this thing called training,

back into the model, and make certain adjustments to the model to

Let's talk a little bit about the training, what exactly goes on

Let's go back and take a look at our L2 loss function. This is

looks like it's a quadratic formula, right? Well, up here, the

large. And our goal is to get somewhere down here, where the loss

means that our predicted value is closer to our true value. So

this way. Okay. And thanks to a lot of properties of math,

gradient descent, in order to follow this slope down this way.

different slopes with respect to some value. Okay, so the loss

w zero, versus w one versus w n, they might all be different.

think about it is, to what extent is this value contributing to

figure that out through some calculus, which we're not going to

But if you want to learn more about neural nets, you should

and figure out what exactly back propagation is doing, in order to

how much do we have to backstep by. So the thing is here, you

this curve at all of these different points. And the closer we get

this step becomes. Now stick with me here. So my new value, this

I'm going to take w zero, and I'm going to set some new value for

set for that is the old value of w zero, plus some factor, which

times whatever this arrow is. So that's basically saying, okay,

and just decrease it this way. So I guess increase it in this

this direction. But this alpha here is telling us, okay, don't

just in case we're wrong, take a small step, take a small step in

closer. And for those of you who, you know, do want to look more

the reason why I use a plus here is because this here is the

just the if you were to use the actual gradient, this should be a

Now this alpha is something that we call the learning rate. Okay,

we're taking steps. And that might, you know, tell our that that

how long it takes for our neural net to converge. Or sometimes if

diverge. But with all of these weights, so here I have w zero, w

update to all of them after we calculate the loss, the gradient of

weight. So that's how back propagation works. And that is

calculate the loss, we're calculating gradients, making

all the all the weights to something adjusted slightly. And then

gradient. And then we're saying, Okay, let's take the training set

again, and go through this loop all over again. So for machine

libraries that we use, right, we've already seen SK learn. But

networks, this is kind of what we're trying to program. And it's

do this from scratch, because not only will we probably have a lot

not going to be fast enough, right? Wouldn't it be great if there

full time professionals that are dedicated to solving this

just give us their code that's already running really fast? Well,

And that's why we use TensorFlow. So TensorFlow makes it really

we also have enough control over what exactly we're feeding into

this line here is basically saying, Okay, let's create a

just, you know, what we've seen here, it just goes one layer to

a dense layer means that all of them are interconnected. So here,

nodes, and this one's all these, and then this one gets connected

So we're going to create 16 dense nodes with relu activation

to create another layer of 16 dense nodes with relu activation.

to be just one node. Okay. And that's how easy it is to define

is an open source library that helps you develop and train your ML

for a neural net. So we're using a neural net for classification.

we are going to use TensorFlow, and I don't think I imported that

that down here. So I'm going to import TensorFlow as TF. And

is going to be, I'm going to use this. So essentially, this is

things that I'm about to pass in. So yeah, layer them linear stack

And what that means, nope, not that. So what that means is I can

some sort of layer, and I'm just going to use a dense layer.

Oops, dot dense. And let's say we have 32 units. Okay, I will

set the activation as really. And at first we have to specify the

and comma. Alright. Alright, so that's our first layer. Now our

another dense layer of 32 units all using relu. And that's it. So

just going to be my output layer, it's going to just be one node.

be sigmoid. So if you recall from our logistic regression, what

a sigmoid, it looks something like this, right? So by creating a

we're essentially projecting our predictions to be zero or one,

And that's going to help us, you know, we can just round to zero

Okay. So this is my neural net model. And I'm going to compile

we have to compile it. It's really cool, because I can just

I want, and it'll do it. So here, if I go to optimizers, I'm

And you'll see that, you know, the learning rate is 0.001. So I'm

So 0.001. And my loss is going to be binary cross entropy. And the

include on here, so it already will consider loss, but I'm, I'm

So we can actually see that in a plot later on. Alright, so I'm

And one thing that I'm going to also do is I'm going to define

actually copying and pasting this, I got these from TensorFlow. So

tutorial, they actually have these, this like, defined. And that's

So I'm actually going to move this cell up, run that. So we're

over all the different epochs. epochs means like training cycles.

means like training cycles. And we're going to plot the accuracy

Alright, so we have our model. And now all that's left is, let's

So I'm going to say history. So TensorFlow is great, because it

of the training, which is why we can go and plot it later on. Now

this neural net model. And fit that with x train, y train, I'm

equal to let's say just let's just use 100 for now. And the batch

let's say 32. Alright. And the validation split. So what the

here somewhere. Okay, so yeah, this validation split is just the

to be used as validation data. So essentially, every single epoch,

saying, leave certain if this is point two, then leave 20% out.

model performs on that 20% that we've left out. Okay, so it's

set. But TensorFlow does it on our training data set during the

outside of just our validation data set to see, you know, what's

I'm going to make that 0.2. And we can run this. So if I run that,

to set verbose equal to zero, which means, okay, don't print

for 100 epochs might get kind of annoying. So I'm just going to

and then we'll see what happens. Cool, so it finished training.

because you know, I've already defined these two functions, I can

oops, loss of that history. And I can also plot the accuracy

So this is a little bit ish what we're looking for. We definitely

decreasing loss and an increasing accuracy. So here we do see

accuracy improves from around point seven, seven or something all

point, maybe eight one. And our loss is decreasing. So this is

loss and accuracy is performing worse than the training loss or

our model is training on that data. So it's adapting to that data.

you know, stuff that it hasn't seen yet. So, so that's why. So in

we could change a bunch of the parameters, right? Like I could

a row of 64 nodes, and then 32, and then one. So I can change some

And a lot of machine learning is trying to find, hey, what do we

So what I'm actually going to do is I'm going to rewrite this so

known as a grid search. So we can search through an entire space

we have 64 nodes and 64 nodes, or 16 nodes and 16 nodes, and so

we can, you know, we can change this learning rate, we can change

you know, the batch size, all these things might affect our

I'm also going to add what's known as a dropout layer in here. And

saying, hey, randomly choose with at this rate, certain nodes, and

in a certain iteration. So this helps prevent overfitting. Okay,

define this as a function called train model, we're going to pass

the number of nodes, the dropout, you know, the probability that

learning rate. So I'm actually going to say lr batch size. And we

right? I mentioned that as a parameter. So indent this, so it goes

I'm going to set this equal to number of nodes. And now with the

to set dropout prob. So now you know, the probability of turning

is equal to dropout prob. And I'm going to keep the output layer

but this here is now going to be my learning rate. And I still

accuracy. We are actually going to train our model inside of this

epochs equal epochs, and this is equal to whatever, you know,

y train belong right here. Okay, so those are getting passed in as

end, I'm going to return this model and the history of that model.

is let's just go through all of these. So let's say let's keep

do is I can say, hey, for a number of nodes in, let's say, let's

happens for the different dropout probabilities. And I mean, zero

Also, to see what happens. You know, for the learning rate in

maybe we want to throw on 0.1 in there as well. And then for the

64 as well. Actually, and let's also throw in 128. Actually, let's

so 128 in there. That should be 01. I'm going to record the model

train model here. So we're going to do x train y train, the number

you know, the number of nodes that we've defined here, dropout,

Okay. And then now we have both the model and the history. And

I want to plot the loss for the history. I'm also going to plot

Probably should have done them side by side, that probably would

Okay, so what I'm going to do is split up, split this up. And that

the subplots. So now this is just saying, okay, I want one row and

plots. Okay, so I'm going to plot on my axis one, the loss. I

work. Okay, we don't care about the grid. Yeah, let's let's keep

So now on here, I'm going to plot all the accuracies on the second

I might have to debug this a bit.

We should be able to get rid of that. If we run this, we already

in here. So if I just run it on this, okay, it has no attribute x

it's like set x label or something. Okay, yeah, so it's, it's set

So let's see if that works. All right, cool. Um, and let's

Okay, so we can actually change the figure size that I'm gonna

set that to. Oh, that's not the way I wanted it. Okay, so that

And that's just going to be my plot history function. So now I can

Here, I'm going to plot the history. And what I'm actually going

I'm going to print out all these parameters. So I'm going to print

the F string to print out all of this stuff. So here, I'm going to

Uh, all of this stuff. So here, I'm printing out how many nodes,

And we already know how many you found, so I'm not even going to

So once we plot this, uh, let's actually also figure out what the,

losses on our validation set that we have that we created all the

Alright, so remember, we created three data sets. Let's call our

validation data with the validation data sets loss would be. And I

let's say I want to record whatever model has the least validation

first, I'm going to initialize that to infinity so that you know,

So if I do float infinity, that will set that to infinity. And

track of the parameters. Actually, it doesn't really matter. I'm

the model. And I'm gonna set that to none. So now down here, if

less than the least validation loss, then I am going to simply

Hey, this validation for this least validation loss is now equal

And the least loss model is whatever this model is that just

So we are actually just going to let this run for a while. And

last model after that. So let's just run. All right, and now we

All right, so we've finally finished training. And you'll notice

actually gets to like 0.29. The accuracy is around 88%, which is

okay, why is this accuracy in this? Like, these are both the

is on the validation data set that we've defined at the beginning,

this is actually taking 20% of our tests, our training set every

and saying, Okay, how much of it do I get right now? You know,

train with any of that. So they're slightly different. And

that I probably you know, probably what I should have done is over

the model fit, instead of the validation split, you can define the

And you can pass in the validation data, I don't know if this is

that's probably what I should have done. But instead, you know,

we have here. So you'll see at the end, you know, with the 64

performance 64 nodes with a dropout of 0.2, a learning rate of

And it does seem like yes, the validation, you know, the fake

loss is decreasing, and then the accuracy is increasing, which is

so finally, what I'm going to do is I'm actually just going to

this model, which we've called our least loss model, I'm going to

and I'm going to predict x test on that. And you'll see that it

are really close to zero and some that are really close to one.

output. So if I do this, and what I can do is I can cast them. So

greater than 0.5, set that to one. So if I actually, I think what

Oh, okay, so I have to cast that as type. And so now you'll see

actually going to transform this into a column as well. So here

I didn't mean to do that. Okay, no, I wanted to just reshape it to

Okay. And using that we can actually just rerun the classification

neural net output. And you'll see that okay, the the F ones are

seems like what happened here is the precision on class zero. So

but the recall decreased. But the F one score is still at a good

class, it looked like the precision decreased a bit the recall

That's also been increased. I think I interpreted that properly. I

work and we got a model that performs actually very, very

had earlier. And the whole point of this exercise was to

define your models. But it's also to say, hey, maybe, you know,

powerful, as you can tell. But sometimes, you know, an SVM or some

appropriate. But in this case, I guess it didn't really matter

accuracy score is still pretty good. So yeah, let's now move on to

We just saw a bunch of different classification models. Now let's

the other type of supervised learning. If we look at this plot

data points. And here we have our x value for those data points.

value, which is now our label. And when we look at this plot,

the line of best fit that best models this data. Essentially,

some new value of x that we don't have in our sample, we're trying

prediction for y be for that given x value. So that, you know,

I don't know. But remember, in regression that, you know, given

we're trying to predict some continuous numerical value for y.

In linear regression, we want to take our data and fit a linear

our linear model might look something along the lines of here.

considered as maybe our line of best fit. And this line is modeled

it down here, y equals b zero, plus b one x. Now b zero just means

extend this y down here, this value here is b zero, and then b one

line, defines the slope of this line. Okay. All right. So that's

for linear regression. And how exactly do we come up with that

with this linear regression? You know, we could just eyeball where

not very good at eyeballing certain things like that. I mean, we

better at giving us a precise value for b zero and b one. Well,

something known as a residual. Okay, so residual, you might also

And what that means is, let's take some data point in our data

far off is our prediction from a data point that we already have.

this is 12345678. So this is y eight, let's call it, you'll see

I in order to represent, hey, just one of these points. Okay. So

would be the prediction. Oops, this here would be the prediction

with this hat. Okay, if it has a hat on it, that means hey, this

my prediction for you know, this specific value of x. Okay. Now

here between y eight and y hat eight. So y eight minus y hat

give us this here. And I'm just going to take the absolute value

the line, right, then you would get a negative value, but distance

just going to put a little hat, or we're going to put a little

And that gives us the residual or the error. So let me rewrite

to all the points, I'm going to say the residual can be calculated

So this just means the distance between some given point, and its

prediction on the line. So now, with this residual, this line of

decrease these residuals as much as possible. So now that we have

our line of best fit is trying to decrease the error as much as

data points. And that might mean, you know, minimizing the sum of

here, this is the sum symbol. And if I just stick the residual

it looks something like that, right. And I'm just going to say,

data set, so for all the different points, we're going to sum up

to try to decrease that with my line of best fit. So I'm going to

me the lowest value of this. Okay. Now in other, you know,

we might attach a squared to that. So we're trying to decrease the

And what that does is it just, you know, it adds a higher penalty

you know, points that are further off. So that is linear

this equation, some line of best fit that will help us decrease

with respect to all the data points that we have in our data set,

the best prediction for all of them. This is known as simple

And basically, that means, you know, our equation looks something

multiple linear regression, which just means that hey, if we have

think of our feature vectors, we have multiple values in our x

look something more like this. Actually, I'm just going to say

up with some coefficient for all of the different x values that I

might have noticed that I have some assumptions over here. And you

what in the world do these assumptions mean? So let's go over

So let's go over them. The first one is linearity.

And what that means is, let's say I have a data set. Okay.

Linearity just means, okay, my does my data follow a linear

increases? Or does y decrease at as x increases? Does so if y

rate as x increases, then you're probably looking at something

nonlinear data set? Let's say I had data that might look something

visually judging this, you might say, okay, seems like the line of

curve like this. Right. And in this case, we don't satisfy that

So with linearity, we basically just want our data set to follow

And independence, our second assumption

just means this point over here, it should have no influence on

or this point over here, or this point over here. So in other

all the samples in our data set should be independent. Okay, they

one another, they should not affect one another.

Okay, now, normality and homoscedasticity, those are concepts

I have a plot that looks something like this, and I have a plot

something like this. And my line of best fit is somewhere here,

In order to look at these normality and homoscedasticity

the residual plot. Okay. And what that means is I'm going to keep

of plotting now where they are relative to this y, I'm going to

going to plot y minus y hat like this. Okay. And now you know,

so it might be here, this one down here is negative, it might be

it's literally just a plot of how you know, the values are

fit. So it looks like it might, you know, look something like

residual plot. And what normality means, so our assumptions are

I might have butchered that spelling, I don't really know. But

saying, okay, these residuals should be normally distributed.

it should follow a normal distribution. And now what

of these points should remain constant throughout. So this spread

same as this spread over here. Now, what's an example of where you

not held? Well, let's say that our original plot actually looks

Okay, so now if we looked at the residuals for that, it might look

like that. And now if we look at this spread of the points, it

is not constant, which means that homoscedasticity, this

might not be appropriate to use linear regression. So that's just

we have a bunch of data points, we want to predict some y value

up with this line of best fit that best describes, hey, given some

guess of what y is. So let's move on to how do we evaluate a

measure that I'm going to talk about is known as mean absolute

for short, okay. And mean absolute error is basically saying, all

all the errors. So all these residuals that we talked about, let's

for all of them, and then take the average. And then that can

we. So the mathematical formula for that would be, okay, let's

Alright, so this is the distance. Actually, let me redraw a plot

suppose I have a data set, look like this. And here are all my

say my line looks something like that. So my mean absolute error

values. This was a mistake. So summing up all of these, and then

I have. So what would be all the residuals, it would be y i,

minus y hat i, so the prediction for that on here. And then we're

all of the different i's in our data set. Right, so i, and then we

we have. So actually, I'm going to rewrite this to make it a

whatever the first data point is all the way through the nth data

it by n, which is how many points there are. Okay, so this is our

telling us, okay, in on average, this is the distance between our

actual value in our training set. Okay. And mae is good because it

get this value here, we can literally directly compare it to

So let's say y is we're talking, you know, the prediction of the

dollars. Once we have once we calculate the mae, we can literally

price, the average, how much we're off by is literally this many

mean absolute error. An evaluation technique that's also closely

squared error. And this is MSE for short. Okay. Now, if I take

and move it down here, well, the gist of mean squared error is

of the absolute value, we're going to square. So now the MSE is

okay, let's sum up something, right, so we're going to sum up all

So now I'm going to do y i minus y hat i. But instead of absolute

I'm going to square them all. And then I'm going to divide by n in

basically, now I'm taking all of these different values, and I'm

them to one another. And then I divide by n. And the reason why we

is that it helps us punish large errors in the prediction. And

because of differentiability, right? So a quadratic equation is

if you're familiar with calculus, a quadratic equation is

value function is not totally differentiable everywhere. But if

don't worry about it, you won't really need it right now. And now

error is that once I calculate the mean squared error over here,

want to compare the values. Well, it gets a little bit trickier to

error is in terms of y squared, right? It's this is now squared.

you know, how many dollars off am I I'm talking how many dollars

you know, to humans, it doesn't really make that much sense. Which

something known as the root mean squared error. And I'm just going

because it's very, very similar to mean squared error. Except now

Okay, so this is our messy, and we take the square root of that

term in which you know, we're defining our error is now in terms

So that's a pro of root mean squared error is that now we can say,

to this metric is this many dollar signs off from our predictor.

which is one of the pros of root mean squared error. And now

of determination, or r squared. And this is a formula for r

to one minus RSS over TSS. Okay, so what does that mean?

of the squared residuals. So maybe it should be SSR instead, but

RSS sum of the squared residuals, and this is equal to if I take

and I take y i minus y hat, i, and square that, that is my RSS,

residuals. Now TSS, let me actually use a different color for

So TSS is the total sum of squares.

And what that means is that instead of being with respect to this

take each y value and just subtract the mean of all the y values,

actually, let's use a different color. Let's use green. If this

so RSS is giving me this measure here, right? It's giving me some

our regressor that we predicted. Actually, I'm gonna take this

and actually, I'm going to use red for that. Well, TSS, on the

how far off are these values from the mean. So if we literally

line of best fit, if we just took all the y values and average all

this is the average value for every single x value, I'm just going

instead, then it's asking, okay, how far off are all these points

Okay, and remember that this square means that we're punishing

they look somewhat close in terms of distance, the further a few

the larger our total sum of squares is going to be. Sorry, that

squares is taking all of these values and saying, okay, what is

any regressor, and I literally just calculated the average of all

and for every single x value, I'm just going to predict that

like, that means that maybe y and x aren't associated with each

best thing that I can do for any new x value, just predict, hey,

And this total sum of squares is saying, okay, well, with respect

what is our error? Right? So up here, the sum of the squared

our what what is our error with respect to this line of best fit?

saying what is the error with respect to, you know, just the

of best fit is a better fit, then this total sum of squares, that

that means that this numerator is going to be smaller than this

And if our errors in our line of best fit are much smaller, then

of the RSS over TSS is going to be very small, which means that R

one. And now when R squared is towards one, that means that that's

good predictor. It's one of the signs, not the only one. So over

that there's this adjusted R squared. And what that does, it just

So x1, x2, x3, etc. It adjusts for how many extra terms we add,

you know, add an extra term, the R squared value will increase

y some more. But the value for the adjusted R squared increase if

improves this model fit more than expected, you know, by chance.

R squared is. I'm not, you know, it's out of the scope of this one

And now that's linear regression. Basically, I've covered the

And, you know, how do we use that in order to find the line of

our computer can do all the calculations for us, which is nice.

it's trying to minimize that error, right? And then we've gone

ways of actually evaluating a linear regression model and the pros

So now let's look at an example. So we're still on supervised

talk about regression. So what happens when you don't just want to

What happens if you actually want to predict a certain value? So

learning repository. And here I found this data set about bike

So this data set is predicting rental bike count. And here it's

hour. So what we're going to do, again, you're going to go into

to download this CSV file. And we're going to move over to collab

this FCC bikes and regression. I don't remember what I called the

regression. Now I'm going to import a bunch of the same things

I'm going to also continue to import the oversampler and the

also just going to let you guys know that I have a few more things

library that lets us copy things. Seaborn is a wrapper over a

to plot certain things. And then just letting you know that we're

TensorFlow. Okay, so one more thing that we're also going to be

sklearn linear model library. Actually, let me make my screen a

awesome. Run this and that'll import all the things that we need.

you know, give some credit to where we got this data set. So let

And I will also give credit to this here.

Okay, cool. All right, cool. So this is our data set. And again,

attributes that we have right here. So I'm actually going to go

Feel free to copy and paste this if you want me to read it out

It's byte count, hour, temp, humidity, wind, visibility, dew

snow, and functional, whatever that means. Okay, so I'm going to

by dragging and dropping. All right. Now, one thing that you guys

you might actually have to open up the CSV because there were, at

characters in mine, at least. So you might have to get rid of

but my computer wasn't recognizing it. So I got rid of that. So

and get rid of some of those labels that are incorrect. I'm going

after we've done that, we've imported in here, I'm going to create

all right, so now what I can do is I can read that CSV file and I

So so like data dot CSV. Okay, so now if I call data dot head,

various labels, right? And then I have the data in there. So I'm

going to get rid of some of these columns that, you know, I don't

I'm going to, when I when I type this in, I'm going to drop maybe

holiday, and the various seasons. So I'm just not going to care

one means drop it from the columns. So now you'll see that okay,

I guess you don't really notice it. But if I set the data frames

and I look at, you know, the first five things, then you'll see

It's a lot easier to read. So another thing is, I'm actually going

df functional. And we're going to create this. So remember that

at language, we want it to be in zeros and ones. So here, I will

Well, if this is equal to yes, then that that gets mapped as one.

Great. Cool. So the thing is, right now, these by counts are for

to make this example simpler, I'm just going to index on an hour,

we're only going to use that specific hour. So I'm just going to

going to use an hour. So here, let's say. So this data frame is

the hour, let's say it equals 12. Okay, so it's noon. All right.

equal to 12. And I'm actually going to now drop that column. Our

so we run this cell. Okay, so now we got rid of the hour in here.

the temperature, humidity, wind, visibility, and yada, yada, yada.

is I'm going to actually plot all of these. So for i in all the

whatever its data frame is, and all the columns, because I don't

actually, it's my first thing. So what I'm going to do is say for

columns, everything after the first thing, so that would give me

onwards. So these are all my features, right? I'm going to just

label how that specific data, how that affects the by count. So

the y axis. And I'm going to plot, you know, whatever the specific

And I'm going to title this, whatever the label is. And, you know,

at noon. And the x label as just the label. Okay, now, I guess we

We don't even need the legend. So just show that plot. All right.

not really doesn't really give us any utility. So then snow rain

you know, is fairly linear dew point temperature, visibility, wind

much humidity, kind of maybe like an inverse relationship. But the

looks like there's a relationship between that and the number of

going to do is I'm going to drop some of the ones that don't don't

maybe wind, you know, visibility. Yeah, so I'm going to get rid of

So now data frame, and I'm going to drop wind, visibility, and

axis again is the column. So that's one. So if I look at my data

temperature, the humidity, the dew point temperature, radiation,

what I want to do is I want to split this into my training, my

just as we talked before. Here, we can use the exact same thing

numpy dot split, and sample, you know that the whole sample, and

of the data frame. And we're going to do that. But now set this to

So I don't really care about, you know, the the full grid, the

use an underscore for that variable. But I will get my training x

have a function for getting the x and y's. So here, I'm going to

get x y. And I'm going to pass in the data frame. And I'm actually

of the y label is, and what the x what specific x labels I want to

then I'm just like, like, I'm only going to I'm going to get

not the wildlife. So here, I'm actually going to make first a deep

And that basically means I'm just copying everything over. If, if

so if not x labels, then all I'm going to do is say, all right, x

data frame is. And I'm just going to take all the columns. So C

if C does not equal the y label, right, and I'm going to get the

is the x labels, well, okay, so in order to index only one thing,

one thing in here, then my data frame is, so let me make a case

labels is equal to one, then what I'm going to do is just say that

and add that just that label values, and I actually need to

So I'm going to pass in negative one comma one there. Now,

specific x labels that I want to use, then I'm actually just going

frame of those x labels, dot values. And that should suffice.

extracting x. And in order to get my y, I'm going to do y equals

label. And at the very end, I'm going to say data equals NP dot h

one next to each other. And I'll take x and y, and return that.

And I'm actually going to reshape this to make it 2d as well so

And I will return data x, y. So now I should be able to say, okay,

frame. And the y label, so my y label is byte count. And actually,

going to let's just do like one dimension right now. And earlier,

had seen that maybe, you know, the temperature dimension does

to use that to predict why. So I'm going to label this also that,

temperature. And I am also going to do this again for, oh, this

validation. And this should be a test. Because oh, that's Val.

And this should be test. Alright, so we run this and now we have

data sets for just the temperature. So if I look at x train temp,

Okay, and I'm doing this first to show you simple linear

create a regressor. So I can say the temp regressor here. And then

linear regression model. And just like before, I can simply fix

in order to train train this linear regression model. Alright, and

this regressor is coefficients and the intercept. So if I do that,

for whatever the temperature is, and then the the x intercept,

right. And I can, you know, score, so I can get the the r squared

and y test. All right, so it's an r squared of around point three

zero, which would mean, hey, there's absolutely no association.

good, it depends on the context. But, you know, the higher that

the two variables would be correlated, right? Which here, it's all

maybe some association between the two. But the reason why I want

you, you know, if we plotted this, this is what it would look

and let's take the training. So this is our data. And then let's

also plotted, so something that I can do is say, you know, the x

is when space, and this goes from negative 20 to 40, this piece of

let's take 100 things from there. So I'm going to plot x, and I'm

this, like, regressor, and predict x with that. Okay, and this

the fit. And this color, let's make this red. And let's actually

I can change how thick that value is. Okay. Now at the very end,

all right, let's also create, you know, title, all these things

here, let's just say, this would be the bikes, versus the

would be number of bikes. And the x label would be the

might cause an error. Yeah. So it's expecting a 2d array. So we

Okay, there we go. So I just had to make this an array and then

we see that, all right, this increases. But again, remember those

linear regression, like this, I don't really know if this fits

wanted to show you guys though, that like, all right, this is what

data would look like. Okay. Now, we can do multiple linear

and do that as well. Now, if I take my data set, and instead of

my current data set right now. Alright, so let's just use all of

right. So I'm going to just say for the x labels, let's just take

remove the byte count. So does that work? So if this part should

this should work now. Oops, sorry. Okay, so I have Oh, but this

temperature anymore, we should actually do this, let's say all,

rerun this piece here so that we have our temperature only data

all data set. Okay. And this regressor, I can do the same thing.

And I'm going to make this the linear regression. And I'm going to

train all. Okay. Alright, so let's go ahead and also score this

R squared performs now. So if I test this on the test data set,

square seems to improve it went from point four to point five,

And I can't necessarily plot, you know, every single dimension.

to say, okay, this is this is improved, right? Alright, so one

tensorflow is you can actually do regression, but with the neural

to we already have our our training data for just the temperature

different columns. So I'm not going to bother with splitting up

ahead and start building the model. So in this linear regression

it does help if we normalize it. So that's very easy to do with

normalizer layer. So I'm going to do tensorflow Keras layers, and

And the input shape for that will just be one because let's just

temperature and the access I will make none. Now for this temp

an equal sign there. I'm going to adapt this to X train temp, and

So that should work great. Now with this model, so temp neural net

you know, dot keras, sequential. And I'm going to pass in this

going to say, hey, just give me one single dense layer with one

is saying, all right, well, one single node just means that it's

sort of activation function to it, the output is also linear. So

Keras layers dot dense. And I'm just going to have one unit. And

So with this model, let's compile. And for our optimizer, let's

let's use the atom again, dot atom, and we have to pass in the

and our learning rate, let's do 0.01. And now, the loss, we

loss, I'm going to do mean squared error. Okay, so we run that

And just like before, we can call history. And I'm going to fit

if I call fit, I can just fit it, and I'm going to take the x

but reshape it. Y train for the temperature. And I'm going to set

that it doesn't, you know, display stuff. I'm actually going to

1000. And the validation data should be let's pass in the

as a tuple. And I know I spelled that wrong. So let's just run

And up here, I've copied and pasted the plot loss from our

to MSC. Because now we're talking we're dealing with mean squared

the loss of this history after it's done. So let's just wait for

plot. Okay, so this actually looks pretty good. We see that the

this actually looks pretty good. We see that the values are

I'm going to go back up and take this plot. And we are going to

here, instead of this temperature regressor, I'm going to use the

And if I run that, I can see that, you know, this also gives me a

you'll notice that this this fit is not entirely the same as the

up here. And that's due to the training process of, you know, of

different ways to try and try to find the best linear regressor.

propagation to train a neural net node, whereas in the other one,

Okay, they're probably just trying to actually compute the line of

well, we can repeat the exact same exercise with our with our

but I'm actually going to skip that part. I will leave that as an

so now what would happen if we use a neural net, a real neural net

one single node in order to predict this. So let's start on that

normalizer. So I'm actually going to take the same setup here. But

one dense layer, I'm going to set this equal to 32 units. And for

Relu. And now let's duplicate that. And for the final output, I

want one cell. And this activation is also going to be Relu,

zero bytes. So I'm just going to set that as Relu. I'm just going

Okay. And at the bottom, I'm going to have this neural net model.

net model, I'm going to compile. And I will actually use the same

instead of a learning rate of 0.01, I'll use 0.001. Okay. And I'm

So the history is this neural net model. And I'm going to fit that

temp, and valid validation data, I'm going to set this again equal

Now, for the verbose, I'm going to say equal to zero epochs, let's

size, actually, let's just not do a batch size right now. Let's

here. And again, we can plot the loss of this history after it's

run this. And that's not what we're supposed to get. So what is

we have our temperature normalizer, which I'm wondering now if we

Do that. Okay, so we do see this decline, it's an interesting

So this is our loss, which all right, if decreasing, that's a good

And actually, what's interesting is let's just let's plot this

And you'll see that we actually have this like, curve that looks

what if I got rid of this activation? Let's train this again. And

Alright, so even even when I got rid of that really at the end, it

it's not the best model, if we had maybe one more layer in here,

to play around with. When you're, you know, working with machine

know what the best model is going to be. For example, this also is

it's okay. So my point is, though, that with a neural net, I mean,

there's like no data down here, right? So it's kind of hard for

we probably should have started the prediction somewhere around

with this neural net model, you can see that this is no longer a

get an estimate of the value, right? And we can repeat this exact

do that. Right. And we can repeat this exact same exercise with

if I now pass in all of the data, so this is my all normalizer

and I should just be able to pass in that. So let's move this to

I'm going to pass in my all normalizer. And let's compile it.

Great. So here with the history, when we're trying to fit this

we're going to use our larger data set with all the features. And

And of course, we want to plot the loss.

Okay, so that's what our loss looks like. So an interesting curve,

So before we saw that our R squared score was around point five,

that with a neural net anymore. But one thing that we can measure

error, right? So if I come down here, and I compare the two mean

so I can predict x test all right. So these are my predictions

will linear multiple multiple linear regressor. So these are my

Okay. I'm actually going to do that at the bottom. So let me just

it down here. So now I'm going to calculate the mean squared error

and the neural net. Okay, so this is my linear and this is my

model, and I predict x test all, I get my two, you know, different

the mean squared error, right? So if I want to get the mean

and y real, I can do numpy dot square, and then I would need the y

real. So this this is basically squaring everything. And this

this entire thing and take the mean of that, that should give me

And the y real is y test all, right? So that's my mean squared

And this is my mean squared error for the neural net. So that's

I guess. So my guess is that it's probably coming from this

shape is probably just six. And okay, so that works now. And the

my inputs are only for every vector, it's only a one dimensional

have I should have just had six, comma, which is a tuple of size

a tuple containing one element, which is a six. Okay, so it's

net results seem like they they have a larger mean squared error

One thing that we can look at is, we can actually plot the real

results versus what the predictions are. So if I say, some access,

axes and make these equal, then I can scatter the the y, you know,

values are on the x axis, and then what the prediction are on the

label this as the linear regression predictions. Okay, so then let

x axis, I'm going to say is the true values. The y axis is going

Or actually, let's plot. Let's just make this predictions.

And then at the end, I'm going to plot. Oh, let's set some

Because I think that's like approximately the max number of

So I'm going to set my x limit to this and my y limit to this.

So here, I'm going to pass that in here too. And all right, this

linear regressor. You see that actually, they align quite well, I

probably too much 2500. I mean, looks like maybe like 1800 would

And I'm actually going to label something else, the neural net

Let's add a legend. So you can see that our neural net for the

it's a little bit more spread out. And it seems like we tend to

here in this area. Okay. And for some reason, these are way off as

But yeah, so we've basically used a linear regressor and a neural

sometimes where a neural net is more appropriate and a linear

I think that it just comes with time and trying to figure out, you

like, hey, what works better, like here, a linear, a multiple

better than a neural net. But for example, with the one

never be able to see this curve. Okay. I mean, I'm not saying this

just saying like, hey, you know, sometimes it might be more

linear. So yeah, I will leave regression at that. Okay, so we just

And in supervised learning, we have data, we have some a bunch of

different samples. But each of those samples has some sort of

a category, a class, etc. Right, we were able to use that label in

right, we were able to use that label in order to try to predict

we haven't seen yet. Well, now let's move on to unsupervised

learning, we have a bunch of unlabeled data. And what can we do

anything from this data? So the first algorithm that we're going

clustering. What k means clustering is trying to do is it's trying

So in this example below, I have a bunch of scattered points. And

is x zero and x one on the two axes, which means I'm actually

right of each point, but we don't know what the y label is for

at these scattered points, we can kind of see how there are

right. So depending on what we pick for k, we might have different

right, then we might pick, okay, this seems like it could be one

another cluster. So those might be our two different clusters. If

for example, then okay, this seems like it could be a cluster.

cluster. And maybe this could be a cluster, right. So we could

data set. Now, this k here is predefined, if I can spell that

the model. So that would be you. All right. And let's discuss how

goes through and computes the k clusters. So I'm going to write

Now, the first step that happens is we actually choose well, the

points on this plot to be the centroids. And by centuries, I just

Okay. So three random points, let's say we're doing k equals

random points to be the centroids of the three clusters. If it

random points. Okay. So maybe the three random points I'm choosing

Here, here, and here. All right. So we have three different

the distance for each point to those centroids. So between all the

So basically, I'm saying, all right, this is this distance, this

all of these distances, I'm computing between oops, not those two,

centroids themselves. So I'm computing the distances for all of

Okay. And that comes with also assigning those points to the

What do I mean by that? So let's take this point here, for

this distance, this distance, and this distance. And I'm saying,

is the closest. So I'm actually going to put this into the red

all of these points, it seems slightly closer to red, and this one

right? Now for the blue, I actually wouldn't put any blue ones in

actually, that first one is closer to red. And now it seems like

closer to green. So let's just put all of these into green here,

have, you know, our two, three, technically centroid. So there's

this group here. And then blue is kind of just this group here, it

of the points yet. So the next step, three that we do is we

centroid. So we compute new centroids based on the points that we

And by that, I just mean, okay, well, let's take the average of

new centroid? That's probably going to be somewhere around here,

any points in there. So we won't touch and then the screen one, we

over here, oops, somewhere over here. Right. So now if I erase all

I can go and I can actually redo step two over here, this

Alright, so I'm going to go back and I'm going to iterate through

and I'm going to recompute my three centroids. So let's see, we're

these are definitely all red, right? This one still looks a bit

this part, we actually start getting closer to the blues.

So this one still seems closer to a blue than a green, this one as

would belong to green. Okay, so now our three centroids are three,

would be this, this, and then this, right? Those are our three

and we compute the new sorry, those would be the three clusters.

the three centroids. So I'm going to get rid of this, this and

red be centered, probably closer, you know, to this point here,

up here. And then this green would probably be somewhere. It's

before. But it seems like it'd be pulled down a bit. So probably

All right. And now, again, we go back and we compute the distance

and the centroids. And then we assign them to the closest

it's very clear. Actually, let me just circle that. And this it

it actually seemed like this point is closer to this blue now. So

be maybe this point looks like it'd be blue. So all these look

And the greens would probably be this cluster right here. So we go

bam. This one probably like almost here, bam. And then the green

here ish. Okay. And now we go back and we compute the we compute

So red, still this blue, I would argue is now this cluster here.

Okay, so we go and we recompute the centroids, bam, bam. And, you

to go and assign all the points to clusters again, I would get the

that's when we know that we can stop iterating between steps two

converged on some solution when we've reached some stable point.

these points are really changing out of their clusters anymore, we

and say, Hey, these are our three clusters. Okay. And this

expectation maximization. This part where we're assigning the

this is something this is our expectation step. And this part

centroids, this is our maximization step. Okay, so that's

And we use this in order to compute the centroids, assign all the

according to those centroids. And then we're recomputing all that

some stable point where nothing is changing anymore. Alright, so

of unsupervised learning. And basically, what this is doing is

some pattern in the data. So if I came up with another point, you

I can say, Oh, it looks like that's closer to if this is a, b, c,

cluster B. And so I would probably put it in cluster B. Okay, so

in the data based on just how, how the points are scattered

the second unsupervised learning technique that I'm going to

principal component analysis. And the point of principal component

used as a dimensionality reduction technique. So let me write that

reduction. And what do I mean by dimensionality reduction is if I

x1 x2 x3 x4, etc. Can I just reduce that down to one dimension

about how all these points are spread relative to one another. And

principal component analysis. Let's say I have some points in the

Okay, so these points might be spread, you know, something like

Okay. So for example, if this were something to do with housing

this here might be x zero might be hey, years since built, right,

and x one might be square footage of the house. Alright, so like

right now it's been, you know, 22 years since a house in 2000 was

analysis is just saying, alright, let's say we want to build a

you know, display something about our data, but we don't we don't

How do we display, you know, how do we how do we demonstrate that

this point than this point. And we can do that using principal

take what you know about linear regression and just forget about

you might get confused. PCA is a way of trying to find direction

variance. So this principal component, what that means is

So some direction in this space with the largest variance, okay,

data set without the two different dimensions. Like, let's say we

mentions, and somebody's telling us, hey, you only get one

What dimension do you want to show us? Okay, so let's say we want

what dimension like what do we do, we want to project our data

Alright, so that in this case might be a dimension that looks

this. And you might say, okay, we're not going to talk about

We don't have a y value. So linear regression, this would be why

have a label for that. Instead, what we're doing is we're taking

all of these take that's not very visible. But take this right

And what PCA is doing is saying, okay, map all of these points

So the transformed data set would be here.

This one's on the data sets are on the line. So we just put that

new one dimensional data set. Okay, it's not our prediction or

If somebody came to us said you only get one dimension, you only

each of these 2d points. What number would you give us? What

So this would be our new one dimensional data set. Okay, it's not

What number would you give me? This would be the number that we

this is where our points are the most spread out. Right? If I took

and let me actually duplicate this so I don't have to rewrite

Or so I don't have to erase and then redraw anything. Let me get

And I just got rid of a point there too. So let me draw that

Alright, so if this were my original data point, what if I had

the PCA dimension? Okay, well, I then would have points that let

color. So if I were to draw a right angle to this for every point,

like this. And so just intuitively looking at these two different

we can see that the points are squished a little bit closer

variance that's not the space with the largest variance. The thing

is that this will give us the most discrimination between all of

variance, the further spread out these points will likely be. Now,

dimension that we should project it on a different way to actually

dimension with the largest variance. It's actually it also happens

to be the dimension that decreases that minimizes the residuals.

we take the residual from that the XY residual, so in linear

we were looking only at this residual, the differences between the

y and y hat, it's not that here in principal component analysis,

from our current point in two dimensional space, and then it's

taking that dimension. And we're saying, alright, how much, you

between that projection residual, and we're trying to minimize

actually equates to this largest variance dimension, this

you can either look at it as minimizing, minimize, let me get rid

the projection residuals. So that's the stuff in orange.

Or to maximizing the variance between the points.

Okay. And we're not really going to talk about, you know, the

calculate out the principal components, or like what that

need to understand linear algebra for that, especially

I'm not going to cover in this class. But that's how you would

now, with this two dimensional data set here, sorry, this one

from a 2d data set, and we now boil it down to one dimension.

dimension, and we can do other things with it. Right, we can, like

then we can now show x versus y, rather than x zero and x one in

Now we can just say, oh, this is a principal component. And we're

the y. Or for example, if there were 100 different dimensions, and

them, well, you could go and you could find the top five PCA

more useful to you than 100 different feature vector values.

analysis. Again, we're taking, you know, certain data that's

some sort of estimation, like some guess about its structure from

wanted to take, you know, a 3d thing, so like a sphere, but we

on. Well, what's the best approximation that we can make? Oh, it's

the same thing. It's saying if we have something with all these

show all of them, how do we boil it down to just one dimension?

information from that multiple dimensions? And that is exactly

residuals, or you maximize the variance. And that is PCA. So we'll

Now, finally, let's move on to implementing the unsupervised

Here, again, I'm on the UCI machine learning repository. And I

you know, I have a bunch of kernels that belong to three different

comma, Rosa and Canadian. And the different features that we have

geometric parameters of those wheat kernels. So the area

width, asymmetry, and the length of the kernel groove. Okay, so

which is easy to work with. And what we're going to do is we're

or I guess we're going to try to cluster the different varieties

So let's get started. I have a colab notebook open again. Oh,

go to the data folder, download this. And so I'm going to go to

and let's get started. So the first thing to do is to import our

notebook. So I've done that here. Okay, and then we're going to

so pandas. And then I'm also going to import seedborn because I'm

specific class. Okay. Great. So now our columns that we have in

the perimeter, the compactness, the length, with asymmetry,

to call it groove. And then the class, right, the wheat kernels

I'm going to do that using pandas read CSV. And it's called seeds

that into a data frame. And the names are equal to the columns

do that? Oops, what did I call this seeds data set text? Alright,

data frame right now, you'll notice something funky. Okay. And

stuff under area. And these are all our numbers with some dash t.

haven't actually told pandas what the separator is, which we can

just a tab. So in order to ensure that like all whitespace gets

we can actually this is for like a space. So any spaces are going

separators. So if I run that, now our this, you know, this is a

So now let's actually go and like visualize this data. So what I'm

each of these against one another. So in this case, pretend that

class, right? Pretend that so this class here, I'm just going to

that like, hey, we can predict our classes using unsupervised

in unsupervised learning, we don't actually have access to the

plot these against one another and see what happens. So for some I

the columns minus one because the classes in the columns. And I'm

so take everything from I onwards, you know, so I like the next

So this will give us basically a grid of all the different like

going to be columns I our y label is going to be the columns j. So

And I'm going to use seaborne this time. And I'm going to say

to be our x label. Or y is going to be our y label. And our data

we're passing in. So what's interesting here is that we can say

like if I give this class, it's going to separate the three

hues. So now what we're doing is we're basically comparing the

and the compactness. But we're going to visualize, you know, what

ahead and I might have to show. So great. So basically, we can see

we get these three groups. The area compactness, we get these

kind of look honestly like somewhat similar. Right, so Wow, look

we have the compactness and the asymmetry. And it looks like

it just looks like they're blobs, right? Sure, maybe class three

one and two kind of look like they're on top of each other. Okay.

might look slightly better in terms of clustering. But let's go

clustering examples that we talked about, and try to implement

going to do is just straight up clustering. So what we learned

So from SK learn, I'm going to import k means. Okay. And just for

you know, any x and any y, I'm just going to say, hey, let's use

I mean, perimeter asymmetry could be a good one. So x could be

Okay. And for this, the x values, I'm going to just extract those

Alright, well, let's make a k means algorithm, or let's, you know,

and in this specific case, we know that the number of clusters is

I'm going to fit this against this x that I've just defined right

create this clusters, so one thing, one cool thing is I can

say k mean dot labels. And it'll give give me if I can type

predictions for all the clusters are. And our actual, oops, not

and we get the class, and the values from those, we can actually

like, you know, everything in general, most of the zeros that it's

And in general, the twos are the twos here. And then this third

to three. Now remember, these are separate classes. So the labels,

really matter. We can say a map zero to one map two to two and map

you know, our mapping would do fairly well. But we can actually

that, I'm going to create this cluster cluster data frame. So I'm

And I'm going to pass in a horizontally stacked array with x, so

the clusters that I have here, but I'm going to reshape them. So

Okay. And the columns, the labels for that are going to be x, y,

to go ahead and do that same seaborne scatter plot. Again, where x

the hue is again the class. And the data is now this cluster data

this here is my k means like, I guess classes.

So k means kind of looks like this. If I come down here and I

this is my original classes with respect to this specific x and y.

like it doesn't do too poorly. Yeah, there's I mean, the colors

For the most part, it gets information of the clusters, right. And

higher dimensions. So with the higher dimensions, if we make x

except for the last one, which is our class, we can do the exact

We can do the exact same thing. So here, and we can

predict this. But now, our columns are equal to our data frame

And then with this class, actually, so we can literally just say

And we can fit all of this. And now, if I want to plot the k means

Alright, so this was my that's my clustered and my original. So

get these on the same page. So yeah, I mean, pretty similar to

actually really cool is even something like, you know, if we

where they were like on top of each other? Okay, so compactness

Right. So if I come down here, and I say compactness and

this is what my scatterplot. So this is what you know, my k means

dimensions for compactness and asymmetry, if we just look at those

right? And we know that the original looks something like this.

alike? No. Okay, so now if I come back down here, and I rerun this

but actually, this clusters, I need to get the labels of the k

Okay, so if I rerun this with higher dimensions

well, if we zoom out, and we take a look at these two, sure, the

there are the three groups are there, right? This does a much

what group is what. So, for example, we could relabel the one in

And then we could make sorry, okay, this is kind of confusing. But

were projected onto this darker pink here, and then this dark one

and this light one was this dark one, then you kind of see like

right? Like even these two up here are the same class as all the

the same in the same color. So you don't want to compare the two

you want to compare which points are in what colors in each of the

application. So this is how k means functions, it's basically

All right, where are my clusters given these pieces of data? And

talked about is PCA. So PCA, we're reducing the dimension, but

you know, seven dimensions. I don't know if there are seven, I

mapping multiple dimensions into a lower dimension number. Right.

So from SK learn decomposition, I can import PCA and that will be

So if I do PCA component, so this is how many dimensions you want

And you know, for this exercise, let's do two. Okay, so now I'm

And my transformed x is going to be PCA dot fit transform, and the

And the same x that I had up here. Okay, so all the other all the

perimeter, compactness, length, width, asymmetry, groove. Okay. So

transformed it. So let's look at what the shape of x used to be.

I had 210 samples, each seven, seven features long, basically. And

is 210 samples, but only of length two, which means that I only

I'm plotting. And we can actually even take a look at, you know,

Okay, so now we see each each one is a two dimensional point

each sample is now a two dimensional point in our new in our new

So what's cool is I can actually scatter these

zero and transformed x. So I actually have to

take the columns here. And if I show that

basically, we've just taken this like seven dimensional thing, and

single or I guess to a two dimensional representation. So that's a

And actually, let's go ahead and do the same clustering exercise

the k means this PCA data frame, I can let's construct data frame

frame is going to be H stack. I'm going to take this transformed x

So actually, instead of clusters, I'm going to use k means dot

So it's 2d. So we can do the H stack. And for the columns, I'm

and the class. All right. So now if I take this, I can also do the

But instead of the k means labels, I want from the data frame the

And I'm just going to take the values from that. And so now I have

with PCA and then a data frame for the truth with also the PCA.

to how I plotted these up here. So let me actually take these

Instead of the cluster data frame, I want the this is the k means

to be class, but now x and y are going to be the two PCA

dimensions. And you can see that the data frame is going to be the

So these are my two PCA dimensions. And you can see that, you

out. And then here, I'm going to go to my truth classes. Again,

of k means this should be truth PCA data frame. So you can see

along these two dimensions, we actually are doing fairly well in

seem like this is slightly more separable than the other like

up here. So that's a good sign. And up here, you can see that hey,

another. I mean, for the most part, our algorithm or unsupervised

to give us is able to spit out, you know, what the proper labels

specific labels to the different types of kernels. But for

kernel kernels and same here. And then these might all be the

be the Canadian kernels. So it does struggle a little bit with,

But for the most part, our algorithm is able to find the three

fairly good job at predicting them without without any information

algorithm any labels. So that's a gist of unsupervised learning. I

this course. I hope you know, a lot of these examples made sense.

that I have done, and you know, you're somebody with more

in the comments and we can all as a community learn from this