All language subtitles for Machine Learning for Everybody – Full Course

kylie ying has worked at many

interesting places such as mit cern and

free code camp she's a physicist

engineer and basically a genius and now

she's gonna teach you about machine

learning in a way that is accessible to

absolute beginners

what's up you guys so welcome to machine

learning for everyone

if you are someone who is interested in

machine learning and you think you are

considered as everyone then this video

is for you

in this video we'll talk about

supervised and unsupervised learning

models we'll go through maybe a little

bit of the logic or math behind them and

then we'll also see how we can program

it on google collab

if there are certain things that i have

done and you know you're somebody with

more experience than me please feel free

to correct me in the comments and we can

all as a community learn from this

together

so with that let's just dive right in

without wasting any time let's just dive

straight into the code and i will be

teaching you guys concepts as we go

so this here is the

uci machine learning repository and

basically they just have a ton of data

sets that we can access and i found this

really cool one called the magic gamma

telescope data set

so in this data set if you don't want to

read all this information to summarize

what i what i think is going on is

there's this gamma telescope and we have

all these high energy particles hitting

the telescope now there's a camera

there's a detector that actually records

certain patterns of you know how this

light hits the camera

and we can use properties of those

patterns in order to predict what type

of particle caused that radiation so

whether it was a gamma particle

or some other head like hadron

down here these are all of the

attributes of those patterns that we

collect in the camera so you can see

that there's you know some length width

size

asymmetry etc

now we're going to use all these

properties to help us discriminate the

patterns and whether or not they came

from a gamma particle or a hadron

so in order to do this we're going to

come up here go to the data folder

and you're going to click this magic04

data and we're going to download that

now over here i have a colab notebook

open so you go to

collab.research.google.com you start a

new notebook

and i'm just going to call this

um the magic data set

so actually i'm going to call this for

code camp magic example

okay

so with that i'm going to first start

with some imports so i will import you

know i always import numpy

i always import pandas

and i always import matplotlib

and then we'll import other things as we

go

so yeah

we run that in order to run the cell you

can either click this play button here

or you can on my computer it's just

shift enter and that that will run the

cell

and here i'm just going to or i'm just

going to you know

let you guys know okay this is where i

found the data set

um so i've copied and pasted this

actually but this is just where i found

the data set

and in order to import that downloaded

file that we we got from the computer

we're going to go over here to this

folder thing

and i am literally just going to drag

and drop that file into here

okay

so in order to take a look at you know

what does this file consist of do we

have the labels do we not i mean we

could open it on our computer but we can

also just do

pandas

read csv

and we can pass in the name of this file

and let's see what it returns so it

doesn't seem like we have

the label so let's go back to

here

i'm just going to make the

columns

uh the column labels all of these

attribute names over here so i'm just

going to take these values and make that

the column names

all right how do i do that

so basically

i will come back here and i will create

a list called calls

and i will type in all of those things

with

f size

f

conch

and we also have an f conc 1

we have f

symmetry

fm3 long

fm3 trans

f

alpha

what else do we have f dist and class

and class okay great

now in order to label those as these

columns down here in our data frame so

basically this command here just reads

some csv file that you pass in csv has

come about comma separated values

um and turned that into a pandas data

frame object

so now if i pass in a names

here then it basically assigns these

labels to the columns of this data set

so

i'm going to set this data frame equal

to df and then if we call so head is

just like give me the first five things

now you'll see that we have labels for

all of these

okay all right great so one thing that

you might notice is that over here the

class labels we have g and h so if i

actually go down here and i do data

frame class dot unique

you'll see that i have either g's or h's

and these stand for gammas or hadrons

um and our computer is not so good at

understanding letters right our

computer's really good at understanding

numbers so what we're going to do is

we're going to convert this to 0 for g

and 1 for h so

here i'm going to set this equal

to

this

whether or not that equals g

and then i'm just going to say as type

int so what this should do is convert

this entire column

if it equals g then this is true so i

guess that would be one and then if it's

h it would be false so that would be

zero but i'm just converting g and h to

one and zeros it doesn't really matter

like

if g is one and h is zero or vice versa

let me just take a step back right now

and talk about this data set so here i

have some data frame and i have

all of these different values for each

entry

now this is a you know each of these is

one sample it's one

example it's one item in our data set

it's one data point all these things are

kind of the same thing when i mention oh

this is one example or this is one

sample or whatever

now each of these samples they have

you know one quality for each or one

value for each of these

labels up here and then it has the class

now what we're going to do in this

specific example is try to predict for

future you know samples whether the

class is g for gamma or h for hadron

and that is something known as

classification

now all of these up here these are known

as our features and features are just

things that we're going to pass into our

model in order to help us predict the

label which in this case is the class

column

so

for you know sample 0 i have

10 different features so i have 10

different values

that i can pass into some model

and i can spit out you know the class

the label and i know the true label here

is g so this is this is actually

supervised learning

all right

so before i move on let me just give you

a quick little crash course on what i

just said this is machine learning for

everyone

well the first question is what is

machine learning

well machine learning is a sub-domain of

computer science that focuses on certain

algorithms which might help a computer

learn from data

without a programmer being there telling

the computer exactly what to do that's

what we call explicit programming

so you might have heard of ai and ml and

data science what is the difference

between all of these

so ai is artificial intelligence and

that's an area of computer science where

the goal is to enable computers and

machines to perform human-like tasks and

simulate human behavior

now machine learning is a subset of ai

that tries to solve

one specific problem and make

predictions using certain data

and data science is a field that

attempts to find patterns and draw

insights from data and that might mean

we're using machine learning

so all of these fields kind of overlap

and all of them might use machine

learning

so there are a few types of machine

learning

the first one is supervised learning

and in supervised learning we're using

labeled inputs so this means whatever

input we get we have a corresponding

output label in order to train models

and to learn outputs of different new

inputs that we might feed our model

so for example i might have these

pictures okay to a computer all these

pictures are are pixels they're pixels

with a certain color

now in supervised learning all of these

inputs have a label associated with them

this is the output that we might want

the computer to be able to predict so

for example over here this picture is a

cat

this picture is a dog and this picture

is a lizard

now there's also unsupervised learning

and in unsupervised learning we use

unlabeled data

to learn about patterns in the data

so here are here are my input

data points again they're just images

they're just pixels

well okay let's say i have a bunch of

these different pictures

and what i can do is i can feed all

these to my computer and i might not you

know my computer's not gonna be able to

say oh this is a cat dog and

lizard in terms of you know the output

but it might be able to cluster all

these pictures it might say hey all of

these have something in common

all of these have something in common

and then these down here have something

in common that's finding some sort of

structure in our unlabeled data

and finally we have reinforcement

learning and reinforcement learning well

they usually there's an agent that is

learning in some sort of interactive

environment based on rewards and

penalties

so let's think of a dog

we can train our dog

but there's not necessarily you know any

wrong or right output at any given

moment right

well let's pretend that dog is a

computer

essentially what we're doing is we're

giving rewards to our computer and

telling our computer hey this is

probably something good that you want to

keep doing well computer agent yeah

terminology

but in this class today we'll be

focusing on supervised learning and

unsupervised learning and learning

different models for each of those

all right so let's talk about supervised

learning first

so this is kind of what a machine

learning model looks like you have a

bunch of inputs

that are going into some model and then

the model is spitting out an output

which is our prediction

so all these inputs this is what we call

the feature vector

now there are different types of

features that we can have we might have

qualitative features

and qualitative means categorical data

there's either a finite number of

categories or groups

so

one example of a qualitative feature

might be gender

and in this case there's only two here

it's for the sake of the example i know

this might be a little bit

outdated here we have a girl and a boy

there are two genders there are two

different categories

that's a piece of qualitative data

another example might be okay we have

you know a bunch of different

nationalities maybe a nationality or a

nation or a location

that might also be an example of

categorical data

now in both of these there's no inherent

order it's not like you know we can

rate us one and

uh france two japan three etcetera right

there's

not really any inherent order built into

either of these categorical data sets

that's why we call this nominal data

now for nominal data the way that we

want to feed it into our computer is

using something called one hot encoding

so let's say that you know i have a data

set some of the items in our data some

of the inputs

might be from the us some might be from

india then canada than france

now how do we get our computer to

recognize that we have to do something

called one hot encoding and basically

one hot encoding is saying okay well if

it matches some category

make that a one and if it doesn't just

make that a zero

so for example if your input were from

the us

you would you might have one zero zero

zero

india you know zero one zero zero canada

okay well the item representing canada

is one and then france the item

representing france is one

and then you can see that the rest are

zeros that's one hot encoding

now there are also a different type of

qualitative feature

so here on the left there are different

age groups there's babies toddlers

teenagers young adults

adults

and so on right and on the right hand

side we might have different ratings so

maybe bad

not so good

mediocre

good and then like great

now these are known as ordinal pieces of

data because they have some sort of

inherent order

right like

being a toddler is a lot closer to being

a baby than being an elderly person

right or good is closer to great than it

is to really bad

so these have some sort of inherent

ordering system

and so for these types of data sets we

can actually just mark them

from you know one to five or we can just

say hey for each of these let's give it

a number

and this makes sense because

like

for example the thing that i just said

how good is closer to great

then good is close to not good at all

well four is closer to five then four is

close to one so this actually kind of

makes sense and it'll make sense for the

computer as well

all right there are also quantitative

pieces of data and quantitative

pieces of data are numerical valued

pieces of data so this could be discrete

which means you know they might be

integers or it could be continuous which

means

all real numbers

so for example the length of something

is a quantitative piece of data

it's a quantitative feature

the temperature of something is a

quantitative feature and then maybe how

many easter eggs i collected in my

basket this easter egg hunt

that is an example of discrete

quantitative feature

okay so these are continuous and this

over here is discrete

so those are the things that go into our

feature vector those are our features

that we're feeding this model because

our computers are really really good

at understanding math right at

understanding numbers they're not so

good at understanding things that humans

might be able to understand

well

what are the types of predictions that

our model can output

so

in supervised learning there are some

different tasks there's one

classification

and basically classification just saying

okay predict discrete classes

and that might mean you know this is a

hot dog

this is a pizza and this is ice cream

okay so there are three distinct classes

and any other pictures of hot dogs pizza

or ice cream

i can put under these labels

hot dog pizza ice cream

this is something known as multi-class

classification

but there's also binary classification

and binary classification you might have

hot dog

or not hot dog so there's only two

categories that you're working with

something that is something and

something that isn't

binary classification

okay so yeah other examples

so if something has positive or negative

sentiment that's binary classification

maybe you're predicting your pictures if

they're cats or dogs that's binary

classification

maybe you know you are writing an email

filter and you're trying to figure out

if an email is spam or not spam

so that's also binary classification

now for multi-class classification you

might have you know cat dog lizard

dolphin shark

rabbit etc

um we might have different types of

fruits so like orange apple pear etc and

then maybe different plant species but

multi-class classification just means

more than two okay and binary means

we're predicting between two things

there's also something called regression

when we talk about supervised learning

and this just means we're trying to

predict

continuous values so instead of just

trying to predict different categories

we're trying to come up with a number

that you know is on some sort of scale

so some examples

so some examples might be the price of

ethereum tomorrow

or it might be okay what is going to be

the temperature

or it might be what is the price of this

house

right so these things don't really fit

into discrete classes

we're trying to predict a number that's

as close to the true value as possible

using

different features of our data set

so that's exactly what our model looks

like in supervised learning

now let's talk about the model itself

how do we make this model learn

or how can we tell whether or not it's

even learning

so before we talk about the models

let's talk about how can we actually

like evaluate these models or how can we

tell whether something is a good model

or a bad model

so

let's take a look at this data set so

this data set has this is from

a diabetes a pima indian diabetes data

set

and here we have different number of

pregnancies different glucose levels

blood pressure skin thickness

insulin bmi age and then the outcome

whether or not they have diabetes one

for they do zero for they don't

so here

um all of these are

quantitative

features right because they're all on

some scale

so each row is a different sample in the

data so it's a different example it's

one person's data and each row

represents one person in this data set

now this column

each column represents a different

feature so this one here is some measure

of blood pressure levels

and this one over here as we mentioned

is the output label so this one is

whether or not they have diabetes

and as i mentioned this is what we would

call a feature vector because these are

all of our features

in one sample

and this is what's known as the target

or the output for that feature vector

that's what we're trying to predict

and all of these together is our

features matrix

x

and over here this is our labels or

targets vector y

so i've condensed this to a chocolate

bar to kind of

talk about some of the other concepts in

machine learning

so over here we have our x our features

matrix and over here this is our label y

so

each row of this will be fed into our

model right

and our model will make some sort of

prediction

and what we do is we compare that

prediction to the actual

value of y that we have in our labeled

data set because that's the whole point

of supervised learning is we can compare

what our model is outputting to oh what

is the truth actually and then we can go

back and we can adjust some things so

the next iteration we get closer

to

what the true value is

so that

whole process here the tinkering the

okay what's the difference where did we

go wrong

that's what's known as training the

model

all right so take this whole

you know chunk right here do we want to

really put

our entire chocolate bar into the model

to train our model

not really right because if we did that

then how do we know that our model can

do well on new data that we haven't seen

like if i were to create a model

to

predict whether or not someone has

diabetes

let's say that i just train all my data

and i see that on my training data it

does well i go to some hospital i'm like

here's my model

i think you can use this to predict if

somebody has diabetes

do we think that would be effective or

not

probably not right because

we haven't

assessed

how well our model can generalize

okay it might do well after you know our

model has seen this data over and over

and over again but what about new data

can our model handle new data

well

how do we how do we get our model to

assess that

so we actually break up our whole data

set that we have

into three different types of data sets

we call it the training data set the

validation data set and the testing data

set

and you know you might have sixty

percent here twenty percent and twenty

percent or eighty ten and ten um it

really depends on how many statistics

you have i think either of those would

be acceptable

so what we do is then we feed the

training data set into our model

we come up with you know this might be a

vector of predictions corresponding with

each

sample that we put into our model

we figure out okay what's the difference

between our prediction and the true

values this is something known as loss

loss is you know what's the difference

here

in some numerical quantity of course

and then we make adjustments and that's

what we call training

okay

so then

once you know we've made a bunch of

adjustments

we can put our validation set through

this model

and the validation set is kind of used

as a reality check

during or after training to ensure that

the model can handle unseen data still

so every single time after we train one

iteration we might

stick the validation set in and see hey

what's the loss there

and then after our training is over we

can assess the validation set and ask

hey what's the loss there

but one key difference here is that we

don't have that training step this

loss never gets fed back into the model

right that feedback loop is not closed

all right so let's talk about loss

really quickly

so here i have four different types of

models i have some sort of data that's

being fed into the model and then some

output

okay so

this output here is pretty far from you

know this

truth that we want

and so

this loss is going to be high

in model b

again this is pretty far from what we

want so this loss is also going to be

high let's give it 1.5

now this one here it's pretty close i

mean maybe not almost but pretty close

to this one

so that might have a loss of 0.5

and then this one here is

maybe further than this but still better

than these two so that loss might be 0.9

okay so which of these model performs

the best

well

model c has the smallest loss so it's

probably model c

okay now let's take model c after you

know we've come up with these all these

models and we've seen okay model c is

probably the best model

we take model c and we run our test set

through this model

and this test set is used as a final

check to see how generalizable

that chosen model is so if i you know

finished training my diabetes data set

then i could run it through some chunk

of the data and i can say oh like this

is how it perform on data that it's

never seen before at any point during

the training process okay

and that loss that's the final reported

performance of

my test set or

this would be the final reported

performance of my model

okay

so let's talk about this thing called

loss because i think i kind of just

glossed over it right

so loss is the difference between your

prediction and the actual

like label

so this would give a slightly higher

loss than this

and this would even give a higher loss

because it's even more off

in computer science we like formulas

right we like formulaic ways

of describing things

so here are some examples of loss

functions and how we can actually come

up with numbers

this here is known as l1 loss

and basically l1 loss just takes the

absolute value

of

whatever your

you know real

value is whatever the real output label

is

subtracts the predicted value

and takes the absolute value of that

okay

so

the absolute value is

a function that looks something like

this

so the further off you are the greater

your losses

right in either direction so

if your real value is off from your

predicted value by 10 then your loss for

that point would be 10 and then this sum

here just means hey we're taking all the

points in our data set and we're trying

to figure out the sum of how far

everything is

now we also have something called l2

loss so

this loss function is quadratic which

means that if it's close the penalty is

very minimal and if it's off by a lot

then the penalty is much much higher

okay

and this instead of the absolute value

we just square

the um

the difference between the two

now there's also something called binary

cross entropy loss

it looks something like this and this is

for uh binary classification this this

might be the loss that we use

so this loss

you know i'm not going to really go

through it too much but you just need to

know that loss decreases as the

performance gets better

so there are some other measures of

accuracy as well so for example accuracy

what is accuracy

so let's say that these are pictures

that i'm feeding my model okay

and these predictions might be

apple orange orange apple

okay but the actual is apple orange

apple apple

so

three of them were correct and one of

them was incorrect so the accuracy of

this model is three quarters or 75

percent

all right coming back to our

collab notebook i'm going to close this

a little bit

again we've imported stuff up here um

and we've already created our data frame

right here and this is this is all of

our data this is what we're going to use

to train our models

so down here

again if we now take a look at our data

set

you'll see that our classes are now

zeros and ones so now this is all

numerical which is good because our

computer can now understand that

okay

and you know it would probably be a good

idea to maybe

kind of plot hey do these things have

anything to do

with the class

so

here

i'm going to go through all the labels

so for label in

the columns of this data frame so this

just gets me the list actually we have

the list right it's called so let's just

use that it might be less confusing

of everything up till the last thing

which is the class so i'm going to take

all these 10 different features

and i'm going to plot them

as a histogram

um

so

and now i'm gonna plot them as a

histogram so basically if i take that

data frame and i say okay for everything

where

the class

is equal to one so these are all of our

gammas remember

now

for that portion of the data frame if i

look at this label so now these

okay what this part here is saying

is

inside the data frame get me everything

where the class is equal to one so

that's all all of these would fit into

that category right

and now let's just look at the label

column so

the first label would be f length which

would be this column

so this command here is getting me

all the different values that belong to

class 1 for this specific label

and that's exactly what i'm going to put

into the histogram

and now i'm just going to tell you know

matplotlib make the color blue make

oops

label this as you know gamma

um

set alpha why do i keep doing that alpha

equal to 0.7 so that's just like the

transparency and then i'm going to set

density equal to true so that when we

compare it to

the hadrons here

we'll have a baseline for comparing them

okay so the density being true just

basically normalizes these distributions

so you know if you have

200 and

of one type and then 50 of another type

well if you drew the histograms it would

be hard to compare because one of them

would be a lot bigger than the other

right but by normalizing them we kind of

are distributing them over how many

samples there are

all right and then i'm just going to put

a title on here make that the label

uh the y label

so because it's density the y label is

probability

and the x label

is just going to be the label

um

what is going on

and i'm going to

include a legend and

plt.show just means okay display the

plot

so if i run that

just be

up to the last item so we want a list

right not just the last item

and now we can see that we're plotting

all of these so here we have the length

oh and i made this gamma so

uh this should be had on

okay

so the gamma's in blue the hadrons are

in red so here we can already see that

you know maybe if the length is smaller

it's probably more likely to be gamma

right

and we can kind of you know these all

look somewhat similar

but here okay clearly if there's more

asymmetry

or if you know this

asymmetry measure is larger

um then it's probably a hadron

okay oh this one's a good one so

f alpha

seems like

hadrons are pretty evenly distributed

whereas if this is smaller it

looks like there's more gammas in that

area

okay so this is kind of what the data

that we're working with we can kind of

see what's going on

okay so the next thing that we're going

to do here

is we are going to create our

train

our validation

and our test data sets

i'm going to set train valid and test

to be equal to

this

so numpy.split i'm just splitting up the

data frame

and if i do this sample where i'm

sampling everything this will basically

shuffle my data

um now if i

i want to pass in where exactly i'm

splitting my data set so

the first split is going to be maybe at

60 percent

so i'm just going to say 0.6 times the

length of this data frame

so and then casa 10 integer

that's going to be the first place where

you know i cut it off and that will be

my training data

now if i then go to 0.8

this basically means everything between

60 and 80 of the length of the data set

will go towards validation

and then

like everything from 80 to 100 is going

to be my test data

so i can run that

and

now

if we go up here and we inspect this

data we'll see that these columns seem

to have values in like the 100s whereas

this one

is 0.03

right so the scale of all these numbers

is way off

and sometimes that will affect our

results

so one thing that we would want to do is

scale these

so that they are you know

so that it's now relative to maybe

the mean and the standard deviation of

that specific column

i'm going to create a function called

scale data set

and i'm going to pass in the data frame

um

and that's what i'll do for now okay so

the x

values are going to be you know i take

the data frame

and

let's assume that

the columns

um

are going to be you know that the label

will always be the last thing in the

data frame so what i can do is say

dot dataframe.com all the way up to the

last

item

and get those values

now for my y

well it's the last column so i can just

do this i can just index into that last

column

and then get those values

now

in

so

i'm actually going to import something

known as uh the standard scalar from

sklearn so if i come up here i can go to

sklearn.preprocessing

and i'm going to import um standard

scalar

i have to run that cell

i'm going to come back down here

and now i'm going to create a scalar and

use that skill or so standard

scalar

and with the scalar what i can do is

actually just fit and transform x

so here i can say x is equal to

scalar dot fit

fit transform

x so what that's doing is saying okay

take x and fit the standard scalar to x

and then transform all those values and

what would it be and that's going to be

our new x

all right

and then i'm also going to just create

you know the whole data as one uh huge

2d numpy array

and in order to do that i'm going to

call

hsac so h stack is saying okay take an

array and another array and horizontally

stack them together that's what the h

stands for

so by horizontally stacked them together

just like put them side by side okay not

on top of each other

so what am i stacking well i have to

pass in something

so that it can stack

x and y

and now

okay so numpy is very particular about

dimensions right so in this specific

case rx is a two-dimensional object but

y is only a one-dimensional thing it's

only a vector

of values

so in order to now reshape it into a 2d

item we have to call numpy.reshape

and we can pass in the dimensions of its

reshape

so

if i pass a negative 1 comma 1 that just

means okay make this a 2d array where

the negative 1 just means infer what

what this dimension value would be which

ends up being the length of y this would

be the same as literally doing this

but the negative one is easier because

we're making the computer do the hard

work

so if i stack that i'm going to then

return the data x and

y

okay

so one more thing is that if we go into

our training data set

okay again this is our training data set

and we get the length of the training

data set

um but where the training data sets

class

is one

so remember that this is the gammas

and then if we print

that and we do the same thing but zero

we'll see that

you know there's around

seven thousand of the gammas but only

around four thousand of the hadrons so

that might actually become an issue

and instead what we want to do

is we want to over sample our

our training data set

so that means that we want to increase

the number of

these values

so that these kind of match better

and surprise surprise there is something

that we can import that will help us do

that it's so

i'm going to go to from mblearn.org

and i'm going to import this random over

sampler

run that cell

and come back down here

so i will actually add in this parameter

called over sample

and set that to false

for default

um

and if i do want to over sample then

what i'm going to do

and by over sample so if i do want to

over sample

then i'm going to create this ros and

set it equal to this random over sampler

and then for x and y i'm just going to

say okay just fit

and resample

x and y and what that's doing is saying

okay take more of the less

class

so take take the less class and keep

sampling from there

to increase the size of our data set of

that smaller class so that they now

match

so if i do this and i scale

data set

and i pass in the training data set

where over sample is true

so this let's say this is train and then

x train

y train

oops

what's going on

oh these should be columns

so basically

what i'm doing now is i'm just saying

okay what is the length of y train

okay now it's 14 800 whatever and now

let's take a look at

um how many of these are type one

so actually we can just

sum that up

and then we'll also see that if we

instead switch the label and ask how

many of them are the other type

it's the same value so now these have

been

evenly you know rebalanced

okay well

okay

so here i'm just going to make

this the validation data set

and then the next one

uh i'm going to make this the test data

set

all right and we're actually going to

switch over sample here to false

now the reason why i'm switching that to

false is because

my validation and my test sets are

for the purpose of you know if i have

data that i haven't seen yet how does my

sample perform on those

and i don't want to over sample for that

right now like i i don't care about

balancing those i'm i want to know if i

have a random set of data that's

unlabeled can i trust my model right

so that's why i'm not oversampling

i run that and

again

what is going on oh it's because we

already have this train

so i have to go come up here and split

that data frame again

and now let's run these

okay

so now we have our data properly

formatted and we're going to move on to

different models now and i'm going to

tell you guys a little bit about each of

these models and then i'm going to show

you how we can do that in our code

so the first model that we're going to

learn about is k n or k nearest

neighbors

okay

so here i've already drawn a plot on the

y-axis i have the number of kids

that a family might have and then on

the x-axis i have their income in terms

of thousands per year so

you know if uh if someone's making 40

000 a year that's where this would be

and if somebody making 320 that's where

that would be somebody has zero kids

it'd be somewhere along this axis

somebody has five it'd be somewhere over

here okay

and now i have

these plus signs and these minus signs

on here

so what i'm going to represent here is

the plus sign

means that they

own a car

and the minus sign

is going to represent no car

okay

so

your initial thought should be okay i

think this is binary classification

because all of our

points all of our samples

have labels so this is a

sample with

the plus label

and this here is another sample

with

the minus label

this is an abbreviation for width that

i'll use

all right

so we have this entire data set and

maybe around half the people own a car

and maybe

around half the people don't own a car

okay well what if i had some new point

let me use choose a different color i'll

use this nice green

well what if i have a new point over

here so let's say that somebody makes 40

000 a year and has two

kids what do we think that would be

well just logically looking at this plot

you might think okay it seems like

they wouldn't have a car right because

that kind of matches the pattern of

everybody else around them

so that's a whole concept of this

nearest neighbors is you look at okay

what's around you

and then you're basically like okay i'm

going to take the label of the majority

that's around me

so the first thing we have to do is we

have to define a distance function

and a lot of times in you know 2d plots

like this our distance function is

something known as

euclidean distance

and euclidean distance

is basically just

this straight line distance like this

okay

so this would be the euclidean distance

it seems like

there's this point

there's this point

there's that point

etcetera so the length of this line this

green line that i just drew that is

what's known as euclidean distance

if we want to get technical with that

this exact formula

is the distance here let me zoom in

the distance is equal to the square root

of one point

x

minus the other points x

squared

plus extend that square root

the same thing for y so y one of one

minus y two of the other

squared

okay so we're basically

trying to find the length the distance

is the difference between x and y

and then

square each of those sum it up and take

the square root

okay so i'm going to erase this so it

doesn't clutter

my drawing

but anyways now going back to this plot

so here

in the nearest neighbor algorithm we see

that there is a

k

right

and this k is basically telling us okay

how many neighbors do we use in order to

judge what the label is

so usually we use a k of maybe you know

three or five

depends on how big our data set is but

here i would say

maybe

a logical number would be three or five

so let's say that we take k

to be equal to three

okay well of this data point that i drew

over here

let me use green to highlight this okay

so of this data point that i drew over

here looks like the three closest points

are definitely this one

this one

and then this one has a length of four

and this one

seems like it'd be a little bit further

than four so actually this

would be our these would be our three

points

well all those points are blue

so chances are

my prediction for this point is going to

be blue

it's going to be

probably don't have a car

all right now what if my point is

somewhere

what if my point is somewhere over here

let's say that

a couple has

four kids and they make 240 000 a year

all right well now my closest points are

this one

probably a little bit over that one

and then this one right okay still all

pluses

well

this one is more than likely to be

a plus

right now let me get rid of some of

these just so that it looks a little bit

more clear

all right let's go through

one more

what about a point that might be

right

here

okay let's see well definitely this is

the closest

right this one's also closest

and then

[Music]

it's really close between the two of

these

but if we actually do the mathematics it

seems like

if we zoom in

this one is right here and this one is

in between these two so

this one here is actually shorter than

this one

and that means that that top one is the

one that we're going to take

now what is the majority of the points

that are close by

well we have one plus here we have one

plus here and we have one minus here

which means that the pluses

are the majority

and that means that this

label

is probably somebody with a car

okay

so this is how k nearest neighbors would

work

it's that simple

and this can be extrapolated to further

dimensions to higher dimensions you know

if you have

here we have two different features we

have the income

and then we have the number of kids

but

let's say we have 10 different features

we can expand our distance function so

that it includes all 10 of those

dimensions we take the square root of

everything and then we figure out which

one is the closest to the point that we

desire

to classify okay

so that's k-nearest neighbors

so now we've learned about uh k-nearest

neighbors

let's see how we would be able to do

that within our code

so here i'm going to label the section k

nearest neighbors

and we're actually going to use a

package from sklearn so the reason why

we you know use these packages so that

we don't have to manually code all these

things ourself because it would be

really difficult and chances are the way

that we would code it either would have

bugs or it'd be really slow or i don't

know a whole bunch of issues

so what we're going to do is hand it off

to the pros

from here i can say

okay from sklearn which is this package

dot neighbors

i'm going to import k

neighbors classifier because we're

classifying

okay

so i run that

and our k n model is going to be this k

neighbors classifier and we can pass in

a parameter of how many neighbors you

know we want to use so first let's see

what happens if we just use one

so now if i do k

and then model dot fit

i can pass in my x training set and my

weight y train

data okay so that effectively fits this

model

and let's get all the predictions so why

k n or i guess yeah let's do y

predictions

and my y predictions are going to be k n

model dot predict

um

so let's use the test set x test

okay

all right so if i

call by predict you'll see that we have

those but if i get my truth values for

that test set you'll see that this is

what we actually do so just looking at

this we got five out of six of them okay

great so let's actually take a look at

something

called the classification report that's

offered by sklearn so if i go to from

sklearn.metrics import

classification report

what i can actually do is say hey print

out this classification

report for me

and let's check you know

i'm giving you the y test and the y

prediction

we run this and we see we get this whole

entire chart so i'm going to tell you

guys a few things on this chart

all right this accuracy is 82 which is

actually pretty good that's just saying

hey if we just look at you know what

each of these new points what it's

closest to

then we actually get an 82 percent

accuracy

which means

how many do we get right versus how many

total are there

now precision is saying okay you might

see that we have it for class one or

class zero and class

what precision is saying was let's go to

this wikipedia diagram over here because

i actually kind of like this diagram

so here

this is our entire data set and on the

left over here we have everything that

we know is positive so everything that

is actually truly positive that we've

labeled positive in our original data

set

and over here this is everything that's

truly negative

now in the circle we have

things that are paused that were labeled

positive by our model

on the left here we have things that are

truly positive because you know this

side is

the positive side and the side is the

negative side so these are truly

positive

whereas all these ones out here well

they should have been positive but they

are labeled as negative

and in here these are the ones that

we've labeled positive but they're

actually negative and out here these are

truly negative

so precision is saying okay

out of all the ones we've labeled as

positive how many of them are true

positives

and recall is saying okay out of all the

ones that we know are truly positive how

many did we actually get right

okay

so going back to this over here our

precision score so

again precision out of all the ones that

we've labeled as

the specific class how many of them are

actually that class it's 77 and 84

now recall how out of all the ones that

are actually this class how many of

those that we get this is 68

and eighty nine percent

all right

so not too shabby we can clearly see

that this recall and precision for

like this the class zero is worse than

class one

right so that means for hadrons it's

worked for hadrons and for our gammas

this f1 score over here is kind of a

combination of the precision and recall

score so

we're actually going to mostly look at

this one because we have an unbalanced

test data set

so here we have a measure of 72 and 87

or 0.72 and 0.87 which is not too shabby

all right

well what if we you know made this three

so we actually see that

okay so what was it originally with one

we see that our f1 score

you know is now it was 0.72 and then

0.87

and then our accuracy was 82 so if i

change that to three

all right so

we've kind of increased zero at the cost

of one and then our overall average

accuracy is 81. so let's actually just

make this five

all right so you know again very similar

numbers we have 82 percent accuracy

which is pretty decent for a model

that's

relatively simple

the next type of model that we're going

to talk about is something known as

naive bayes

now in order to understand the concepts

behind naive bayes we have to be able to

understand conditional probability and

bayes rule

so

let's say i have some sort of data set

that's shown in this table right here

people who have covid are over here in

this red row and people who do not have

covet are down here in this green row

now what about the cova test well people

who have tested positive

are over here in this column

and people who have tested negative are

over here in this

column

okay

yeah so basically our categories are

people who have coven and test positive

people who don't have covid but test

positive so a false false positive

people who have covet and test negative

which is a false negative

and people who don't have covid and test

negative which

good means you don't have coven

okay so let's make this slightly more

legible

and here in the margins i've written

down the sums

of

whatever it's referring to so this here

is the sum of this entire row

and this here might be the sum of this

column over here

okay

so the first question that i have is

what is the probability of having covid

given that you have a positive test

and in probability we write that out

like this so the probability

of covid

given so this line that vertical line

means

given that you know some condition

so given a positive test

okay

so

what is the probability of having covid

given a positive test

so what this is asking is saying okay

let's go into

this condition so the condition of

having a positive test that is

this slice of the data right that means

if you're in this slice of data you have

a positive test so given that we have a

positive test given in this condition in

this circumstance we have a positive

test

so what's the probability that we have

covid

well

if we're just using this data the number

of people that have covid is 531.

so i'm going to say that there's 531

people that have copied

and then now we divide that by the total

number of people that have a positive

test which is 551

okay so that's the probability and

doing a quick

division

we get that this is equal

to around

96.4 percent

so according to this data set which is

data that i made up off the top of my

head so it's not actually real cova data

but according to this data

uh the probability of encoded given that

you tested positive

is

96.4

all right now with that let's talk about

bayes rule

which is this section here

let's ignore this bottom part for now

so

base rule is asking okay what is the

probability of some event a happening

given that b happen so this

we already know has happened this is our

condition right

well what if we don't have data for that

right like what if we don't know what

the probability of a given b is

well bayes rule is saying okay well you

can actually go and calculate it as long

as you have the probability of b given a

the probability of a and the probability

of b

okay and this is just a mathematical

formula for that

all right so here we have bayes rule

and let's actually see bayes rule in

action let's use it on an example

so here let's say that we have some

um disease statistics okay

so knock over different disease

and we know that the probability of

obtaining a false positive is 0.05

probability of obtaining a false

negative is 0.01 and the probability of

the disease is 0.1

okay what is the probability of the

disease given that

we got a positive test

how do we even go about solving this

so what what do i mean by false positive

what's a different way to rewrite that

a false positive is when you test

positive but you don't actually have the

disease so this here is a probability

that you'd have a positive test given

no disease

right

and similarly for the false negative

it's the probability that you test

negative given that you actually have

the disease so if i put that into a

chart

for example

and this might be my positive and

negative tests and this might be

my diseases

disease and no disease

well the probability that i test

positive but actually have no disease

okay that's 0.05 over here

and then the false negative is up here

for 0.01 so i'm testing negative but i

don't actually have the disease

this so the probability that you test

positive and you don't have the disease

plus the probability that you test

negative given that you don't have the

disease

that should sum up to one

okay because if you don't have the

disease then you should have some

probability that you're testing positive

and some probably that you're testing

negative but that probability

in total should be one

so that means that

the probability of negative and no

disease this should be the reciprocal

this would be the opposite so it should

be

0.95 because it's 1 minus whatever this

probability is

and then similarly

oops

up here this should be 0.99 because

the probability that we

you know test negative and have the

disease plus the probability that we

test positive and have the disease

should equal one

so this is our probability chart and now

this probability of disease

being point zero point one just means i

have a ten percent probability of

actually of having the disease right

like

in the general population the

probability that i have the disease is

0.1

okay so what is the probability that i

have the disease given that i got a

positive i got a positive

test well remember that we can write

this out in terms of bayes rule right so

if i use this rule up here

this is the probability of a positive

test given that i have the disease

times the probability

of the disease

divided by the probability of the

evidence which is my positive test

all right now let's plug in some numbers

for that the probability of having a

positive test given that i have disease

is

0.99

and then the probability that i have the

disease

is this value over here

0.1

okay

and then

the probability that i have a positive

test at all

should be okay what is the probability

that i have a positive test given that i

actually have the disease

and then having having the disease

and then the other case

where the probability of me having a

negative test given or sorry positive

test giving no disease

times the probability of not actually

having a disease

okay so i can expand that

probability of having positive tests out

into these two different cases

i have a disease and then i don't

and then

what's the probability of having

positive tests in either one of those

cases

so that expression would become

0.99

times 0.1

plus

0.05 so that's the probability that i'm

testing positive but don't have the

disease

and the times the probability that i

don't actually have the disease so

that's 1 minus 0.1 the probability that

the population doesn't have the disease

is 90

so

0.9 and let's do that multiplication

and i get an answer

of 0.6875

or

68.75 percent

okay

all right so we can actually expand that

we can expand bayes rule

and apply it to classification

and this is what we call naive bayes

so first a little terminology so the

posterior

is this over here because it's asking

hey what is the probability

of some class ck so by ck i just mean

you know the different categories so c

for category or class or whatever so

category one might be cats category two

dogs category three lizards

all the way we have k categories k is

just some number

okay

so

what is the probability of having

of

this specific sample x so this is our

feature factor

of this one sample

what is the probability of x fitting

into category one two three four

whatever right so that that's what this

is asking what is the probability that

you know it's actually from this class

given all this evidence that we see the

x's

so

the likelihood

is this quantity over here it's saying

okay

well given that you know assume assume

we are

assumed that this class is class ck okay

assume that this is a category

well what is the likelihood of actually

seeing x

all these different features from that

category

and then this here is the prior so like

in the entire population of things

what what is what are the probabilities

what is the probability of this class in

general like if i have

you know in my entire data set what is

the percentage what is the chance that

this image is a cat how many cats do i

have

right

and then this down here is called the

evidence because

what we're trying to do

is we're changing our prior we're

creating this new posterior probability

built upon the prior by using some sort

of evidence right and that evidence is

the probability of x

so that's some vocab

and

this here

is a rule for naive bayes

whoa okay let's digest

that

a little bit okay

so what is let me use a different color

what is this

side of the equation asking

it's asking

what is the probability that we are in

sum class k ck

given that you know this is my first

input this is my second input this is

you know my third fourth this is my nth

input

so let's say that our classification is

do we play soccer today or not

okay and let's say our x's are okay is

it how much wind is there how much

uh rain is there and what day of the

week is it so let's say that it's

raining it's not windy but it's

wednesday do we play soccer do we not

so let's use bayes rule on this so this

here

is equal to the probability of x1

x2 all these joint probabilities given

class k

times the probability of that class

all over the probability of this

evidence

okay

so what is this fancy symbol over here

this means proportional

to

so how our equal sign means it's equal

to this like little squiggly sign means

that this is proportional to

okay

and this denominator over here

you might notice that it has no impact

on the class like this that number

doesn't depend on the class right so

this is going to be constant for all of

our different classes

so what i'm going to do is make things

simpler so i'm just going to say that

this probability

x1 x2 all the way to xn

this is going to be proportional to the

numerator i don't care about the

denominator because it's the same for

every single class

so this is proportional to x1 x2

xn

given

class k times the probability of that

class

okay

all right

so in naive bayes the

point of it being naive

is that we're actually

this joint probability we're just

assuming that all of these different

things

are all independent so in my soccer

example

you know

the probability that we're playing

soccer

um

or the probability that you know it's

windy and it's rainy and and it's

wednesday all these things are

independent we're assuming that they're

independent

so that means that i can actually write

this part of the equation here

as

this so each term in here

i can just multiply

all them together so the probability of

the first

feature given that it's class k

times the probability

of the second feature given that's

probably like class k all the way up

until

you know the nth

feature

of

given that it's class k

so this expands to all of this

all right which means that this here is

now proportional

to the thing that we just expanded

times this

so i'm going to write

that out so the probability

of that class

and i'm actually going to use this

symbol so what this means is it's a huge

multiplication it means multiply

everything

to the right of this so this probability

x

given

some class k

but do it for all the i so i

what is i okay we're going to go from

the first

x i all the way to the end so that means

for every single i we're just

multiplying

these probabilities together

and that's where

this up here comes from

so

to wrap this up oops this should be a

line to wrap this up in plain english

basically what this is saying is the

probability that you know we're in some

category given that we have all these

different features

is proportional to the probability of

that class in general

times the probability of each of those

features

given that we're in this one class that

we're testing

so the probability

of it you know of us playing soccer

today given that it's rainy not windy

and

um and it's wednesday

is proportional to okay well what is

what is the probability that we play

soccer anyways

and then times the probability that it's

rainy given that we're playing soccer

times the probability that it's not

windy given that we're playing soccer so

how many times are we playing soccer

when it's windy you know

and then how many times or what's the

probability that's wednesday given that

we're playing soccer

okay

so how do we use this in order to make a

classification

so that's where this comes in our y hat

our predicted y

is going to be equal to something called

the arg max

and then this expression over here

because we want to take the arg max well

we want

so okay if i

write out this

again this means the probability of

being in some class c k given all of our

evidence

well we're going to take the k

that maximizes

this expression on the right

that's what arc max means so if k is in

zero oops

one through

k so this is how many categories there

are we're going to go through each k

and we're going to solve

this expression over here and find the k

that makes that the largest

okay

and

remember that instead of writing this we

have now a formula thanks to bayes rule

for helping us

approximate that right and something

that maybe we can

we maybe we have like the evidence for

that we have the answers for that based

on our training set

so this principle of going through each

of these and finding whatever class

whatever category maximizes

this expression on the right this is

something known as m

ap for short

or maximum

a

posteriori

uh pick the hypothesis so pick the k

that is the most probable so that we

minimize the probability of

misclassification

all right

so that is m ap that is

naive bayes

back to the notebook so

just like how i imported k-nearest name

uh k

neighbors classifier up here for naive

bayes i can go to sklearn.naive

bayes

and i can import gaussian

naive bayes

all right

and here i'm going to say my naive bayes

model is equal this is very similar to

what we had above

and i'm just going to

say with this model

we are going to fit

x train

and y train

all right just like above

so this i might actually have to

so i'm going to set that

and uh

exactly just like above i'm going to

make my prediction

so here i'm going to instead use my

naive bayes model

and of course i'm going to run

the classification report again

so i'm actually just going to put these

in the same cell

but here we have the y the new y

prediction and then y test is still our

original test data set

so if i run this

you'll see that

okay what's going on here we get worse

scores right our precision

for all of them they look slightly worse

and

our um

you know for

our precision our recall our f1 score

they look slightly worse for all the

different categories and our total

accuracy i mean it's still 72 which is

not too shabby

but it's still 72

okay

um

which you know is not not that great

okay so let's move on to logistic

regression

here i've drawn a plot i have y so this

is my label

on one axis and then this is maybe one

of my features so let's just say i only

have one feature in this case

deck zero right

well

we see that

you know i have a few of one class type

down here

and we know it's one class type because

it's zero and then we have our other

class type one up here

okay

so many of you guys are familiar with

regression so let's start there

if i were to draw a regression line

through this

it might look something

like

like this

right

well this doesn't seem to be a very good

model like why would we use this

specific line to predict why

right

it's it's

iffy

okay

um for example we might say

okay well it seems like you know

everything from here downwards would be

one class type in here upwards would be

another class type

but when you look at this you're just

you you

visually can tell okay like

that line doesn't make sense things are

not those dots are not along that line

and the reason is because we are doing

classification not regression

okay

well first of all let's start here we

know that this

model

if we just use this line it equals m x

so whatever this let's just say it's x

plus b which is the y intercept right

and m is the slope

but when we use a linear regression is

it actually y hat

no it's not right so when we're working

with linear regression what we're

actually estimating in our model

is a probability what's the probability

between 0 and 1 that is class 0 or class

1.

so here let's rewrite this

as p equals mx

plus b

okay well mx plus b that can range

you know from negative infinity to

infinity right for any for any value of

x it goes from negative infinity to

infinity

but probability we know probably one of

the rules of probability is that

probability has to stay between zero and

one

so how do we fix this

well maybe instead of

just setting the probability equal to

that we can set the odds

equal to this so by that i mean okay

let's do probability divided by 1 minus

the probability okay so now it becomes

this ratio

now this ratio is allowed to take on

infinite values

but there's still one issue here

let me move this over a bit

the one issue here is that

mx plus b that can still be negative

right like if you know i have a negative

slope if i have a negative b if i have

some negative x's in there i don't know

but that can be that's allowed to be

negative

so how do we fix that

we do that by actually taking the log

of the odds

okay

so now i have the log of you know some

probability divided by 1 minus the

probability and now that is on a range

of negative infinity to infinity which

is good because the range of log should

be negative infinity to infinity

now how do i solve for p the probability

well the first thing i can do is take

you know

i can remove the log by taking the not

the um

e to the whatever is on both sides

so that gives me the probability

over the one minus the probability

is now equal to e to the m x plus b

okay

so let's multiply that out so the

probability is equal to

one minus probability e to the m x plus

b

so p is equal to e to the m x plus b

minus p

times e to the m x plus b

and now we have we can move like terms

to one side so if i do p

uh so basically i'm moving this over so

i'm adding p so now p one plus

e to the m x plus b

is equal to

e to the m x plus b and let me change

this parenthesis make it a little bigger

so now my probability can be e to the mx

plus b

divided by 1 plus e to the mx

plus b

okay

well

let me just rewrite this really quickly

i want a numerator of one on top

okay so what i'm going to do is i'm

going to multiply

this by negative mx plus b

and then also the bottom by negative mx

plus b and i'm allowed to do that

because

this over this is 1.

so now my probability is equal to 1 over

1 plus

e to the negative mx plus b and now why

do i rewrite it like that it's because

this is actually a form of a special

function

which is called the sigmoid

function

and for the sigmoid function

it looks something like this so s of x

sigmoid you know at with that sum x

is equal to 1 over

1 plus e to the negative

x

so essentially what i just did up here

is rewrite this

in some sigmoid function

where the x value is actually mx plus b

so maybe i'll change this to y just to

make that a bit more clear it doesn't

matter what the variable name is

but this is our sigmoid function

and

visually what our sigmoid function looks

like

is it goes from zero so this here is

zero

to one

and it looks something

like this curved s which i didn't draw

too well let me try that again

this is hard to draw

something if i can draw this right

like

that

okay so it goes in between zero and one

and you might notice that this form

fits our

shape up here

oops

let's

draw it sharper but it fits our shape up

there a lot better right

all right so that is

what we call

logistic regression we're basically

trying to fit our data

to the sigmoid function

okay

and when we only have you know one

um

data point so if we only have one

feature x then that's what we call

simple

logistic regression

but then if we have you know so that's

only x0 but then if we have x0 x1

all the way to xn

we call this multiple

logistic regression because there are

multiple features that we're considering

when we're building our model

logistic regression

so i'm going to put that here and again

from sklearn

this linear model we can import logistic

regression

right

and just like how we did above we can

repeat all of this so here instead of nb

i'm going to call this

the log model or lg logistic regression

i'm going to change this to logistic

regression

so i'm just going to use the default

logistic regression

but actually if you look here you see

that you can use different penalties so

right now we're using

um an l2 penalty

but

l2 is our quadratic formula okay so that

means that for you know outliers it

would really

penalize that

for all these other things you know you

can toggle

these

different parameters and you might get

slightly different results if i were

building a production level logistic

regression model that i would want to go

and i would want to figure out you know

what are the best

parameters to pass into here based on my

validation data

but for now we'll just we'll just use

this out of the box

so again i'm going to fix

the x train and the y train

and i'm just going to predict again so i

can just call this again

um and instead of l g uh nb i'm going to

use lg so here this is decent precision

65

recall 71

f1 68

or 82

uh total accuracy of 77 okay so it

performs slightly better than night bass

but it's still not as good as knn

all right so the last model for

classification that i wanted to talk

about is something called

support vector machines

or svms for short

so

what exactly

is an svm

model i have two different features x0

and x1 on the axis

and then i've told you if it's you know

class 0 or class 1 based on the blue and

the red

labels

my goal is to find some sort of

line

between these two labels that best

divides the data

all right so this line is our svm model

so i call it a line here because in 2d

it's a line but in 3d it would be a

plane and then you can also have more

and more dimensions so the proper term

is actually i want to find the

hyperplane

that best differentiates these two

classes

let's

see a few examples okay so

first

between these

three

lines

let's say a

b

and c

which one is the best divider of the

data which one has you know all the data

on one side or the other or at least if

it doesn't which one divides it the most

right like which one is has the most

defined boundary between the two

different groups

so

this this question should be pretty

straightforward

it should be a

right because a has that clear distinct

line

between where you know everything on

this side of a is one label it's

negative and everything on this side of

a is the other label it's positive

so what if i have a but then what if i

had drawn my b

like this

and my c

maybe like this

sorry they're kind of the labels are

kind of close together

but now which one is the best

so i would argue that it's still a

right and why is it still a

because in these other

two

look at how close this is to that to

these points

right so if i had some new point

that i wanted to estimate okay say i

didn't have a or b

so let's say we're just working with c

let's say i have some new point that's

right here

or maybe a new point that's right there

well it seems like just logically

looking at this i mean without the

boundary that

would probably go under the positives

right

i mean it's pretty close to that other

positive

so one thing that we care about in svms

is something known as the margin

okay

so

not only do we want to separate the two

classes really well we also care about

the boundary in between

where the points in those classes in our

data set are

and the line that we're drawing so

in a line like this

the closest values to this line

might be

like here

i'm trying to draw these perpendicular

right and so this effectively

if i switch over to these dotted lines

if i can draw this right

so these effectively are what's known as

the margins

okay

so these both here

these are our margins

in our svms

and our goal is to maximize those

margins so not only do we want the line

that best separates the two different

classes we want the line that has the

largest margin

and the data points that lie on

the margin lines the data so basically

these are the data points that's helping

us define our divider

these are what we call support

vectors

hence the name support vector machines

okay so the issue with svm sometimes is

that they're not so robust

to outliers

right so for example if i had

one outlier

like this up here

that would totally change where i want

my

support vector to be

even though that might be my only

outlier okay

so that's just something to keep in mind

as you know you're working with svms is

it might not be the best model if there

are outliers in your data set

okay so another example of svms

might be let's say that we have data

like this i'm just going to use a one

dimensional data set for this example

let's say we have a data set that looks

like this

well our you know separator should be

perpendicular to this line

but it should be somewhere along this

line so it could be

anywhere like this

you might argue okay well there's one

here and then you could also just draw

another one over here

right and then maybe you can have two

svms but that's not really how svms work

but one thing that we can do is we can

create some sort of projection

so i realized here that one thing

i forgot to do

was to label where zero was so let's

just say zero

is here

now what i'm going to do is i'm going to

say okay

i'm going to have x and then i'm going

to have x

sorry x0 and x1 so x0 is just going to

be my original x

but i'm going to make x1 equal

to let's say

x

squared so whatever is this squared

right so now

my negatives would be you know maybe

somewhere here

here

just pretend that it's somewhere up here

right and now my pluses might be

something like

that

and i'm going to run out of space over

here so i'm just going to draw these

together use your imagination

but once i draw it like this

well it's a lot easier to apply a

boundary right now our svm could be

maybe something like this

this

and now you see that we've divided our

data set now it's separable where one

class is this way

and the other class

is that way

okay so that's known as svms

um i do highly suggest that you know any

of these models that we just mentioned

if you're interested in them do go more

in depth mathematically into them like

how do we how do we find this hyperplane

right i'm not going to go over that in

this specific course because you're just

learning what an svm is

but it's a good idea to know oh okay

this is the technique behind finding

you know what exactly are the are the

um how do you define the hyperplane that

we're going to use

so anyways this transformation that we

did down here

this is known as the kernel trick

so when we go from x to some coordinate

x and then x squared

what we're doing is we are applying a

kernel so that's why it's called the

kernel trick

so svms are actually really powerful and

you'll see that here so

from

sklearn.svm we are going to import

svc and svc is our support vector

classifier

so with this

so with our sbm model

we are going to you know create an svc

model

and we are going to

uh

again fit this to x train i could have

just copy and pasted this i should have

probably done that

okay

taking a bit longer

all right

let's predict using our svm model and

here

let's see if i can hover over this

all right so again you see a lot of

these different parameters here that you

can go back and change if you were

creating a production level model

okay but

in this specific case

we'll just use it out of the box again

so

if i make predictions you'll note that

wow the accuracy actually jumps to 87

with the svm

and even with class 0 there's nothing

less than you know 0.8 which is great

and for class one i mean everything's at

0.9 which is higher than anything that

we had seen to this point

so so far we've gone over four different

classification models we've done svms

logistic regression naive bayes and k n

and these are just simple ways on how to

implement them each of these they have

different you know

they have different hyper parameters

that you can go and you can toggle and

you can try to see

if that helps later on or not

but

for the most part they perform

they give us around 70 to 80 percent

accuracy

okay with svm being the best now let's

see if we can actually beat that using a

neural net now the final type of model

that i wanted to talk about is known as

a neural net or neural network

and neural nets look something like this

so you have an input layer this is where

all your features would go

and

they have all these arrows pointing to

some sort of hidden layer

and then all these arrows point to some

sort of output layer

so what is what does all this mean each

of these layers in here this is

something known as a neuron

okay so that's a neuron

in a neural net

these are all of our features that we're

inputting into the neural net so that

might be x0 x1 all the way through

xn

right and these are the features that we

talked about there they might be you

know the pregnancy the bmi the

age etc

and now all these get weighted by some

value

so they get multiplied by some w number

that applies to that one specific

category that one specific feature so

these two get multiplied

and the sum of all of these goes into

that neuron

okay so basically i'm taking w0 times x0

and then i'm adding x1 times w1 and then

i'm adding you know x2 times w2 etc all

the way to xn times

n and that's getting

input into the neuron

now i'm also adding this bias term which

just means okay i might want to shift

this by a little bit so i might add 5 or

i might add 0.1 or i might subtract 100

i don't know but we're going to add this

bias term

and the output of all these things so

the sum of this this this and this

go into something known as an activation

function okay

and then after applying this activation

function we get an output

and this is what a neuron would look

like

now a whole network of them would look

something like

this so i kind of gloss over this

activation function

what exactly is that

this is how a neural net looks like if

we have all our inputs here and let's

say all of these arrows represent some

sort of addition

right

then what's going on is we're just

adding a bunch of times

right we're adding the some sort of

weight times these input layer

a bunch of times and then if we were to

go back and factor that all out

then this entire neural net

is just a linear combination of these

input layers

which i don't know about you but that

just seems kind of useless right because

we could literally just write that out

in a formula why would we need to set up

this entire neural network we

wouldn't

so the activation function is introduced

right so without an activation function

this just becomes a linear model

an activation function might look

something like this and as you can tell

these are not linear and the reason why

we introduce these

is so that our entire model doesn't

collapse on itself and become a linear

model

so over here this is something known as

a sigmoid function it runs between zero

and one

tan runs between negative one all the

way to one

and this is relu which anything less

than zero is zero and that anything

greater than zero is linear

so with these activation functions

every single output of a neuron

is no longer just the linear combination

of these it's some sort of altered

linear state which means that the input

into the next neuron

is

you know

it doesn't it doesn't collapse on itself

it doesn't become linear because we've

introduced all these non-linearities

so this is the training set the model

the loss right and then we do this thing

called training where we have to feed

the loss back into the model

and make certain adjustments the model

to improve

this predicted output

let's talk a little bit about the

training what exactly goes on during

that step

let's go back and take a look at our l2

loss function

this is what our l2 loss function looks

like it's a quadratic formula right

well up here the error is really really

really really large

and

our goal is to get somewhere down here

where the loss is decreased right

because that means that our predicted

value is closer to our true value

so that means that we want to go

this way

okay

and thanks to a lot of properties of

math something that we can do is called

gradient descent

in order to follow this

slope

down this way

this

quadratic

is it has different um

slopes with respect to some value

okay so

the loss with respect to some weight

w0

versus w1 versus wn

they might all be different

right so

some way that i kind of think about it

is to what extent is this value

contributing to our loss and we can

actually figure that out through some

calculus which we're not going to touch

up on in this specific course but

if you want to learn more about neural

nets you should probably also learn some

calculus and figure out what exactly

backpropagation is doing in order to

actually calculate you know how much do

we have to backstep by

so the thing is here you might notice

that this follows this curve at all

these different points and the closer we

get to the bottom the smaller this step

becomes

now stick with me here

so

my new value this is what we call a

weight update i'm going to take w0

and i'm going to set some new value for

w0

and what i'm going to set for that is

the old value of w0

plus

some

factor which i'll just call alpha for

now

times

whatever this arrow is so that's

basically saying okay take our old

w0 our old weight

and just decrease it

this way so i guess increase it in this

direction right like take a step in this

direction but this alpha here is telling

us okay don't don't take a huge step

right just in case we're wrong take a

small step take a small step in that

direction see if we get any closer

and for those of you who you know do

want to look more into the mathematics

of things the reason why i use a plus

here is because

this here is the negative gradient right

if this were just the if you were to use

the actual gradient this should be a

minus

now this alpha is something that we call

the learning rate

okay and that adjusts how quickly we're

taking steps

and that might you know tell our that

that will ultimately control

how long it takes for our neural net to

converge

or sometimes if you set it too high it

might even diverge

but with all of these weights so here i

have w0 w1 and then wn

we make the same update to all of them

after we calculate

the

loss

the gradient of the loss with respect to

that weight

so that's how backpropagation works

and that is everything that's going on

here after we calculate the loss we're

calculating gradients

making adjustments in the model so we're

setting all the all the weights to

something adjusted slightly

and then

we're saying okay let's take the

training set and run it through the

model again and go through this loop all

over again so for machine learning we

already have seen some libraries that we

use right we've already seen sk learn

but

when we start going into neural networks

this is kind of what we're trying to

program

and

it's not

very fun

to try to program this from scratch

because

not only will we probably have a lot of

bugs but also it's probably not going to

be fast enough

right

wouldn't it be great if there are some

you know

full-time professionals that are

dedicated to solving this problem and

they could literally just give us their

code that's already running really fast

well the answer is yes that exists

and that's why we use tensorflow so

tensorflow makes it really easy to

define these models

but we also have enough control

over what exactly we're feeding into

this model

so for example this line here is

basically saying okay let's create

a sequential neural net

so sequential is just you know what

we've seen here it just goes one layer

to the next

and a dense layer means that all them

are interconnected so here this is

interconnected with all of these nodes

and this one's all these and then this

one gets connected to all of

the next ones and so on so we're going

to create 16

dense nodes

with relu activation functions and then

we're going to create another layer of

16

dense nodes with relu activation and

then our output layer is going to be

just one node okay

and that's how easy it is to define

something in tensorflow

so tensorflow is an open source library

that helps you develop and train your ml

models

let's implement this for a neural net so

we're using a neural net for

classification

now

so our neural net model

we are going to use tensorflow and i

don't think i imported that up here so

we are going to import that down here

so i'm going to import

tensorflow as tf

and enter

cool

so my

neural net model

is going to be

i'm going to use this

um

[Music]

so essentially this is saying

layer all these things that i'm about to

pass in

so yeah

layer them linear stack of layers

layer them as a model

and what that means nope not that so

what that means is i can pass in

um some sort of layer

and i'm just going to use a dense layer

uh oops dot dense

and let's say we have 32

units

okay

i will also

um

set the activation as relu

and at first we have to specify the

input shape

so here we have 10 comma

all right

all right so that's our first layer now

our next layer i'm just going to have

another

a dense layer of 32 units all using relu

and

that's it so for the final layer this is

just going to be my output layer

it's going to just be one node

and the activation is going to be

sigmoid

so

if you recall from our logistic

regression what happened there was when

we had a sigmoid it looks something like

this

right so by creating a sigmoid

activation on our last layer we're

essentially projecting our predictions

to be zero or one

just like in logistic

regression

and that's going to help us

you know we can just round to zero or

one and classify that way

so this is my neural net model and i'm

going to

compile this so in tensorflow we have to

compile it

it's really cool because i can just

literally pass in what type of optimizer

i want and it'll do it

um so here if i go to optimizers i'm

actually going to use atom

and you'll see that you know the

learning rate is

0.001 so i'm just going to use that

default so 0.001

and my loss is going to be

binary cross

entropy

and

the metrics that i'm also going to

include on here so it already will

consider loss but i'm i'm also going to

tack on accuracy so we can actually see

that in a plot later on

all right so i'm going to run this

um

and

one thing that i'm going to also do is

i'm going to define these plot

definitions so i'm actually copying

pasting this

i got these from tensorflow so if you go

on to some tensorflow tutorial they

actually have these this like

defined

uh and that's exactly what i'm doing

here so i'm actually going to move this

cell up

run that so we're basically plotting the

loss over all the different epochs

epochs means like training cycles and

we're going to plot the accuracy over

all the epochs

all right so we have our model

and now

all that's left is let's train it okay

so i'm going to say history so

tensorflow is great because it keeps

track of the history of the training

which is why we can go and plot it later

on

now i'm going to set that equal to this

neural net model

and fit that

with x train

y train

uh i'm going to

make the number of epochs equal to let's

say just let's just use 100 for now

and the batch size i'm going to set

equal to let's say 32.

all right um

and the validation split

so what the validation split does if

it's down here somewhere

okay so yeah this validation split is

just the fraction the training data to

be used as validation data

so essentially every single epoch what's

going on is

uh tensorflow saying leave certain if if

this is 0.2 then leave 20

out and we're going to test how the

model performs on that 20 that we've

left out

okay so it's basically like our

validation data set but

um tensorflow does it on our training

data set during the training so we have

now a measure outside of just our

validation data set to see you know

what's going on

so validation split i'm going to make

that 0.2

and we can run this so if i run that

all right and i'm actually going to

set verbose

equal to zero which means okay don't

print anything because printing

something for 100 epochs might get kind

of annoying

so i'm just gonna let it run

let it train and then we'll see what

happens

cool so it finished training and now

what i can do is because you know i've

already defined these two functions

i can go ahead and i can plot the loss

oops

plot loss of that history

and i can also plot the accuracy

throughout

the training

so

this is a little bit ish what we're

looking for we definitely are looking

for a steadily decreasing loss

and an increasing accuracy so here we do

see that you know our validation

accuracy improves from

uh around point seven

seven or something all the way up to

somewhere around point maybe eight one

and our loss is decreasing so this is

good

it is expected that the validation loss

and accuracy is performing worse than um

the training loss or accuracy and that's

because our model is training on that

data so it's adapting to that data

whereas the validation stuff is you know

stuff that it hasn't seen yet so

so that's why

so in machine learning as we saw above

we could change a bunch of the

parameters right like i could change

this to 64. so now it'd be a row of 64

nodes and then 32 and then one

so i can change some of these parameters

and

a lot of machine learning is trying to

find hey what do we set these hyper

parameters to

so what i'm actually going to do is i'm

going to rewrite this so that

we can do something what's known as a

grid search so we can search through an

entire space of hey what happens if

you know we

have 64 nodes and 64 nodes or 16 nodes

and 16 nodes

and so on

um and then on top of all that we can

you know we can change

this uh

learning rate we can change how many

epochs we can change

you know the batch size all these things

might affect our training

and

just for kicks i'm also going to add

what's known as a dropout layer in here

and what dropout is doing is saying hey

randomly choose

with at this rate certain nodes

and don't train them

in you know a certain iteration so this

helps prevent overfitting

okay

so

i'm actually going to

define this

as a function called train model we're

going to pass an x train y train

um the number of

nodes

uh the drop out

you know the probability that we just

talked about

um

learning rate

so i'm actually going to say lr

batch size

and

we can also pass the number of epochs

right i mentioned that as a parameter

so indent this so it goes under here and

with these two i'm going to set this

equal to number of nodes

and now with the two dropout layers i'm

going to set dropout prob so now you

know the probability of

turning off a node during the training

is equal to dropout prop

um and i'm going to keep the output

layer the same

now i'm compiling it but this here is

now going to be my learning rate

and i still want binary cross entropy

and accuracy

we are actually going to train

our model inside of

this

function

but here we can do the epochs equals

epochs and this is equal to whatever you

know we're passing in

uh x train y train belong right here

okay so those are getting passed in as

well

and finally at the end i'm going to

return this model and the history of

that model

okay

so

now what i'll do

is let's just go through all of these so

let's say let's keep the epochs at 100.

and now what i can do is i can say hey

for a number of nodes in

let's say let's do 16 32 and 64 to see

what happens

for the different dropout

probabilities in

i mean zero would be nothing let's use

0.2 also to see what happens

um

you know for the learning rate in

uh

0.005

0.001

and you know maybe we want to throw on

0.1 in there as well

and then for the batch size uh let's do

16 32 64 as well actually and let's also

throw in 128. actually let's get rid of

16. sorry

let's throw 128 in there

that should be zero one

i'm going to

record the model in history using this

train model here

so

we're going to do x train y

train

the number of nodes is going to be you

know the number of nodes that we've

defined here

dropout

prob lr

batch size

and epochs okay

and then now we have both the model and

the history and what i'm going to do is

again i want to plot

the loss

for the history i'm also going to plot

the accuracy

probably should have done them side by

side that probably would have been

easier

okay so

what i'm going to do is

split up split this up

and that will be

subplots so now this is just saying okay

i want one row and two columns in that

row for my plots

okay

so

i'm going to plot on my axis one

the loss

i don't actually know this is gonna work

okay we don't care about the grid uh

yeah let's let's keep the grid

and then now on my other

so now on here i'm going to plot all the

accuracies on the second plot

i might have to debug this a bit

but we should be able to get rid of that

if we run this we already have history

saved as a variable in here so if i just

run it on this okay it has no

attribute x label

oh i think it's because it's like set x

label or something

okay yeah so it's it's set instead of

just x label y label

so let's see if that works

all right cool

um and let's actually make this a bit

larger

okay so we can actually change the

figure size and i'm gonna set

let's see what happens if i set that to

oh that's not the way i wanted it um

okay so that looks reasonable

and that's just going to be my plot

history function so now i can plot them

side by side

here i'm going to plot the history

and what i'm actually going to do is i

so here first i'm going to print out all

these parameters so i'm going to print

out

use the f string to print out uh all of

this stuff

so here i'm printing out how many nodes

um

the dropout probability

uh the learning rate

and we already know how many emails so

i'm not even gonna bother with that

so

once we plot

this

uh let's actually also

figure out what the um

what the validation loss is on our

validation set that we have

that we created all the way back up here

all right so remember we created three

data sets

let's call our model and evaluate

what the

validation

data what the validation data sets loss

would be

and i actually want to record

let's say i want to record

whatever model has the least validation

loss so

first i'm going to initialize that to

infinity so that you know any model will

beat that score

so if i do float infinity that will set

that to infinity

and um maybe i'll keep track of the

parameters actually it doesn't really

matter

i'm just going to keep track of the

model

and i'm going to set that to none

so now down here

if the validation loss is ever less than

the least validation loss

then i am going to simply come down here

and say hey

this validation

or this lease validation loss is now

equal to the validation loss

and

the least

loss model

is whatever this model

is that just earned that validation loss

okay

so

we are actually just going to let this

run

um for a while and then we're going to

get our least

loss model after that

so

let's just run

all right and now we wait

all right so we have finally finished

training

and you'll notice that okay down here

the loss actually gets to like 0.29

the accuracy is around 88 which is

pretty good

so you might be wondering okay why is

this accuracy in this

like these are both the validation

so this accuracy here is on the

validation data set that we've defined

at the beginning right and this one here

this is actually taking 20 of our test

our training set every time during the

training and saying okay how much of it

do i get right now

you know after this one step where i

didn't train with any of that

so they're slightly different and

actually i realized later on that i

probably you know probably what i should

have done

is over here when we were defining

the model fit instead of the validation

split you can define the validation data

and you can pass in the validation data

i don't know if this is the proper

syntax but

that's probably what i should have done

but instead you know we'll just stick

with what we have here

so you'll see at the end you know with

the 64 nodes it seems like this is our

best performance 64 nodes with a dropout

of 0.2 a learning rate of

0.001 and a batch size of 64.

and it does seem like yes the validation

you know the fake validation but the

validation um

loss is decreasing and then the accuracy

is increasing which is a good sign okay

so finally

what i'm going to do is i'm actually

just going to predict so i'm going to

take

this model which we've called our least

loss model

i'm going to take this model and i'm

going to predict

x test

on that

and you'll see that it gives me some

values that are really close to zero and

some that are really close to one and

that's because we have a sigmoid output

so

if i

do this

what i can do is i can cast them

so i'm going to say anything that's

greater than 0.5

set that to 1. so if i

actually i think what happens if i do

this

oh okay so i have to cast that as type

and so now you'll see that it's ones and

zeros

and i'm actually going to transform this

into a column as well

so here i'm going to

oh oops uh i didn't mean to do that okay

no i wanted to just reshape it to

that so now

[Music]

it's one dimensional okay

and

using that we can actually

just rerun the classification report

based on these this neural net output

and you'll see that okay the the f1 or

the accuracy gives us 87

so it seems like what happened here is

the precision

on uh class 0 so the hadrons has

increased a bit but the recall decreased

but the f1 score is still at a good 0.81

and um

for the other class it looked like the

precision decreased a bit the recall

increased for an overall f1 score

that's also been increased

i think i interpreted that properly i

mean we went through all this work and

we got a model that performs actually

very very similarly to the svm model

that we had earlier

and the whole point of this exercise was

to demonstrate okay these are how you

can define your models but it's also to

say hey maybe

you know neural nets are very very

powerful as you can tell

but sometimes you know an svm or some

other model

might actually be more appropriate

but in this case i guess it didn't

really matter which one we used at the

end um

an 87 percent accuracy accuracy score is

still pretty good

so yeah let's now move on to regression

we just saw a bunch of different

classification models now let's shift

gears into regression the other type of

supervised learning

if we look at this plot over here we see

a bunch of scattered data points

and here we have our x

value for those data points and then we

have the corresponding y value which is

now our label

and when we look at this plot

well our goal in regression is to find

the line of best fit that best models

this data

essentially we're trying to let's say

we're given some new value of x that we

don't have in our sample we're trying to

say okay what would my prediction for y

b

for that given x value so that you know

might be somewhere around there

i don't know

but remember in regression that you know

given certain features we're trying to

predict some continuous numerical value

for y

in linear regression

we want to take our data and fit a

linear model to this data so in this

case our linear model might look

something along the lines of here

right

so this here would be considered as

maybe our line of

best

fit

and this line

is modeled by the equation i'm going to

write it down here

y

equals

b 0

plus b 1 x

now b0 just means it's this y-intercept

so if we extend this y down here

this value here is b0 and then b1

defines the slope

of this line

okay

all right so that's the that's the

formula

for linear regression

and how exactly do we come up with that

formula what are we trying to do with

this linear regression

you know we could just eyeball

where should the line be but humans are

not very good at eyeballing certain

things like that i mean we can get close

but a computer is better at giving us a

precise value for b0 and b1

well let's introduce the concept of

something known as a residual

okay so

residual

you might also hear this being called

the error

and what that means is let's take some

data point in our data set

and we're going to evaluate how far off

is our prediction

from

a data point that we already have

so this here is our y let's say

this is one two three four five six

seven

eight so this is y eight let's call it

you'll see that i use this y i in order

to represent hey just one of these

points

okay

so this here is y and this here would be

the prediction oops this here would be

the prediction for y

8

which i've labeled with this hat okay if

it has a hat on it that means hey this

is what this is my guess this is my

prediction

for

you know this specific

value of x

okay now the residual

would be

this distance here

between y

eight and y hat eight so

y eight

minus y hat eight

all right because that would give us

this

here and i'm just going to take the

absolute value of this because what if

it's below the line

right then you would get a negative

value but distance can't be negative so

we're just going to put a little hat or

we're going to put a little absolute

value around this quantity

and that gives us

the residual or the error

so let me rewrite that

and you know to generalize to all the

points i'm going to say the residual can

be calculated

as y i

minus y hat

of i

okay

so this just means the distance between

some given point

and its prediction its corresponding

prediction on the line

so now

with this residual

this line of best fit

is generally trying to decrease these

residuals as much as possible

so

now that we have some value for the

error our line of best fit is trying to

decrease the error as much as possible

for all of the different data points

and that might mean you know minimizing

the sum of all the residuals so this

here this is the sum

symbol and if i just stick the residual

calculation

in there

it looks something like that right and

i'm just going to say okay for all of

the i's in our data set so for all the

different points we're going to sum up

all the residuals

and i'm going to try to decrease that

with my line of best fit so i'm going to

find the b0 and b1 which gives me the

lowest value of this

okay

now in other you know sometimes in

different circumstances we might

attach a squared to that so we're trying

to decrease the sum of the squared

residuals

and what that does is it just

you know it adds a higher penalty for

how far off we are from you know points

that are further off so that is linear

regression we're trying to find

this equation some line of best fit

that will help us decrease

this measure of error

with respect to all the data points that

we have in our data set and try to come

up with the best prediction for all of

them

this is known as simple

linear

regression

basically that means you know our

equation looks something

like this

now there's also multiple

linear regression

which just means that hey if we have

more

than one value for x so like think of

our feature vectors we have multiple

values in our x vector

then our predictor might look something

more

like this

actually i'm just going to say etc plus

b n x n so now i'm coming up with some

coefficient

for all of the different

x values that i have in my vector now

you guys might have noticed that i have

some assumptions over here and you might

be asking okay kylie what in the world

do these assumptions mean so let's go

over them

the first one is linearity

and what that means is let's say i have

a data set

okay

linearity just means okay my does my

data follow a linear pattern does

y increase as x increases or does y

decrease at as x increases does so if y

increases or decreases at a constant

rate as x increases

then you're probably looking at

something linear so what's an example of

a non-linear data set let's say i had

data that might look something like that

okay

so now just visually judging this you

might say okay seems like the line of

best fit might actually be some curve

like this

right

and in this case we don't satisfy that

linearity

assumption anymore

so

with linearity we basically just want

our data set to follow some sort of

linear

trajectory

and independence

our second assumption

just means

this point over here

it should have no influence on this

point over here or this point over here

or this point over here so in other

words all the points

all the samples in our data set

should be independent

okay they should not rely on one another

they should not affect one another

okay now

normality and

homoscedasticity those are concepts

which use this residual okay

so if i have a plot

that looks

something

like

this

and my line of best fit

is somewhere

here

maybe it's something like that

in order to look at these normality and

homoscedasticity

assumptions let's look at the residual

plot okay

and what that means is i'm going to keep

my same x axis

but instead of plotting now where they

are relative to this y i'm going to plot

these errors so now i'm going to plot y

minus y hat

like this

okay

and now you know this one is slightly

positive so it might be here this one

down here is negative it might be here

so our

residual plot

it's literally just a plot of how you

know the values are distributed around

our line of best fit

so it looks like

it might you know look something

like this

okay

so this might be our residual plot and

what normality means so our assumptions

are normality

and homo

schedasticities

good ass

i might have butchered that spelling i

don't really know

but

what

normality is saying is saying okay these

residuals should be normally distributed

okay

around this line of best fit it should

follow a normal distribution

and now what

homoscedasticity says okay our variance

of these points

should remain constant

throughout so this spread here should be

approximately the same as this spread

over here

now what's an example of where you know

homo

schedasticity is not held

well let's say that our

original plot actually looks something

like

this

okay so now if we looked at the

residuals for that

it might look something

like that

and now if you we look at this spread of

the points

it decreases right so now the spread is

not constant which means that

homeostasis

homoscedasticity

um

this this assumption would not be

fulfilled and it might not be

appropriate to use linear regression

so that's just linear regression

basically we have a bunch of data points

we want to predict some y value

for those

and we're trying to come up with this

line of best fit that best describes hey

given some value x

what would be my best guess of what y is

so let's move on to how do we evaluate a

linear regression model

so the first

measure that i'm going to talk about is

known as mean absolute error or m-a-e

for short

okay

and mean absolute error

is basically saying all right let's take

all the errors so all these residuals

that we talked about

let's sum up

the distance for all of them and then

take the average and then that can

describe you know how far off are we

so the

mathematical formula for that would be

okay let's take all the residuals

all right so this is the distance

actually let me redraw a plot down here

so

suppose i have a data set look like this

and

here are all of my

data points right

and now let's say my line looks

something like

that

so

my mean absolute error would be summing

up

all of these values

this was a mistake

so summing up all of these

and then dividing by how many data

points i have

so what would be all the residuals it

would be y i right so every single point

minus y hat i so the prediction for that

on here

and then we're going to sum over all of

the different i's in our data set

right so

i

and then we divide by the number of

points we have so actually i'm going to

rewrite this to make it a little clearer

so i is equal to whatever the first data

point is all the way through the nth

data point

and then we divide it by n which is how

many points there are

okay so this is our measure of m a e

and this is basically telling us okay in

on average

this is the distance

between

our predicted value and the actual value

in our training set

okay

and mae is good because it allows us to

you know when we get this value here we

can literally directly compare it to

whatever

units the y value is in so let's say y

is

we're talking you know the prediction of

the price of a house

right

in dollars

once we have once we calculate the mae

we can literally say oh the average you

know price

the average

um how much we're off by

is literally this many dollars

okay

so that's the mean absolute error

an evaluation technique that's also

closely related to that

is called the mean squared error

and this is mse

for short

okay

now

if i take this plot again

and i duplicate it and move it down here

well the gist of mean squared error is

kind of the same but instead of the

absolute value we're going to square

so now the mse

is something along the lines of okay

let's sum up

something right so we're going to sum up

all of our errors

so now i'm going to do y i minus y hat i

but instead of absolute valuing them i'm

going to square them all and then i'm

going to divide by n in order to find

the mean

so basically now i'm taking

all of these

different values and i'm squaring them

first before i add them to one another

and then i divide by n

and the reason why we like using mean

squared error is that

it helps us punish large errors in the

prediction

and later on mse might be important

because of differentiability right so a

quadratic equation is differentiable you

know if you're familiar with calculus a

quadratic equation is different

differentiable whereas the absolute

value function is not totally

differentiable everywhere

but if you don't understand that don't

worry about it you won't really need it

right now

and now one downside of mean squared

error is that once i calculate the mean

squared error over here

and i go back over to y and i want to

compare the values

well it gets a little bit trickier to do

that because

now my mean squared error is in terms of

y squared right it's

this is now squared so instead of just

dollars how you know how many dollars

off am i i'm talking how many dollars

squared

off

am i

and that you know to humans it doesn't

really make that much sense which is why

we have created something known as the

root mean square error

and

i'm just going to copy

this diagram over here because it's very

very similar

to mean squared error

except

now we take a big squared root

okay so this is rmse and we take the

square root

of that

mean squared error and so now the term

in which you know we're defining

our error

is now in terms of that dollar sign

symbol again so that's a pro of rooting

squared error is that now we can say

okay our error according to this metric

is this many dollar signs off from our

predictor

okay so it's in the same unit which is

one of the pros of root mean squared

error

and now finally there is the coefficient

of determination or r squared

and this is the formula for r squared so

r squared is equal to 1 minus rss

over

tss

okay so what does that mean

basically rss

stands for the sum

of the squared

residuals

so maybe it should be ssr instead but

rss sum of the squared residuals and

this

is equal

to

if i take the sum of all the values

and i take y i minus y hat

i

and square that

that is my rss right it's the sum of the

squared residuals

now tss let me actually use a different

color for that

so tss is the total

sum

of squares

and what that means is that instead of

being with respect to

this prediction

we are instead

going to

take each y value and just subtract

the mean of all the y values

and square that

okay so if i drew this out

and if this were my

[Music]

actually let's use a different color

let's use green

if this were my predictor

so rss is giving me this measure

here

right it's giving me some estimate of

how far off we are from our regressor

that we predicted

actually

i'm going to use red for that

well

tss on the other hand is saying okay how

far off are these values from the mean

so if we literally didn't do any

calculations for the line of best fit if

we

just took all the y values and averaged

all of them

and said hey this is the average value

for every single x value

i'm just going to predict that average

value instead

then it's asking okay how far off are

all these points from that line

okay and remember that this square means

that we're punishing

larger errors

right so even if they look somewhat

close in terms of distance

the further a few data points are

then the further the larger our total

sum of squares is going to be

sorry that was my dog

so the total sum of squares is taking

all of these values and saying okay what

is the sum of squares if i didn't do any

regressor and i literally just

calculated the average

of all the y values in my data set and

for every single x value i'm just going

to predict that average

which means that okay like that means

that maybe y and x aren't associated

with each other at all like the best

thing that i can do for any new x value

just predict hey this is the average of

my data set

and this total sum of squares is saying

okay well with respect to that average

what is our error

right so up here the sum of the squared

residuals

this is telling us what is our what what

is our error with respect to

this line and best fit while our total

sum of square is saying what is the

error with respect to you know just the

average y value

and

if our line of best fit is a better fit

then

this total sum of squares

that means that you know this

numerator

that means that this numerator is going

to be smaller than this denominator

right and if our errors in our

mind and best fit are much smaller

then that means that this ratio of the

rss over tss is going to be very small

which means that r squared is going to

go towards one

and now when r squared is towards one

that means that that's usually a sign

that we have a good

predictor

it's one of the signs not the only one

so over here i also have you know that

there's this adjusted r squared and what

that does it just adjusts for the number

of terms so x1 x2 x3 etc it adjusts for

how many extra terms we add because

usually when we

um you know

add an extra term the r squared value

will increase because that'll help us

predict why some more

but the value for the adjusted r squared

increases if the new term actually

improves this model fit more than

expected you know by chance so that's

what adjusted r squared is i'm not you

know it's out of the scope of this one

specific course and now that's linear

regression basically

i've covered the concept of residuals or

errors

and

you know how do we use that in order to

find the line of best fit

and you know our computer can do all the

calculations for us which is nice but

behind the scenes it's trying to

minimize that error right

and then we've gone through all the

different ways of actually evaluating a

linear regression model and the pros and

cons of each one

so now let's look at an example so we're

still on supervised learning but now

we're just going to talk about

regression so what happens when you

don't just want to predict you know type

one two three what happens if you

actually wanna predict a certain value

so again i'm on the uci machine learning

repository

and

here i found this data set about

bike sharing in seoul

south korea

so this data set is predicting rental

bike count and here it's the account of

bikes rented at each hour

so what we're going to do again you're

going to go into the data folder and

you're going to

download this csv file

and we're going to move over to colab

again

and here i'm going to name this fcc

bikes

and regression

i don't remember what i called the last

one but yeah fcc bikes regression

now i'm going to import a bunch of the

same things that i did earlier

um

and you know i'm gonna also continue to

import the oversampler

and the standard scaler

and then i'm actually also just going to

let you guys know that i have a few more

things i wanted to import

so this is a library that lets us copy

things uh seaborne is a wrapper over

matplotlib so

it also allows us to plot certain things

and then just letting you know that

we're also going to be using tensorflow

okay so one more thing that we're also

going to be using we're going to use the

sklearn linear model library and

actually let me make my screen a little

bit bigger

so yeah

awesome

run this and

that'll import all the things that we

need

so again i'm just going to you know give

some credit to where we got this data

set

so let me copy and paste um

this uci

thing

and i will also give credit to this

here

okay

cool all right cool

so this is our data set and again it

tells us all the different attributes

that we have right here so i'm actually

going to go ahead

and paste this in here

um

feel free to copy and paste this if you

want me to read it out loud so you can

type it it's by count

hour

temp

humidity

wind visibility dew point temp

radiation rain snow

and

functional whatever that means

okay so i'm going to come over here and

import my data by dragging and dropping

all right

now one thing that you guys might

actually need to do is you might

actually have to open up the csv because

there were

at first a few um like forbidden

characters in mine at least

so you might have to get rid of like i

think there was a degree here but my

computer wasn't recognizing it so i got

rid of that so you might have to go

through and get rid of some of those

labels that are incorrect

i'm gonna

do this okay

but

after we've done that we've imported in

here i'm going to

create a data a data frame from that so

all right so now what i can do is i can

read that csv file and i can get the

data into here

so sold by data.csv

okay so now if i call data.head

you'll see that i have all the various

labels right and then i have the data in

there

so i'm going to from here um

i'm actually going to get rid of some of

these columns that you know i don't

really care about so here i'm going to

when i when i type this in i'm going to

drop maybe the date

whether or not it's a holiday

and the various seasons

so i'm just

not going to care about these things

axis equals one means drop it from the

columns

so now you'll see that okay we still

have i mean i guess you don't really

notice it but

if i set the data frames columns equal

to data set calls

and i

look at you know the first five things

then you'll see that this is now our

data set it's a lot easier to read so

another thing is i'm actually going to

df functional

and we're going to create this so

remember that our computers are not very

good at language we want it to be in

zeros and ones so here i will convert

that

well if this is equal to yes

then that

that gets mapped as one so then set type

integer

all right

great

cool so the thing is right now these

byte counts are for whatever hour so to

make this example simpler i'm just going

to index on an hour and i'm going to say

okay we're only going to use that

specific hour

so

here let's say um

so this data frame is only going to be

data frame where the hour

let's say it equals 12 okay so it's noon

all right

so now you'll see that all the hours are

equal to 12 and i'm actually going to

now drop that column

our

axis equals one

all right so if we run this cell okay so

now we got rid of the hour in here

and we just have the bike count the

temperature humidity wind visibility and

yada yada yada

all right so what i want to do is i'm

going to actually plot all of these so

for

i and all the columns so the range

length of uh whatever this data frame is

and all the columns because i don't have

byte count as

actually it's my first thing so what i'm

going to do is say for a label and data

frame

columns everything after the first thing

so that would give me the temperature

and onwards so these are all my features

right

uh i'm going to just scatter

so

i want to see how that label how that

specific data

um how that affects the byte count so

i'm going to plot the byte count on the

y-axis

and i'm going to plot you know whatever

the specific label is on the x-axis

and i'm going to title this

uh whatever the label is

and

you know make my y label the bike count

at noon

and the x label

as just the label

okay now

i guess we don't even need the legend

so just show that plot

all right

so it seems like functional is not

really uh

doesn't really give us any utility

so then snow

rain

um

seems like this radiation

you know is fairly linear

dew point temperature

visibility

uh wind doesn't really seem like it does

much

humidity kind of maybe like an inverse

relationship

but the temperature definitely looks

like there's a relationship between that

and the number of bikes right so what

i'm actually going to do is i'm going to

drop some of the ones that don't don't

seem like they really matter so

maybe wind

you know visibility

yeah so i'm going to get rid of wind

visibility and functional

so

let me now data frame

and i'm going to drop

wind visibility

and functional

all right

and the axis again is the column so

that's one

so if i look at my data set now

i have just the temperature the humidity

the dew point temperature radiation rain

and snow

so again what i want to do is i want to

split this into my training

my validation and my test data set

just as we talked before

here

uh we can use the exact same thing that

we just did and we can say numpy.split

and sample you know that the whole

sample

um and then create our splits

of the data frame

and we're going to do that but now set

this to 8.

okay

so i don't really care about you know

the the full

grid the full array so i'm just gonna

use

an underscore for that variable

but i will get my training

x and y's and actually i don't have a um

function

for getting the x and y's so here

i'm going to write a function to find

get x y

and

uh

i'm going to pass in the data frame and

i'm actually going to pass in what the

name of the y label is and

what the x what specific x labels i want

to

look at

so

here if that's none then i'm just not

like i'm only going to i'm going to get

everything from the data set that's not

the while it

so here i'm actually going to

make first a deep copy

of my data frame

and

that basically means i'm just copying

everything over

if

uh if like x labels is none so if not x

labels

then all i'm going to do is say all

right x is going to be whatever this

data frame is

and i'm just going to take all the

columns so c for c and

data frame

dot columns

if c does not equal the y label

all right and i'm gonna get the values

from that

but if there is

the x labels

well okay so

in order to index only one thing so like

let's say i pass in only one thing in

here um

then my data frame is

so

let me make a case for that so if the

length of x labels is equal to one

then what i'm going to do is just say

that

this

is going to be

uh x labels and add that just that label

um

values and i actually need to reshape to

make this 2d

so i'm going to pass a negative 1 comma

1

there

now otherwise if i have like a list of

specific x labels that i want to use

then i'm actually just going to say x is

equal to data frame of those x labels

dot values and that should suffice

all right so now that's just me

extracting x

and in order to get my y i'm going to do

y equals data frame

and then pass in the y label

and at the very end i'm going to say

data equals np

dot h stack so i'm stacking them

horizontally one next to each other

and i'll take x and y

and return that oh but

uh this needs to be values and i'm

actually going to reshape this to make

it 2d as well so that we can do this h

stack

and i will return data x y

so now i should be able to say okay get

x y

and

take that data frame

and the y label so my our y label is by

count

and actually so for the x label i'm

actually going to

let's just do like one dimension right

now and earlier i got rid of the plots

but we had seen that maybe you know the

temperature dimension does really well

and we might be able to use that to

predict why

so

i'm going to label this also that you

know it's just using the temperature

and i am also going to do this again

for

oh this should be

and this should be a validation and

there should be a test

um because oh that's val

all right

but here

it should be

val this should be test

all right so we run this and now we have

our training validation and test

data sets for just the temperature so if

i look at x train

temp it's literally just the temperature

okay and i'm doing this first to show

you simple linear regression

all right so right now i can create a

regressor

so i can say

the temp regressor here

and then i'm going to you know make a

linear regression model and just like

before i can

simply fix

fit my x train temp y train

temp in order to train train this linear

regression model

all right and then i can also

i can print

this

regressor's coefficients

and

the intercept so

if i do that

okay this is the coefficient for

whatever the temperature is and then the

the x-intercept

okay

or the y-intercept sorry

all right

and i can you know score so i can get

the um

the r squared

score

so i can score

x

test

and y test

all right so it's an r squared of around

0.38 which is better than zero which

would mean hey there's absolutely no

association but it's also not you know

like

a

good it depends on the context but

you know the higher that number it means

the higher that the two variables would

be correlated right which

here it's

all right it just means there's maybe

some association between the two

but uh the reason why i wanted to do

this one d was to show you

you know if we plotted this this is what

it would look like so if i

uh create a scatter plot

and

let's take the training

um

so this is our data and then let's make

it blue

and then if i also plotted so

something that i can do is say you know

the x range that i'm going to plot it

is

when space um and this goes from

negative 20 to 40 this piece of data so

i'm going to say let's take 100

things from there

so i'm going to

plot x and i'm going to take this

temp this like regressor and predict

x

with that

okay and this label

i'm going to label that

um

the

fit

and this color let's make this red

and let's actually

[Music]

set the line with so i can i can change

how thick

that value is

okay

now at the very end uh let's create a

legend

and let's

all right let's also create you know

title

all these things that matter

in some sense

so here let's just say um

this would be the bikes

versus the temperature

right and the y label would be

number of bikes

and the x label would be the temperature

so i actually think that this might

cause an error yeah

so it's expecting a 2d array so we

actually have to

reshape

this

let's

[Applause]

[Music]

okay there we go

so i just had to make this an array and

then reshape it so it was 2d now we see

that all right this

increases but again remember those

assumptions that we had about linear

regression like this i don't really know

if this

fits those assumptions

right i just wanted to show you guys

though that like

all right this is what a line of best

fit through this data would look like

okay

now

we can do multiple linear regression

right

so i'm going to go ahead and do that as

well

now

if i take

my data set

and instead of the labels

so actually what's my current data set

right now

all right so let's just use all of these

except for the byte count right so i'm

going to just say for the x labels

let's just take the data frames columns

and

just remove the byte count

so does that work

so if this part should be affects labels

is none

and then this should work now

oops sorry

okay so i have

oh but this here because it's not just

the temperature

anymore

we should actually do this um let's say

all

right so i'm just going to quickly rerun

this

piece here so that we have our

temperature only data set and now we

have our all data set

okay

and this regressor i can do the same

thing so i can do the all regressor

and i'm going to make this the linear

regression

and

i'm going to fit this to x train all and

y

train all

okay

all right so let's go ahead and also

score this regressor and let's see how

the r squared performs now so if i test

this

on the test data set what happens

all right so our r squared seems to

improve it went from 0.4 to 0.5 which is

a good sign

okay

and i can't necessarily plot you know

every single dimension but this just

this is just to say okay this has this

is improved right all right so one cool

thing that you can do with tensorflow is

you can actually

do regression

but with a neural net

so

here i'm going

to um

we already have our our training data

for just the temperature and just you

know for all the different columns so

i'm not going to bother with splitting

up the data again

i'm just going to go ahead and start

building the model so

in this linear regression model uh

typically

you know it does help if we normalize it

so that's very easy to do with

tensorflow i can just create some

normalizer layer so i'm going to do

tensorflow keras layers

and get the normalization

layer

and the input shape

for that

will just be one because let's just do

it again on just the temperature

and the axis i will

make none

now for this temp normalizer

and i should have had an equal sign

there

um i'm going to adapt this to x

train

temp

and

reshape this to just a single vector

so that should work great now with this

model so temp neural net model what i

can do is i can do

you know diacharis

that's sequential

and i'm going to pass in this normalizer

layer

and then i'm going to say hey just give

me one single dense layer with one

single unit and what that's doing is

saying all right

well one single node just means that

it's linear and if you don't add any

sort of activation function to it the

output is also linear

so here i'm going to have tensorflow

keras

layers.dense

and i'm just

gonna have one unit

and that's going to be my model

okay

so

uh with this

model

let's compile

and for our optimizer um let's

use

let's use adam again

answers

dot atom and we have to pass in the

learning rate

so learning rate and our learning rate

let's do 0.01

and now

the loss we

actually let's give this one 0.1 and the

loss i'm going to do mean squared error

okay so we run that we've compiled it

okay great

and just like before we can call history

and i'm going to fit this model so

here if i call fit

i can just fit it and i'm going to take

the

uh x train with the temperature

but reshape it

um y train for the temperature

and i'm going to set verbose equal to

zero so that it doesn't you know display

stuff

i'm actually going to set epochs equal

to let's do 1000

um

and the validation

data should be let's pass in the

validation data set here

as a tuple

and i know i spelled that wrong

so let's just run this

and up here i've copy and pasted the

plot loss from our previous but change

the y label to msc because now we're

talking we're dealing with mean squared

error

and

i'm going to plot the loss of this

history after it's done so let's just

wait for this to finish training and

then to plot

okay so this actually looks pretty good

we see that the values are converging

so now what i can do is i'm going to

go back up and take this plot

and we are going to just run that plot

again so

here

um

instead of

this temperature regressor i'm going to

use the neural net regressor

this neural net model

and if i run that i can see that you

know this also gives me a linear

regressor

you'll notice that this this fit is not

entirely the same as the one

up here and that's due to the training

process

of

you know of this neural net so just two

different ways to try and try to find

the best linear regressor

okay but here we're using back

propagation to train a neural net node

whereas in the other one they probably

are not doing that okay they're probably

just trying to actually compute

the line of best fit so

okay given this

well we can repeat the exact same

exercise

with our

um

with our multiple linear regressions

okay

but i'm actually going to skip that part

i will leave that as an exercise to the

viewer okay so now what would happen if

we use a neural net a real neural net

instead of just you know one single node

in order to predict this so

let's start on that code we already have

our normalizer so i'm actually going to

take the same

uh setup here but instead of you know

this one dense layer i'm going to set

this equal to 32 units and for my

activation i'm going to use relu

and now let's

duplicate that

and for the final output i just want one

answer so i just want one cell

and this activation is also going to be

relu because i can't ever have less than

zero bikes so i'm just going to set that

as relu

i'm just going to name this the neural

net model okay

and at the bottom i'm going to have this

um neural net model

i'm going to have to know that model i'm

going to compile

um

[Music]

and i will actually use the same

compiler here

but instead of

instead of a learning rate of 0.01 i'll

use 0.001

okay

and i'm going to train this here so the

history

is this

neural net model um and i'm going to fit

that against x train temp y train temp

and

valid

validation

data i'm going to set this again equal

to

xval

temp and y

vowel

temp

now

for the verbose i'm going to set that

equal to zero

epochs let's do 100 and

here for the batch size actually let's

just not do a batch size right now

let's just try let's see what happens

here

and again we can plot the loss of this

history after it's done training

so let's just run this

and that's not what we're supposed to

get so what is going on

here is sequential we have our

temperature

normalizer

which i'm wondering now if we have to

re-do that

okay so we do see this

decline it's an interesting curve but we

do we do see it eventually

um

so this is our loss which all right it's

decreasing that's a good sign and

actually what's interesting is let's

just let's plot this model again

so here instead of that

and you'll see that we actually have

this like

curve that looks something like this so

actually what if i got rid of this

activation

let's train this again

and see what happens

all right so even even when i got rid of

that relu at the end

it kind of knows hey you know if

it's not the best model

if we had maybe one more layer in here

these are just things that you have to

play around with

when you're you know working with

machine learning it's like you don't

really know

what the best model is going to be

um for example this also is not

brilliant

but

i guess it's okay so my point is though

that with a neural net

i mean this is not brilliant but also

there's like no data down here right so

it's kind of hard for our model to

predict in fact we probably should have

started the prediction somewhere around

here

my point though is that with this neural

net model you can see that this is no

longer a linear predictor but yet we

still get an estimate of the value

right and we can repeat this exact same

exercise

with the multiple

uh

inputs

so here

if i now pass in all the data

so

this is my all

normalizer

and i should just be able to pass in

that

so

let's move this to the next cell

um

here i'm going to pass in my all

normalizer

and let's compile it yeah those

parameters look good

great so

here with the history when we're trying

to

fit this model instead of temp we're

going to use

our larger data set with all the

features

and let's just train that

and of course we want to plot the loss

okay so that's what our loss looks like

um

it's an interesting curve but it's

decreasing

so before we saw that our r squared

score was around 0.52 well we don't

really have that with a neural net

anymore but one thing that we can

measure is hey what is the mean squared

error

right so if i come down here

um

and i compare the two mean squared

errors so

so i can predict

x test

all right

so these are my predictions using that

linear regressor will linear multiple

multiple linear regressor so these are

my my predictions

linear regression

okay

i'm actually going to do

that at the bottom so

let me just copy and paste that cell

and bring it down here so now i'm going

to calculate the mean squared error

for both

um the linear

regressor and the neural net okay so

this is

my

linear and this is my neural net so

if i use my neural net model and i

predict

x test all

i get my two you know different y

predictions

and

um

i can calculate the mean squared error

right so

if i want to get the mean squared error

and i have y

prediction and y

real

i can do

numpy dot square and then i would need

the y prediction minus you know the real

so this this is basically squaring

everything

um and

this should be

a vector so if i

just take this entire thing and take the

mean

of that

that should give me the mse so let's

let's just try that out

and the y real is why test

all

right so that's my mean squared error

for the linear regressor

and this is my mean squared error for

the neural net

so that's interesting uh i will debug

this live i guess

so my guess is that it's probably coming

from

this normalization

layer

um

because this input shape is

probably just

six

and

okay so

that works now

and the reason why is because

like my inputs are only

for every vector it's only a one

dimensional vector of length six so i

should have i should have just had six

comma which is a tuple of size six from

the start or it's a it's a tuple

containing one element which is a six

okay so it's actually interesting that

my

uh neural net results seem like they

they have a larger mean squared error

than my linear aggressor

um one thing that we can look at is

we can actually plot the real versus you

know the the actual results versus

what the predictions are so

if i say

some axis and i use

plt.axes

and make these

equal

then i can scatter

the the y

you know the test so what the actual

values are on the x-axis and then what

the predictions

are on the

x-axis

okay

uh and i can label this as the linear

regression predictions

okay so then let me just label my axes

so the x-axis i'm going to say is the

true values

the y-axis is going to be my linear

regression predictions

or actually let's plot

let's just make this predictions

and then at the end

i'm going to plot

oh let's set some limits

uh

because i think that's like

approximately the max number of bikes

so

i'm going to set my x limit to this and

my y limit

to this

so here i'm going to pass that in here

too

and

all right

this is what we actually get for our

linear regressor

you see that actually they align quite

well i mean to some extent so 2000 is

probably too much

2500 i mean

looks like

maybe like 1800 would be enough here for

our limits

um

and i'm actually going to

label something else the neural net

predictions

let's add a legend

so you you can see that our neural net

for the

larger values it seems like it's a

little bit more spread out and it seems

like

we tend to underestimate a little bit

down here in this area

okay

and for some reason these are way off as

well

but yeah so we've basically used a

linear regressor and a neural net um

honestly there are some times where a

neural net is more appropriate and a

linear regressor is more appropriate

i think that it just comes with time and

trying to figure out you know and just

literally seeing like hey what works

better like here a linear a multiple

linear regressor might actually work

better than a neural net but for example

with the one-dimensional case

a linear regressor would never be able

to see this curve

okay

i mean i'm not saying this is a great

model either but i'm just saying

like hey you know

sometimes it might be more appropriate

to use something that's not linear

so yeah i will leave regression at that

okay so we just talked about supervised

learning

and in supervised learning we have data

we have some a bunch of features and for

a bunch of different samples but each of

those samples has some sort of label on

it whether that's a number a category a

class etc right we were able to use that

label in order to try to predict new

labels of other points that we haven't

seen yet

well now let's move on to unsupervised

learning

so with unsupervised learning we have a

bunch of unlabeled data

and what can we do with that you know

can we learn anything from this data

so the first algorithm that we're going

to discuss is known as k means

clustering what k means clustering is

trying to do is it's trying to compute

k

clusters

from the data

so in this example below i have a bunch

of scattered points and you'll see that

this is

x0 and x1 on the two axes which means

i'm actually plotting two different

features

right of each point but we don't know

what the y label is for those points

and now just looking at these scattered

points

we can kind of see how there are

different clusters in the data set right

so depending on what we pick for k we

might have different clusters

let's say k equals two right then we

might pick okay this seems like it could

be one cluster but this here is also

another cluster so those might be our

two different clusters

if we have k equals three

for example then okay this seems like it

could be a cluster

this seems like it could be a cluster

and maybe this could be a cluster right

so we could have three different

clusters in the data set

now this k here is predefined

if i can spell that correctly

by the person who's running the model so

that would be you

all right

and let's discuss how you know the

computer actually goes through and

computes

the k clusters

so i'm going to write those steps down

here

now the first step that happens is we

actually

choose well the computer

chooses three random points

on this plot

to be the centroids

and by centroids i just mean the center

of the clusters okay

so three random points let's say we're

doing k equals three so we're choosing

three random points to be the centroids

of the three clusters if it were two

we'd be choosing two random points

okay

so maybe the three random points i'm

choosing might be here

here

[Music]

and here

all right

so we have

three different points

and the second thing that we do

is we actually calculate

the distance

for each point

to those centroids

so between all the points

and the centroid

so basically i'm saying all right this

is this distance this is this distance

this is this distance

all of these distances i'm computing

between oops not those two

between the points not the centroids

themselves

so i'm computing the distances for all

of these plots to each of the centroids

okay

and that

comes with also

assigning

those points to the closest centroid

what do i mean by that

so

let's take

this point here for example so i'm

computing this distance this distance

and this distance and i'm saying okay it

seems like the red one is the closest so

i'm actually going to put this into the

red

centroid

so if i do that for all of these points

this one seems slightly closer to red

and this one seems slightly closer to

red

right

now for the blue

i actually wouldn't

put any blue ones in here but

we would probably actually that first

one is closer to red

and now it seems like the rest of them

are probably closer to green

so let's just put all of these into

green here

like that

and

cool so now we have

you know our two three technically

centroid so there's this group here

there's this group here

and then blue is kind of just this group

here it hasn't really touched any of the

points yet

so the next step

three that we do

is we actually go and we recalculate the

centroid so we compute

new centroids

based on the points that we have in all

the centroids

and by that i just mean okay well

let's take the average of all these

points and where is that new centroid

that's probably going to be somewhere

around here right the blue one we don't

have any points in there so we won't

touch and then the screen one

we can put that

hmm probably somewhere over here oops

somewhere over here

right so now if i erase

all of the

previously computed centroids

i can go and i can actually redo step

two

over here this calculation

all right so i'm going to go back and

i'm going to iterate through everything

again and i'm going to recompute my

three centroids so

let's see we're going to take this red

point these are definitely all red right

this one still looks a bit

red

now

this part we actually start getting

closer to the blues

so this one still seems closer to a blue

than a green

this one as well

and

i think the rest would belong to green

okay so now are three centroids or three

sorry our three clusters would be this

is

this and then

this

right

those are our three centroids

and so now we go back and we compute the

new sorry those would be the three

clusters so now we go back and we

compute the three centroids so i'm going

to get rid of this this and this

and now where would this red be centered

probably closer you know to this point

here

this blue might be closer to up here

and then this green would probably be

somewhere

it's pretty similar to what we had

before

but it seems like it'd be pulled down a

bit so probably somewhere around there

for green

all right and now again we go back and

we compute

the distance between all the points and

the centroids and then we assign them to

the closest centroid okay

so

the reds are all here it's very clear

actually let me just circle that

and this

um

it actually seems like this point is

closer to this blue now

so

the

blues seem like they would be maybe

this point looks like it'd be blue so

all these look like they would be blue

now

and the greens would probably be this

cluster right here

so we go back we compute

the uh centroids

bam

this one

probably like almost here bam

and then the green

looks like it would be probably

um here-ish

okay

and now we go back and we compute

the

we

compute the clusters again

so

red still this

blue

i would argue is now this cluster here

and green

is this cluster here okay so we go and

we recompute

the centroids bam

bam

and

you know bam

and now if i were to go

and assign all the points to clusters

again i would get the exact same thing

right and so that's when we know that we

can stop iterating between steps two and

three is when we've converged on some

solution when we've reached some stable

point and so now because none of these

points are really changing out of their

clusters anymore we can go back to the

user and say hey these are our three

clusters

okay and this process

something known as

expectation maximization

this part where we're assigning the

points the closest centroid this is

something this is our

expectation

step

and this part where we're computing the

new centroids

this is our

maximization

step

okay so that's

expectation maximization

and we use this in order to

compute

the centroids assign all the points to

clusters according to those centroids

and then we're recomputing all that over

again until we reach some stable point

where nothing is changing anymore

all right so that's our first example of

unsupervised learning and basically what

this is doing is trying to find some

structure some pattern in the data so if

i came up with another point

you know might be somewhere here i can

say oh looks like that's closer to

um

if this is a b c it looks like that's

closest to cluster b and so i would

probably put it in cluster b

okay so we can find some structure in

the data based on just how

how the points are

scattered relative to one another

now the second unsupervised learning

technique that i'm going to discuss with

you guys something known as principle

component analysis

and the point of principle component

analysis is

very often it's used as a dimensionality

reduction technique

so

let me write that down

it's used for dimensionality reduction

and what do i mean by dimensionality

reduction is if i have a bunch of

features like x1 x2 x3 x4 etc can i just

reduce that down to one

dimension that gives me the most

information about how all these points

are spread relative to one another

and that's what pca is for so pca

principal component analysis

let's say i have

some points

in

the x0 and x1 feature space

okay so

uh these points might be spread you know

something like

this

okay

so

for example if this were

um

something to do with housing prices

right

this here might be x0 might be hey uh

years

since

built right since the house was built

and x1 might be square footage

of the house

all right so like years since built i

mean like right now it's been you know

22 years since a house in 2000 was built

now principal component analysis is just

saying all right let's say we want to

build a model or let's say we want to

you know display something about

our data but we don't we don't have two

axes to show it on

how do we display you know

how do we how do we demonstrate that

this point is a further away from this

point than this point

and we can do that using principle

component analysis so

take what you know about linear

regression and just forget about it for

a second otherwise you might get

confused pca is a way of trying to

find direction in the space

with the largest variance so this

principle component what that means

is basically the component

so some direction

in this space

with the largest

variance

okay

it tells us the most about

our data set without the two different

dimensions like let's say we have these

two different dimensions and somebody's

telling us hey you only get one

dimension in order to show your data set

what dimension like what do we do we

want to project our data onto a single

dimension

all right

so that in this case might be a

dimension that looks something like

this and you might say okay

we're not going to talk about linear

regression okay

we don't have a y value so in linear

regression this would be y this is not y

okay we don't have a label for that

instead what we're doing is we're taking

the right angle projection so all of

these take that's not very visible

but take this right angle projection

onto this line

and what pca is doing is saying okay map

all of these points onto this

one-dimensional space

so the

transformed data set would be here

this one's on the data set so or on the

line so we just put that there

but now this would be our new

one-dimensional data set

okay it's not our prediction or anything

this is our new data set if somebody

came to us said you only get one

dimension you only get one number to

represent each of these 2d points

what number would you give me

this

would be the number

that we gave

okay

this

in this direction this is where our

points are the most spread out

right if i took this plot

and let me actually duplicate this so i

don't have to

rewrite

anything so i don't have to erase and

then redraw anything

um let me get rid of some of this stuff

and i just got rid of a point there too

so let me draw that back

all right

so if this were my original data point

what if i had taken you know

this

to be

the pca dimension

okay

well

i

then would have points

that

let me actually do that in different

color

so if i were to draw a right angle to

this

for every point

my points would look something like this

and so just intuitively looking at these

two different plots this top one and

this one we can see that the points are

squished a little bit closer together

right which means that the variance

that's not the space with the largest

variance

the thing about the largest variance

is that this will give us the most

discrimination between all of these

points

the larger the variance the further

spread out these points will likely be

now

and so that's the that's the dimension

that we should project it on

a different way to actually look at that

like what is the dimension with the

largest variance it's actually it also

happens to be the dimension that

decreases

that minimizes

the residuals so

if we take all the points and we take

the residual from that the xy residual

so in linear regression

in linear regression we were looking

only at this residual the differences

between the predictions right between y

and y hat it's not that

here in principle component analysis

we're taking the difference from

our current point in two-dimensional

space

and then its projected point

okay so we're taking that

dimension

and we're saying

all right how much

you know

how much distance is there between

that projection residual and we're

trying to minimize that for all of these

points

so that actually equates to

this largest variance dimension

this dimension here

the pca dimension

you can either look at it as minimizing

minimize

let me get rid of this

the projection residuals so that's the

stuff in orange

or two

maximizing the variance

between the points

okay

and we're not really going to talk about

you know the method that we need in

order to calculate out the principal

components or like what that projection

would be

because you will need to understand

linear algebra for that especially

um eigenvectors and eigenvalues which

i'm not going to cover in this class

but that's how you would find the

principal components okay

now with this two-dimensional data set

here sorry this one-dimensional data set

we started from a 2d data set and we

now boil it down to one dimension well

we can go and take that dimension and we

can do other things with it

right we can like if there were a y

label then we can now show x versus y

rather than x 0 and x 1

in different plots with that y now we

can just say oh this is a principal

component and we're going to plot that

with the y or for example if there were

a hundred different dimensions and you

only wanted to take

five of them well you could go and you

could find the top five pca dimensions

and

that might be a lot more useful to you

than 100 different feature factor values

right

so that's principle component analysis

again we're taking

you know certain data that's unlabeled

and we're trying to

make some sort of estimation

like some guess about its structure

from

that original data set if we wanted to

take you know a 3d thing so like a

sphere

but we only have a 2d surface to draw it

on

well what's the best approximation that

we can make oh it's a circle right pca

is kind of the same thing it's saying if

we have something with all these

different dimensions but we can't show

all of them how do we boil it down to

just one dimension how do we extract the

most information

from that multiple dimensions

and that is exactly either you minimize

the projection residuals or you maximize

the variance and that is pca so we'll go

through an example of that now finally

let's move on to implementing the

unsupervised learning part of this class

here again i'm on the uci machine

learning repository and i have a seeds

data set where

you know i have a bunch of kernels that

belong to three different types of wheat

so there's comma rosa and canadian

and

the different um features that we have

access to are you know geometric

parameters of those weak kernels so the

area perimeter compactness

length width

with asymmetry and the length of the

kernel groove

okay so all these are real values which

is easy to work with and what we're

going to do is we're going to try to

predict

or i guess we're going to try to cluster

the different varieties of the wheat

so let's get started i have a colab

notebook open again oh you're gonna have

to you know go to the data folder

download this

and

let's get started

so the first thing to do is to

import our seeds data set

into our colab notebook

so i've done that here

okay and then

we're going to import all the classics

again so pandas

um and then i'm also going to import

seaborne because i'm going to want that

for this specific class

okay

great so now our columns that we have in

our seed data set are the area

the perimeter

um the compactness

the length

with

asymmetry

groove

lengths i mean i'm just going to call it

group and then the class right the weak

kernels class so now we have to import

this um

i'm going to do that using pandas read

csv

and

it's called seeds data.csv

so

i'm going to turn that into a data frame

and the names are equal to the columns

over here

so what happens if i just do that

oops what did i call this seeds

dataset.text

all right

so if we actually look at our data frame

right now

you'll notice something funky okay and

here you know we have all the stuff

under area and these are all our numbers

with some dash t

so the reason is because we haven't

actually

told pandas what the separator is which

we can do

like this and this t that's just a tab

so in order to ensure that like all

white space gets recognized as a

separator

we can actually

this is for like a space so any spaces

are going to get recognized as data

separators so if i run that

now are um

this you know this is a lot better okay

okay

so now let's actually go and like

visualize this data so

what i'm actually going to do is plot

each of these against one another so

in this case pretend that we don't have

access to the class right pretend that

so this class here i'm just going to

show you in this example that like hey

we can predict our classes using

unsupervised learning

but for this example in unsupervised

learning we don't actually have access

to the class

so i'm going to just try to plot these

against one another and see what happens

so for some i in range

you know the columns minus one because

the class is in the columns

and i'm just going to say for j in range

so take everything from i

onwards you know so i like the next

thing after i

uh until

the end of this so this will give us

basically a grid

of all the different like combinations

and our x label is going to be

columns i our y label

is going to be the columns

j so those are our labels up here

and i'm going to use

seaborne this time

and i'm going to say

scatter my data so our x is going to be

our x label

our y is going to be our y label

um

and

our data is going to be the data frame

that we're passing in

so what's interesting here is that we

can say hue

and what this will do is say

like if i give this class it's going to

separate the three different classes

into three different queues so now what

we're doing is we're basically comparing

the area and the perimeter or the area

and the compactness

but we're going to visualize you know

what classes they're in

so let's go ahead and i might have to

show

so

great

so basically we can see perimeter and

area we give we get these three

groups

um

the area compactness we get these three

groups

and so on so these all kind of look

honestly like somewhat similar

right so

wow look at this one so this one we have

the compactness and the asymmetry and it

looks like there's not really i mean it

just looks like they're blobs right

sure maybe class three is over here more

but one and two kind of look like

they're on top of each other

okay

i mean there are some that might look

slightly better in terms of clustering

but let's go through some of the some of

the clustering examples that we talked

about and try to implement those

the first thing that we're going to do

is just straight up clustering

so

uh what we learned about was k-means

clustering so from

learn

i'm going to import

uh k means

okay

and

just for the

sake of being able to run

you know any x in any y

i'm just gonna say hey let's use some

x um

what's a good one maybe

i mean perimeter asymmetry could be a

good one

so x could be perimeter y could be

asymmetry

okay

and for this the the x values i'm going

to just extract

those specific

values right

well

let's make a k-means

uh algorithm or let's you know define

this so k means

and in this specific case we know that

the number of clusters

is three so let's just use that

and i'm going to fit this against this x

that i've just defined right here

right

so

um

you know if i create this clusters so

one thing one cool thing is i can

actually go to these clusters and i can

say k-mean dot labels

and it'll get give me

if i can type correctly it'll give me

what its predictions for all the

clusters are

and our actual

oops

not that um if we go to the data frame

and we get the class

and the values from those we can

actually compare these two and say hey

like you know everything in general most

of the zeros that it's predicted

are the ones right and in general the

twos are the twos here and then the

third class one okay that corresponds to

three

now remember these are separate classes

so the labels what we actually call them

don't really matter we can say oh map

zero to one map two to two and map one

to three

okay and our you know our mapping would

do fairly well

but we can actually visualize this and

in order to do that i'm going to create

this cluster

cluster data frame

so i'm going to create a data frame and

i'm going to pass in

um a horizontally stacked

array with x

so my values for x and y

and then um the clusters that i have

here

but i'm going to reshape them so it's 2d

okay

and the columns

the labels for that are going to be x y

and

plus

okay

so

i'm going to go ahead and do that same

seabourn scatter plot

again

where x is x y is y

and now uh the hue is again the class

and the data is now this cluster data

frame

all right so this here

this here is my

um k means

like i guess classes

so k-means kind of looks like this

if i come down here and i

plot you know my original data frame

this is my original

classes with respect to this specific x

and y

and you'll see that honestly like it

doesn't do too poorly

yeah there's i mean the colors are

different but that's fine

um

for the most part it it

gets information of the clusters

right

and now we can do that with higher

dimensions

so with the higher dimensions if we make

x equal to you know all the columns

except for the last one which is our

class

we can do the exact same thing

so here

and

we can

predict

this

uh but now

our columns are equal to

our data frame

columns all the way to the last one and

then with this class actually so we can

literally just say data frame columns

and we can fit all of this

and now

if i want to plot the k-means classes

all right so this was my

uh that's my clustered and my original

so actually let me see if i can

get these on the same page

so yeah i mean pretty similar to what we

just saw but what's actually really cool

is

even something like you know if we

change

so what's one of them where they were

like on top of each other

ah okay so compactness and asymmetry

this one's messy

right so if i come down here

and i say

uh

compactness and asymmetry

and i'm trying to do this in 2d

this is what my scatter plot so this is

what you know

my k means is telling me for these two

dimensions for compactness and asymmetry

if we just look at those two

these are our three classes right and we

know that the original looks something

like this and are these two

remotely

alike no

okay so now if i come back down here and

i rerun this higher dimensions one but

actually

this clusters

i need to

get the labels of the k-means again

okay so if i rerun this

with

higher dimensions

well if we zoom out and we take a look

at these two sure the colors are mixed

up

but

in general there the three groups

are there right this does a much better

job at assessing okay what group

is what

so

for example we could relabel uh the one

in the original class to two

and then we could make

sorry okay this is kind of confusing but

for example if

this light pink were projected onto

this darker pink here and then this dark

one was actually the light pink and this

light one was this dark one then you

kind of see like these correspond to one

another right like even these two up

here

are the same classes all the other ones

over here which are the same in the same

color

so you don't want to compare the two

colors between the plots you want to

compare which points are in what colors

in each of the plots

so

that's one cool application so this is

how

k-means

functions it's basically taking all the

data sets and saying all right

where are my clusters given these pieces

of data and then the next thing that we

talked about

is pca so pca we're reducing the

dimension but we're

mapping

all these like you know seven dimensions

i don't know if there are seven i made

that number up but we're mapping

multiple dimensions into a lower

dimension number

right and so let's see how that works

so from sklearn

decomposition i can import pca and that

will be my pca model

so if i do pca um

component so this is how many dimensions

you want to map it into and you know for

this exercise let's do two

okay so now i'm taking the top two

dimensions

and my transformed x

is going to be pca dot fit transform

and the same x that i had up here okay

so all the other all the values

basically

uh area perimeter compactness length

with asymmetry groove

okay

so let's run that

and we've transformed it so

let's look at

what the shape of x used to be so there

okay so 7 was right i had 210 samples

each seven

seven features long basically

and now my transformed

x

is 210 samples but only of length two

which means that i only have two

dimensions now that i'm plotting

and we can actually even take a look at

you know

the first five things

okay so now we see each each one is a

two dimensional point each sample is now

a two dimensional point

in our new

um

in our new dimensions

so

what's cool is i can actually scatter

these

zero

and transformed

x

so i actually have to

take the columns here

and if i show that

basically we've just taken this like

seven dimensional

thing and we've made it into a single or

i guess two a two dimensional

representation so that's a point of pca

and

actually let's go ahead and do the same

clustering exercise as we did up here

if i take the

k-means

this pca data frame i can let's

construct

data frame out of that

and

the data frame is going to be h stack

i'm going to take this transformed x

and the clusters

dot reshape so actually instead of

clusters i'm going to use uh k-means dot

labels

and i need to reshape this

so it's 2d so we can do the h stack

um

and for the columns i'm going to set

this

to pca1

pca2

and class

all right

so now if i take this

i can also do the same for the truth

but instead of the k means labels i want

from the data frame the original classes

and i'm just going to take the values

from that

and so now i have

a

data frame for the k-means with pca and

then a data frame for the truth with

also the pca

and i can now plot these similarly to

how i plotted these up here

so let me actually take these two

instead of the cluster data frame i want

the

uh this is the k means

pca data frame

this is still going to be class but now

x and y are going to be the two pca

dimensions

okay

so these are my two pca dimensions and

you can see that

you know they're

they're pretty spread out

and then here

i'm going to go to my truth classes

again it's pca1 pca2 but instead of

k-means this should be truth pca data

frame

so you can see that like in the truth

data frame

along these two dimensions

we actually are doing fairly well in

terms of separation right it does seem

like this is slightly more separable

than

the other like dimensions that we had

been looking at up here

so that's a good sign

um

and up here you can see that hey some of

these correspond to one another i mean

for the most part our algorithm our

unsupervised clustering algorithm

is able to give us is able to spit out

you know what the proper

uh labels are i mean if you map these

specific labels to the different types

of kernels but for example this one

might all be the comic kernels and same

here and then these might all be the

canadian kernels and these might all be

the canadian kernels

so it does struggle a little bit with

you know where they overlap

but for the most part our algorithm is

able to find the three different

categories

and do a fairly good job at predicting

them without without any information

from us we haven't given our algorithm

any labels so that's the gist of

unsupervised learning

i hope you guys enjoyed this course i

hope you know a lot of these examples

made sense um if there are certain

things that i have done

and you know you're somebody with more

experience than me please feel free to

correct me in the comments and we can

all as a community learn from this

together so thank you all for watching