All language subtitles for 6. Types of Evaluation

Now we've gone through Step 1 into problem definition and data.

It's time for step 3 evaluation every machine learning problem you come across.

We'll have the similar goal of finding insights in data to predict the future in some way an evaluation

metric is a measure of how well a machine learning algorithm predicts the future.

And in this step the question you'll want to answer is what defines success for us.

For example if your problem is to use patient medical records to classify whether someone has heart

disease or not you might start by saying for this project to be valuable we need a machine learning

model with over ninety nine percent accuracy because predicting whether or not a patient has heart disease

is an important task.

So you'll want a highly accurate model and as you could imagine there are different evaluation metrics

for different problems for classification or predicting whether something is one thing or another.

Accuracy precision and recall a common for regression or predicting a number such as how much a car

will sell for.

You'll probably want to minimize how different the number your model predicts to the actual sale price

for this mean absolute error and main square area are common options or for recommendation problems.

You may have thousands of different products to recommend to someone but in reality you only care about

the top 10 recommendations and how well they align to a customer's potential interest.

To measure this you could use precision at K where in our case k is 10 sitting down and thinking about

an evaluation metric.

At the start of a project ensures everyone on it has a similar goal to work towards but it's important

to remember these don't have to be exact either as you find out about more about the data you might

find the evaluation metric changes as the project goes on for now don't worry if you're not too sure

about each of these we'll see different examples of these.

As we build projects as an example of an evaluation metric being used in practice we had a project where

we wanted to use the text from car insurance claims to predict who caused the accident the person submitting

the claim or the other person involved the car insurance company we partnered with wanted at least a

95 percent accurate model to consider the project worth continuing.

This meant the model I was building had to be able to read a car insurance claim and predict with 95

percent accuracy who caused the accident.

This meant it was only allowed to get it wrong 1 out of 20 claims.

Again we'll learn more about evaluation metrics when we get hands on for different projects.

But the thing to remember is as you go you'll start to define a problem like in Step 1 then you'll start

to remember all this is a classification problem I should use accuracy as my evaluation metric to get

an idea of how my model is doing before we go on to Step Four have a think about different things you

measure everyday.

How do you measure them.

Are there different kinds of measurements for different tasks.