All language subtitles for 12. The Naive Forecast and the Importance of Baselines

In this lecture, we are going to discuss a method of forecasting called the naive forecast before we

discuss what the new forecast is.

Let's talk about the importance of establishing baselines in machine learning.

The purpose of a baseline is to have a relevant point of comparison.

To give you an example, suppose that you went to your professor tomorrow and say, hey, I just made

a great discovery.

I built a model using deep learning and I got 99 percent accuracy.

Unfortunately, the statement is meaningless because you don't have a context of a baseline.

Your professor might say, but student, we already know that a simple linear model achieves 100 percent

accuracy.

In this case, your model is worse than what currently exists.

And furthermore, it's also slower.

So how can we make sure that we don't make this mistake?

The answer is using a baseline.

You'll notice that when you're reading machine learning papers, they often compare their results against

the existing state of the art.

Now, it's important to realize that you don't have to constantly try to beat the state of the art.

In fact, that's kind of antithetical to science because all you're doing is chasing numbers.

A good a real life example of that is when students study very hard to memorize their notes so that

they can ace their exam without truly understanding the material or how it's applied when you're chasing

only numbers.

Sometimes it's easy to forget why those numbers matter in the first place.

Anyway, to get back to the point.

It's not always true that you want to compare to the state of the art.

Sometimes if you just want to test whether or not something is working, for example, as a proof of

concept, then the simplest model possible should suffice.

In Time series analysis, the simplest model possible happens to be the naive forecast, what is the

new forecast?

Well, it's just another name for something we've already discussed that is to copy the previous known

value forward in time.

One interesting phenomena that happens in a Time series analysis is that a lot of bad models end up

looking like naive forecasts.

When you look at the model predictions from afar, they seem to be pretty close to the true values.

But when you zoom in, you can see that they seem pretty close only because they're just copying the

previous value or close to it.

A really bad situation is this suppose that you fit a model and you say, aha, I beat the naive forecast

because my accuracy is better than the Niyi forecast.

But of course, you have to ask, are you talking about the accuracy on the sample data or the out of

sample data?

Note that in machine learning we often refer to these as the training data and the test data.

So I would use these terms interchangeably.

Just be aware.

Well, one common mistake is that people believe because they got good accuracy on their in sample data,

that the same will be true for their out of sample data.

And also, quite commonly, it turns out that the opposite is true.

You might beat the night forecast on the sample data, but on the out of sample data, the naive forecast

beats you.

Why does this happen?

It happens when your model over fits to the noise in the training data, but doesn't actually generalize

to the true underlying pattern in the Time series if one exists.

So that's why it's really important to compare your model to a baseline such as the naive forecast.

It's not good enough to say I got 80 percent classification rate on my train and 75 percent on my test

set.

If some other method achieves 70 percent on the test, then your model isn't as good.

There's one application I've seen a lot, which I think is a very interesting consequence of the times

that we live in today.

There are marketers everywhere on the Internet.

The name of the game is SEO search engine optimization.

Everyone is trying to get clicks for their blog or to get people to sign up for their course or whatever.

And of course, one of the obvious topics that many beginners care about is stock price prediction.

Now, if you're taking this course, then you know better.

But let's continue the story.

So imagine what happens.

You take one of the most popular topics that would appeal to someone who doesn't know about finance

like stock predictions.

You take one of the most popular machine learning algorithms that would appeal to someone who may not

know a lot about machine learning, but has heard many buzzwords like LSM.

LSM has been a very popular model for a sequence model.

And then what do you do?

Well, you combine these two, you make a blog post on stock predictions with LSD.

In fact, many people have done so.

It's obviously a very appealing idea.

No one would blame you for clicking on an article or buying a course that claims to be able to do this.

I won't name any names, but there are some courses that make the very mistake I'm talking about in

this lecture.

Maybe you're watching this and you know where you've seen something like this yourself.

There is another even worse mistake that these marketers make.

But we'll discuss that later when we talk about forecasting in general for a time series.

Models like Arima, if you want, you're encouraged to skip over to the forecasting lecture so we can

continue this discussion.

To get back to the naive forecast, let's recall what we learned about random walks.

As you recall, a random walk is where on every step of a time series I flip a coin or pick a number

from a distribution.

And that number is added to my current position in order to go to the next position.

If my noise distribution is a zero centered Gaussian with variants Sigma Square, which is not unreasonable,

then the best forecast is the naive forecast.

I can do no better than predicting the last known value.

Another way to think about this is if you build a model that you think is good but it cannot beat the

night forecast, then it might suggest that your model is actually worse than a random walk model.

In other words, a random walk model describes the data better than your model.