All language subtitles for 6. Pandas Cleaning and selection

Hi, everyone, and welcome in this new video.

In this video, we're going to see the power of pandas for cleaning and do a little transformation in

our data.

So first, we need to import the CAC file attached to this video.

Or you can find on the GitHub named Assets Point.

CAC.

To put it in Google Calendar, you just need to do a drag and drop here.

Then we are going to use pandas to open this CSB file to do it.

We are going to use the Reed Point system function.

Then we need to put.

The path to have access to this sea is defined, so if you walk in the local environment, I assure

you that you have the good puff here.

Then we first I will show you that.

I say I think it's way better.

As we can see, we have a lot of stock price, the volume, etc. and the dates.

And usually it is really better to put the date as the index.

So to do it, we need to specify specific dates equal to and we need to give the name of the index column.

So it is better, but Hugh, I have put a database with a lot of issue.

Too onerous to clean these datasets together.

So first, the first things that we need to fix is that Biton read in chronological order.

So if?

Here we have the date.

We need to have the latest date first.

And here it is not that we want because here we have two thousand twenty and here we have 2000.

So here it is.

Older in descending order, and we want an ascending order.

So

I'm going to fix this using the SALT index function.

And Putin ascending equal to.

So we are fixed this issue now.

We have a lot of missing value.

So we are going to delete all this missing value.

But before we are just going to select some specific Collins to have, though to have a very huge dataset.

So now we are going to delete all this missing value to do it.

It is very simple.

You just have to

apply the drop any function which will delete all the while containing at least one known value.

But this function is more complex than that.

If you want to use all the possibilities of this function, for example, just delete if there or at

least three value or a certain threshold, etc. I will invite you to go on the panda's documentation

because it is equivalent for all this calls.

So

now we can see that we don't have any missing value.

So now we are going to see.

How to reset the index, because sometimes we need to reset the index because dates is not would be

good for some situation, and if we want to reset the index, we just have to use the reset index function

and.

With this function, we can choose to keep the date, for example, all.

To drop it using drop equal to so I have not put this data frame in or viable assets.

So

asset is still with the dates is in index.

So now I will show you a very interesting function to create our own technical indicators.

So to show you this function.

I will create a simple moving average.

So first, we need to select.

At DataFrame.

So we choose the close S&P 500 columns, but we can choose a lot of other column if we want.

Then we applied the warning function, and the warning function doesn't walk alone because the warning

function allows us to do one bit, but we need to apply a function to this warning function and to us

is the mean function.

Now, let me show you the shift function, the shift function is very interesting when you want to create

the percentage of variation.

For example,

when you want to compute the return of our strategy, you have the percentage of variation of the assets

and you have your signal to buy or sell a stock, for example.

But these two columns are in the same day, but it is not true to just multiply these two columns together

because if you think the return of strategy as a percentage of operations, for example, from eight

a.m. to eight p.m. but you take just the decision at eight pm, you have an issue because.

You are not in the markets during the period that you take the percentage of variation.

So I don't know, it's very clear, but it will be very clear when we are going to compute the percentage

of aviation in the next chapter.

But it is important to at least show this function to be a little bit more comfortable with this function

later.

So if I put shift equal one, for example,

we can see that we have just shift the value by one rule if I put 10, for example.

We shift the value with 10 rule, so it is very important to understand what this function does because

it will be the key to do a back this properly.

Now we are going to talk about the group by function, the group buying function is a very interesting

tool, even if we are not going to use it in this courses, but it is very important to know it because

sometimes it can be very useful.

So I will use the group function on my data frame.

I want to group by the column none.

So all the number here.

And then as for the running function, Roubaix doesn't work well and you need to apply something.

And for example, if I put me, I will have.

For one, the mean of all the venue here, associate to the them one, for example, here, if I change

the venue, I will have the mean of this three venue for the number three.

So it is very interesting.

In some way we can do exactly the same with the same function, for example.

So we have four here, 11 plus 10, so 21.

And for the number of three, we have 15 plus 15 plus plus 10.

So 15, 15 percent plus 10 is equal to 40.

So we have the same.

Of the value associate to each member, so it can be very interesting in some case, and we can also

do the standard deviation to have idea of the dispersion of the value from the mean.

So, Hugh, I cannot compute it with my mind quickly, like the Mean or the Sun.

But it is exactly the same thing.

We're going to take the mean.

So this number, for example, for the number three, then we compute the distance between the mean

and 15, the mean and 15 and the mean and 10.

And then we compute the standard deviation and we do exactly the same for the number one, we compute

the difference between the mean and this value, the mean and this value and we compute the standard

deviation.

So it is all for this video.

I will invite you again to play with this because Master Penders is a very necessary skill to master

algorithmic trading.