All language subtitles for 021 Aggregation Pipeline_ Matching and Grouping_Downloadly.ir_en

Welcome back.

In this video and the next one

I want to introduce you to the MongoDB

aggregation pipeline which is an extremely powerful

and extremely useful MongoDB framework

for data aggregation.

And the idea is that we basically

define a pipeline that all documents

from a certain collection go through

where they are processed step by step

in order to transform them into aggregated results.

For example, we can use the aggregation pipeline

in order to calculate averages

or calculating minimum and maximum values

or we can calculate distances even,

and we can really do all kinds of stuff.

It's really amazing how powerful

this aggregation pipeline is.

All right, but enough talk.

Let's now actually start using the aggregation pipeline

and I'm gonna create a new header function here

because later on we will then define a new route

and then use that function for that.

But that new route, I'm gonna create it later.

So I want to create a function here that's gonna

calculate a couple of statistics about our tours.

So I'm gonna call this one getTourStats.

Okay, so Stats for statistics,

and it's gonna be an async function

and so a vs code now automatically prefilled off this.

All right, let's create or try catch block here.

Actually, I'm gonna copy this from right up here,

and now in here I will create

a variable called stats.

Now the aggregation pipeline really is a MongoDB feature.

But Mongoose, of course, gives us access to it,

so that we can use it in the Mongoose driver, right?

So using our tour model in order to access

the tour collection, we say

tour.aggregate.

And so the aggregation pipeline is a bit

like a regular query and so using the aggregation pipeline

it's a just a bit like doing a regular query.

The difference here is that in aggregations,

as I already mentioned, we can manipulate the data

in a couple of different steps

and so let's now actually define these steps.

And for that, we pass in an array of so-called stages.

So we pass in an array, and then here we will

then have a lot of stages.

And again the documents then pass through these stages

one by one, step by step in the define sequence

as we define it here.

So each of the elements in this array

will be one of the stages.

And there are a ton of different stages

that we can choose from, but I will just tell you

the most common ones in this lecture

and also in the next one.

But anyway, I now want to take a minute

and take a quick look at the MongoDB documentation

because I think we didn't do that yet.

So MongoDB and we already know this website.

So on our MongoDB website we go here to learn,

then documentation and then in here we select

the MongoDB Manual and this documentation is huge.

It really, it has a lot of stuff in here

and if you were gonna go through it all,

well, then you'd become really a MongoDB master.

So it gives you a nice introduction to MongoDB,

you have the installation here.

You have the CRUD Operations, so a nice overview

about a lot of the stuff

that we already learned before, right?

Then you have stuff about the aggregation

and in particular, the aggregation pipeline

which we're gonna talk about now,

but what I want to really show you here

is actually this reference that we have at the bottom

because in the reference we have here for example

the operators and that's where we have

a lot of important stuff that we've been using all the time.

So let's take a look here at the query operators,

for example and here you see, we actually have

these operators that we already used.

So for example, greater than or equal,

or less than and equal, but if someday

in your own applications you need some other operator,

well, now you know where you can find them.

Now of course, it's also always very helpful

to just do a quick Google search, probably find

some stack overflow post about it,

but believe me, many times it's easier

and faster to just quickly jump into the documentation

and quickly search for whatever you're needing.

You just need to know how the documentation works

and once you know that, it's then gonna be quite easy

for you to find what you're looking for, all right?

Here you also have some logical operators

for example, the OR operator we already used.

Again, really a lot of stuff, geospatial operators

and we're gonna use some actually, later,

and yeah, really a lot of stuff.

Then what I wanted to show you after this lecture actually,

are the aggregation pipeline stages.

So here you have a lot of them

and what we're gonna use in this lecture here is match,

for example and group,

where is that?

Yeah group, that is a very important one,

and so as I mentioned, I'm gonna show you

the most important ones in this lecture and the next one.

But again, if you in some case need something else,

then just go ahead, come here to this documentation

and you will find what you're looking for.

All right?

Anyway, let's now learn how we can actually define

one of these stages, all right?

And I'm gonna start with match.

And match is basically to select

or to filter certain documents.

And so it's very simple.

It's really just like a filter object in MongoDB,

such like we've been using so many times.

So each of the stages is an object

and then here comes the name of the stage.

So this one is the match stage, all right?

And as I mentioned, it's really just a query.

And so let's say that for starters,

we only want to select documents

which have a ratings average greater or equal than 4.5.

So can you do that?

I'm just gonna give you a second to do that here.

So I hope you did it.

So ratings average, should be

greater or equal than 4.5.

And so that's it, that's the first stage.

And usually this match stage is just a preliminary stage

to then prepare for the next stages which come ahead.

So the next one is now the group stage.

So group and then in there

we need to always pass just another object.

So it looks kinda weird with all these objects,

but, yeah, you've seen that before

and MongoDB just works this way.

It's always objects, inside of objects, inside of objects.

And this group here is where the real magic happens

because as the name says,

it allows us to group documents together,

basically using accumulators.

And an accumulator is for example,

even calculating an average.

So if we have five tours, each of them has a rating,

we can then calculate the average rating using group.

And so let's do exactly that right here.

Now the first thing, is we always need to specify

is the id because this is where we're gonna specify

what we want to group by.

For now, we say null here because we want

to have everything in one group so that we can calculate

the statistics for all of the tours together

and not separate it by groups.

We will, a bit later then also group by different stuff,

for example we can group by the difficulty

and we can then, for example calculate the average

for the easy tours, the average for the medium tours

and the average for the difficult tours.

So again, we can group by one of our fields

and that field, we are gonna specify in here,

but for now, as I said, we want to calculate these averages

for all the tours together in one big group.

So in that case we say _id and set it to null.

Now let's actually calculate the average rating.

In order to do that, we simply specify a new field,

so let's simply call it the average rating,

so like this, and this will be

well, the average,

which is yet another MongoDB operator, so this one here.

You will find it in the reference if you look it up.

So this is a mathematical operator calculating

the average and now the name of the field.

And again, I know that this is gonna look very weird,

but in order to specify the field which we want

to calculate the average from, we need to use

the dollar sign, but in quotes here

and then the name of the field.

So ratings average in this case.

And let's also calculate the average price.

So average price, and again, did it wrong here,

and so

the average and then again,

the name of the field and in quotes

and with this dollar sign.

Right, so it looks kind of weird again,

but this is just the way

that the aggregation pipeline works.

So we have the average price.

Let's actually also calculate the minimum price,

so the smallest price, and the largest price.

So minPrice

and from now on, it's gonna all look kind of similar,

so now we use the min operator

and then again the field name like this.

And a comma here and let's now just duplicate this one

in order to calculate the maximum price.

And now, let's actually take a look

at what we already got at this point.

So we now need to actually send out this response,

and so again, I'm just gonna go head

and grab some of these already done responses here.

So the data that we want to send out is called stats.

And now all we need to do is actually add a new route

here in our tour routes and again,

I'm gonna edit here, right at the top.

So router.route

/gettour, or actually just tour-stats, just like this

because the get is already defined in the HTTP method.

So no need to repeat it in the name.

So tourController.getTourStats.

Give it a safe and let's actually test it out now.

So I'm really exciting to test it out now, actually

to see if it works because again it's really an amazing

and really powerful tool

which allows you to do so much stuff.

So tour-stats, send it

and that's not really the result

that we were waiting for, so that's just the pipeline

that we defined, but I already know why that is.

So we did not await the result.

So this, basically, just like a normal query

is gonna return an aggregate object.

So .find is gonna return a query, and .aggregate

is gonna return an aggregate object.

And then only when we await it,

it actually comes back with the result.

So that's also why we defined this function here

as an async function, so that we can then use await

in there and so this is the right place.

Let's try it again.

And here we go!

So great!

We have the average rating of all our tours,

we have the average price which you see is

1563 and then the minimum and the maximum, right?

And from our data, we can actually confirm

that this is true.

So great!

Really, really great.

Now, one more thing that I want to do here

is to actually calculate the total number

of ratings that we have and also the total number of tours.

So we have the average rating here

and let's actually do it before,

so numRatings,

like this

and you can probably guess

that this one is called sum and then the ratingsAverage,

or actually not ratingsAverage, but ratingsQuantity, right?

So that's where the number of ratings is stored

and so the number of ratings, the total,

will be the sum of all of these together.

And now, the last one, is the number of tours

and that one is a bit trickier,

and so that's a nice one to show you here.

So we still want to sum, so to add everything together,

basically, so we still use sum,

but what are we gonna add together?

Well, we basically add one for each document,

and so we say 1, and that's it.

So basically for each of the document

that's gonna go through this pipeline,

one will be added to this num counter.

Let's call it numTours.

And so, yeah, let's test it out now, again.

And indeed we get our nine tours

and we already know that we have nine.

So these are all the statistics for all the tours together,

but let's now take it to the next level.

As I said before, we can now group our results

for different fields.

And let's actually start out with the difficulty.

And so that's quite similar to specifying the fields

down there which is simply the dollar sign,

and then the name of the field.

So $difficulty, give it a safe and that's it.

Send it here, and now we have the statistics

all defined for each of these three difficulties.

So that's really, really amazing.

So this is the part that blows my mind.

It's absolutely fantastic.

I mean, you can start to see all the kinds of stuff,

all the kinds of data manipulation

that you can do using this pipeline, right?

So we really can now start to analyze stuff,

for example, we see that the easiest tours

are the ones who get the poorest ratings

with a 4.6, while the medium tours

are the ones who get the highest ratings.

We can also see that the difficult tours

are the most expensive ones with an average price

of 1997, while the easiest ones are also the cheapest ones

with a price of 1272, right?

We also see that the easiest tours

are the ones which we have the most,

so four easy ones and only two difficult ones, right?

So we can use this, of course,

to get all kinds of insights into our data.

Let's comment this one out here

and let's just try another one,

for example, by ratings, why not,

let's just try out, see what we get.

So for example, we have two tours,

so basically with a rating of 4.5,

we have two tours with 4.7, one tour with 4.8,

and three tours with 4.9.

Oh, and also down here, one tour with 4.6,

and so for example, one insight that we can get here

is that the very expensive tours with this average price

of 1997 get extremely good ratings.

So all kinds of stuff, really that we can do here.

Let's get rid of this one, let's add this one back,

and we can now actually even,

do some operations with this one.

So just for fun, let's put this difficulty to uppercase.

So this of course, is gonna be another object here

and then here in front is the operator.

And so the operator in this case, is called $toUpper.

So just, yet another MongoDB operator.

So check it out, and now it's spelled uppercase.

Great!

So we have the group stage here which is quite complete.

So let's try another one which is a sort stage.

So another object in our array for yet another stage

and this one is called sort.

So sort and then in there, we need yet another object,

and it kind of auto-completed this one, I don't know why,

and so here we can now specify which field we want

to sort this by and let's actually use the average price.

And now here in the sorting we actually need to use

the field names that we specified up here in the group.

We can no longer use the old names because at this point

they are already gone.

They no longer exist.

So at this point, in the aggregation pipeline,

we really already have these results.

So these are now our documents basically.

So if you want to sort by the average price,

then this is the field name we gotta use.

So we can say, the average price

and then we can say 1 for ascending.

So let's try that out.

And yeah, indeed.

That's the lowest one and that's the highest one.

Great!

So we did a lot of stuff.

Let me just show you that we can also repeat stages.

So let's actually do another match here

and this one is really just to show you

that we actually can repeat stages

and I also want to show you another operator.

So let's now select by id and remember that id

is now the difficulty, right?

So we just specified that up here.

So we want the id and now a new operator

that we didn't use yet.

We want it to be not equal to, and let's say easy.

And so, just like this we're gonna select

all the documents that are not easy.

So in this case, the one that says

medium and difficult, right?

So basically, excluding the one that says easy, right?

So again, it's not really useful here

because that will take away a lot of our meaningful data,

but just to show you that we can of course,

match multiple times.

So in this case, we matched once before we actually

did the group and then we matched once we were ready,

doing the grouping.

Now, let's actually keep this here,

out of our aggregation pipeline,

but I'm gonna leave it here,

just so that you keep it as a reference.

Give it another save, give it another try,

and so these are our results

for getting our tour statistics.

So, really nice, really great.

I hope you enjoyed this video

and in the next one, we're actually gonna keep using

the aggregation pipeline to do some more calculations.

And to show you some more different stages.

So keep tuned for that one.