Original subtitles

One of the most important steps

in building data intensive apps is to actually model

all this data in MongoDB.

And so that's what we're gonna talk about in this lecture.

So it's really crucial that you follow it

through even at first its a lot to take in. All right.

Anyway, lets now get started.

Now, data modeling is probably a very new concept to you.

So before we start;

lets make clear what we're actually gonna talk about.

So, data modeling is the process of taking unstructured data

generated by a real world scenario

and then structure it into a logical data model

in a database.

And we do that according to a set of criteria

which we're gonna learn about in this video.

For example; lets say that we want to design

an online shop data model.

There will be initially a ton of unstructured data

that we know we need.

Right.

Stuff like products, categories,

customer's orders, shopping carts, suppliers.

And so on and so forth.

Our goal with data modeling is to then structure

this data into a logical way.

Reflecting the real-world relationships

that exists between some of these data sets.

A bit like you can see in this example.

And this is of course just a kind of imaginary situation

but you get the idea.

Right.

Now, many backend developers say that data modeling

is where we have to think the most.

That its the most demanding part of building

an entire application.

Because it really is not always straight-forward.

And sometimes there are simply no right answers.

So not just one unique correct way of structuring the data.

But anyway I will do my best to lay down the process

in this video.

And for that we're gonna go through four steps.

So in the first step; we learned about how to identify

different types of relationships between data.

Then we're gonna understand the difference

between referencing or normalization

and embedding or denormalization.

In the next and most important step;

I will show you my framework for deciding

whether we should embed documents

or reference to other documents

based on a couple of different factors.

Also, we have to quickly talk about

different types of referencing.

Because that's important if that is the type of design

that we choose for our data.

So this is gonna be in fact a quite theoretical lecture.

But also an absolutely essential one for your progress

as a back-end developer.

Because the way we design data so the way we model our data

can make or break our entire application.

And there will be a lot of examples along the way

to make this process easier.

All right.

And the first thing that we are gonna talk about

is the different types of relationships

that can exist between data.

So there are three big types of relationships.

One to one, one to many, and many to many.

And I'm gonna use a movie application

as an example in this slide.

Okay?

So first a one to one relationship

between data is basically

when one field can only have one value.

So in our movie application example; one movie only ever

have one name.

And so this is a simple example

of a one to one relationship.

But these relationships are not really that important

in terms of data modeling.

Now the most important relationships

are the one to many relationships.

And they are so important that in MongoDB

we actually distinguish between three types

of one to many relationships.

One to a few, one to many, and one to a ton or to a million

or something like that. So the difference here is based

on the relative amount of the many. All right.

So an example to a one to a few relationship is that

one movie can win many awards but actually just a few.

So movie is not gonna win a thousand awards

but it can win some.

And so this is a typical one to few relationship.

So you see that in general a one to many relationship

means that one document can relate to many other documents.

Now this might look a bit abstract without the JSON data

but that's actually the purpose here.

I just wanna show you a conceptual overview

of these different types of relationships.

Anyway, any one to many relationship

one document can relate to hundreds or thousands

of other documents.

For example; one movie can have thousands of reviews

in our application.

And so this not really a one to few

but one to many relationship. Okay?

And finally we have the one to ton relationship.

Imagine we wanted to implement some logging functionality

in our app. So basically to know exactly what's going on

on our server.

This logs can then easily grow to millions of documents.

And so this is a very typical example

of a one to tons a relationship.

And the difference between many and a ton is of course

a bit fuzzy but just think that if something can grow

almost to infinity then its definitely

a one to a ton relationship.

So again the one to many relationships

are the most important ones to know.

By the way; in relational databases

there is just one to many without quantifying

how much that many actually is.

In MongoDB databases though

it is an extremely important difference.

Because its one of the factors that we're gonna use

to decide if we should denormalize or normalize data

as you will learn a bit later.

Anyway, the less type of relationship is the many to many

where one movie can have many actors.

But at the same time one actor can play in many movies.

And so here the relationship basically

goes in both directions.

Where before in the other types

it was only in one direction.

For example one movie can have many reviews

but one specific is only for that one movie. Right.

And the same goes for the awards.

So one specific award like for the best actor

goes to only one movie not multiple ones.

But with movies and actors it is indeed different.

So again one movie stars many actors

but one actor plays many movies

and so its a many to many relationship.

Okay.

So keep all this in mind as we now move forward

in this lecture.

And probably the most important aspect that we need to learn

about MongoDB databases is referencing

and embedding two datasets.

And we actually already talked a little bit

about this before but lets review it here

and go a bit deeper also.

So each time we have two related datasets

we can either represent that related data in a reference

or normalized form or in an embedded or denormalized form.

And I keep using the two related terms together

like referencing and normalizing

because you will see them both being used

and so its important that you know all of them.

Anyway, in the referenced form we keep two related

datasets and all the documents separated.

So again all the data is nicely separated

which is exactly what normalized means.

So continuing, the movie database example from before

we would have one movie document and one actor document

for each actor. Now how would we then make the connection

between movie and the actors so that later in our app

we can show which actors played in a particular movie.

Because if they are all completely different document

the movie has no way of knowing about the actors. Right.

Well that's where the IDs come in.

So we use the actor IDs in order to create references

on the movie document.

Effectively connecting movies with actors.

So you see that in a movie document we have an array

where we stored the IDs of all the actors

so that when we request data about a certain a movie

we can easily identify its actors. Does that make sense?

Now this type of referencing is called child referencing

because its the parent in this case the movie

who references its children. In this case the actors.

So we're really creating some sort of hierarchy here. Right.

Now there is also parent referencing

and we are gonna talk about that a bit later.

And by the way in relational databases; all data is always

represented in normalized form like this.

But in a no sequel database like MongoDB

we can denormalize data into a denormalized form

simply by embedding the related document

right into the main document.

So now we have all the relevant data about actors

right inside in one main movie document without the need

for separate documents, collections, and IDs.

So again, if we choose to denormalize or to embed our data

we will have one main document containing all the main data

as well as the related data. All right.

And the result of this is that our application

will need to fewer queries to the database.

Because we can get all the data

about movies and actors all at the same time

which will of course increase our performance.

Now the downside here is of course

that we can't really query the embedded data on its own.

And so if that's a requirement for the application

you would have to choose a normalized design

and since we're talking about pros and cons

of the denormalized form; lets do the same

about the normalized design.

And basically its kind of the opposite

of what we just talked about.

So there is an improvement in performance

when we often need to query the related data on it's own

because we then can just query the data that we need

and not always movies and actors together.

But on the other hand; when we need to actually query

movies and actors together we then are gonna need

many queries to the database.

So first the query for the movie and then from there

we will also need a query for the actor

and that is of course works for performance.

So when designing your database; this is the kind of stuff

that you need to keep in mind. All right.

And now just as a side note; we could of course begin

our thought process with denormlized data

and then come to the conclusion

that its best to actually normalize the data.

So when thinking about our data model

this way of organizing data works of course in both ways.

Now, how do we actually decide if we should

normalize or denormalize the data?

Well that's exactly what we're gonna learn next.

So when we have two related datasets; we have to decide

if we're gonna embed the datasets or if we're gonna

keep them separated and reference them

from one dataset to the other.

And I kind of developed this decision framework

which I'm gonna show you where we use three criteria

to take that decision.

First we look at the type of relationships

that exists between datasets.

Second we try to determine the data access pattern

of the dataset that we want to either embed or reference.

And this just means to analyze how often data is read

and written in that dataset.

Then we also look at something that I call data closeness.

And data closeness is term that I actually just made up

but what it means is how much the data is really related

and how we want to query the data from the database.

And this will make more sense

when we talk about it in a moment.

Now to actually take the decision; we need to combine

all of these three criteria

and not just use one of them in isolation.

So for example; just because criteria number one

says to embed it doesn't mean that we don't need to look

at the other two criteria.

All right and lets start with the relationship type.

So usually when we have one to few relationship

we will always embed the related dataset

into the main dataset just like we learned

in the last slide. Right.

Now in a one to many relationship; things are a bit

more fuzzy so its okay to either embed or reference.

In that case we will have to decide

according to the other two criteria.

Now on the other hand, on a one to a ton

or a many to many relationship we usually always reference

the data. That's because if we actually did embed

in this case we could quickly create way too large document.

Even potentially surpassing the maximum of 16 megabytes.

And so the solution for that is of course referencing

or normalizing the data. And as a quick example;

lets say that in our movie database example

we have around 100 images associated to each movie.

So we could say its a one to many relationship

but are we gonna embed the dataset or should we rather

reference them here. Well we don't really know.

So lets take a look at the other two criteria.

So the second one is about data access patterns

where its just a fancy description

for evaluating whether a certain dataset

is mostly written to or mostly read from.

So if the dataset that we're deciding about is mostly read

and the data is not updated a lot

then we should probably embed that dataset.

So a high read/write ratio just means

that there is a lot more reading than writing.

And a again, a dataset like that is a good candidate

for embedding.

The reason for this is that by embedding we only need

one trip to the database per query.

While for referencing we need two trips. Right.

So if we embed data that is read a lot;

in each query we save one trip to the database

making the entire process way more performant.

So I think that our movie image example

would actually be a good candidate for embedding.

Because once the 100 image are saved to the database

they are not really updated anymore

because there is not really anything to update

about an image. Right, so its all about reading

and therefore based on this criteria

we would embed the imaged documents.

Now on the other hand, if our data is updated a lot

then we should consider referencing or normalizing the data.

That's because its more work for the database engine

to update and embed a document

than a more simple standalone document.

And since our main goal is performance;

we just normalize the dataset.

In our example lets say each movie has many reviews

and each review can be marked as helpful by the user.

So each time someone clicks on this review was helpful

in our application. We need to update

the corresponding document.

And this means that the data can change all the time

and so this is a great candidate for normalizing.

Again because we don't want to be querying the movies

all the time if all we really wanna update is the reviews

by marking them as helpful.

Okay, does that make sense?

And finally the last criteria I call data closeness;

which is just like a measure for how much the data

is related. So if the two datasets really

intrinsically belong together then they should

probably be embedded into one another.

In our example; all users can have many email addresses

on their account and since they are so intrinsically

connected to the user, there is no doubt emails

should be embedded into the document.

Now if we frequently need to query both of datasets

on their own then that's a very good reason

to normalize the data into two separate datasets.

Even if they are closely related.

So imagine that in our app we have a quiz

where users have to identify a movie based on images.

This means that we're gonna query a lot of images

on their own.

So without necessarily querying for the movies themselves.

And so if we apply this third criteria;

we come to the conclusion that we should actually normalize

the image dataset. All right.

Because again if we implement this quiz functionality;

images are gonna be queried on their own all the time.

So, all of this shows that we should really look

all the three criteria together

rather than just one of them in isolation.

Because that might lead to less optimal decisions.

And I say less optimal instead of wrong

because they are not really completely right

or completely wrong ways of modeling our data.

There are no hard rules; these are just like guidelines

that you can follow to find the probably

most correct way of structuring your data.

But again, it's hard to be really really wrong.

Okay?

Now, lets say that we have chosen to normalize

our datasets.

So in other words to reference data.

Then after that we still have to choose

between three different types of referencing.

Child referencing, parent referencing

and two-way referencing.

So the first type is child referencing.

Which is the referencing type I actually showed you before.

Okay?

And lets not take the error logging example

that I mentioned earlier. Where we could potentially

have millions of locked documents.

So in child referencing; we basically keep references

to the related child documents in a parent document.

And they are usually stored in an array.

So you see that each log has an ID

and then in the app document there is that array

with all of these IDs. Right?

However, the problem here is that this array

of IDs can become very large if there are lots of children.

And this is an anti-pattern in MongoDB.

So something that we should avoid at all costs.

Also, child referencing makes it

so that parents and children are very tightly coupled.

Which is not always ideal. But that's exactly

why we have parent referencing.

So in parent referencing; it actually works

the other way around.

So here in each child document we keep a reference

to the parent element.

Therefore the name parent referencing.

In this example the app ID is 23

and so in each log there is the app field

with the 23 ID in it.

So that the child always knows its parent.

And so in this case the parent actually knows nothing

about the children.

Not who they are and not how many they are.

So, they are way more isolated and more standalone.

In that, it can sometimes be beneficial.

So which of these two types is actually better

for this data relationship.

And remember how I said that there

could be millions of logs and so lets suppose

there is two million logged documents.

In a case of child referencing, that would mean

that there are two million ID references

in the app document.

Right? Now also remember how I said that

there is 16 megabyte limit on documents.

So if we kept adding and adding these child IDs

into the array on the parent; then we would

pretty quickly hit that 16 megabytes limit

that each Bson document can hold.

Simply because that array will grow so much.

So that's not really gonna work.

Is it?

On the other hand with parent referencing

that problem is not gonna happen.

We will simply have two million locked documents

just like before but each of them holds ID of its parent.

But there is no array that will grow indefinitely

and therefore parent referencing

would be best solution here.

So the conclusion of all this is that in general

child referencing is best used

for one to a few relationships. Where we know before hand

that the array of child documents won't grow that much.

On the other hand, parent referencing is best used

for one to many and one to a ton relationships

like this one. Okay?

So again always keep in mind that one of the most

important principals of MongoDB data modeling

is that array should never be allowed to grow indefinitely.

In order to never break that 16 megabyte limit.

We also don't want to send our users an array

with thousands of IDs each time

they request a parent dataset. Okay?

So did this logic make sense to you?

Then lets move on to third type of referencing

which is two-way referencing.

And this time with the movie and actor example

I showed you when we talked about

many to many relationships. Remember that?

So again, each movie has many actors

and each actor plays in many movies.

And so that's a typical many to many relationship.

And we usually use this two-way referencing to design

many to many relationships. And it works like this;

in each movie we will keep references to all the actors

that star in that movie. So a bit like in child referencing.

However and at the same time in each actor

we also keep references to all the movies

that the actor played in.

So movies and actors are connected in both directions.

In therefore the name two-way referencing.

And this makes it really easy to search

for both movies and actors completely independently.

While also making it easy to find the actors

associated to each movie and the movies associated

to each actor.

(deep breath)

This was quite a long lecture indeed.

With a lot of new concepts and principals

and guidelines to remember.

So in order to help you with that; here goes a quick

summary and some more general guidelines that you can

take a look at when you need it.

So the most important principal is: structure your data

to match the ways that your application queries

and updates data.

Or in other words: identify the questions that arise

from your application's use cases first, and then model

your data so that the questions can get answered

in the most efficient way.

For example; when I need to query movies and actors

always together or are there scenarios where I only

query movies or only actors.

That kind of questions is what your data model

will be based on.

In general, always favor embedding unless there is a good

reason not to embed.

Especially on one to a few and one to many relationships.

Next up, a one to a ton or a many to many relationship

is usually a good reason to reference instead of embedding.

Also, favor referencing when data is updated a lot

and if you need to frequently access a dataset on its own.

Use embedding when data is mostly read but rarely updated

and when two dataset belong intrinsically together.

Don't allow arrays to grow indefinitely.

Therefore, if you want to normalize; use child referencing

for one to many relationships and parent referencing

for one to a ton relationships.

And finally use two-way referencing

for many to many relationships.

All right?

And that pretty much sums it up.

I would actually recommend you watching this video

twice if you can, just because of how important

this material really is. All right?

Anyway, see you in the next video.