Original subtitles

Let's start off with sort of a 30,000 foot view of the elastic stack, and the components within it and

how they fit together. So, elastic search is just one piece of this system.

It started off as basically a scalable version of the Lucene open source search framework, and it

just added the ability to horizontally scale Lucene indici, so we'll talk about shards of elastic

search, and each shard in elastic search is just a single Lucene inverted index of documents, so every shard

is an actual Lucene instance of its own. However, elastic search has evolved to be much more than just

Lucene spread out across a cluster,

it can be used for much more than full text search now, and it can actually handle structure data and

aggregate data very quickly. So, it's not just for search, you can handle structure data of any type and

you'll see it's often used for things like aggregating logs and things like that.

And what's really cool, is that it's often a much faster solution than things like Hadoop or Spark or

Flink; you're actually building in new things into the elastic search all the time, with things like graph

visualization and machine learning that actually make elastic search a competitor for things like Hadoop

and Spark and Flink, only it can give you an answer in milliseconds, instead of in hours.

So for the right sorts of use cases, elastic search can be a very powerful tool and not just for search.

So let's zoom in and see what elastic search is really about. At a low level, it's really just about handling

JSON requests, so we're not talking about pretty UIs or graphical interfaces, when we're

just talking about elastic search itself, we're talking about a server that can process JSON requests

and give you back JSON data, and it's up to you to actually do something useful with that.

So for example, we're using curl here to actually issue an arrest request with a GET verb for a given index

called "tags", and we're just searching everything that's in it and you can see the results come back in

JSON format here, and it's up to you to pass all this.

So for example, we did get one result here called, for the movie, "Swimming to Cambodia", which has a given

user I.D. and a tag of "Cambodia".

So if this is part of a tags index that we're searching, this is what a result might actually look like.

So, just to make it real,

that's a sort of output you can expect from elastic search itself. But, there's more to it than just elastic

search, there's also Kibana, which sits on top of elastic search and that's what gives you a pretty web

UI.

So if you're not building your own application on top of elastic search or your own web application, Kibana

can be used just for searching and visualizing what's in your search index graphically, and it can do

very complex aggregations of data, it can graph your data, it can create charts and it's often used to do

things like log analysis - so if you're familiar with things like Google Analytics, the combination of

elastic search and Kibana can be used as sort of a way to roll your own Google Analytics at a very large

scale.

Let's zoom in and take a look at what it might look like.

So here's an actual screenshot from Kibana looking at some real log data.

You can see there's multiple dashboards you can actually look at that are built into Kibana

and this lets you visualize things like; where the hits on my web site are coming from, and where are the error

response codes and how are they all broken down, and what's my distribution of URLs, whatever you

can dream up.

So, there are a lot of specialized dashboards for certain kinds of data, and it kind of brings home

the point that elastic search is not just for searching text anymore; you can actually use it for aggregating

things like apache access logs, which is what this view in Kibana does.

But you can also use Kibana for pretty much anything else you want to. Later on this course we'll use

it to visualize the complete works of William Shakespeare for example, and you can see how it can also

be used for text data as well; it's a very flexible tool and a very powerful UI. We can also have something

called logstash in the Beats framework, and these are ways of actually publishing data into elastic

search, in real time, in a streaming format.

So if you have for example, a collection of web server logs coming in that you just want to feed into

your search index over time automatically, FileBeat can just sit on your web servers and look for new

log files and parse them out, structure them in the way that elastic search wants, and then feed them into

your elastic search cluster as they come in. Logstash does much the same thing, it can also be used to

push data around between your servers and elastic search, but often it's used sort of an intermediate step,

so you have a very lightweight FileBeat client that would sit on your web servers, logstash would accept

those and sort of collect them and pool them up for feeding into elastic search over time. But it's

not just made for log files, and it's not just made for elastic search and web servers either, these are

all very general purpose systems that allow you to tie different systems together and publish data

to wherever it needs to go, which might be elastic search, might be something else, but it's all part of the

elastic stack still. But, it can also collect data from things like Amazon S3 or Kafka, or pretty

much anything else you can imagine, databases, and we'll look at all of those examples later in this course.

Finally, another piece of the elastic stack is called X-pack.

This is actually a paid add on offered by elastic.co, and it offers things like security and alerting

and monitoring and reporting, features like that.

It also contains some of the more advanced features that are just starting to make it into elastic search

now, such as machine learning and graph exploration, so you can see that with X-Pack, elastic search starts

to become a real competitor for much more complex and heavy weight systems like Flink and Spark.

But that's another piece of the elastic stack when we talk about this larger ecosystem. You can see

here that there are free parts of X-Pack, like the monitoring framework, that lets you quickly visualize

what's going on with your cluster, you know,

what's my cpu utilization system load?

How much memory you have available?

Things like that.

So, when things start to go wrong with your cluster,

this is a very useful tool to have for understanding the health of your cluster.

So that's it at a high level, the elastic stack. Obviously elastic search can still be used for a powering

search on a web site like Wikipedia or something, but with these components it can be used for so much

more.

It's actually a larger framework for publishing data from any source you can imagine, and visualizing

it as well through things like Kibana,

and it also has operational capabilities through X-Pack; so that is the elastic stack at a high level.

Lets dive in more into elastic search itself and learn more about how it works.

Let's start off with sort of a 30,000 foot view of the elastic stack, and the components within it. And

how they fit together. So, elastic search is just one piece of this system.

It started off as basically a scalable version of the Lucene open source search framework, and it

just added the ability to horizontally scale Lucene indici, so we'll talk about shards of elastic

search, and each shard in elastic search is just a single Lucene inverted index of documents, so every shard

is an actual Lucene instance of its own. However, elastic search has evolved to be much more than just

Lucene spread out across a cluster,

it can be used for much more than full text search now, and it can actually handle structure data and

aggregate data very quickly. So, it's not just for search, you can handle structure data of any type and

you'll see it's often used for things like aggregating logs and things like that.

And what's really cool, is that it's often a much faster solution than things like Hadoop or Spark or

Flink; you're actually building in new things into the elastic search all the time, with things like graph

visualization and machine learning that actually make elastic search a competitor for things like Hadoop

and Spark and Flink, only it can give you an answer in milliseconds, instead of in hours.

So for the right sorts of use cases, elastic search can be a very powerful tool and not just for search.

So let's zoom in and see what elastic search is really about. At a low level, it's really just about handling

JSON requests, so we're not talking about pretty UIs or graphical interfaces, when we're

just talking about elastic search itself, we're talking about a server that can process JSON requests

and give you back JSON data, and it's up to you to actually do something useful with that.

So for example, we're using curl here to actually issue an arrest request with a GET verb for a given index

called "tags", and we're just searching everything that's in it and you can see the results come back in

JSON format here, and it's up to you to pass all this.

So for example, we did get one result here called, for the movie, "Swimming to Cambodia", which has a given

user I.D. and a tag of "Cambodia".

So if this is part of a tags index that we're searching, this is what a result might actually look like.

So, just to make it real,

that's a sort of output you can expect from elastic search itself. But, there's more to it than just elastic

search, there's also Kibana, which sits on top of elastic search and that's what gives you a pretty web

UI.

So if you're not building your own application on top of elastic search or your own web application, Kibana

can be used just for searching and visualizing what's in your search index graphically, and it can do

very complex aggregations of data, it can graph your data, it can create charts and it's often used to do

things like log analysis - so if you're familiar with things like Google Analytics, the combination of

elastic search and Kibana can be used as sort of a way to roll your own Google Analytics at a very large

scale.

Let's zoom in and take a look at what it might look like.

So here's an actual screenshot from Kibana looking at some real log data.

You can see there's multiple dashboards you can actually look at that are built into Kibana

and this lets you visualize things like; where the hits on my web site are coming from, and where are the error

response codes and how are they all broken down, and what's my distribution of URLs, whatever you

can dream up.

So, there are a lot of specialized dashboards for certain kinds of data, and it kind of brings home

the point that elastic search is not just for searching text anymore; you can actually use it for aggregating

things like apache access logs, which is what this view in Kibana does.

But you can also use Kibana for pretty much anything else you want to. Later on this course we'll use

it to visualize the complete works of William Shakespeare for example, and you can see how it can also

be used for text data as well; it's a very flexible tool and a very powerful UI. We can also have something

called logstash in the Beats framework, and these are ways of actually publishing data into elastic

search, in real time, in a streaming format.

So if you have for example, a collection of web server logs coming in that you just want to feed into

your search index over time automatically, FileBeat can just sit on your web servers and look for new

log files and parse them out, structure them in the way that elastic search wants, and then feed them into

your elastic search cluster as they come in. Logstash does much the same thing, it can also be used to

push data around between your servers and elastic search, but often it's used sort of an intermediate step,

so you have a very lightweight FileBeat client that would sit on your web servers, logstash would accept

those and sort of collect them and pool them up for feeding into elastic search over time. But it's

not just made for log files, and it's not just made for elastic search and web servers either, these are

all very general purpose systems that allow you to tie different systems together and publish data

to wherever it needs to go, which might be elastic search, might be something else, but it's all part of the

elastic stack still. But, it can also collect data from things like Amazon S3 or Kafka, or pretty

much anything else you can imagine, databases, and we'll look at all of those examples later in this course.

Finally, another piece of the elastic stack is called X-pack.

This is actually a paid add on offered by elastic.co, and it offers things like security and alerting

and monitoring and reporting, features like that.

It also contains some of the more advanced features that are just starting to make it into elastic search

now, such as machine learning and graph exploration, so you can see that with X-Pack, elastic search starts

to become a real competitor for much more complex and heavy weight systems like Flink and Spark.

But that's another piece of the elastic stack when we talk about this larger ecosystem. You can see

here that there are free parts of X-Pack, like the monitoring framework, that lets you quickly visualize

what's going on with your cluster, you know,

what's my cpu utilization system load?

How much memory you have available?

Things like that.

So, when things start to go wrong with your cluster,

this is a very useful tool to have for understanding the health of your cluster.

So that's it at a high level, the elastic stack. Obviously elastic search can still be used for a powering

search on a web site like Wikipedia or something, but with these components it can be used for so much

more.

It's actually a larger framework for publishing data from any source you can imagine, and visualizing

it as well through things like Kibana,

and it also has operational capabilities through X-Pack; so that is the elastic stack at a high level.

Lets dive in more into elastic search itself and learn more about how it works.

Let's start off with sort of a 30,000 foot view of the elastic stack, and the components within it and

how they fit together. So, elastic search is just one piece of this system.

It started off as basically a scalable version of the Lucene open source search framework, and it

just added the ability to horizontally scale Lucene indici, so we'll talk about shards of elastic

search, and each shard in elastic search is just a single Lucene inverted index of documents, so every shard

is an actual Lucene instance of its own. However, elastic search has evolved to be much more than just

Lucene spread out across a cluster,

it can be used for much more than full text search now, and it can actually handle structure data and

aggregate data very quickly. So, it's not just for search, you can handle structure data of any type and

you'll see it's often used for things like aggregating logs and things like that.

And what's really cool, is that it's often a much faster solution than things like Hadoop or Spark or

Flink; you're actually building in new things into the elastic search all the time, with things like graph

visualization and machine learning that actually make elastic search a competitor for things like Hadoop

and Spark and Flink, only it can give you an answer in milliseconds, instead of in hours.

So for the right sorts of use cases, elastic search can be a very powerful tool and not just for search.

So let's zoom in and see what elastic search is really about. At a low level, it's really just about handling

JSON requests, so we're not talking about pretty UIs or graphical interfaces, when we're

just talking about elastic search itself, we're talking about a server that can process JSON requests

and give you back JSON data, and it's up to you to actually do something useful with that.

So for example, we're using curl here to actually issue an arrest request with a GET verb for a given index

called "tags", and we're just searching everything that's in it and you can see the results come back in

JSON format here, and it's up to you to pass all this.

So for example, we did get one result here called, for the movie, "Swimming to Cambodia", which has a given

user I.D. and a tag of "Cambodia".

So if this is part of a tags index that we're searching, this is what a result might actually look like.

So, just to make it real,

that's a sort of output you can expect from elastic search itself. But, there's more to it than just elastic

search, there's also Kibana, which sits on top of elastic search and that's what gives you a pretty web

UI.

So if you're not building your own application on top of elastic search or your own web application, Kibana

can be used just for searching and visualizing what's in your search index graphically, and it can do

very complex aggregations of data, it can graph your data, it can create charts and it's often used to do

things like log analysis - so if you're familiar with things like Google Analytics, the combination of

elastic search and Kibana can be used as sort of a way to roll your own Google Analytics at a very large

scale.

Let's zoom in and take a look at what it might look like.

So here's an actual screenshot from Kibana looking at some real log data.

You can see there's multiple dashboards you can actually look at that are built into Kibana

and this lets you visualize things like; where the hits on my web site are coming from, and where are the error

response codes and how are they all broken down, and what's my distribution of URLs, whatever you

can dream up.

So, there are a lot of specialized dashboards for certain kinds of data, and it kind of brings home

the point that elastic search is not just for searching text anymore; you can actually use it for aggregating

things like apache access logs, which is what this view in Kibana does.

But you can also use Kibana for pretty much anything else you want to. Later on this course we'll use

it to visualize the complete works of William Shakespeare for example, and you can see how it can also

be used for text data as well; it's a very flexible tool and a very powerful UI. We can also have something

called logstash in the Beats framework, and these are ways of actually publishing data into elastic

search, in real time, in a streaming format.

So if you have for example, a collection of web server logs coming in that you just want to feed into

your search index over time automatically, FileBeat can just sit on your web servers and look for new

log files and parse them out, structure them in the way that elastic search wants, and then feed them into

your elastic search cluster as they come in. Logstash does much the same thing, it can also be used to

push data around between your servers and elastic search, but often it's used sort of an intermediate step,

so you have a very lightweight FileBeat client that would sit on your web servers, logstash would accept

those and sort of collect them and pool them up for feeding into elastic search over time. But it's

not just made for log files, and it's not just made for elastic search and web servers either, these are

all very general purpose systems that allow you to tie different systems together and publish data

to wherever it needs to go, which might be elastic search, might be something else, but it's all part of the

elastic stack still. But, it can also collect data from things like Amazon S3 or Kafka, or pretty

much anything else you can imagine, databases, and we'll look at all of those examples later in this course.

Finally, another piece of the elastic stack is called X-pack.

This is actually a paid add on offered by elastic.co, and it offers things like security and alerting

and monitoring and reporting, features like that.

It also contains some of the more advanced features that are just starting to make it into elastic search

now, such as machine learning and graph exploration, so you can see that with X-Pack, elastic search starts

to become a real competitor for much more complex and heavy weight systems like Flink and Spark.

But that's another piece of the elastic stack when we talk about this larger ecosystem. You can see

here that there are free parts of X-Pack, like the monitoring framework, that lets you quickly visualize

what's going on with your cluster, you know,

what's my cpu utilization system load?

How much memory you have available?

Things like that.

So, when things start to go wrong with your cluster,

this is a very useful tool to have for understanding the health of your cluster.

So that's it at a high level, the elastic stack. Obviously elastic search can still be used for a powering

search on a web site like Wikipedia or something, but with these components it can be used for so much

more.

It's actually a larger framework for publishing data from any source you can imagine, and visualizing

it as well through things like Kibana,

and it also has operational capabilities through X-Pack; so that is the elastic stack at a high level.

Lets dive in more into elastic search itself and learn more about how it works.

Let's start off with sort of a 30,000 foot view of the elastic stack, and the components within it. And

how they fit together. So, elastic search is just one piece of this system.

It started off as basically a scalable version of the Lucene open source search framework, and it

just added the ability to horizontally scale Lucene indici, so we'll talk about shards of elastic

search, and each shard in elastic search is just a single Lucene inverted index of documents, so every shard

is an actual Lucene instance of its own. However, elastic search has evolved to be much more than just

Lucene spread out across a cluster,

it can be used for much more than full text search now, and it can actually handle structure data and

aggregate data very quickly. So, it's not just for search, you can handle structure data of any type and

you'll see it's often used for things like aggregating logs and things like that.

And what's really cool, is that it's often a much faster solution than things like Hadoop or Spark or

Flink; you're actually building in new things into the elastic search all the time, with things like graph

visualization and machine learning that actually make elastic search a competitor for things like Hadoop

and Spark and Flink, only it can give you an answer in milliseconds, instead of in hours.

So for the right sorts of use cases, elastic search can be a very powerful tool and not just for search.

So let's zoom in and see what elastic search is really about. At a low level, it's really just about handling

JSON requests, so we're not talking about pretty UIs or graphical interfaces, when we're

just talking about elastic search itself, we're talking about a server that can process JSON requests

and give you back JSON data, and it's up to you to actually do something useful with that.

So for example, we're using curl here to actually issue an arrest request with a GET verb for a given index

called "tags", and we're just searching everything that's in it and you can see the results come back in

JSON format here, and it's up to you to pass all this.

So for example, we did get one result here called, for the movie, "Swimming to Cambodia", which has a given

user I.D. and a tag of "Cambodia".

So if this is part of a tags index that we're searching, this is what a result might actually look like.

So, just to make it real,

that's a sort of output you can expect from elastic search itself. But, there's more to it than just elastic

search, there's also Kibana, which sits on top of elastic search and that's what gives you a pretty web

UI.

So if you're not building your own application on top of elastic search or your own web application, Kibana

can be used just for searching and visualizing what's in your search index graphically, and it can do

very complex aggregations of data, it can graph your data, it can create charts and it's often used to do

things like log analysis - so if you're familiar with things like Google Analytics, the combination of

elastic search and Kibana can be used as sort of a way to roll your own Google Analytics at a very large

scale.

Let's zoom in and take a look at what it might look like.

So here's an actual screenshot from Kibana looking at some real log data.

You can see there's multiple dashboards you can actually look at that are built into Kibana

and this lets you visualize things like; where the hits on my web site are coming from, and where are the error

response codes and how are they all broken down, and what's my distribution of URLs, whatever you

can dream up.

So, there are a lot of specialized dashboards for certain kinds of data, and it kind of brings home

the point that elastic search is not just for searching text anymore; you can actually use it for aggregating

things like apache access logs, which is what this view in Kibana does.

But you can also use Kibana for pretty much anything else you want to. Later on this course we'll use

it to visualize the complete works of William Shakespeare for example, and you can see how it can also

be used for text data as well; it's a very flexible tool and a very powerful UI. We can also have something

called logstash in the Beats framework, and these are ways of actually publishing data into elastic

search, in real time, in a streaming format.

So if you have for example, a collection of web server logs coming in that you just want to feed into

your search index over time automatically, FileBeat can just sit on your web servers and look for new

log files and parse them out, structure them in the way that elastic search wants, and then feed them into

your elastic search cluster as they come in. Logstash does much the same thing, it can also be used to

push data around between your servers and elastic search, but often it's used sort of an intermediate step,

so you have a very lightweight FileBeat client that would sit on your web servers, logstash would accept

those and sort of collect them and pool them up for feeding into elastic search over time. But it's

not just made for log files, and it's not just made for elastic search and web servers either, these are

all very general purpose systems that allow you to tie different systems together and publish data

to wherever it needs to go, which might be elastic search, might be something else, but it's all part of the

elastic stack still. But, it can also collect data from things like Amazon S3 or Kafka, or pretty

much anything else you can imagine, databases, and we'll look at all of those examples later in this course.

Finally, another piece of the elastic stack is called X-pack.

This is actually a paid add on offered by elastic.co, and it offers things like security and alerting

and monitoring and reporting, features like that.

It also contains some of the more advanced features that are just starting to make it into elastic search

now, such as machine learning and graph exploration, so you can see that with X-Pack, elastic search starts

to become a real competitor for much more complex and heavy weight systems like Flink and Spark.

But that's another piece of the elastic stack when we talk about this larger ecosystem. You can see

here that there are free parts of X-Pack, like the monitoring framework, that lets you quickly visualize

what's going on with your cluster, you know,

what's my cpu utilization system load?

How much memory you have available?

Things like that.

So, when things start to go wrong with your cluster,

this is a very useful tool to have for understanding the health of your cluster.

So that's it at a high level, the elastic stack. Obviously elastic search can still be used for a powering

search on a web site like Wikipedia or something, but with these components it can be used for so much

more.

It's actually a larger framework for publishing data from any source you can imagine, and visualizing

it as well through things like Kibana,

and it also has operational capabilities through X-Pack; so that is the elastic stack at a high level.

Lets dive in more into elastic search itself and learn more about how it works.