Would you like to inspect the original subtitles? These are the user uploaded subtitles that are being translated:
1
00:00:00,270 --> 00:00:04,410
Let's start off with sort of a 30,000 foot view of the elastic stack, and the components within it and
2
00:00:04,410 --> 00:00:08,530
how they fit together. So, elastic search is just one piece of this system.
3
00:00:08,550 --> 00:00:13,740
It started off as basically a scalable version of the Lucene open source search framework, and it
4
00:00:13,740 --> 00:00:19,020
just added the ability to horizontally scale Lucene indici, so we'll talk about shards of elastic
5
00:00:19,020 --> 00:00:24,870
search, and each shard in elastic search is just a single Lucene inverted index of documents, so every shard
6
00:00:24,870 --> 00:00:30,120
is an actual Lucene instance of its own. However, elastic search has evolved to be much more than just
7
00:00:30,180 --> 00:00:32,420
Lucene spread out across a cluster,
8
00:00:32,549 --> 00:00:36,780
it can be used for much more than full text search now, and it can actually handle structure data and
9
00:00:36,780 --> 00:00:42,030
aggregate data very quickly. So, it's not just for search, you can handle structure data of any type and
10
00:00:42,030 --> 00:00:45,260
you'll see it's often used for things like aggregating logs and things like that.
11
00:00:45,480 --> 00:00:49,740
And what's really cool, is that it's often a much faster solution than things like Hadoop or Spark or
12
00:00:49,780 --> 00:00:54,570
Flink; you're actually building in new things into the elastic search all the time, with things like graph
13
00:00:54,720 --> 00:00:59,970
visualization and machine learning that actually make elastic search a competitor for things like Hadoop
14
00:00:59,970 --> 00:01:04,519
and Spark and Flink, only it can give you an answer in milliseconds, instead of in hours.
15
00:01:04,530 --> 00:01:10,660
So for the right sorts of use cases, elastic search can be a very powerful tool and not just for search.
16
00:01:10,920 --> 00:01:15,780
So let's zoom in and see what elastic search is really about. At a low level, it's really just about handling
17
00:01:15,780 --> 00:01:20,520
JSON requests, so we're not talking about pretty UIs or graphical interfaces, when we're
18
00:01:20,520 --> 00:01:25,800
just talking about elastic search itself, we're talking about a server that can process JSON requests
19
00:01:25,800 --> 00:01:29,790
and give you back JSON data, and it's up to you to actually do something useful with that.
20
00:01:30,210 --> 00:01:36,990
So for example, we're using curl here to actually issue an arrest request with a GET verb for a given index
21
00:01:36,990 --> 00:01:40,890
called "tags", and we're just searching everything that's in it and you can see the results come back in
22
00:01:40,890 --> 00:01:43,620
JSON format here, and it's up to you to pass all this.
23
00:01:43,620 --> 00:01:49,080
So for example, we did get one result here called, for the movie, "Swimming to Cambodia", which has a given
24
00:01:49,080 --> 00:01:50,820
user I.D. and a tag of "Cambodia".
25
00:01:50,820 --> 00:01:55,080
So if this is part of a tags index that we're searching, this is what a result might actually look like.
26
00:01:55,080 --> 00:01:56,790
So, just to make it real,
27
00:01:56,910 --> 00:02:02,070
that's a sort of output you can expect from elastic search itself. But, there's more to it than just elastic
28
00:02:02,100 --> 00:02:06,810
search, there's also Kibana, which sits on top of elastic search and that's what gives you a pretty web
29
00:02:06,810 --> 00:02:07,330
UI.
30
00:02:07,770 --> 00:02:12,930
So if you're not building your own application on top of elastic search or your own web application, Kibana
31
00:02:12,960 --> 00:02:17,820
can be used just for searching and visualizing what's in your search index graphically, and it can do
32
00:02:17,820 --> 00:02:22,740
very complex aggregations of data, it can graph your data, it can create charts and it's often used to do
33
00:02:22,740 --> 00:02:27,120
things like log analysis - so if you're familiar with things like Google Analytics, the combination of
34
00:02:27,120 --> 00:02:32,220
elastic search and Kibana can be used as sort of a way to roll your own Google Analytics at a very large
35
00:02:32,220 --> 00:02:32,740
scale.
36
00:02:32,940 --> 00:02:35,100
Let's zoom in and take a look at what it might look like.
37
00:02:35,100 --> 00:02:38,430
So here's an actual screenshot from Kibana looking at some real log data.
38
00:02:38,430 --> 00:02:42,860
You can see there's multiple dashboards you can actually look at that are built into Kibana
39
00:02:42,940 --> 00:02:47,580
and this lets you visualize things like; where the hits on my web site are coming from, and where are the error
40
00:02:47,580 --> 00:02:51,210
response codes and how are they all broken down, and what's my distribution of URLs, whatever you
41
00:02:51,210 --> 00:02:52,330
can dream up.
42
00:02:52,350 --> 00:02:56,730
So, there are a lot of specialized dashboards for certain kinds of data, and it kind of brings home
43
00:02:56,730 --> 00:03:02,160
the point that elastic search is not just for searching text anymore; you can actually use it for aggregating
44
00:03:02,160 --> 00:03:05,710
things like apache access logs, which is what this view in Kibana does.
45
00:03:05,800 --> 00:03:09,780
But you can also use Kibana for pretty much anything else you want to. Later on this course we'll use
46
00:03:09,780 --> 00:03:15,060
it to visualize the complete works of William Shakespeare for example, and you can see how it can also
47
00:03:15,060 --> 00:03:21,800
be used for text data as well; it's a very flexible tool and a very powerful UI. We can also have something
48
00:03:21,800 --> 00:03:26,920
called logstash in the Beats framework, and these are ways of actually publishing data into elastic
49
00:03:26,930 --> 00:03:29,800
search, in real time, in a streaming format.
50
00:03:29,870 --> 00:03:33,980
So if you have for example, a collection of web server logs coming in that you just want to feed into
51
00:03:33,980 --> 00:03:39,110
your search index over time automatically, FileBeat can just sit on your web servers and look for new
52
00:03:39,110 --> 00:03:43,640
log files and parse them out, structure them in the way that elastic search wants, and then feed them into
53
00:03:43,640 --> 00:03:49,460
your elastic search cluster as they come in. Logstash does much the same thing, it can also be used to
54
00:03:49,460 --> 00:03:54,440
push data around between your servers and elastic search, but often it's used sort of an intermediate step,
55
00:03:54,590 --> 00:03:59,810
so you have a very lightweight FileBeat client that would sit on your web servers, logstash would accept
56
00:03:59,810 --> 00:04:05,930
those and sort of collect them and pool them up for feeding into elastic search over time. But it's
57
00:04:05,930 --> 00:04:09,920
not just made for log files, and it's not just made for elastic search and web servers either, these are
58
00:04:09,920 --> 00:04:15,020
all very general purpose systems that allow you to tie different systems together and publish data
59
00:04:15,050 --> 00:04:19,310
to wherever it needs to go, which might be elastic search, might be something else, but it's all part of the
60
00:04:19,310 --> 00:04:24,950
elastic stack still. But, it can also collect data from things like Amazon S3 or Kafka, or pretty
61
00:04:24,950 --> 00:04:29,660
much anything else you can imagine, databases, and we'll look at all of those examples later in this course.
62
00:04:31,460 --> 00:04:34,680
Finally, another piece of the elastic stack is called X-pack.
63
00:04:34,700 --> 00:04:39,650
This is actually a paid add on offered by elastic.co, and it offers things like security and alerting
64
00:04:39,650 --> 00:04:42,210
and monitoring and reporting, features like that.
65
00:04:42,230 --> 00:04:46,460
It also contains some of the more advanced features that are just starting to make it into elastic search
66
00:04:46,450 --> 00:04:51,770
now, such as machine learning and graph exploration, so you can see that with X-Pack, elastic search starts
67
00:04:51,770 --> 00:04:56,590
to become a real competitor for much more complex and heavy weight systems like Flink and Spark.
68
00:04:57,200 --> 00:05:01,130
But that's another piece of the elastic stack when we talk about this larger ecosystem. You can see
69
00:05:01,130 --> 00:05:06,440
here that there are free parts of X-Pack, like the monitoring framework, that lets you quickly visualize
70
00:05:06,470 --> 00:05:07,910
what's going on with your cluster, you know,
71
00:05:08,000 --> 00:05:09,980
what's my cpu utilization system load?
72
00:05:09,980 --> 00:05:11,150
How much memory you have available?
73
00:05:11,150 --> 00:05:11,810
Things like that.
74
00:05:11,810 --> 00:05:14,150
So, when things start to go wrong with your cluster,
75
00:05:14,150 --> 00:05:18,290
this is a very useful tool to have for understanding the health of your cluster.
76
00:05:18,290 --> 00:05:23,090
So that's it at a high level, the elastic stack. Obviously elastic search can still be used for a powering
77
00:05:23,090 --> 00:05:27,800
search on a web site like Wikipedia or something, but with these components it can be used for so much
78
00:05:27,800 --> 00:05:28,180
more.
79
00:05:28,190 --> 00:05:33,380
It's actually a larger framework for publishing data from any source you can imagine, and visualizing
80
00:05:33,380 --> 00:05:34,860
it as well through things like Kibana,
81
00:05:34,880 --> 00:05:40,540
and it also has operational capabilities through X-Pack; so that is the elastic stack at a high level.
82
00:05:40,610 --> 00:05:44,060
Lets dive in more into elastic search itself and learn more about how it works.
83
00:00:00,270 --> 00:00:04,410
Let's start off with sort of a 30,000 foot view of the elastic stack, and the components within it. And
84
00:00:04,410 --> 00:00:08,530
how they fit together. So, elastic search is just one piece of this system.
85
00:00:08,550 --> 00:00:13,740
It started off as basically a scalable version of the Lucene open source search framework, and it
86
00:00:13,740 --> 00:00:19,020
just added the ability to horizontally scale Lucene indici, so we'll talk about shards of elastic
87
00:00:19,020 --> 00:00:24,870
search, and each shard in elastic search is just a single Lucene inverted index of documents, so every shard
88
00:00:24,870 --> 00:00:30,120
is an actual Lucene instance of its own. However, elastic search has evolved to be much more than just
89
00:00:30,180 --> 00:00:32,420
Lucene spread out across a cluster,
90
00:00:32,549 --> 00:00:36,780
it can be used for much more than full text search now, and it can actually handle structure data and
91
00:00:36,780 --> 00:00:42,030
aggregate data very quickly. So, it's not just for search, you can handle structure data of any type and
92
00:00:42,030 --> 00:00:45,260
you'll see it's often used for things like aggregating logs and things like that.
93
00:00:45,480 --> 00:00:49,740
And what's really cool, is that it's often a much faster solution than things like Hadoop or Spark or
94
00:00:49,780 --> 00:00:54,570
Flink; you're actually building in new things into the elastic search all the time, with things like graph
95
00:00:54,720 --> 00:00:59,970
visualization and machine learning that actually make elastic search a competitor for things like Hadoop
96
00:00:59,970 --> 00:01:04,519
and Spark and Flink, only it can give you an answer in milliseconds, instead of in hours.
97
00:01:04,530 --> 00:01:10,660
So for the right sorts of use cases, elastic search can be a very powerful tool and not just for search.
98
00:01:10,920 --> 00:01:15,780
So let's zoom in and see what elastic search is really about. At a low level, it's really just about handling
99
00:01:15,780 --> 00:01:20,520
JSON requests, so we're not talking about pretty UIs or graphical interfaces, when we're
100
00:01:20,520 --> 00:01:25,800
just talking about elastic search itself, we're talking about a server that can process JSON requests
101
00:01:25,800 --> 00:01:29,790
and give you back JSON data, and it's up to you to actually do something useful with that.
102
00:01:30,210 --> 00:01:36,990
So for example, we're using curl here to actually issue an arrest request with a GET verb for a given index
103
00:01:36,990 --> 00:01:40,890
called "tags", and we're just searching everything that's in it and you can see the results come back in
104
00:01:40,890 --> 00:01:43,620
JSON format here, and it's up to you to pass all this.
105
00:01:43,620 --> 00:01:49,080
So for example, we did get one result here called, for the movie, "Swimming to Cambodia", which has a given
106
00:01:49,080 --> 00:01:50,820
user I.D. and a tag of "Cambodia".
107
00:01:50,820 --> 00:01:55,080
So if this is part of a tags index that we're searching, this is what a result might actually look like.
108
00:01:55,080 --> 00:01:56,790
So, just to make it real,
109
00:01:56,910 --> 00:02:02,070
that's a sort of output you can expect from elastic search itself. But, there's more to it than just elastic
110
00:02:02,100 --> 00:02:06,810
search, there's also Kibana, which sits on top of elastic search and that's what gives you a pretty web
111
00:02:06,810 --> 00:02:07,330
UI.
112
00:02:07,770 --> 00:02:12,930
So if you're not building your own application on top of elastic search or your own web application, Kibana
113
00:02:12,960 --> 00:02:17,820
can be used just for searching and visualizing what's in your search index graphically, and it can do
114
00:02:17,820 --> 00:02:22,740
very complex aggregations of data, it can graph your data, it can create charts and it's often used to do
115
00:02:22,740 --> 00:02:27,120
things like log analysis - so if you're familiar with things like Google Analytics, the combination of
116
00:02:27,120 --> 00:02:32,220
elastic search and Kibana can be used as sort of a way to roll your own Google Analytics at a very large
117
00:02:32,220 --> 00:02:32,740
scale.
118
00:02:32,940 --> 00:02:35,100
Let's zoom in and take a look at what it might look like.
119
00:02:35,100 --> 00:02:38,430
So here's an actual screenshot from Kibana looking at some real log data.
120
00:02:38,430 --> 00:02:42,860
You can see there's multiple dashboards you can actually look at that are built into Kibana
121
00:02:42,940 --> 00:02:47,580
and this lets you visualize things like; where the hits on my web site are coming from, and where are the error
122
00:02:47,580 --> 00:02:51,210
response codes and how are they all broken down, and what's my distribution of URLs, whatever you
123
00:02:51,210 --> 00:02:52,330
can dream up.
124
00:02:52,350 --> 00:02:56,730
So, there are a lot of specialized dashboards for certain kinds of data, and it kind of brings home
125
00:02:56,730 --> 00:03:02,160
the point that elastic search is not just for searching text anymore; you can actually use it for aggregating
126
00:03:02,160 --> 00:03:05,710
things like apache access logs, which is what this view in Kibana does.
127
00:03:05,800 --> 00:03:09,780
But you can also use Kibana for pretty much anything else you want to. Later on this course we'll use
128
00:03:09,780 --> 00:03:15,060
it to visualize the complete works of William Shakespeare for example, and you can see how it can also
129
00:03:15,060 --> 00:03:21,800
be used for text data as well; it's a very flexible tool and a very powerful UI. We can also have something
130
00:03:21,800 --> 00:03:26,920
called logstash in the Beats framework, and these are ways of actually publishing data into elastic
131
00:03:26,930 --> 00:03:29,800
search, in real time, in a streaming format.
132
00:03:29,870 --> 00:03:33,980
So if you have for example, a collection of web server logs coming in that you just want to feed into
133
00:03:33,980 --> 00:03:39,110
your search index over time automatically, FileBeat can just sit on your web servers and look for new
134
00:03:39,110 --> 00:03:43,640
log files and parse them out, structure them in the way that elastic search wants, and then feed them into
135
00:03:43,640 --> 00:03:49,460
your elastic search cluster as they come in. Logstash does much the same thing, it can also be used to
136
00:03:49,460 --> 00:03:54,440
push data around between your servers and elastic search, but often it's used sort of an intermediate step,
137
00:03:54,590 --> 00:03:59,810
so you have a very lightweight FileBeat client that would sit on your web servers, logstash would accept
138
00:03:59,810 --> 00:04:05,930
those and sort of collect them and pool them up for feeding into elastic search over time. But it's
139
00:04:05,930 --> 00:04:09,920
not just made for log files, and it's not just made for elastic search and web servers either, these are
140
00:04:09,920 --> 00:04:15,020
all very general purpose systems that allow you to tie different systems together and publish data
141
00:04:15,050 --> 00:04:19,310
to wherever it needs to go, which might be elastic search, might be something else, but it's all part of the
142
00:04:19,310 --> 00:04:24,950
elastic stack still. But, it can also collect data from things like Amazon S3 or Kafka, or pretty
143
00:04:24,950 --> 00:04:29,660
much anything else you can imagine, databases, and we'll look at all of those examples later in this course.
144
00:04:31,460 --> 00:04:34,680
Finally, another piece of the elastic stack is called X-pack.
145
00:04:34,700 --> 00:04:39,650
This is actually a paid add on offered by elastic.co, and it offers things like security and alerting
146
00:04:39,650 --> 00:04:42,210
and monitoring and reporting, features like that.
147
00:04:42,230 --> 00:04:46,460
It also contains some of the more advanced features that are just starting to make it into elastic search
148
00:04:46,450 --> 00:04:51,770
now, such as machine learning and graph exploration, so you can see that with X-Pack, elastic search starts
149
00:04:51,770 --> 00:04:56,590
to become a real competitor for much more complex and heavy weight systems like Flink and Spark.
150
00:04:57,200 --> 00:05:01,130
But that's another piece of the elastic stack when we talk about this larger ecosystem. You can see
151
00:05:01,130 --> 00:05:06,440
here that there are free parts of X-Pack, like the monitoring framework, that lets you quickly visualize
152
00:05:06,470 --> 00:05:07,910
what's going on with your cluster, you know,
153
00:05:08,000 --> 00:05:09,980
what's my cpu utilization system load?
154
00:05:09,980 --> 00:05:11,150
How much memory you have available?
155
00:05:11,150 --> 00:05:11,810
Things like that.
156
00:05:11,810 --> 00:05:14,150
So, when things start to go wrong with your cluster,
157
00:05:14,150 --> 00:05:18,290
this is a very useful tool to have for understanding the health of your cluster.
158
00:05:18,290 --> 00:05:23,090
So that's it at a high level, the elastic stack. Obviously elastic search can still be used for a powering
159
00:05:23,090 --> 00:05:27,800
search on a web site like Wikipedia or something, but with these components it can be used for so much
160
00:05:27,800 --> 00:05:28,180
more.
161
00:05:28,190 --> 00:05:33,380
It's actually a larger framework for publishing data from any source you can imagine, and visualizing
162
00:05:33,380 --> 00:05:34,860
it as well through things like Kibana,
163
00:05:34,880 --> 00:05:40,540
and it also has operational capabilities through X-Pack; so that is the elastic stack at a high level.
164
00:05:40,610 --> 00:05:44,060
Lets dive in more into elastic search itself and learn more about how it works.
165
00:00:00,270 --> 00:00:04,410
Let's start off with sort of a 30,000 foot view of the elastic stack, and the components within it and
166
00:00:04,410 --> 00:00:08,530
how they fit together. So, elastic search is just one piece of this system.
167
00:00:08,550 --> 00:00:13,740
It started off as basically a scalable version of the Lucene open source search framework, and it
168
00:00:13,740 --> 00:00:19,020
just added the ability to horizontally scale Lucene indici, so we'll talk about shards of elastic
169
00:00:19,020 --> 00:00:24,870
search, and each shard in elastic search is just a single Lucene inverted index of documents, so every shard
170
00:00:24,870 --> 00:00:30,120
is an actual Lucene instance of its own. However, elastic search has evolved to be much more than just
171
00:00:30,180 --> 00:00:32,420
Lucene spread out across a cluster,
172
00:00:32,549 --> 00:00:36,780
it can be used for much more than full text search now, and it can actually handle structure data and
173
00:00:36,780 --> 00:00:42,030
aggregate data very quickly. So, it's not just for search, you can handle structure data of any type and
174
00:00:42,030 --> 00:00:45,260
you'll see it's often used for things like aggregating logs and things like that.
175
00:00:45,480 --> 00:00:49,740
And what's really cool, is that it's often a much faster solution than things like Hadoop or Spark or
176
00:00:49,780 --> 00:00:54,570
Flink; you're actually building in new things into the elastic search all the time, with things like graph
177
00:00:54,720 --> 00:00:59,970
visualization and machine learning that actually make elastic search a competitor for things like Hadoop
178
00:00:59,970 --> 00:01:04,519
and Spark and Flink, only it can give you an answer in milliseconds, instead of in hours.
179
00:01:04,530 --> 00:01:10,660
So for the right sorts of use cases, elastic search can be a very powerful tool and not just for search.
180
00:01:10,920 --> 00:01:15,780
So let's zoom in and see what elastic search is really about. At a low level, it's really just about handling
181
00:01:15,780 --> 00:01:20,520
JSON requests, so we're not talking about pretty UIs or graphical interfaces, when we're
182
00:01:20,520 --> 00:01:25,800
just talking about elastic search itself, we're talking about a server that can process JSON requests
183
00:01:25,800 --> 00:01:29,790
and give you back JSON data, and it's up to you to actually do something useful with that.
184
00:01:30,210 --> 00:01:36,990
So for example, we're using curl here to actually issue an arrest request with a GET verb for a given index
185
00:01:36,990 --> 00:01:40,890
called "tags", and we're just searching everything that's in it and you can see the results come back in
186
00:01:40,890 --> 00:01:43,620
JSON format here, and it's up to you to pass all this.
187
00:01:43,620 --> 00:01:49,080
So for example, we did get one result here called, for the movie, "Swimming to Cambodia", which has a given
188
00:01:49,080 --> 00:01:50,820
user I.D. and a tag of "Cambodia".
189
00:01:50,820 --> 00:01:55,080
So if this is part of a tags index that we're searching, this is what a result might actually look like.
190
00:01:55,080 --> 00:01:56,790
So, just to make it real,
191
00:01:56,910 --> 00:02:02,070
that's a sort of output you can expect from elastic search itself. But, there's more to it than just elastic
192
00:02:02,100 --> 00:02:06,810
search, there's also Kibana, which sits on top of elastic search and that's what gives you a pretty web
193
00:02:06,810 --> 00:02:07,330
UI.
194
00:02:07,770 --> 00:02:12,930
So if you're not building your own application on top of elastic search or your own web application, Kibana
195
00:02:12,960 --> 00:02:17,820
can be used just for searching and visualizing what's in your search index graphically, and it can do
196
00:02:17,820 --> 00:02:22,740
very complex aggregations of data, it can graph your data, it can create charts and it's often used to do
197
00:02:22,740 --> 00:02:27,120
things like log analysis - so if you're familiar with things like Google Analytics, the combination of
198
00:02:27,120 --> 00:02:32,220
elastic search and Kibana can be used as sort of a way to roll your own Google Analytics at a very large
199
00:02:32,220 --> 00:02:32,740
scale.
200
00:02:32,940 --> 00:02:35,100
Let's zoom in and take a look at what it might look like.
201
00:02:35,100 --> 00:02:38,430
So here's an actual screenshot from Kibana looking at some real log data.
202
00:02:38,430 --> 00:02:42,860
You can see there's multiple dashboards you can actually look at that are built into Kibana
203
00:02:42,940 --> 00:02:47,580
and this lets you visualize things like; where the hits on my web site are coming from, and where are the error
204
00:02:47,580 --> 00:02:51,210
response codes and how are they all broken down, and what's my distribution of URLs, whatever you
205
00:02:51,210 --> 00:02:52,330
can dream up.
206
00:02:52,350 --> 00:02:56,730
So, there are a lot of specialized dashboards for certain kinds of data, and it kind of brings home
207
00:02:56,730 --> 00:03:02,160
the point that elastic search is not just for searching text anymore; you can actually use it for aggregating
208
00:03:02,160 --> 00:03:05,710
things like apache access logs, which is what this view in Kibana does.
209
00:03:05,800 --> 00:03:09,780
But you can also use Kibana for pretty much anything else you want to. Later on this course we'll use
210
00:03:09,780 --> 00:03:15,060
it to visualize the complete works of William Shakespeare for example, and you can see how it can also
211
00:03:15,060 --> 00:03:21,800
be used for text data as well; it's a very flexible tool and a very powerful UI. We can also have something
212
00:03:21,800 --> 00:03:26,920
called logstash in the Beats framework, and these are ways of actually publishing data into elastic
213
00:03:26,930 --> 00:03:29,800
search, in real time, in a streaming format.
214
00:03:29,870 --> 00:03:33,980
So if you have for example, a collection of web server logs coming in that you just want to feed into
215
00:03:33,980 --> 00:03:39,110
your search index over time automatically, FileBeat can just sit on your web servers and look for new
216
00:03:39,110 --> 00:03:43,640
log files and parse them out, structure them in the way that elastic search wants, and then feed them into
217
00:03:43,640 --> 00:03:49,460
your elastic search cluster as they come in. Logstash does much the same thing, it can also be used to
218
00:03:49,460 --> 00:03:54,440
push data around between your servers and elastic search, but often it's used sort of an intermediate step,
219
00:03:54,590 --> 00:03:59,810
so you have a very lightweight FileBeat client that would sit on your web servers, logstash would accept
220
00:03:59,810 --> 00:04:05,930
those and sort of collect them and pool them up for feeding into elastic search over time. But it's
221
00:04:05,930 --> 00:04:09,920
not just made for log files, and it's not just made for elastic search and web servers either, these are
222
00:04:09,920 --> 00:04:15,020
all very general purpose systems that allow you to tie different systems together and publish data
223
00:04:15,050 --> 00:04:19,310
to wherever it needs to go, which might be elastic search, might be something else, but it's all part of the
224
00:04:19,310 --> 00:04:24,950
elastic stack still. But, it can also collect data from things like Amazon S3 or Kafka, or pretty
225
00:04:24,950 --> 00:04:29,660
much anything else you can imagine, databases, and we'll look at all of those examples later in this course.
226
00:04:31,460 --> 00:04:34,680
Finally, another piece of the elastic stack is called X-pack.
227
00:04:34,700 --> 00:04:39,650
This is actually a paid add on offered by elastic.co, and it offers things like security and alerting
228
00:04:39,650 --> 00:04:42,210
and monitoring and reporting, features like that.
229
00:04:42,230 --> 00:04:46,460
It also contains some of the more advanced features that are just starting to make it into elastic search
230
00:04:46,450 --> 00:04:51,770
now, such as machine learning and graph exploration, so you can see that with X-Pack, elastic search starts
231
00:04:51,770 --> 00:04:56,590
to become a real competitor for much more complex and heavy weight systems like Flink and Spark.
232
00:04:57,200 --> 00:05:01,130
But that's another piece of the elastic stack when we talk about this larger ecosystem. You can see
233
00:05:01,130 --> 00:05:06,440
here that there are free parts of X-Pack, like the monitoring framework, that lets you quickly visualize
234
00:05:06,470 --> 00:05:07,910
what's going on with your cluster, you know,
235
00:05:08,000 --> 00:05:09,980
what's my cpu utilization system load?
236
00:05:09,980 --> 00:05:11,150
How much memory you have available?
237
00:05:11,150 --> 00:05:11,810
Things like that.
238
00:05:11,810 --> 00:05:14,150
So, when things start to go wrong with your cluster,
239
00:05:14,150 --> 00:05:18,290
this is a very useful tool to have for understanding the health of your cluster.
240
00:05:18,290 --> 00:05:23,090
So that's it at a high level, the elastic stack. Obviously elastic search can still be used for a powering
241
00:05:23,090 --> 00:05:27,800
search on a web site like Wikipedia or something, but with these components it can be used for so much
242
00:05:27,800 --> 00:05:28,180
more.
243
00:05:28,190 --> 00:05:33,380
It's actually a larger framework for publishing data from any source you can imagine, and visualizing
244
00:05:33,380 --> 00:05:34,860
it as well through things like Kibana,
245
00:05:34,880 --> 00:05:40,540
and it also has operational capabilities through X-Pack; so that is the elastic stack at a high level.
246
00:05:40,610 --> 00:05:44,060
Lets dive in more into elastic search itself and learn more about how it works.
247
00:00:00,270 --> 00:00:04,410
Let's start off with sort of a 30,000 foot view of the elastic stack, and the components within it. And
248
00:00:04,410 --> 00:00:08,530
how they fit together. So, elastic search is just one piece of this system.
249
00:00:08,550 --> 00:00:13,740
It started off as basically a scalable version of the Lucene open source search framework, and it
250
00:00:13,740 --> 00:00:19,020
just added the ability to horizontally scale Lucene indici, so we'll talk about shards of elastic
251
00:00:19,020 --> 00:00:24,870
search, and each shard in elastic search is just a single Lucene inverted index of documents, so every shard
252
00:00:24,870 --> 00:00:30,120
is an actual Lucene instance of its own. However, elastic search has evolved to be much more than just
253
00:00:30,180 --> 00:00:32,420
Lucene spread out across a cluster,
254
00:00:32,549 --> 00:00:36,780
it can be used for much more than full text search now, and it can actually handle structure data and
255
00:00:36,780 --> 00:00:42,030
aggregate data very quickly. So, it's not just for search, you can handle structure data of any type and
256
00:00:42,030 --> 00:00:45,260
you'll see it's often used for things like aggregating logs and things like that.
257
00:00:45,480 --> 00:00:49,740
And what's really cool, is that it's often a much faster solution than things like Hadoop or Spark or
258
00:00:49,780 --> 00:00:54,570
Flink; you're actually building in new things into the elastic search all the time, with things like graph
259
00:00:54,720 --> 00:00:59,970
visualization and machine learning that actually make elastic search a competitor for things like Hadoop
260
00:00:59,970 --> 00:01:04,519
and Spark and Flink, only it can give you an answer in milliseconds, instead of in hours.
261
00:01:04,530 --> 00:01:10,660
So for the right sorts of use cases, elastic search can be a very powerful tool and not just for search.
262
00:01:10,920 --> 00:01:15,780
So let's zoom in and see what elastic search is really about. At a low level, it's really just about handling
263
00:01:15,780 --> 00:01:20,520
JSON requests, so we're not talking about pretty UIs or graphical interfaces, when we're
264
00:01:20,520 --> 00:01:25,800
just talking about elastic search itself, we're talking about a server that can process JSON requests
265
00:01:25,800 --> 00:01:29,790
and give you back JSON data, and it's up to you to actually do something useful with that.
266
00:01:30,210 --> 00:01:36,990
So for example, we're using curl here to actually issue an arrest request with a GET verb for a given index
267
00:01:36,990 --> 00:01:40,890
called "tags", and we're just searching everything that's in it and you can see the results come back in
268
00:01:40,890 --> 00:01:43,620
JSON format here, and it's up to you to pass all this.
269
00:01:43,620 --> 00:01:49,080
So for example, we did get one result here called, for the movie, "Swimming to Cambodia", which has a given
270
00:01:49,080 --> 00:01:50,820
user I.D. and a tag of "Cambodia".
271
00:01:50,820 --> 00:01:55,080
So if this is part of a tags index that we're searching, this is what a result might actually look like.
272
00:01:55,080 --> 00:01:56,790
So, just to make it real,
273
00:01:56,910 --> 00:02:02,070
that's a sort of output you can expect from elastic search itself. But, there's more to it than just elastic
274
00:02:02,100 --> 00:02:06,810
search, there's also Kibana, which sits on top of elastic search and that's what gives you a pretty web
275
00:02:06,810 --> 00:02:07,330
UI.
276
00:02:07,770 --> 00:02:12,930
So if you're not building your own application on top of elastic search or your own web application, Kibana
277
00:02:12,960 --> 00:02:17,820
can be used just for searching and visualizing what's in your search index graphically, and it can do
278
00:02:17,820 --> 00:02:22,740
very complex aggregations of data, it can graph your data, it can create charts and it's often used to do
279
00:02:22,740 --> 00:02:27,120
things like log analysis - so if you're familiar with things like Google Analytics, the combination of
280
00:02:27,120 --> 00:02:32,220
elastic search and Kibana can be used as sort of a way to roll your own Google Analytics at a very large
281
00:02:32,220 --> 00:02:32,740
scale.
282
00:02:32,940 --> 00:02:35,100
Let's zoom in and take a look at what it might look like.
283
00:02:35,100 --> 00:02:38,430
So here's an actual screenshot from Kibana looking at some real log data.
284
00:02:38,430 --> 00:02:42,860
You can see there's multiple dashboards you can actually look at that are built into Kibana
285
00:02:42,940 --> 00:02:47,580
and this lets you visualize things like; where the hits on my web site are coming from, and where are the error
286
00:02:47,580 --> 00:02:51,210
response codes and how are they all broken down, and what's my distribution of URLs, whatever you
287
00:02:51,210 --> 00:02:52,330
can dream up.
288
00:02:52,350 --> 00:02:56,730
So, there are a lot of specialized dashboards for certain kinds of data, and it kind of brings home
289
00:02:56,730 --> 00:03:02,160
the point that elastic search is not just for searching text anymore; you can actually use it for aggregating
290
00:03:02,160 --> 00:03:05,710
things like apache access logs, which is what this view in Kibana does.
291
00:03:05,800 --> 00:03:09,780
But you can also use Kibana for pretty much anything else you want to. Later on this course we'll use
292
00:03:09,780 --> 00:03:15,060
it to visualize the complete works of William Shakespeare for example, and you can see how it can also
293
00:03:15,060 --> 00:03:21,800
be used for text data as well; it's a very flexible tool and a very powerful UI. We can also have something
294
00:03:21,800 --> 00:03:26,920
called logstash in the Beats framework, and these are ways of actually publishing data into elastic
295
00:03:26,930 --> 00:03:29,800
search, in real time, in a streaming format.
296
00:03:29,870 --> 00:03:33,980
So if you have for example, a collection of web server logs coming in that you just want to feed into
297
00:03:33,980 --> 00:03:39,110
your search index over time automatically, FileBeat can just sit on your web servers and look for new
298
00:03:39,110 --> 00:03:43,640
log files and parse them out, structure them in the way that elastic search wants, and then feed them into
299
00:03:43,640 --> 00:03:49,460
your elastic search cluster as they come in. Logstash does much the same thing, it can also be used to
300
00:03:49,460 --> 00:03:54,440
push data around between your servers and elastic search, but often it's used sort of an intermediate step,
301
00:03:54,590 --> 00:03:59,810
so you have a very lightweight FileBeat client that would sit on your web servers, logstash would accept
302
00:03:59,810 --> 00:04:05,930
those and sort of collect them and pool them up for feeding into elastic search over time. But it's
303
00:04:05,930 --> 00:04:09,920
not just made for log files, and it's not just made for elastic search and web servers either, these are
304
00:04:09,920 --> 00:04:15,020
all very general purpose systems that allow you to tie different systems together and publish data
305
00:04:15,050 --> 00:04:19,310
to wherever it needs to go, which might be elastic search, might be something else, but it's all part of the
306
00:04:19,310 --> 00:04:24,950
elastic stack still. But, it can also collect data from things like Amazon S3 or Kafka, or pretty
307
00:04:24,950 --> 00:04:29,660
much anything else you can imagine, databases, and we'll look at all of those examples later in this course.
308
00:04:31,460 --> 00:04:34,680
Finally, another piece of the elastic stack is called X-pack.
309
00:04:34,700 --> 00:04:39,650
This is actually a paid add on offered by elastic.co, and it offers things like security and alerting
310
00:04:39,650 --> 00:04:42,210
and monitoring and reporting, features like that.
311
00:04:42,230 --> 00:04:46,460
It also contains some of the more advanced features that are just starting to make it into elastic search
312
00:04:46,450 --> 00:04:51,770
now, such as machine learning and graph exploration, so you can see that with X-Pack, elastic search starts
313
00:04:51,770 --> 00:04:56,590
to become a real competitor for much more complex and heavy weight systems like Flink and Spark.
314
00:04:57,200 --> 00:05:01,130
But that's another piece of the elastic stack when we talk about this larger ecosystem. You can see
315
00:05:01,130 --> 00:05:06,440
here that there are free parts of X-Pack, like the monitoring framework, that lets you quickly visualize
316
00:05:06,470 --> 00:05:07,910
what's going on with your cluster, you know,
317
00:05:08,000 --> 00:05:09,980
what's my cpu utilization system load?
318
00:05:09,980 --> 00:05:11,150
How much memory you have available?
319
00:05:11,150 --> 00:05:11,810
Things like that.
320
00:05:11,810 --> 00:05:14,150
So, when things start to go wrong with your cluster,
321
00:05:14,150 --> 00:05:18,290
this is a very useful tool to have for understanding the health of your cluster.
322
00:05:18,290 --> 00:05:23,090
So that's it at a high level, the elastic stack. Obviously elastic search can still be used for a powering
323
00:05:23,090 --> 00:05:27,800
search on a web site like Wikipedia or something, but with these components it can be used for so much
324
00:05:27,800 --> 00:05:28,180
more.
325
00:05:28,190 --> 00:05:33,380
It's actually a larger framework for publishing data from any source you can imagine, and visualizing
326
00:05:33,380 --> 00:05:34,860
it as well through things like Kibana,
327
00:05:34,880 --> 00:05:40,540
and it also has operational capabilities through X-Pack; so that is the elastic stack at a high level.
328
00:05:40,610 --> 00:05:44,060
Lets dive in more into elastic search itself and learn more about how it works.
41006
Can't find what you're looking for?
Get subtitles in any language from opensubtitles.com, and translate them here.
Afrikaans
Albanian
Amharic
Arabic
Armenian
Azerbaijani
Basque
Belarusian
Bengali
Bosnian
Bulgarian
Catalan
Cebuano
Chichewa
Chinese (Simplified)
Chinese (Traditional)
Corsican
Croatian
Czech
Danish
Dutch
English
Esperanto
Estonian
Finnish
French
Frisian
Galician
Georgian
German
Greek
Gujarati
Haitian Creole
Hawaiian
Hungarian
Icelandic
Indonesian
Irish
Italian
Japanese
Kazakh
Khmer
Korean
Kurdish (Kurmanji)
Kyrgyz
Lao
Latvian
Lithuanian
Luxembourgish
Macedonian
Malagasy
Malay
Maltese
Maori
Mongolian
Myanmar (Burmese)
Nepali
Norwegian
Pashto
Polish
Portuguese
Romanian
Russian
Samoan
Serbian
Sesotho
Shona
Sinhala
Slovak
Slovenian
Somali
Spanish
Swahili
Swedish
Tajik
Thai
Turkish
Ukrainian
Urdu
Uzbek
Vietnamese