All language subtitles for Apache Spark Full Course - Learn Apache Spark in 8 Hours - Apache Spark Tutorial - Edureka - YouTube

For the past five years Spark
has been on an absolute tear

becoming one of the most widely
used Technologies in big data

and AI. Today's cutting-edge
companies like Facebook app

will Netflix Uber

and many more have deployed

spark at massive scale
processing petabytes of data

to deliver Innovations ranging

from detecting
fraudulent Behavior

to delivering personalized
experiences in real.

Lifetime and many such
innovations that are

transforming every industry.

Hi all I welcome you
all to this full court session

on Apache spark a complete
crash course consisting

of everything you need
to know to get started

with Apache Spark from scratch.

But before we get into details,

let's look at our agenda for
today for better understanding

and ease of learning.

The entire crash course
is divided into 12 modules

in the first module introduction
to spark will try to understand

what exactly Is and how it
performs real time processing

in second module will dive deep
into different components

that constitute spark will also
learn about Spark architecture

and its ecosystem next up
in the third module.

We will learn what exactly

relational distributed data
sets are in spark.

Fourth module is all about
data frames in this module.

We will learn what
exactly data frames are

and how to perform different
operations in data frames

moving on in the fifth.

Module we will
discuss different ways

that spark provides
to perform SQL queries

for accessing and processing
data in the six module.

We will learn

how to perform streaming
on live data streams using spark

where and in the seventh
module will discuss

how to execute different machine
learning algorithms using

spark machine learning library
8 module is all

about spark Graphics
in this module.

We are going to learn what
graph processing is and

how to perform graph processing
using Bob Graphics library

in the ninth module will discuss
the key differences between

two popular data processing
Paddock rooms mapreduce

and Spark talking
about 10 module will integrate

to popular James spark

and Kafka. 11th module is
all about pyspark in this module

will try to understand

how by spark exposes
spark programming model

to python lastly
in the 12 module.

We'll take a look at most
frequently Asked interview.

Options on spark

which will help you
Ace your interview

with flying colors.

Thank you guys
while you are at it,

please do not
forget to subscribe

and Edureka YouTube channel

to stay updated with
current training Technologies.

There has been -

underworld that spark is
a future of Big Data platform,

which is hundred times faster

than mapreduce and is also
a go-to tool for all solutions.

But what exactly is
Apache spark and what?

It's so popular.

And in the session I will give
you a complete Insight

of Apache spark

and its fundamentals
without any further due.

Let's quickly.

Look at the topics to be covered

in this session
first and foremost.

I will tell you

what is Apache spark
and its features next.

I will take you
to the components

of spark ecosystem

that makes Park as a future
of Big Data platform.

After that.

I will talk

about the fundamental
data structure of spark

that is rdd I will also tell you

about its features its Asians
the ways to create rdd Etc

and at the last either wrap
up the session by giving

a real-time use case of spark.

So let's get started

with the very first
topic and understand

what is spark spark
is an open-source

killable massively parallel

in memory execution
environment for running

analytics applications.

You can just think
of it as an in-memory layer

that sits about the
multiple data stores

where data can be loaded
into the memory

and analyzed in parallel
across the cluster.

Into big data processing much
like mapreduce Park Works

to distribute the data
across the cluster

and then process
that data in parallel.

The difference here is
that unlike mapreduce

which shuffles the files around
the disc spark Works in memory,

and that makes it much
faster at processing

the data than mapreduce.

It is also said to be

the Lightning Fast unified
analytics engine for big data

and machine learning.

So now let's look
at the interesting features

of Apache Spark.

Coming to speed you
can cause Park as

a swift processing framework.

Why because it is
hundred times faster in memory

and 10 times faster on the disk
on comparing it with her.

Do not only

that it also provides
High data processing speed

next powerful cashing.

It has a simple
programming layer

that provides powerful caching

and disk persistence
capabilities and Spark

can be deployed through mesos.

How do PI on or Sparks
own cluster manager

as you all know?

That's Park itself was designed

and developed for
real-time data processing.

So it's obvious fact

that it offers
real-time competition

and low latency because of
in memory competitions

next polyglot spark
provides high level apis

in Java Scala Python

and our spark code
can be written in any

of these four languages.

Not only that it also provides
a shell in Scala and python.

These are the various
features of spark now,

let's see the The various
components of spark ecosystem.

Let me first tell you
about the spark or component.

It is the most vital component
of Spartacus system,

which is responsible

for basic I/O functions
scheduling monitoring Etc.

The entire Apache spark
ecosystem is built on the top

of this core execution engine

which has extensible apis
in different languages

like Scala python are and Chava

as I have already
mentioned the spark

and the departs from essos.

How do you feel John
or Sparks own cluster manager

the spark ecosystem
library is composed

of various components

like spark SQL spark streaming
machine learning library.

Now, let me explain
you each of them.

The spark SQL component
is used to Leverage The Power

of declarative queries

and optimize storage
by executing SQL queries

on spark data,

which is present in the rdds

and other external sources
next Sparks trimming

component allows developers
to perform batch.

Processing and streaming of data

in the same application and come
into machine learning library.

It eases the deployment

and development of scalable
machine learning pipelines,

like summary statistics
correlations feature extraction

transformation functions
optimization algorithms Etc

and graph x component lets
the data scientist to work

with graph are non rough sources
to achieve flexibility

and resilience and graph
construction and transformation

and now talking

about the programming
languages spark supports car.

I just a functional
programming language in which

the spark is written.

So spark supports Colour as
the interface then spark also

supports python interface.

You can write the program
in Python and execute it

over the spark again.

If you see the code
in Python and Scala,

both are very similar then our
is very famous for data analysis

and machine learning.

So spark has also added
the support for our

and it also supports Java

so you can go ahead

and write the code in Java
and Giggle with this park

next the data can be stored

in hdfs local file
system Amazon S3 cloud

and it also supports SQL
and nosql database as well.

So this is all about the various
components of spark ecosystem.

Now, let's see what's next

when it comes to iterative
distributed computing

that is processing the data
over multiple jobs

and competitions.

We need to reuse

or share the data
among multiple jobs

in earlier Frameworks
like Hadoop there were problems

while dealing with multiple.

Operations or jobs here.

We need to store the data

and some intermediate stable
distributed storage such as hdfs

and multiple I/O operations
makes the overall computations

of jobs much slower
and they were replications

and civilizations

which in turn made
the process even more slower

and our goal here was
to reduce the number

of I/O operations to hdfs

and this can be achieved only
through in-memory data sharing

the in-memory data sharing
the stent 200 times faster.

Of the network and disk sharing

and rdds try to solve all
the problems by enabling

fault-tolerant distributed
in memory competitions.

So now let's understand

what our rdds it stands for
resilient distributed data set.

They are considered to be
the backbone of spark

and is one of the fundamental
data structure of spark.

It is also known as
the schema-less structures

that can handle both structured
and unstructured data.

So in spark anything
you do is around rdd.

You're reading the
data in spark.

When it is read
into our daily again,

when you're transforming
the data, then you're performing

Transformations on old rdd
and creating a new one.

Then at last you will perform
some actions on the rdd

and store that data
present in an rdd

to a persistent storage
resilient distributed data set

has an immutable distributed
collection of objects.

Your objects can be anything

like strings lines
Rose objects collections

Etc rdds can contain
any type of python Java

or Scala objects.

Even including user
defined classes as

And talking about
the distributed environment.

Each data set present
in an rdd is divided

into logical partitions,

which may be computed
on different nodes

of the cluster due to this you
can perform Transformations

or actions on the complete data

parallely and I don't have
to worry about the distribution

because spark takes care of that

are they these are
highly resilient that is

they are able to recover
quickly from any issues

as a same data chunks

are replicated across
multiple executor notes thus

so even if one executor
fails another will still

process the data.

This allows you to perform
functional calculations

against a data set very quickly

by harnessing the power
of multiple nodes.

So this is all about rdd now.

Let's have a look at some

of the important features of
our dbe's rdds have a provision

of in memory competition

and all transformations
are lazy.

That is it does not compute
the results right away

until an action is applied.

So it supports
in memory competition

and lazy evaluation
as well next.

Fault tolerant in case of rdds.

They track the data
lineage information

to rebuild the last data
automatically and this is

how it provides fault tolerance
to the system.

Next immutability data
can be created

or received any time

and once defined its value
cannot be changed.

And that is the reason why

I said are they these are
immutable next partitioning

at is the fundamental
unit of parallelism

and Spark rdd

and all the data chunks
are divided into partitions

and already next persistence.

So users can reuse rdd

and choose a storage stategy for
them coarse-grained operations

applies to all elements
in datasets through Maps

or filter or
group by operations.

So these are the various
features of our daily.

Now, let's see
the ways to create rdd.

There are three ways to create
rdds one can create rdd

from paralyzed Collections
and one can also create rdd

from the existing card ID

or other are DTS
and it can also be created

from external data sources
as well like hdfs.

Amazon S3 hbase Etc.

Now let me show you
how to create rdds.

I'll open my terminal
and first check

whether my demons
are running or not.

Cool here.

I can see that Hadoop

and Spark demons
both are running.

So now at the first let's start

the spark shell it will take
a bit time to start the shell.

Cool.

Now the spark shall has started

and I can see the version of
spark as two point one point one

and we have a scholar
shell over here.

Now.

I will tell you
how to create rdds

in three different ways using
Scala language at the first.

Let's see how to create an rdd

from paralyzed collections
SC dot paralyzes the method

that I use to create a paralyzed
collection of oddities

and this method is

a spark context paralyzed method
to create a palace collection.

So I will give a seedot bad.

Lice and here I will paralyze
one 200 numbers.

In five different partitions
and I will apply collect

as action to start the process.

So here in the result,

you can see an array
of fun 200 numbers.

Okay.

Now let me show you

how the partitions appear
in the web UI of spark.

So the web UI port for spark is
localhost four zero four zero.

So here you have just
completed one task.

That is St. Dot
paralyzed collect.

Correct here.

You can see all the five stages
that are succeeded

because we have divided the task
into five partitions.

So let Show you the partitions.

So this is a dag
which realization

that is the directed acyclic
graph visualization wherein

you have applied only
paralyzed as a method

so you can see only
one stage here.

So here you can see the rdd
that is been created

and coming to even timeline
you can see the task

that has been executed
in five different stages

and the different colors imply.

The scheduler delayed tasks

these sterilization Time shuffle
rate Time shuffle right time.

I'm execute a Computing
time Etc here.

You can see the summary metrics
for the created rdd here.

You can see

that the maximum time it
took to execute the tasks

in five partitions parallely is
just 45 milliseconds.

You can also see the executor ID
the host ID the status

that is succeeded
duration launch time Etc.

So this is one way
of creating an rdd

from paralyzed collections.

Now, let me show you

how to create an rdd
from the I think our DD okay

here I'll create
an array called Aven

and assign numbers one to ten.

One two, three,
four five six seven.

Okay, so I got the result here.

That is I have created
an integer array of 1 to 10

and now I will paralyze
this a day one.

Sorry, I got an error.

It is a seedot pass
the lies of a one.

Okay, so I created an rdd
called parallel collection cool.

Now I will create a new Oddity
from the existing already.

That is Val new are d d is equal

to a 1 dot map data
present in an rdd.

I will create a new ID
from existing rdd.

So here I will take a one.

As a difference and map
the data and multiply

that data into two.

So what should be our output

if I Mark the data present
in an rdd into two,

so it would be like
2 4 6 8 up to 20, correct?

So, let's see how it works.

Yes, we got the output
that is multiple of 1 to 10.

That is two four
six eight up to 20.

So this is one of the method

of creating a new ID
from an old rdt.

And I have one more method that
is from external file sources.

So what I will do here is I
will give that test is equal

to SC dot txt file here.

I will give the path
to hdfs file location

and Link the path that is hdfs
who localhost 9000 is a path

and I have a folder.

Called example and in that
I have a file called sample.

Cool, so I got one
more already created here.

Now.

Let me show you this file

that I have already kept
in hdfs directory.

I will browse the file system

and I will show you
the / example directory

that I have created.

So here you can see the example

that I have created as
a directory and here I

have sample as input file
that I have been given.

So here you can see
the same path location.

So this is

how I can create an rdd
from external file sources.

In this case.

I have used hdfs as
an external file source.

So this is how we can create
rdds from three different ways

that is paralyzed collections
from external RDS

and from an existing rdds.

So let's move further and see
the various rdd.

It's actually supports

two men operations namely
Transformations and actions

as have already set.

Our treaties are immutable.

So once you create an rdd,

you cannot change
any content in the Hardy,

so you might be wondering

how our did he applies
those Transformations?

Correct?

When you run
any Transformations,

it runs those Transformations
on all our DD

and create a new body.

This is basically done
for optimization reasons.

Transformations are
the operations

which are applied

on a An rdd to create a new rdd
now these Transformations work

on the principle
of lazy evaluations.

So what does it mean it means

that when we call
some operation in rdd

at does not execute immediately
and Spark montañés,

the record of the operation
that is being called

since Transformations
are lazy in nature

so we can execute the operation

any time by calling
an action on the data.

Hence in lazy evaluation
data is not loaded

until it is necessary now these

Since analyze the RTD

and produce result
simple action can be count

which will count the rows and
rdd and then produce a result

so I can say

that transformation produced new
rdd and action produced results

before moving further
with the discussion.

Let me tell you about
the three different workloads

that spark it is they are
batch mode interactive mode

and streaming mode
in case of batch mode.

We run a batch
of you write a job

and then schedule it
it works through a queue

or a batch of separate.

Jobs without manual
intervention then in case

of interactive mode.

It is an interactive shell

where you go and execute
the commands one by one.

So you will execute
one command check the result

and then execute
other command based

on the output result and so

on it works similar
to the SQL shell

so she'll is the one

which executes a driver program
and in the Shell mode.

You can run it
on the cluster mode.

It is generally used
for development work

or it is used
for ad hoc queries,

then comes the streaming mode

where the program
is continuously running.

As invented data
comes it takes a data

and do some Transformations

and actions on the data
and get some results.

So these are the three
different workloads

that spark 8 us now.

Let's see a real-time
use case here.

I'm considering Yahoo!

As an example.

So what are
the problems of Yahoo!

Yahoo!

Properties are

highly personalized
to maximize relevance.

The algorithms used
to provide personalization.

That is the
targeted advertisement

and personalized content
are highly sophisticated.

It and the relevance model
must be updated frequently

because stories news feed and
ads change in time and Yahoo,

has over 150 petabytes of data

that the stored
on 35,000 node Hadoop cluster,

which should be access
efficiently to avoid latency

caused by the data movement

and to gain insights
from the data

and cost-effective manner.

So to overcome
these problems Yahoo!

Look to spark to
improve the performance

of this iterative
model training here.

Machine learning algorithm for
news personalization required

15,000 lines of C++ code

on the other hand the machine
learning algorithm has just

won 20 lines of Scala code.

So that is
the advantage of spark

and this algorithm was ready
for production use

in just 30 minutes of training
on a hundred million datasets

and Sparks Rich API is available

in several programming
languages and has resilient

in memory storage
options and a scum.

Potable with Hadoop through yarn
and the spark yarn project.

It uses Apache spark
for personalizing It's

News web pages and for
targeted advertising.

Not only that it also uses
machine learning algorithms

that run an Apache spark
to find out what kind

of news user are
interested to read

and also for categorizing
the new stories to find

out what kind of users
would be interested

in Reading each category of news

and Spark runs over Hadoop Ian
to use existing data.

And clusters and
the extensive API of spark

and machine learning library
is the development

of machine learning algorithms
and Spar produces the latency

of model training.

We are in memory rdd.

So this is how spark has helped
Yahoo to improve the performance

and achieve the targets.

So I hope you understood
the concept of spark

and its fundamentals.

Now, let me just give
you an overview

of the Spark architecture
Apache spark has a well-defined

layered architecture

where all the components
and layers are Loosely coupled

and integrated with various
extensions and libraries.

This architecture is based
on two main abstractions.

First one resilient
distributed data sets

that is rdd

and the next one directed
acyclic graph called DAC

or th e in order to understand
this park architecture.

You need to first know
the components of the spark

that the spark.

System and its fundamental
data structure rdd.

So let's start by understanding
the spark ecosystem

as you can see from the diagram.

The spark ecosystem is composed
of various components

like spark SQL spark

screaming machine learning
library Graphics spark

our and the code a pi component
talking about spark SQL.

It is used to Leverage The Power
of declarative queries

and optimize storage
by executing SQL queries

on spark data,

which is present in rdds.

And other external sources
next Sparks remain component

allows developers
to perform batch processing

and trimming of the data

and the same application coming
to machine learning library.

It eases the development

and deployment of scalable
machine learning pipelines,

like summary statistics
cluster analysis methods

correlations dimensionality
reduction techniques

feature extractions

and many more now
Graphics component.

Let's the data scientist to work

with graph and non graph
sources to achieve.

Security and resilience
and graph construction

and transformation coming
to spark our it is an r package

that provides a light weighted
front end to use Apache spark.

It provides a distributed
data frame implementation

that supports operations like
selection filtering aggregation,

but on large data sets,

it also supports
distributed machine learning

using machine learning library.

Finally the spark or component.

That is the most vital component
of spark ecosystem,

which is responsible.

Possible for basic
I/O functions scheduling

and monitoring the entire spark
ecosystem is built on the top

of this code execution engine

which has extensible apis
in different languages

like Scala python
are and Java now,

let me tell you
about the programming languages

at the first Spark support

Scala Scala is a functional
programming language

in which spark is written

and Spark suppose Carla
as an interface then

spark also supports python.

Face, you can write program
in Python and execute it

over the spark again.

If you see the code
and Scala and python,

both are very similar then
coming to our it is very famous

for data analysis
and machine learning.

So spark has also added
the support for our

and it also supports Java

so you can go ahead
and write the Java code

and execute it over the spark

against Park also provides
you interactive shell

for Scala Python and are
very can go ahead

and Execute the commands
one by one.

So this is all about
the sparkle ecosystem.

Next.

Let's discuss the fundamental
data structure of spark

that is rdd called as
resilient distributed data sets.

So and Spark anything
you do is around rdd,

you're reading the data
and Spark then it is read

into R DT again.

When you're transforming
the data, then you're performing

Transformations on an old rdd
and creating a new one.

Then at the last you
will perform some actions

on the data and store.

Dataset present in an rdd

to a persistent storage
resilient distributed data

set as an immutable distributed
collection of objects.

Your objects can be anything

like string lines
Rose objects collections Etc.

Now talking about
the distributed environment.

Each data set in rdd is divided
into logical partitions,

which may be computed
on different nodes

of the cluster due to this you
can perform Transformations

and actions on the
complete data parallelly.

And you don't have to worry
about the distribution

because part takes care

of that next as I said our
did these are immutable.

So once you create
an rdd you cannot change

any content in the Rd

so you might be wondering

how our did the applies
those Transformations correct?

Then you run any Transformations
at runs those Transformations

on all our DD
and create a new Oddity.

This is basically done
for optimization reasons.

So, let me tell you
one thing here are decals.

The cached and persistent

if you want to save an rdd
for the future work,

you can cash it

and it will improve
the spark performance rdd

is a fault-tolerant
collection of elements

that can be operated
on in parallel.

If our DD is lost
it will automatically

be recomputed by using
the original Transformations.

This is House Park
provides fault tolerance.

There are two ways to create
rdds first one by paralyzing

an existing collection
in your driver program

and the second one
by Referencing a data set

in the external storage system

such as shared file
system hdfs hbase Etc.

Now Transformations
are the operations

that you perform an rdd
which will create a new body.

For example, you
can perform filter on an rdd

and create a new rdd.

Then there are actions
which analyzes the rdd

and produced result
simple action can be count

which will count
the rows in our D

and producer isn't so I can say

that transformation produced
new ID Actions produce results.

So this is all

about the fundamental
data structure of spark

that is already now.

Let's dive into the core topic
of today's discussion

that the Spark architecture.

So this is
the Spark architecture

in your master node.

You have the driver program
which drives your application.

So the code that you're writing
behaves as a driver program or

if you are using
the interactive shell the shell

acts as a driver program
inside the driver program.

The first thing

that you do is you create
a spark context assume

that the spark context
is a gateway to allspark

functionality at a similar
to your database connection.

So any command you execute
in a database goes

through the database connection
similarly anything you do

on spark goes through
the spark context.

Now this park on text works

with the cluster manager
to manage various jobs,

the driver program

and the spark context takes care
of executing the job

across the cluster
a job is splitted the

And then these tasks

are distributed over
the work or not.

So anytime you create the rtt.

In the spark context
that rdd can be distributed

across various notes

and can be cashed their so rdd
set to be taken partitioned

and distributed across various
notes now worker knows are

the slave nodes whose job is
to basically execute the tasks.

The task is then executed

on the partition rdds
in the worker nodes

and then Returns the result back
to the spark context spot.

Our context takes the job breaks
the shop into the task

and distribute them
on the worker nodes

and these tasks works
on partition rdds perform,

whatever operations you
wanted to perform

and then collect the result

and give it back
to the main Spar context.

If your increase
the number of workers,

then you can divide jobs

and more partitions and execute
them para Leo multiple systems.

This will be actually
lot more faster.

Also if you increase
the number of workers,

it will also
increase your memory.

And you can catch the jobs

so that it can be executed
much more faster.

So this is all
about Spark architecture.

Now.

Let me give you
an infographic idea

about the Spark architecture.

It follows master-slave
architecture here.

The client submits
Park user application code

when an application code
is submitted driver

implicitly converts a user code

that contains Transformations

and actions into a logically
directed graph called DHE

at this stage it also

Performs optimizations such as
pipelining Transformations,

then it converts
a logical graph called DHE

into physical execution plan

with many stages

after converting into
physical execution plan.

It creates a physical
execution units called tasks

under each stage.

Then these tasks are bundled

and sent to the cluster
now driver talks

to the cluster manager
and negotiates a resources

and cluster manager launches
the needed executors

at this point driver
be Also send the task

to the executors based
on the placement

when executor start to register
themselves with the drivers,

so that driver will have
a complete view of the executors

and executors now start
executing the tasks

that are assigned by
the driver program

at any point of time

when the application is running
driver program will monitor

the set of executors

that runs and the driver note

also schedules future tasks
Based on data placement.

So this is how the internal
working takes place in space.

Architecture, there are
three different types

of workloads that spark

and cater first batch mode
in case of batch mode.

We run a bad shop here
you write the job

and then schedule it.

It works through a queue
or batch of separate jobs

through manual intervention
next interactive mode.

This is an interactive shell

where you go and execute
the commands one by one.

So you'll execute
one command check the result

and then execute
the other command based

on the output result and so

on it works similar to the SQL.

Action social is the one
which executes a driver program.

So it is generally used
for development work

or it is also used
for ad hoc queries,

then comes the streaming mode

where the program
is continuously running as

and when the data
comes it takes a data

and do some Transformations
and actions on the data

and then produce output results.

So these are the three
different types of workloads

that spark actually caters now,

let's move ahead and see
a simple demo here.

Let's understand how
to create a spark up.

Location in spark
shell using Scala.

So let's understand

how to create a spark
application in spark shell

using Scala assume

that we have a text file
in the hdfs directory

and we are counting the number
of words in that text file.

So, let's see how to do it.

So before I start running,
let me first check

whether all my demons
are running or not.

So I'll type sudo JPS

so all my spark demons
and Hadoop elements are running

that I have master/worker
as Park demon son named notice.

Manager non-manager everything
as Hadoop team it.

So the first thing

that I do here is
I run the spark shell

so it takes bit time
to start in the meanwhile.

Let me tell you the web UI port

for spark shell is
localhost for 0 4 0.

So this is a web
UI first Park like

if you click on jobs right now,
we have not executed anything.

So there is
no details over here.

So there you have job stages.

So once you execute the chops

If you'll be having
the records of the tasks

that you have executed here.

So here you can see
the stages of various jobs

and tasks executed.

So now let's check

whether our spark
shall have started or not.

Yes.

So you have your spark version
as two point one point one

and you have a scholar
shell over here.

So before I start the code,

let's check the content
that is present

in the input text file
by running this command.

So I'll write

where test is equal
to SC dot txt file

because I have saved
a text file over there

and I'll give
the hdfs part location.

I've stored my text file
in this location.

And Sample is the name
of the text file.

So now let me give
test dot collect

so that it collects the data

and displays the data that
is present in the text file.

So in my text file,

I have Hadoop research analysts
data science and science.

So this is my input data.

So now let me map

the functions and apply
the Transformations and actions.

So I'll give our map is equal
to SC dot txt file

and I will specify

my but location So this
is my input part location

and I'll apply
the flat map transformation

to split the data.

There are separated by space

and then map the word count to
be given as word comma one now.

This would be executed.

Yes.

Now, let me apply the action

for this to start
the execution of the task.

So let me tell you one thing
here before applying an action.

This park will not start
the execution process.

So here I have applied
produced by key

as the action to start
counting the number

of words in the text file.

So now we are done
with applying Transformations

and actions as well.

So now the next step is

to specify the output location
to store the output file.

So I will give
as counts dot save as text file

and then specify
the location form output file.

I'll sort it
in the same location

where I have my input file.

Never specify my output
file name as output 9 cool.

I forgot to give
a double quotes.

And I will run this.

So it's completed now.

So now let's see the output.

I will open my Hadoop web UI

by giving local lost Phi
double zero seven zero

and browse the file system
to check the output.

So as I have said,

I have example asthma director
that I have created

and in that I have specified
output 9 as my output.

So I have the two part
files been created.

Let's check each
of them one by one.

So we have the data count

as one analyst count
as one and science

count as two so this is
a first part file now.

Let me open the second
part file for you.

So this is the second
part file there you

have Hadoop count as one
and the research count as one.

So now let me show
you the text file

that we have specified
as the input.

So as I have told

you Hadoop counters
one research count as

one analyst one data one signs
and signs as 1 1 so

in might be thinking
data science is a one word

no in the program code.

We have asked to count the word
that the separated by a space.

So that is why we have
science count as two.

I hope you got an idea
about how word count works.

Similarly, I will now
paralyzed 1/200 numbers

and divide the tasks

into five partitions to show
you what is partitions of tusks.

So I will write a seedot
paralyzed 1/200 numbers

and divide them
into five partitions

and apply collect action
to collect the numbers

and start the execution.

So it displays you
an array of 100 numbers.

Now, let me explain you the job
stages partitions even timeline.

Dag representation
and everything.

So now let me go
to the web UI of spark

and click on jobs.

So these are the tasks

that have submitted so
coming to word count example.

So this is the
dagger usual ization.

I hope you can see
it clearly first

you collected the text file,

then you applied
flatmap transformation

and mapped it to count
the number of words

and then applied
Reduce by key action

and then save the output file

as save as text file.

So this is Entire
tag visualization

of the number of steps

that we have covered
in our program.

So here it shows
the completed stages

that is two stages
and it also shows the duration

that is 2 seconds.

And if you click
on the event timeline,

it just shows the executor
that is added.

And in this case you
cannot see any partitions

because you have not split the
jobs into various partitions.

So this is how you can see
the even timeline and the -

visualization here you
you can also see

the stage ID descriptions
when you have submitted

that I have just
submitted it now

and in this it also
shows the duration

that it took to execute the task
and the output pipes

that it took the shuffle
rate Shuffle right

and many more now to show
you the partitions see

in this you just applied
SC dot paralyzed, right?

So it is just showing
one stage where you

have applied the parallelized
transformation here.

It shows the succeeded
task as Phi by Phi.

That is you have divided
the task into five stages

and all the five stages has been
executed successfully now here

you can see the partitions
of the five different stages

that is executed in parallel.

So depending on the colors,

it shows the scheduler delay

the shuffle rate time
executor Computing time result

civilization time

and getting result time
and many more

so you can see that duration

that it took to execute
the five tasks in parallel

at the same time as maximum.

Um one milliseconds.

So in memory spark as
much faster computation

and you can see the IDS

of all the five different
tasks all our success.

You can see the locality level.

You can see the executor and
the host IP ID the launch time

the duration it take everything

so you can also see

that we have created our DT
and paralyzed it similarly here

also for word count example,

you can see the rdd

that has been created
and also the Actions

that have applied
to execute the task

and you can see the duration
that it took even here also,

it's just one milliseconds

that it took to execute
the entire word count example,

and you can see the ID is
locality level executor ID.

So in this case,

we have just executed
the task in two stages.

So it is just showing
the two stages.

So this is all about

how web UI looks and what are
the features and information

that you can see
in the web UI of spark

after executing the program
and the Scala shell.

So in this program,

you can see that first gave
the part to the input location

and check the data

that is presented
in the input file.

And then we applied
flatmap Transformations

and created rdd

and then applied action to start
the execution of the task

and save the output file
in this location.

So I hope you got a clear idea

of how to execute a word
count example and check

for the various features

and Spark web UI like
partitions that visualisations

and I hope you found the session
interesting Apache spark.

This word can generate a spark
in every Hadoop Engineers mind.

It is a big data
processing framework,

which is lightning fast
and cluster Computing.

And the core reason behind
its outstanding performance is

the resilient distributed
data set or in short.

They are DD and today I'll focus

on the topic called
rdd using spark

before we get Get started.

Let's have a quick look
on the agenda.

For today's session.

We shall start with
understanding the need for rdds

where we'll learn

the reasons behind which
the rdds were required.

Then we shall learn
what our rdds

where will understand
what exactly an rdd is

and how do they work later?

I'll walk you through
the fascinating features

of rdds such as in
memory computation partitioning

persistence fault tolerance

and many more

once I finished a theory

I'll get your hands on rdds

where will practically create

and perform all possible
operations on a disease

and finally I'll wind

up this session with
an interesting Pokémon use case,

which will help you understand
rdds in a much better way.

Let's get started spark is one

of the top mandatory skills
required by each

and every Big Data developer.

It is used
in multiple applications,

which need real-time processing
such as Google's

recommendation engine credit
card fraud detection.

And many more to understand
this in depth.

We shall consider Amazon's
recommendation engine assume

that you are searching
for a mobile phone

and Amazon and you have certain
specifications of your choice.

Then the Amazon search engine
understands your requirements

and provides you the products

which match the specifications
of your choice.

All this is made possible
because of the most

powerful tool existing
in Big Data environment,

which is none other
than Apache spark

and resilient distributed data.

Is considered to be
the heart of Apache spark.

So with this let's begin
our first question.

Why do we need a disease?

Well, the current world
is expanding the technology

and artificial intelligence

is the face for this Evolution
the machine learning algorithms

and the data needed
to train these computers

are huge the logic

behind all these algorithms
are very complicated

and mostly run in a distributed
and iterative computation method

the machine learning
algorithms could not use

the older mapreduce.

Grams, because the traditional
mapreduce programs needed

a stable State hdfs and we know

that hdfs generates redundancy
during intermediate computations

which resulted in a major
latency in data processing

and in hdfs gathering data

for multiple processing units
at a single instance.

First time consuming along
with this the major issue

was the HTF is did not have
random read and write ability.

So using this old
mapreduce programs

for machine learning
problems would be Then

the spark was introduced

compared to mapreduce spark

is an advanced big data
processing framework resilient

distributed data set

which is a fundamental
and most crucial data structure

of spark was the one

which made it all possible rdds
are effortless to create

and the mind-blowing
property with solve.

The problem was it's in memory
data processing capability

Oddity is not a distributed
file system instead.

It is a distributed
collection of memory

where the data needed
is always stored

and kept available.

Lynn RAM and because of
this property the elevation it

gave to the memory
accessing speed was unbelievable

The Oddities our fault tolerant

and this property bought it
a Dignity of a whole new level.

So our next question would be

what are rdds the resilient
distributed data sets

or the rdds are

the primary underlying
data structures of spark.

They are highly fault tolerant

and the store data
amongst multiple computers

in a network the data is written
into multiple executable notes.

So that in case of a Calamity
if any executing node fails,

then within a fraction
of second it gets back up

from the next executable node

with the same processing speeds
of the current node,

the fault-tolerant property
enables them to roll back

their data to the original state

by applying simple
Transformations on

to the Lost part
in the lineage hard.

It is do not need
anything called hard disk

or any other secondary storage

all that they need
is the main memory,

which is Ram now

that we have understood
the need for our dear.

It is and what exactly

an RTD is so let us see
the different sources

from which the data
can be ingested into an rdd.

The data can be loaded
from any Source

like hdfs hbase high C ql

you name it?

They got it.

Hence.

The collected data
is dropped into an rdd.

And guess what the rdds
a free-spirited they

can process any type of data.

They won't care if the data
is structured unstructured

or semi-structured now,

let me walk you
through the features.

Just of rdds,

which give it an edge
over the other Alternatives

in memory computation the idea

of in memory computation bought
the groundbreaking progress

in cluster Computing it
increase the processing speed

when compared with the hdfs
moving on to Lacey evaluations

the phrase lazy

Explains It

All spark logs all
the Transformations you apply

onto it and will not throw
any output onto the display

until an action is provoked.

Next is Fault tolerance rdds
are Lutely, fault-tolerant.

Any lost partition of an rdd
can be rolled back by applying

simple Transformations on
to the last part

in the lineage speaking

about immutability the data once
dropped into an rdd is immutable

because the access provided
by our DD is just re

only the only way to access

or modified is by applying
a transformation on to an rdd

which is prior
to the present one

discussing about partitioning.

The important reason for Sparks.

Parallel processing is
its part issue.

By default spot determines
the number of Parts

into which your data is divided,

but you can override this
and decide the number of blocks.

You want to split your data.

Let's see what persistence is

Sparks are it is
a totally reusable.

The users can apply

certain number of
Transformations on to an rdd

and preserve the final Oddity

for future use this avoids
all the hectic process

of applying all
the Transformations from scratch

and now last but not the least
course crane operations.

The operations performed
on rdds using Transformations

like map filter flat map
Etc change the arteries

and update them.

Hence.

Every operation applied
onto an RTD is course trained.

These are the features of rdds
and moving on to the next stage.

We shall understand.

The creation of rdds art.

It is can be created
using three methods.

The first method is using
parallelized collections.

Next method is by using external
storage like hdfs hbase.

Hi.

And many more the third one
is using an existing ID,

which is prior
to the present one.

Now, let us see understand

and create an array D
through each method now

Spa can be run on Virtual
machines like spark VM

or you can install
a Linux operating system

like Ubuntu and
run it Standalone,

but we here at Erica use
the best-in-class cloud lab

which comprises of
all the Frameworks.

You needed a single
stop Cloud framework.

No need of any hectic.

Has of downloading any file

or setting up
an environment variables

and looking for
a hardware specification Etc.

All you need is a login ID

and password to the all-in-one
ready to use cloud lab

where you can run
and save all your programs.

Let us fire up our spark shell
using the command spark to -

shell now as partial
is been fired up.

Let's create a new rdd.

So here we are creating
a new RTD with the first method

which is using the
parallelized collections here.

We are creating a new rdt

by the name parallelized
collections are ready.

We are starting a spark context

and we have paralyzing
an array into the rdd

which consists of the data
of the days of a week,

which is Monday Tuesday,

Wednesday, Thursday,
Friday and Saturday.

Now, let's create
this our new rdd

paralyzed collections rdd
is successfully created now,

let's display the data
which is present in our RTD.

So this was the data
which is present in our RTD now,

let's create a new ID
using a second method.

The second method
of creating an rdd

was using an external storage
such as hdfs high SQL

and many more here.

I'm creating a new rdd
by the name spark file

where I'll be loading
a text document into the rdd

from an external storage,

which is hdfs.

And this is the location
where my text file is located.

So the new rdd spark file
is successfully created now,

let's display the data

which is present
in as pack file a TD.

It's the data

which is present in
as pack file ID is

a collection of alphabets
starting from A to Z.

Now.

Let's create a new already
using the third method

which is using
an existing iridium,

which is prior to the present
one in the third method.

I'm creating a new Rd
by the name verts and

I'm creating a spark context

and paralyzing a statement
into the RTD Words,

which is spark is
a very powerful language.

So this is
a collection of Words,

which I have passed
into the new.

You are DD words.

Now.

Let us apply a transformation

on to the RTD and create
a new artery through that.

So here I'm applying
map transformation

on to the previous rdd

that is words and I'm storing
the data into the new ID

which is WordPress.

So here we are applying
map transformation in order

to display the first letter
of each and every word

which is stored
in the RTD words.

Now, let's continue.

The transformation is been
applied successfully now,

let's display the contents
which are present in new ID

which is word pair So

as explained we have displayed
the starting letter of each

and every word

as s is starting letter of spark
is starting letter of East and

so on L is starting
letter of language.

Now, we have understood
the creation of a dedes.

Let us move on

to the next stage where we'll
understand the operations

that are performed
on rdds Transformations

and actions are
the two major operations

that are performed on added.

He's let us understand what
our Transformations we applied.

Summations in order to access
filter and modify the data

which is present in an rdd.

Now Transformations are further
divided into two types

narrow Transformations and
why Transformations now,

let us understand what
our narrow Transformations

we apply narrow Transformations
onto a single partition

of parent ID

because the data required
to process the RTD is available

on a single partition

of parent additi the examples

for neurotransmission
our map filter.

At map partition
and map partitions.

Let us move on to the next
type of Transformations

which is why Transformations.

We apply why Transformations
on to the multiple partitions

of parent a greedy

because the data required
to process an rdd is available

on multiple partitions

of the parent
additi the examples

for why Transformations
are reduced by and Union now,

let us move on to the next part

which is actions actions
on the other hand

are considered to be
the next part of operations,

which are used
to display the final.

The examples for actions
are collect count take

and first till now
we have discussed

about the theory part on rdd.

Let us start
executing the operations

that are performed on a disease.

In a practical part
will be dealing with an example

of IPL match stata.

So here I have a CSV file

which has the IPL match records
and this CSV file is stored

in my hdfs and I'm loading.

My batch is dot CSV file
into the new rdd,

which is CK file as a text file.

So the match is dot CSV file
is been successfully loaded

as a text file into the new ID,

which is CK file now,

let us display the data
which is present

in our seek a file
using an action command.

So collect is the action command

which I'm using in order
to display the data

which is present
in my CK file a DD.

So here we have in total
six hundred and thirty six rows

of data which consists
of IPL match records

from the year 2008 to 2017.

Now, let us see the schema
of a CSV file.

I am using the action command
first in order to display

the schema of a match
is dot CSV file.

So this command will display
the starting line

of the CSV file.

We have so the schema

of a CSV file is the ID
of the match season city

where the IPL match
was conducted date

of the match team one team
two and so on now,

let's perform the further
operations on a CSV file.

Now moving on
to the further operations.

I'm about to split
the second column of my CSV file

which consists the information
regarding the states

which conducted the IPL matches.

So I am using this operation
in order to display

the states where
the matches were conducted.

So the transformation
is been successfully applied

and the data has been stored
into the new ID which is States.

Now, let's display the data
which is stored

in our state's rdd using
the collection action command,

so these with The states

where the matches
were being conducted now,

let's find out the city

which conducted the maximum
number of IPL matches.

Yeah, I'm creating
a new ID again,

which is States count
and I'm using map transformation

and I am counting each
and every city and the number

of matches conducted
in that particular City.

The transformation
is successfully applied

and the data has been stored
into the account ID.

Now.

Let us create a new editing
by name State count em

and apply reduced by

key transformation and map
transformation together

and consider topple one as
the city name and toppled

to as the Number of matches

which were considered
in that particular City

and apply sort

by K transformation
to find out the city

which conducted maximum number
of IPL matches.

The Transformations
are successfully applied

and the data is being stored
into the state count.

Em RTD now let's
display the data

which is present in state count.

Em, I did here I am using

take action command in order
to take the top 10 results

which are stored
in state count MRDD.

So according to the results
we have Mumbai

which Get the maximum number
of IPL matches,

which is 85 since the year
2008 to the year 2017.

Now let us create a new ID
by name fil ardi and use

flat map in order to filter
out the match data

which were conducted
in the city Hydra path

and store the same data
into the file rdd

since transformation is been
successfully applied now,

let us display the data
which is present in our fil ardi

which consists of the matches

which were conducted
excluding the city Hyderabad.

So this is the data

which is present in our fil ardi
D which excludes the matches

which are played
in the city Hyderabad now,

let us create another rdd

by name fil and store
the data of the matches

which were conducted
in the year 2017.

We shall use
filter transformation

for this operation.

The transformation is
been applied successfully

and the data has been stored
into the fil ardi now,

let us display the data
which is present there.

Michelle use collect

action command and now we have
the data of all the matches

which your plate especially
in the year 2070.

similarly, we can find
out the matches

which were played
in the year 2016 and we

can save the same data
into the new rdd

which is fil to Similarly,

we can find out the data
of the matches

which were conducted in the year
2016 and we can store

the same data into our new rdd

which is fil to I
have used filter transformation

in order to filter out
the data of the matches

which were conducted
in the year 2016 and I

have saved the data
into the new RTD

which is a file to now,

let us understand
the union transformation

which will apply

the union transformation on
to the fil ardi and fil to rdd.

In order to combine
both the data is present

in both The Oddities here.

I'm creating a new rdd

by the name Union rdd and I'm
applying Union transformation

on the to Oddities
that we created before.

The first one is fil ardi
which consists of the data

of the matches played
in the year 2017.

And the second one is a file

to which consists
the data of the matches.

Which up late in the year
2016 here I'll be clubbing

both the R8 is together

and I'll be saving the data
into the new rdd.

Which is Union rdd.

Now let us display the data
which is present in a new array,

which is Union rgd.

I am using collect
action command in order

to display the data.

So here we have the data
of the matches

which were played in the u.s.

2016 and 2017.

And now let's continue
with our operations

and find out the player
with maximum number of man

of the match awards
for this operation.

I am applying map transformation

and splitting out
the column number 13,

which consists of the data
of the players who won the man

of the match awards
for that particular match.

So the transformation
is been successfully applied

and the column number
13 is been successfully split

and the data has been
stored into the man

of the match our DD now.

We are creating a new rdd
by the named man

of the match count me applying
map Transformations on

to a previous rdd

and we are counting the number

of awards won by each and
every particular player.

Now, we shall create a new ID
by the named man of the match

and we are applying reduced
by K. Under the previous added

which is man of the match count.

And again, we are applying
map transformation

and considering topple one
as the name of the player

and topple to as
the number of matches.

He played and won the man
of the match Awards,

let us use take action command
in order to print the data

which is stored in our new RTD
which is man of the match.

So according to the result
we have a bws

who won the maximum number
of man of the matches,

which is 15.

So these are the few operations
that were performed on rdds.

Now, let us move on
to our Pokémon use case

so that we can understand
our duties in a much better way.

So the steps to be performed
in Pokémon use cases are loading

the Pokemon data dot CSV file

from an external storage
into an rdd removing the schema

from the Pokémon
data dot CSV file

and finding out the total number
of water type Pokemon

finding the total number
of fire type Pokemon.

I know it's getting interesting.

So let me explain you each
and every step practically.

So here I am creating
a new identity

by name Pokemon data rdd one

and I'm loading my CSV file
from an external storage.

That is my hdfs as a text file.

So the Pokemon data dot CSV file
is been successfully loaded

into our new rdd.

So let us display the data

which is present
in our Pokémon data rdd one.

I am using collect
action command for this.

So here we have 721 rows
of data of all the types

of Pokemons we have So now
let us display the schema

of the data we have

I have used the action command
first in order to display

the first line of a CSV file

which happens to be
the schema of a CSV file.

So we have index

of the Pokemon name
of the Pokémon.

Its type total points
HP attack points defense points

special attack special
defense speed generation,

and we can also find

if a particular Pokemon
is legendary or not.

Here, I'm creating a new RTD

which is no header

and I'm using filter operation
in order to remove the schema

of a Pokemon data dot CSV file.

The schema of Pokemon data
dot CSV file is been removed

because the spark
considers the schema

as a data to be processed.

So for this reason,

we remove the schema now,
let's display the data

which is present
in a no-hitter rdd.

I am using action command

collect in order
to display the data

which is present
in no header rdd.

So this is the data

which is stored
in a no-hitter rdd

without the schema.

So now let us find out
the number of partitions

into which are no header are
ready is been split in two.

So I am using partitions
transformation in order to find

out the number of partitions.

The data was split
in two according to the result.

The no header rdd is been split
into two partitions.

I am here creating a new rdt
by name water rdd

and I'm using filter
transformation in order to find

out what a type Pokemons in
our Pokémon data dot CSV file.

I'm using action command collect
in order to print the data

which is present in water rdd.

So these are the total number
of water type Pokemon

that we have in our
Pokémon data dot CSV.

Similarly.

Let's find out
the fire type Pokemons.

I'm creating a new identity
by the name fire RTD

and applying filter operation
in order to find out

the fire type Pokemon
present in our CSV file.

I'm using collect action command
in order to print the data

which is present in fire rdd.

So these are the fire type
Pokemon which are present

in our Pokémon
data dot CSV file.

Now, let us count the total
number of water type Pokemon

which are present
in a Pokemon data dot CSV file.

I am using count action for this
and we have 112 water type

Pokemon is present in
our Pokémon data dot CSV file.

Similarly.

Let's find out the total number
of fire-type Pokémon

as we have I'm using count
action command for the same.

So we have a total 52 number

of fire type Pokemon Sinnoh
Pokemon data dot CSV files.

Let's continue with
our further operations

where we'll find out a highest
defense strength of a Pokémon.

I am creating a new ID
by the name defense list

and I'm applying
map transformation

and spreading out
the column number six in order

to extract the defense points

of all the Pokemons present in
our Pokémon data dot CSV file.

So the data is been stored
successfully into a new era.

DD which is defenseless.

Now.

I'm using Mac's action command
in order to print out

the maximum different strengths
out of all the Pokemons.

So we have 230 points as
the maximum defense strength

amongst all the Pokemons.

So in our further operations,

let's find out the Pokemons
which come under the category

of having highest
different strengths,

which is 230 points.

In order to find out
the name of the Pokemon

with highest defense strength.

I'm creating a new identity
with the name.

It defense with Pokemon name
and I'm applying

May transformation on
to the previous array,

which is no header

and I'm splitting out

column number six which happens
to be the different strengths

in order to extract the data
from that particular row,

which has the defense
strength as 230 points.

Now I'm creating a new RTD again

with the name maximum defense
Pokemon and I'm applying

group bike a transformation
in order to display the Pokemon

which have the maximum defense
points that is 230 points.

So according to the result.

We have Steelix Steelix
Mega chacal Aggregate

and aggregate Mega

as the Pokemons with
highest different strengths,

which is 230 points.

Now we shall find
out the Pokemon

which is having least
different strengths.

So before we find
out the Pokemon

with least different strengths,

let us find out
the least defense points

which are present
in the defense list.

So in order to find
out the Pokémon

with least different strengths,

I have created a new rdt

by name minimum defense Pokemon
and I have applied distinct

and sort by Transformations
on to the defense list rdd

in order to extract
the least defense points present

in the defense list

and I have used take
action command in order

to display the data

which is present
in minimum defense Pokemon rdd.

So according to the results,

we have five points as
the least defense strength

of a particular Pokémon now,

let us find out
the name of the On

which comes under the category
of having Five Points as

different strengths now,

let us create a new rdd

which is difference Pokemon name
to and apply my transformation

and split the column number 6
and store the data

into our new rdd

which is defense
with Pokemon name, too.

The transformation is
been successfully applied

and the data is now
stored into the new rdd

which is defense

with Pokemon name to the data
is been successfully loaded.

Now, let us apply
the further operations here.

I am creating another rdd with
name minimum defense Pokemon

and I'm applying group bike
a transformation in order

to extract the data from the row

which has the defense
points as 5.0.

The data is been successfully
loaded now and let us display.

The data which is present

in minimum defense Pokemon rdd
now according to the results.

We have to number of Pokemons,

which come under the category
of having Five Points

as that defense strength
the Pokemons chassis knee

and happening at
the to Pokemons,

which I have in the least
definition the world

of Information Technology

and big data processing started
to see multiple potentialities

from spark coming into action.

Such Pinnacle in Sparks
technology advancements is

the data frame.

And today we shall
understand the technicalities

of data frames

and Spark a data frame and Spark
is all about performance.

It is a powerful multifunctional
and an integrated data structure

where the programmer can work
with different libraries

and perform numerous
functionalities without breaking

a sweat to understand apis

and libraries involved

in the process
without wasting any time.

Let us understand a topic
for today's discussion.

I line up the docket
for understanding.

Data frames and Spark is below

which will begin with
what our data frames here.

We will learn what
exactly a data frame is.

How does it look like and what
are its functionalities then we

shall see why do we need
data frames here?

We shall understand
the requirements which led us

to the invention
of data frames later.

I'll walk you through
the important features

of data frames.

Then we should look

into the sources from which
the data frames and Spark

get their data from Once
the theory part is finished.

I will get us involved
into the Practical part

where the creation
of a dataframe happens to be

a first step next we shall work
with an interesting example,

which is related to football

and finally to understand
the data frames

in spark in a much

better way we should work
with the most trending topic

as I use case,

which is none other
than the Game of Thrones.

So let's get started.

What is a data frame
in simple terms a data frame

can be considered as a
distributed collection of data.

The data is organized
under named columns,

which provide us The operations
to filter group process

and aggregate the available data
data frames can also be used

with Sparks equal and we
can construct data frames

from structured data files rdds
or from an external storage

like hdfs Hive Cassandra hbase

and many more with
this we should look

into a more simplified example,

which will give us a basic
description of a data frame.

So we shall deal
with an employee database

where we have entities
and their data types.

So the name of the employee
is a first entity

And its respective data type

is string data type similarly
employee ID has data type

of string employee phone number

which is integer data type

and employ address happens
to be string data type.

And finally the employee salary
is float data type.

All this data is stored
into an external storage,

which may be hdfs Hive

or Cassandra using
the data frame API

with their respective schema,

which consists of the name

of the entity along
with this data type now

that we have understood what
exactly a data frame is.

Let us quickly move on
to our next stage

where we shall understand the
requirement for a data frame.

It provides as

multiple programming
language support ability.

It has the capacity to work
with multiple data sources,

it can process both structured
and unstructured data.

And finally it is
well versed with slicing

and dicing the data.

So the first one is

the support ability for
multiple programming languages.

The IT industry
is required a powerful

and an integrated data structure

which could support

multiple programming languages
and at the same.

Same time without
the requirement of

additional API data frame
was the one stop solution

which supported multiple
languages along with a single

API the most popular languages

that a dataframe could
support our our python.

Skaila, Java and many more
the next requirement

was to support
the multiple data sources.

We all know that in
a real-time approach

to data processing
will never end up

at a single data
source data frame is

one such data structure,

which has the capability
to support and process data.

From a variety of data
sources Hadoop Cassandra.

Json files hbase.

CSV files are the examples
to name a few.

The next requirement was
to process structured

and unstructured data.

The Big Data environment was
designed to store huge amount

of data regardless

of which type exactly

it is now Sparks data frame
is designed in such a way

that it can store a huge
collection of both structured

and unstructured data

in a tabular format
along with its schema.

The next requirement was slicing
In in dicing data now,

the humongous amount

of data stored in Sparks
data frame can be sliced

and diced using the operations
like filter select group

by order by and many

more these operations
are applied upon the data

which are stored in form
of rows and columns

in a data frame these
with a few crucial requirements

which led to the invention
of data frames.

Now, let us get
into the important features

of data frames

which bring it an edge
over the other alternatives.

Immutability lazy
evaluation fault tolerance

and distributed memory storage,

let us discuss about each
and every feature in detail.

So the first one is
immutability similar to

the resilient distributed data
sets the data frames

and Spark are also immutable
the term immutable depicts

that the data was stored

into a data frame
will not be altered.

The only way to alter the data
present in a data frame

would be by applying
simple transformation operations

on to them.

So the next feature
is lazy evaluation.

Valuation lazy evaluation
is the key to the remarkable

performance offered by spark
similar to the rdds data frames

in spark will not throw
any output onto the screen

until and unless an action
command is encountered.

The next feature
is Fault tolerance.

There is no way

that the Sparks data frames
can lose their data.

They follow the principle
of being fault tolerant

to the unexpected calamities

which tend to destroy
the available data.

The next feature is distributed

storage Sparks dataframe
distribute the data.

Most multiple locations

so that in case of a node
failure the next available node

can takes place to continue
the data processing.

The next stage will be
about the multiple data source

that the spark dataframe
can support the spark API

can integrate itself

with multiple programming
languages such as scalar Java

python our MySQL

and many more making
itself capable to handle

a variety of data sources
such as Hadoop Hive hbase

Cassandra, Json file.

As CSV files my SQL
and many more.

So this was the theory part

and now let us move
into the Practical part

where the creation

of a dataframe happens
to be a first step.

So before we begin
the Practical part,

let us load the libraries

which required in order to
process the data in data frames.

So these are the few libraries
which we required

before we process the data
using our data frames.

Now that we have loaded
all the libraries

which we required to process
the data using the data frames.

Let us begin with the creation
of our data frame.

So we shall create a new data
frame with the name employee

and load the data

of the employees present
in an organization.

The details of the employees
will consist the first name

the last name and their mail ID
along with their salary.

So the First Data frame is
been successfully created now,

let us design the schema
for this data frame.

So the schema for this data
frame is been described

as shown the first name is of
string data type and similarly.

The last name is
a string data type

along with the mail address.

And finally the salary
is integer data type

or you can give
flow data type also,

so the schema has been
successfully delivered now,

let us create
the data frame using

Create data frame function here.

I'm creating a new data frame
by starting a spark context

and using the create
data frame method and loading

the data from Employee
and employer schema.

The data frame is
successfully created now,

let's print the data

which is existing
in the dataframe EMP DF.

I am using show method here.

So the data

which is present in EMB DF is
been successfully printed now,

let us move on to the next step.

So the next step for our today's
discussion is working

with an example related
to the FIFA data set.

So the first step
in our FIFA example

would be loading the schema
for the CSV file.

We are working with so

the schema has been
successfully loaded now.

Now let us load the CSV file
from our external storage

which is hdfs
into our data frame,

which is FIFA DF.

The CSV file is been
successfully loaded

into our new data frame,

which is FIFA DF now,

let us print the schema
of a data frame using

the print schema command.

So the schema
is been successfully

displayed here and we have
the following credentials.

Of each and every player
in our CSV file now,

let's move on to a further
operations on a dataframe.

We will count the total number
of records of the play

as we have in our CSV file
using count command.

So we have a total
of eighteen thousand

to not seven players
in our CSV files.

Now, let us find out the details

of the columns on which
we are working with.

So these were the columns
which we are working with which

consists the idea of the player
name age nationality potential

and many more.

Now let us use the column value

which has the value of each

and every player
for a particular T and let

us use describe command in order
to see the highest value

and the least value
provided to a player.

So we have account
of a total number of 18,000

to not seven players

and the minimum worth
given to a player is 0

and the maximum is given
as 9 million pounds.

Now, let us use
the select command

in order to extract
the column name and nationality.

How to find out the name of each

and every player along
with his nationality.

So here we have we can display
the top 20 rows of each

and every player

which we have in our CSV file
along with us nationality.

Similarly.

Let us find out the players
playing for a particular Club.

So here we have
the top 20 Place playing

for their respective clubs

along with their names

for example messy
playing for Barcelona

and Ronaldo for
Juventus and Etc.

Now, let's move
to the next stages.

No, let us find out the players

who are found to be most active
in a particular national team

or a particular club
with h less than 30 years.

We shall use

filter transformation
to apply this operation.

So here we have the details

of the Players whose age
is less than 30 years

and their club and nationality
along with their jersey numbers.

So with this we have finished
our FIFA example now

to understand the data frames
in a much better way,

let us move on
into our use case,

which is about the most Hot
Topic The Game of Thrones.

Similar to our previous example,

let us design the schema
of a CSV file first.

So this is the schema
for a CSV file

which consists the data
about the Game of Thrones.

So, this is a schema
for our first CSV file.

Now, let us create the schema
for our next CSV file.

I have named the schema
for our next CSV file a schema

to and I've defined
the data types for each

and every entity the scheme
has been successfully designed

for the second CSV file also.

Now let us load our CSV files
from our external storage,

which is our hdfs.

The location of the first CSV
file character deaths dot CSV

is our hdfs,

which is defined as above

and the schema is been
provided as schema.

And the header true option
is also been provided.

We are using spark
read function for this

and we are loading this data
into our new data frame,

which is Game
of Thrones data frame.

Similarly.

Let's load the other CSV file

which is battles dot CSV
into another data frame,

which is Game of Thrones
Butters dataframe the CSV file.

Has been successfully
loaded now.

Let us continue
with the further operations.

Now let us print
the schema offer Game

of Thrones data frame using
print schema command.

So here we have the schema

which consists of
the name alliances

death rate book of death
and many more similarly.

Let's print the schema of Game
of Thrones Butters data frame.

So this is a schema
for our new data frame,

which is Game of Thrones
battle data frame.

Now, let's continue
the further operations.

Now, let us display
the data frame

which we have created using
the following command data frame

has been successfully printed
and this is the data

which we have in our data frame.

Now, let's continue
with the further operations.

We know that there are
a multiple number

of houses present in the story
of Game of Thrones.

Now, let us find out

each and every individual house
present in the story.

Let us use the following command
in order to display each

and every house present
in the Game of Thrones story.

So we have the following houses
in the Game of Thrones story.

Now, let's continue
with the further operations

the battles in the Game
of Thrones were fought for ages.

Let us classify the vast waste

with their occurrence
according to the years.

We shall use select
and filter transformation

and we shall access The Columns
of the details of the battle

and the year in which
they were fought.

Let us first find
out the battles

which were fought in the year.

R 298 the following
code consists of

filter transformation

which will provide the details
for which we are looking.

So according to the result.

These were the battles
were fought in the year

298 and we have the details
of the attacker Kings

and the defender Kings

and the outcome of the attacker
along with their commanders

and the location
where the war was fought now,

let us find out the wars
based in the air 299.

So these with the details

of the verse which were fought
in the year 299 and similarly,

let us also find out the bars
which are waged in the year 300.

So these were the words

which were fought
in the year 300.

Now, let's move on
to the next operations

in our use case.

Now, let us find out the tactics
used in the wars waged

and also find out the total
number of vast waste by using

each type of those tactics
the following code must help us.

Here we are using select

and group by operations
in order to find out each

and every type of tactics
used in the war.

So they have used Ambush sees
raising and Pitch type

of tactics inverse

and most of the times they
have used pitched battle type

of tactics inverse.

Now, let us continue
with the further operations

the Ambush type of battles are
the deadliest now,

let us find out the Kings

who fought the battles
using these kind of tactics

and also let us find out
the outcome of the battles

fought here the In code
will help us extract the data

which we need here.

We are using select
and we're commands

and we are selecting
The Columns year attacking

Defender King attacker outcome
battle type attacker Commander

defend the commander now,

let us print the details.

So these were the battles
fought using the Ambush tactics

and these were
the attacker Kings

and the defender Kings along
with their respective commanders

and the wars waste
in a particular year now.

Let's move on
to the next operation.

Now let us focus on the houses

and extract the deadliest house
amongst the rest.

The following code will help us
to find out the deadliest house

and the number
of patents the wage.

So here we have the details
of each and every house

and the battles the waged
according to the results.

We have stuck

and Lannister houses to be
the deadliest among the others.

Now, let's continue
with the rest of the operations.

Now, let us find out
the deadliest king

among the others

which will use the following
command in order to find

the deadliest king
amongst the other kings

who fought in the A
number of Firsts.

So according to the results
we have Joffrey as

the deadliest King

who fought a total number
of 14 battles.

Now, let us continue
with the further operations.

Now, let us find out the houses

which defended most number
of Wars waste against them.

So the following code must help
us find out the details.

So according to the results.

We have Lannister house
to be defending the most number

of paths based against them.

Now, let us find out
the defender King who defend

it most number of battles
which were waste against him

So according to the result drop
stack is the king

who defended most
number of patterns

which waged against him.

Now.

Let's continue with
the further operations.

Since Lannister house
is my personal favorite.

Let me find out the details

of the characters
in Lannister house.

This code will
describe their name

and gender one for male

and 0 for female along with
their respective population.

So let me find out
the male characters

in The Lannister house first.

So here we have used select
and we're commanded.

Ends in order to find out
the details of the characters

present in Lannister house

and the data is been stored
into tf1 dataframe.

Let us print the data
which is present in idea

of one data frame
using show command.

So these are the details

of the characters
present in Lannister house,

which are made now similarly.

Let us find out the female
character is present

in Lannister house.

So these are the characters
present in Lannister house

who are females

so we have a total number of
69 male characters and 12 number

of female characters
in The Lannister house.

Now, let us continue with
the next operations at the end

of the day every episode

of Game of Thrones had
a noble character.

Let us now find out all
the noble characters

amongst all the houses

that we have in our Game
of Thrones CSV file

the following code must help
us find out the details.

So the details

of all the characters
from all the houses

who are considered to be Noble.

I've been saved
into the new data frame,

which is DF 3 now,

let us print the details
from the df3 data frame.

So these are the top 20 members
from all the houses

who are considered to be Noble
along with their genders.

Now, let us count the total
number of noble characters

from the entire game
of thrones stories.

So there are a total
of four hundred and thirty

number of noble characters
existing in the whole game

of throne story.

Nonetheless, we have also

faced a few Communists
whose role in The Game

of Thrones is found
to be exceptional vision

of find out the details
of all those commoners

who were highly dedicated
to their roles in each episode

the data of all,

the commoners is
been successfully loaded

into the new data frame,

which is TFO now let
us print the data

which is present in the DF
for using the show command.

So these are the top
20 characters identified as

common as amongst all the Game
of Thrones stories.

Now, let us find out
the count of total number

of common characters.

So there are a total

of four hundred and
eighty seven common characters

amongst all stories
of Game of Thrones.

Let us continue
with the further operations.

Now they were a few rows

who were considered
to be important

and equally Noble, hence.

They were carried out
under the last book.

So let us filter
out those characters

and find out the details
of each one of them.

The data of all the characters
who are considered to be Noble

and carried out

until the last book are being
stored into the new data frame,

which is TFO now let
us print the data

which is existing

in the data frame for so
according to the results.

We have two candidates

who are considered to be
the noble and their character

is been carried on
until the last book

amongst all the battles.

I found the battles
of the last books

to be generating more
adrenaline in the readers.

Let us find out the details
of those battles using

the following code.

So the following code will help
us to find out the bars

which were fought
in the last year's

of the Game of Thrones.

So these are the details
of the vast which are fought

in the last year's
of the Game of Thrones

and the details of the Kings

and the details
of their commanders

and the location
where the war was fought.

Welcome to this interesting
session of Sparks SQL tutorial

from a drecker.

So in today's session,

we are going to learn about
how we will be working.

Spock sequent now what all you

can expect from this course
from this particular session

so you can expect that.

We will be first learning
by Sparks equal.

What are the libraries

which are present
in Sparks equal.

What are the important
features of Sparkle?

We will also be doing
some Hands-On example

and in the end we will see
some interesting use case

of stock market analysis now

Rice Park sequel is it

like Why we are learning it
why it is really important

for us to know about
this Sparks equal sign.

Is it like really hot in Market?

If yes, then why we want
all those answer from this.

So if you're coming
from her do background,

you must have heard a lot
about Apache Hive now

what happens in Apache.

I also like in Apache
Hive SQL developers

can write the queries in SQL way

and it will be getting converted

to your mapreduce
and giving you the out.

Now we all know

that mapreduce is
lower in nature.

And since mapreduce
is going to be slower

and nature then definitely

your overall high score
is going to be slower in nature.

So that was one challenge.

So if you have let's say
less than 200 GB of data

or if you have
a smaller set of data.

This was actually
a big challenge

that in Hive your performance
was not that great.

It also do not have
any resuming capability stuck.

You can just start it also.

- cannot even drop
your encrypted data bases.

That's was also one
of the challenge

when you deal with
the security side.

Now what sparks equal have done

it Sparks equal have solved
almost all of the problem.

So in the last sessions
you have already learned

about the smart way right House
Park is faster from mapreduce

and not we have already learned

that in the previous
few sessions now.

So in this session,

we are going to kind of take
a live range of all that so

definitely in this case

since This pack is
faster because of

the in-memory computation.

What is in memory competition?

We have already seen it.

So in memory computations
is like whenever we

are Computing anything
in memory directly.

So because of in memory
competition capability because

of arches purpose poster.

So definitely your spark SQL is
also been to become first know

so if I talk

about the advantages
of Sparks equal over Hive

definitely number one it
is going to be faster

in Listen to your hive
so a high quality,

which is let's say
you're taking around 10 minutes

in Sparks equal.

You can finish that same query
in less than one minute.

Don't you think it's
an awesome capability

of subsequent definitely as
right now second thing is

when if let's say you
are writing something and -

now you can take an example

of let's say a company
who is let's say developing -

queries from last 10 years.

Now they were doing it.

There were all happy

that they were able
to process picture.

That they were worried
about the performance

that Hive is not able
to give them a that level

of processing speed what
they are looking for.

Now this fossil.

It's a challenge
for that particular company.

Now, there's a challenge right?

The challenge is
they came to know know

about subsequent fine.

Let's say we came
to know about it,

but they came to know

that we can execute
everything is Park Sequel

and it is going to be
faster as well fine.

But don't you think that

if these companies working
for net set past 10 years?

In Hive they must have already
written lot of Gordon -

now if you ask them to migrate
to spark SQL is will it be

until easy task?

No, right.

Definitely.

It is not going
to be an easy task.

Why because Hive syntax
and Sparks equals and X though.

They boot tackle the sequel way
of writing the things

but at the same time
it is always a very

it carries a big difference,

so there will be a good
difference whenever we talk

about the syntax between them.

So it will take a very
good amount of time

for that company to change
all of the query mode

to the Sparks equal way
now Sparks equal came up

with a smart salvation

what they said is even

if you are writing
the query with -

you can execute
that Hive query directly

through subsequent don't you
think it's again

a very important
and awesome facility, right?

Because even now
if you're a good Hive developer,

you need not worry about

that how you will be now
that migrating to Sparks.

Well, you can still keep on
writing to the hive query

and can your query
will automatically be

getting converted to spot sequel
with similarly in Apache spark

as we have learned
in the past sessions,

especially through spark
streaming that Sparks.

The aiming is going to make
you real time processing right?

You can also perform
your real-time processing

using a purchase. / now.

This sort of facility is you

can take leverage even
you know Sparks ago.

So let's say you can do
a real-time processing

and at the same time
you can also Perform

your SQL query now the type

that was the problem.

You cannot do

that because when we talk
about Hive now in -

it's all about Hadoop is

all about batch
processing batch processing

where you keep historical data

and then later you
process it, right?

So it definitely Hive also
follow the same approach

in this case also high risk

going to just only follow
the batch processing mode,

but when it comes to a purchase,

but it will also be taking care
of the real-time processing.

So how all these things happens

so Our Park sequel always
uses your meta store

Services of your hive
to query the data stored

and managed by -

so in when you were
learning about high,

so we have learned at that time
that in hives everything.

What we do is always
stored in the meta Stone

so that met Esther was
The crucial point, right?

Because using that meta store

only you are able
to do everything up.

So like when you are doing
let's say or any sort of query

when you're creating a table,

everything was getting stored
in that same metal Stone.

What happens Spock sequel

also use the same metal Stone
now is whatever metal store.

You have created with respect
to Hive same meta store.

You can also use it

for your Sparks equal
and that is something

which is really awesome
about this spark sequent

that you did not create
a new meta store.

You need not worry
about a new storage space

and not everything

what you have done with respect
to your high same method

you can use it.

Now.

You can ask me then
how it is faster

if they're using
cymatics don't remember.

But the processing part
why high was lower

because of its processing way

because it is converting
everything to the mapreduce

and this it was making
the processing very very slow.

But here in this case

since the processing is going
to be in memory computation.

So in Sparks equal case,

it is always going to be
the faster now definitely

it just because of
the meta store site.

We are only able
to fetch the data are

not but at the same time
for any other thing

of the processing related stuff,

it is always going to be At the

when we talk about
the processing stage

it is going to be in memory
does it's going to be faster.

So let's talk about some success
stories of Sparks equal.

Let's see some use cases
Twitter sentiment analysis.

If you go through over

if you want sexy remember
our spark streaming session,

we have done a Twitter
sentiment analysis, right?

So there you have seen

that we have first initially
got the data from Twitter and

that to we have got
it with the help

of Sparks Damon and later

what we did later.

We just analyze everything
with the help of spot.

Oxycodone so you can see
an advantage as possible.

So in Twitter sentiment analysis

where let's say
you want to find out

about the Donald Trump, right?

You are fetching the data

every tweet related
to the Donald Trump

and then kind of bring
analysis in checking

that whether it's
a positive with negative

tweet neutral tweet,

very negative with very
positive to it.

Okay, so we have already
seen the same example there

in that particular session.

So in this session,

as you are noticing

what we are doing we
just want to kind of so

that once you're
streaming the data

and the real time
you can also do it.

Also, seeing using
spark sequel just you

are doing all the processing
at the real time similarly

in the stock market analysis.

You can use Park
sequel lot of bullies.

You can adopt the in the banking
fraud case Transitions and all

you can use that.

So let's say your credit
card current is getting swipe

in India and in next 10 minutes

if your credit card
is getting swiped

in let's say in u.s.

Definitely that is not possible.

Right?

So let's say you are doing all
that processing real-time.

You're detecting everything
with respect to sparsely me.

Then you are let's say applying
your Sparks equal to verify

that Whether it's
a user Trend or not, right?

So all those things you want
to match up as possible.

So you can do that similarly

the medical domain
you can use that.

Let's talk about
some Sparks equal features.

So there will be
some features related to it.

Now, you can use

what happens when this sequel
got combined with this path.

We started calling it

as Park sequel now

when definitely we are talking
about SQL be a talking

about either a structure data

or a semi-structured data now

SQL queries cannot deal
with the unstructured data,

so that is definitely one of
Thing you need to keep in mind.

Now your spark sequel also
support various data formats.

You can get a data from pocket.

You must have heard about Market

that it is a columnar
based storage and it

is kind of very much
compressed format of the data

what you have but it's
not human readable.

Similarly.

You must have heard
about Jason Avro

where we keep the value
as a key value pair.

Hi Cassandra, right?

These are nosql TVs

so you can get all the data
from these sources now.

You can also convert
your SQL queries

to your A derivative

so you can you can you
will be able to perform

all the transformation steps.

So that is one thing you can do.

Now if we talk about performance

and scalability definitely
on this red color graph.

If you notice this
is related to your Hadoop,

you can notice

that red color graph is much
more encompassing to blue color

and blue color denotes
my performance with respect

to Sparks equal

so you can notice that spark
SQL is performing much better

in comparison to your Hadoop.

So we are on this Y axis.

We are taking the running.

On the x-axis.

We were considering
the number of iteration

when we talk about
Sparks equal features.

Now few more features
we have for example,

you can create a connection
with simple your jdbc driver

or odbc driver, right?

These are simple
drivers being present.

Now, you can create

your connection with his path
SQL using all these drivers.

You can also create a user
defined function means let's say

if any function is
not available to you

and that gives you can create
your own functions.

Let's say if function
Is available use

that if it is not available,

you can create a UDF means
user-defined function

and you can directly execute

that user-defined function
and get your dessert sir.

So this is one example
where we have shown

that you can convert.

Let's say if you don't have
an uppercase API present

in subsequent how you
can create a simple UDF for a

and can execute it.

So if you notice there

what we are doing
let's get this is my data.

So if you notice in this case,

this is data set is
my data part.

So this is I'm generating
as a sequence.

I'm creating it as a data frame
see this 2df part here.

Now after that we
are creating a / U DF here

and notice we are converting
any value which is coming

to my upper case, right?

We are using this to uppercase
API to convert it.

We are importing this function
and then what we did now

when we came here,
we are telling that okay.

This is my UDF.

So UDF is upper by

because we have created
here also a zapper.

So we are telling

that this is my UDF
in the first step and then Then

when we are using it,

let's say with our datasets
what we are doing so data sets.

We are passing year
that okay, whatever.

We are doing convert it

to my upper developer you DFX
convert it to my upper case.

So see we are telling you
we have created our / UDF

that is what we are passing
inside this text value.

So now it is just
getting converted

and giving you all the output
in your upper case way

so you can notice
that this is your last value

and this is your
uppercase value, right?

So this got converted

to my upper case
in this particular.

Love it.

Now.

If you notice here
also same steps.

We are how to we
can register all of our UDF.

This is not being shown here.

So now this is
how you can do that spark

that UDF not register.

So using this API,

you can just register
your data frames now similarly,

if you want to get the output

after that you can get
it using this following me

so you can use the show API
to get the output

for this Sparks
equal at attacher.

Let's see that so what is Park

sequel architecture now is
Park sequel architecture

if we talked about so
what happens to your let

's say getting the data
of with using

your various formats, right?

So let's say you can get
it from your CSP.

You can get it
from your Json format.

You can also get it
from your jdbc format.

Now, they will be
a data source API.

So using data source API,

you can fetch the data
after fetching the data

you will be converting
to a data frame

where so what is data frame.

So in the last one
we have learned

that that when we were creating
everything is already

what we were doing.

So, let's say this was
my Cluster, right?

So let's say this is machine.

This is another machine.

This is another machine, right?

So let's say these are
all my clusters.

So what we were doing
in this case now

when we were creating all
these things are as were cluster

what was happening here.

We were passing
Oliver values him, right?

So let's say we
were keeping all the data.

Let's say block B1 was there

so we were passing all
the values and work creating it

in the form of in the memory
and we were calling

that as rdd now

when we were walking in SQL
we have to store the the data

which is a table of data, right?

So let's say there is a table

which is let's say
having column details.

Let's say name age.

Let's say here.

I have some value here
are some value here.

I have some value here
at some value, right?

So let's say I have some value
of this table format.

Now if I have to keep
this data into my cluster

what you need to do,

so you will be keeping first
of all into the memory.

So you will be having

let's say name H this column
to test first of all year

and after that you will be
having some details of this.

Perfect.

So let's say this much data,

you have some part
in the similar kind

of table with some other values
will be here also,

but here also you are going
to have column details.

You will be having name H
some more data here.

Now if you notice this
is sounding similar to our DD,

but this is not exactly
like our GD right

because here we are not only
keeping just the data but we

are also studying something
like a column in a storage

right?

We also the keeping
the column in all of it.

Data nodes or we can call it as
if Burke or not, right?

So we are also keeping
the column vectors

along with the rule test.

So this thing is called
as data frames.

Okay, so that is called
your data frame.

So that is what we are going to
do is we are going to convert it

to a data frame API then

using the data frame TSS or by
using Sparks equal to H square

or you will be processing
the results and giving

the output we will learn about
all these things in detail.

So, let's see this Popsicle
libraries now there are

multiple apis available.

This like we have
data source API we

have data frame API.

We have interpreter
and Optimizer and SQL service.

We will explore
all this in detail.

So let's talk about
data source appear

if we talk about data source API
what happens in data source API,

it is used to read

and store the structured
and unstructured data

into your spark SQL.

So as you can notice in Sparks
equal we can give fetch the data

using multiple sources

like you can get it
from hive take Cosette.

Inverse ESP Apache
BSD base Oracle DB so

many formats available, right?

So this API is going

to help you to get all the data
to read all the data store it

where ever you want to use it.

Now after that your data

frame API is going
to help you to convert

that into a named Colin

and remember I
just explained you

that how you store
the data in that

because here you are not keeping
like I did it.

You're also keeping
the named column as

well as Road it is That is
the difference coming up here.

So that is
what it is converting.

In this case.

We are using data
frame API to convert it

into your named column
and rows, right?

So that is what you
will be doing.

So at it also follows the same
properties like your IDs

like your attitude is
Pearl easily evaluated

in all same properties
will also follow up here.

Okay now interpret
an Optimizer and interpreter

and Optimizer step
what we are going to do.

So, let's see if we have
this data frame API,

so we are going to first
create this name.

Column then after that we
will be now creating an rdd.

We will be applying
our transformation step.

We will be doing over action
step right to Output the value.

So all those things

where it is happens it happening
in The Interpreter

and optimizes them.

So this is all happening
in The Interpreter and optimism.

So this is what all
the features you have.

Now, let's talk about
SQL service now in SQL service

what happens it is going
to again help you

so it is just doing the order.

Formation action the last day

after that using
spark SQL service,

you will be getting
your spark sequel outputs.

So now in this case whatever
processing you have done right

in terms of transformations
in all of that so you can see

that your sparkers SQL service
is an entry point for working

along the structure data
in your aperture spur.

Okay.

So it is going to kind of
help you to fetch the results

from your optimize data

or maybe whatever you
have interpreted before

so that is what it's doing.

So this kind of completes.

This whole diagram now,

let us see that how we
can perform a work queries

using spark sequin.

Now if we talk
about spark SQL queries,

so first of all,

we can go to spark cell itself
engine execute everything.

You can also execute
your program using

spark your Eclipse also
directing from there.

Also, you can do that.

So if you are let's say log in
with your spark shell session.

So what you can do,

so let's say you have first
you need to import this

because into point x
you must have heard

that there is something
called as Park session

which came so that is
what we are doing.

So in our last session
we have Have you learned

about all these things are
now Sparkstation is something

but we're importing after that.

We are creating sessions path
using a builder function.

Look at this.

So This Builder API you we
are using this Builder API,

then we are using the app name.

We are providing a configuration
and then we are telling

that we are going to create
our values here, right?

So we had that's why
we are giving get okay,

then we are importing
all these things right

once we imported after that

we can say that okay.

We were want to read
this Json file.

So this implies God
or Jason we want to read up here

and in the end we want
to Output this value, right?

So this d f becomes my data
frame containing store value

of my employed or Jason.

So this decent value
will get converted

to my data frame.

We're now in the end PR just
outputting the result now

if you notice here
what we are doing,

so here we are first

of all importing your spark
session same story.

We just executing it.

Then we are building
our things better in that.

We're going to
create that again.

We are importing it then
we are reading Json file

by using Red Dot Json API.

We are reading
never employed or Jason.

Okay, which is present
in this particular directory

and we are outputting
so can you can see

that Json format will be
the T value format.

But when I'm doing this DF
not show it is just showing

up all my values here.

Now.

Let's see how we
can create our data set.

Now when we talk about data set,

you can notice
what we're doing.

Now.

We have understood all
this stability the how we

can create a data set now
first of all in data set

what we do so So
in data set we can create

the plus you can see we
are creating a case class employ

right now in case class

what we are doing we are done
just creating a sequence

in putting the value Andrew H
like name and age column.

Then we are displaying
our output all this data

set right now.

We are creating a primitive data
set also to demonstrate mapping

of this data frames
to your data sets.

Right?

So you can notice
that we are using

to D's instead of 2 DF.

We are using two DS
in this case.

Now, you may ask me what's
the difference with respect

to data frame, right?

With respect to data frame

in data frame
what we were doing.

We were create
again the data frame

and data set both
exactly looks safe.

It will also be having
the name column in rows

and everything up.

It is introduced lately
in 1.6 versions and later.

And what is it

provides it it provides
a encoder mechanism using

which you can get

when you are let's say
reading the weight data back.

Let's say you are DC
realizing you're not doing

that step, right?

It is going to be faster.

So the performance
wise data set is better.

That's the reason it
is introduced later nowadays.

People are moving from
data frame two data sets Okay.

So now we are just outputting

in the end see the same
thing in the output.

But so we are creating
employ a class.

Then we are putting the value
inside it creating a data set.

We are looking
at the values, right?

So these are the steps we
have just understood them now

how we can read of a Phi so
we want to read the file.

So we will use three dot Json
as employee employee was

what remember case class which
we have created last thing.

This was the classic
we have created

your case class employee.

So we are telling
that we are creating like this.

We are just out
putting this value.

We just within shop
you can see this way.

We can see this output.

Also now, let's see

how we can add the schema
to rdd now in order

to add the schema to rdd
what we are going to do.

So in this case also,

you can look at we
are importing all the values

that we are importing all
the libraries whatever

are required then

after that we are using
this spark context text

by reading the data splitting it

with respect to comma then
mapping the attributes.

We will employ The case
that's what we have done

and putting converting
this values to integer.

So in then we are converting
to to death right after that.

We are going to create
a temporary viewer table.

So let's create
this temporary view employ.

Then we are going
to use part dot Sequel

and passing up our SQL query.

Can you notice that we
have now passing the value

and we are assessing
this employ, right?

We are assessing
this employee here.

Now, what is this employ
this employee was

of a temporary view
which we have created

because the challenge
in Sparks equalist

when Whether you want
to execute any SQL query you

cannot say select aesthetic
from the data frame.

You cannot do that.

There's this is
not even supported.

So you cannot do select extract
from your data frame.

So instead of that

what we need to do is we need
to create a temporary table

or a temporary view
so you can notice here.

We are using this create
or replace temp You by replace

because if it is already
existing override on top of it.

So now we are creating
a temporary table

which will be exactly similar
to mine this data frame now

you You can just directly
execute all the query

on your return preview
Autumn Prairie table.

So you can notice here
instead of using employ DF

which was our data frame.

I am using here temporary view.

Okay, then in the end,

we just mapping
the names and a right

and we are outputting the bells.

That's it.

Same thing.

This is just
an execution part of it.

So we are just showing
all the steps here.

You can see in the end.

We are outputting
all this value now

how we can add
the schema to rdd.

Let's see this transformation
step now in this case you Notice

that we can map
this youngster fact

the we're converting
this map name

into the string for
the transformation part, right?

So we are checking all
this value that okay.

This is the string type name.

We are just showing up
this value right now.

What were you doing?

We are using this map encoder
from the implicit class,

which is available to us
to map the name and Each pie.

Okay.

So this is
what we're going to do

because remember in
the employee is class.

We have the name and age column
that we want to map now.

Now in this case,

we are mapping
the names to the ages.

Has so you can notice

that we are doing for ages
of our younger CF data frame

that what we
have created earlier

and the result is an array.

So the result but you're going
to get will be an array

with the name map
to your respective ages.

You can see this output
here so you can see

that this is getting map.

Right.

So we are getting seeing
this output like name is John

it is 28 that is what
we are talking about.

So here in this case,
you can notice

that it was representing
like this in this case.

The output is coming out
in this particular format now,

let's talk about
how Can add the schema

how we can read the file
we can add a whiskey minor

so we will be first
of all importing the type class

into your passion.

So with this is

what we have done
by using import statement.

Then we are going to import
the row class into this partial.

So rho will be used
in mapping our DB schema.

Right?

So you can notice

we're importing this also then
we are creating an rdd called

as employ a DD.

So in case this case
you can notice

that the same priority
we are creating

and we are creating this
with the help of this text file.

So once we have create this we
are going to Define our schema.

So this is the scheme approach.

Okay.

So in this case,

we are going to Define it
like named and space

than H. Okay,

because they these were
the two I have in my data also

in this employed or tht

if you look at these

are the two data which
we have named NH.

Now what we can do
once we have done

that then we can split it
with respect to space.

We can say

that our mapping value
and we are passing it

all this value inside
of a structure.

Okay, so we are defining a burn
or fields are ready.

That is what we are doing.

See this the fields are ready,

which is going to now output
after mapping the employee ID.

Okay, so that is
what we are doing.

So we want to just do this
into my schema strength,

then in the end.

We will be obtaining this field.

If you notice this field
what we have created here.

We are obtaining
this into a schema.

So we are passing this
into a struct type

and it is getting converted
to be our scheme of it.

So that is what we will do.

You can see all
this execution same steps.

We are just executing
in this terminal now,

Let's see how we are going
to transform the results.

Now, whatever we
have done, right?

So now we have already created
already called row editing.

So let's create

that Rogue additive are going
to Gray and we want

to transform the employee ID
using the map function

into row already.

So let's do that.

Okay.

So in this case

what we are doing so look
at this employed reading

we are splitting it
with respect to coma

and after that we are telling
see remember we have name

and then H like this so

that's what you're telling
me telling that act.

Zero or my attributes

one and why we're trimming
it just inverted to ensure

if there is no spaces
and on which other

so those things we don't want
to unnecessarily keep up.

So that's the reason we are
defining this term statement.

Now after that after we
once we are done with this,

we are going to define
a data frame employed EF

and we are going to store
that rdd schema into it.

So now if you notice
this row ID,

which we have defined here

and schema which we have defined
in the last case right now

if you'll go back
and notice here.

Schema, we have created here
right with respect to my Fields.

So that schema and this value

what we have just
created here rowady.

We are going to pass it and say

that we are going
to create a data frame.

So this will help us
in creating a data frame now,

we can create our temporary view
on the base of employee

of let's create an employee
or temporary View and then

what we can do we can execute
any SQL queries on top of it.

So as you can see
SparkNotes equal we

can create all the SQL queries
and can directly execute

that now what we can do.

We want to Output the values
we can quickly do that.

Now.

We want to let's say display
the names of we can say Okay,

attribute 0 contains the name
we can use the show command.

So this is how we
will be performing the operation

in the scheme away now,

so this is the same output way
means we're just executing

this whole thing up.

You can notice here.

Also, we are just
saying attribute 0.0.

It is representing
or me my output now,

let's talk about Json data.

Now when we talk
about Json data,

let's talk about how we
can load our files and work on.

This so in this case,
we will be first.

Let's say importing
our libraries.

Once we are done with that.

Now after that we can just say

that retort Jason we are
just bringing up our employed

or Jason you see
this is the execution

of this part now similarly,

we can also write
back in the pocket

or we can also read
the value from parque.

You can notice this

if you want to write
let's say this value employee

of data frame to my market way

so I can sit right dot
right dot market.

So this will be created
employed or Park.

Be created and hear all
the values should be converted

to employed or packet.

Only thing is the data.

If you go and see
in this particular directory,

this will be a directory.

We should be getting created.

So in this data,
you will notice

that you will not be able
to read the data.

So in that case
because it's not human readable.

So that's the reason you
will not be able to do that.

So, let's say you want
to read it now so you

can again bring it back

by using Red Dot Market you are
reading this employed at pocket,

which I just created

then you are creating
a temporary view

or temporary table

and then By using
standard SQL you can execute

on your temporary table.

Now in this way.

You can read your pocket file
data and in then we are just

displaying the result see
the similar output of this.

Okay.

This is how we can execute
all these things up now.

Once we have done all this,

let's see how we
can create our data frames.

So let's create this file path.

So let's say we have created
this file employed or Jason

after that we can
create a data frame

from our Json path, right?

So we are creating this
by using retouch Jason then

we can Print the schema.

What does to this is going
to print the schema

of my employee data frame?

Okay, so we are going to use
this print schemer to print

up all the values then we
can create a temporary view

of this data frame.

So we are create doing

that see create or replace
temp you we are creating

that which we have seen
it last time also now

after that we can
execute our SQL query.

So let's say we are executing
our SQL query from employee

where age is between 18
and 30, right?

So this kind of SQL query.

Let's say we want
to do we can get

that And in the end we
can see the output Also.

Let's see this execution.

So you can see

that all the vampires who these
are let's say between 18 and 30

that is showing up
in the output.

Now.

Let's see this
rdd operation way.

Now what you can do

so we are going to create this
add any other employer Nene now

which is going to store
the content of employed George

and New Delhi Delhi.

So see this part,

so here we are creating this
by using make a DD

and we have just this is going
to store the content containing

Such from noodle, right?

You can see this

so New Delhi is my city
named state is the ring.

So that is what we
are passing inside it.

Now what we are doing we
are assigning the content

of this other employee ID
into my other employees.

So we are using
this dark dot RI dot Json

and we are reading at the value

and in the end we
are using this show appear.

You can notice
this output coming up now.

Let's see with the hive table.

So with the hive table
if you want to read that,

so let's do it
with the case class

and Spark sessions.

So first of all,

we are going to import
a guru class and we are going

to use path session
into the Spartan.

So let's do that for a way.

I'm putting this row
this past session

and not after that.

We are going to create a class
record containing this key

which is of integer data type

and a value which is
of string type.

Then we are going
to set our location

of the warehouse location.

Okay to this pathway rows.

So that is what we are doing.

Now.

We are going to build
a spark sessions back

to demonstrate the hive
example in spots equal.

Look at this now,

so we are creating Sparks
session dot Builder again.

We are passing the Any app name

to it we have passing
the configuration to it.

And then we are saying
that we want to enable

The Hive support now

once we have done

that we are importing
this spark SQL library center.

And then you can notice
that we can use SQL

so we can create now a table SRC

so you can see create table
if not exist as RC

with column to stores the data
as a key common value pair.

So that is what we
are doing here.

Now, you can see all
this execution of the same step.

Now.

Let's see the sequel operation
happening here now in this case

what we can do.

We can now load the data
from this example,

which is present to succeed.

Is this KV m dot txt file,

which is available to us

and we want to store it
into the table SRC

which we have just
created and now

if you want to just view the all
this output becomes a sequence

select aesthetic form SRC

and it is going to show up

all the values you
can see this output.

Okay.

So this is the way you can show
up the virus now similarly we

can perform the count operation.

Okay, so we can say
select Counter-Strike

from SRC to select the number
of keys in there.

See tables, and now
select all the records,

right so we can say
that key select key gamma value

so you can see

that we can perform all
over Hive operations here

on this right similarly.

We can create a data set
string DS from spark DF

so you can see this
also by using SQL DF

what we already have
we can just say map

and then provide

the case class in can map
the ski common value pair

and then in the end we
can show up all this value see

this execution of this in then
you can notice this output

which we want it now.

Let's see the result back.

But now we can create
our data frame here.

Right so we can create
our data frame records deaf

and store all the results

which contains the value
between 1 200.

So we are storing all the values
between 1/2 and video.

Then we are creating
a victim Prairie View.

Okay for the records,
that's what we are doing.

So for requires the FAA
creating a temporary view

so that we can have
over Oliver SQL queries now,

we can execute all the values

so you can also notice we
are doing join operation here.

Okay, so we can display
the content of join

between the records
and this is our city.

We can do a joint on this part
so we can also perform all

the joint operations
and get the output.

Now.

Let's see our use case for it.

If we talk about use case.

We are going to analyze
our stock market with the help

of spark sequence
select understand

the problem statement first.

So now in our problem statement,

so what we want to do so we want
to accept definitely everybody

must be aware of this top market
like in stock market.

You can lot
of activities happen.

You want to know analyze it

in order to make some profit
out of it and all those stuff.

Alright, so now
let's say our company

have collected a lot of data
for different 10 companies

and they want to do
some computation.

Let's say they want to compute
the average closing price.

They want to list the companies
with the highest closing prices.

They want to compute the average
closing price per month.

They want to list the number
of big price Rises

and fall and compute
some statistical correlation.

So these things we are going
to do with the help

of our spark SQL statement.

So this is a very common we want
to process the huge data.

We want to handle The input
from the multiple sources,

we want to process
the data in real time

and it should be easy to use.

It should not be
very complicated.

So all this requirement will be
handled by my spots equal right?

So that's the reason
we are going to use

the spacer sequence.

So as I said

that we are going
to use 10 companies.

So we are going to kind
of use this 10 companies

and on those ten companies.

We are going to see

that we are going to perform
our analysis on top of it.

So we will be using
this table data

from Yahoo finance
for all this following stocks.

So for n and a A bit sexist.

So all these companies

we have on on which we
are going to perform.

So this is how my data will look
like which will be having date

opening High rate low rate
closing volume adjusted close.

All this data will
be presented now.

So, let's see how we

can Implement a stock analysis
using spark sequel.

So what we have to do for that,

so this is how many data
flow diagram will sound like

so we have going to initially
have the huge amount

of real-time stock data

that we are going to process it
through this path SQL.

So going to It into
a named column base.

Then we are going
to create an rdd

for functional programming.

So let's do that.

Then we are going to use
a reverse Park sequel

which will calculate
the average closing price

for your calculating.

The company with is closing
per year then buy

some stock SQL queries
will be getting our outputs.

Okay, so that is
what we're going to do.

So all the queries
what we are getting generated,

so it's not only this we
are also going to compute

few other queries what we
have solve those queries.

We're going to execute him.

Now.

This is how the flow
will look like.

So we are going
to initially have this Data

what I have just shown you a now
what you're going to do.

You're going to create
a data frame you

are going to then create
a joint clothes are ready.

We will see what we
are going to do here.

Then we are going
to calculate the average

closing price per year.

We are going to hit
a rough patch SQL query and get

the result in the table.

So this is how my execution
will look like.

So what we are going
to do in this case,

first of all,

we are going to initialize the
Sparks equal in this function.

We are going to import all
the required libraries then we

are going to start

our spark session after
importing all the required.

B we are going to create

our case class whatever
is required in the case class,

you can notice a then
we are going to Define

our past stock scheme.

So because we have already
learnt how to create a schema

as we're going to create
this page table schema

by creating this way.

Well, then we are going
to Define our parts.

I DD so in parts are did

if you notice so
here we are creating.

This parts are ready mix.

We have going to create all

of that by using
this additive first.

We are going to remove
the header files also from it.

Then we are going
to read our CSV file

into Into stocks a a
on DF data frame.

So we are going to read
this as C dot txt file.

You can see we are reading
this file and we are going

to convert it into a data frame.

So we are passing
it as an oddity.

Once we are done then

if you want to print
the output we can do it

with the help of show API.

Once we are done
with this now we want

to let's say display the average

of addressing closing price
for n and for every month,

so if we can do all of that also
by using select query, right

so we can say this data frame
dot select and pass

whatever parameters are required
to get the average know,

You can notice are

inside this we are creating
the Elias of the things as well.

So for this DT,

we are creating
areas here, right?

So we are creating the Elias
for it in a binder

and we are showing
the output also so here

what we are going to do now,

we will be checking

that the closing
price for Microsoft.

So let's say they're going up
by 2 or with greater than 2

or wherever it is going
by greater than 2 and now we

want to get the output
and display the result

so you can notice
that wherever it is going

to be greater than 2 we
are getting the value.

So we are hitting
the SQL query to do that.

So we are hitting
the SQL query now on this

you can notice the SQL query

which we are hitting
on the stocks.

Msft.

Right?

This is the we have data frame

we have created now
on this we are doing

that and we are putting
our query that

where my condition
this to be true means

where my closing price
and my opening price

because let's say
at the closing price

the stock price by let's say
a hundred US Dollars

and at that time in the morning

when it open with
the Lexi 98 used or so,

wherever it is going
to be having a different.

Of to or greater

than to that only output
we want to get so that is

what we're doing here.

Now.

Once we are done then after that
what we are going to do now,

we are going to use
the join operation.

So what we are going to do so
we will be joining the Annan

and except bestop's in order
to compare the closing price

because we want
to compare the prices

so we will be doing that.

So first of all,

we are going to create a union
of all these stocks

and then display
this guy joint Rose.

So look at this

what we're going to do
we're going to use

the spark sequence and

if you notice this closely
what we're doing in this case,

So now in this park sequel,

we are hitting
the square is equal

and all those stuff then
we are saying from this

and here we are using
this joint operation

may see this join oppression.

So this we are joining it on

and then in the end
we are outputting it.

So here you can see
you can do a comparison

of all these clothes price
for all these talks.

You can also include no
for more companies right now.

We have just shown you
an example with to complete

but you can do it
for more companies as well.

Now in this case

if you notice what we're doing
were writing this in the park

a file format and Save Being
into this particular location.

So we are creating
this joint stock market.

So we are storing it as
a packet file format and here

if you want to read
it we can read

that and showed output

but whatever file you
have saved it as a pocket

while definitely you
will not be able to read that up

because that file is going
to be the perfect way

and park it way are the files
which you can never read.

You will not be able
to read them up now,

so you will be seeing this
average closing price per year.

I'm going to show you all
these things running also some

just right to explaining you
how things will be run.

We're doing up here.

So I will be showing
you all these things

in execution as well.

Now in this case,

if you notice
what we are doing again,

we are creating
our data frame here.

Again, we are executing our
query whatever table we have.

We are executing on top of it.

So in this case

because we want to find
the average closing per year.

So what we are doing
in this case,

we are going to create
a new table containing

the average closing price
of let's say an and fxn first

and then we are going
to display all this new table.

So we are in the end.

We are going to
register this table

or The temporary table

so that we can execute
our SQL queries on top of it.

So in this case,

you can notice that we
are creating this new table.

And in this new table,

we have putting
our SQL query right

that SQL query

is going to contains
the average closing Paso

the SQL queries finding out
the average closing price of N

and all these companies
then whatever we have now.

We are going to apply
the transformation step

not transformation
of this new table,

which we have created
with the year

and the corresponding
three company data

what we have created
into the The company

or table select
which you can notice

that we are creating
this company or table

and here first of all,

we are going to create
a transform table company

or and going to display
the output so you can notice

that we are hitting
the SQL query

and in the end we have printing
this output similarly

if we want to let's say
compute the best

of average close we can do that.

So in this case again
the same way now,

if once they have learned
the basic stuff,

you can notice that everything

is following a similar approach
now in this case also,

we want to find out let's say
the best of the average

So we are creating
this best company here now.

It should contain the best
average closing price of an MX

and first so we can just get
this greatest and all battery.

So we creating

that then after that we
are going to display this output

and we will be again registering
it as a temporary table now,

once we have done that then
we can hit our queries now,

so we want to check
let's say best

performing company per year.

Now what we have to do for that.

So we are creating
the final table in which

we are going to compute

all the things we are going
to perform the join or not.

So although SQL query we
are going to perform here

in order to compute

that which company
is doing the best

and then we are going
to display the output.

So this is what the output
is going showing up here.

We are again storing
as a comparative View

and here again the same
story of correlation

what we're going to do here.

So now we will be using
our statistics libraries to find

the correlation between Anand
epochs companies closing price.

So that is what we
are going to do now.

So correlation in finance
and the investment

and industries is a statistics.

Measures the degree

to which to Securities move
in relation to each other.

So the closer the correlation is

to be 1 this is going
to be a better one.

So it is always like

how to variables are correlated
with each other.

Let's say your H is highly
correlated to your salary,

but you're earning like
when you are young you usually

unless and when you

are more Edge definitely
you will be earning more

because you will be more mature
similar way I can say that.

Your salary is also dependent
on your education qualification.

And also on the premium
Institute from where you

have done your education.

Let's say if you are from IIT,

or I am definitely
your salary will be higher

from any other campuses.

Right Miss.

It's a probability.

We what I'm telling you.

So let's say

if I have to correlate now
in this case the education

and the salary but I can easily
create a correlation, right?

So that is
what the correlation go.

So we are going to do all

that with respect
to Overstock analysis.

Now now what we are doing

in this case, so You can notice
we are creating this series one

where we heading
the select query now,

we are mapping all
this an enclosed price.

We are converting to a DD
similar way for Series 2.

Also we are doing that right.

So this is we are doing
for rabbits or earlier.

We have done it for an enclosed
and then in the end we

are using the statistics
dot core to create

a correlation between them.

So you can notice this is how we
can execute everything now.

Let's go to our VM and see
everything in our execution.

Question from at all.

So this VM how we
will be getting you

will be getting all
this VM from a director.

So you need not worry
about all that but

that how I will be
getting all this p.m.

In a so a once you
enroll for the courses

and also you will be getting all
this came from that Erika said

so even if I am working

on Mac operating system
my VM will work.

Yes every operating system.

It will be supported.

So no trouble you
can just use any sort

of VM in all means
any operating system to do that.

So what I would occur do
is they just don't want

You to be troubled
in any sort of stuff here.

So what they do is
they kind of ensure

that whatever is required
for your practicals.

They take care of it.

That's the reason they
have created their own VM,

which is also going to be
a lower size and compassion

to Cloudera hortonworks VM

and this is going to definitely
be more helpful for you.

So all these things
will be provided to

you question from nothing.

So all this project I am going
to learn from the sessions.

Yes.

So once you enroll for so
right now whatever we have seen

definitely we have just Otten
upper level of view of this

how the session looks
like for a purchase.

But but when we actually teach
all these things in the course,

it's usually are much more
in the detailed format.

So in detail format,

we kind of keep on showing
you each step in detail

that how the things are working
even including the project.

So you will be also learning
with the help of project

on each different topic.

So that is the way
we kind of go for it.

Now if I am stuck
in any other project then

who will be helping me

so they will be
a support team 24 by 7

if Get stuck at any moment.

You need to just
give a call and kit

and a call or email.

There is a support ticket
and immediately the technical

team will be helping across
the support team is 24 by 7.

They are they are
all technical people

and they will be assisting
you across on all

that even the trainers
will be assisting you for any

of the technical query great.

Awesome.

Thank you now.

So if you notice this is my data

we have we were executing
all the things on this data.

Now what we want to do
if you notice this is

the same code which I
have just shown you.

Earlier also now let us
just execute this code.

So in order to execute this

what we can do we can connect
to my spa action.

So let's get
connected to suction.

Someone's will be connected
to Spur action.

We will go step by step.

So first we will be
importing our package.

This take some time let
it just get connected.

Once this is connected now,

you can notice

that I'm just importing all
the all the important libraries

we have already
learned about that.

After that, you will be
initialising your spark session.

So let's do

that again the same steps
what you have done before.

Once we will be done.

We will be creating
a stock class.

We could have also directly
executed from Eclipse.

Also, this is just I want

to show you step-by-step
whatever we have learnt.

So now you can see
for company one and then

if you want to do

some computation we want to even
see the values and all right,

so that's what we're doing here.

So if we are just getting
the files creating another did,

you know, so let's execute this.

Similarly for your a back
similarly for your fast

for all this so I'm just copying
all these things together

because there are a lot
of companies for which we

have to do all this step.

So let's bring it
for all the 10 companies

which we are going to create.

So as you can see,

this print scheme has giving
it output right now.

Similarly.

I can execute for a rest
of the things as well.

So this is just giving
you the similar way.

All the outputs will be shown
up here company for company V

all these companies you
can see this in execution.

After that, we will be creating
our temporary view

so that we can execute
our SQL queries.

So let's do it for complaint
and also then after that we

can just create a work all
over temporary table for it.

Once we are done now
we can do our queries.

Like let's say we
can display the average

of existing closing price
for and and for each one

so we can hit this query.

So all these queries will happen
on your temporary view

because we cannot anyway
to all these queries

on our data frames are out

so you can see this this
is getting executed.

Trying it out to Tulsa now
because they've done dot shoe.

That's the reason
you're getting this output.

Similarly.

If we want to let's say list
the closing price for msft

which went up more than $2 way.

So that query also we

can execute now we have already
understood this query in detail.

It is seeing is
execution partner

so that you can appreciate
whatever you have learned.

See this is the output
showing up to you.

Now after that

how you can join all the stack
closing price right similar way

how we can save the joint view
in the packet for table.

You want to read that back.

You want to create a new table

like so let's execute all
these three queries together

because we have
already seen this.

Look at this.

So this in this case,

we are doing the drawing class
basing this output.

Then we want to save it
in the package files.

We are saving it and we want
to again reiterate back.

Then we are creating
our new table, right?

We were doing that join

and on so that is
what we are doing in this case.

Then you want
to see this output.

Then we are against touring
as a temp table or not.

Now.

Once we are done with this step
also then what so we

have done it in Step 6.

Now we want to perform.

Let's have a transformation

on new table corresponding
to the three companies

so that we can compare
we want to create

the best company containing
the best average closing price

for all these three companies.

We want to find the companies

but the best closing
price average per year.

So let's do all that as well.

So you can see best company
of the year now here also

the same stuff we are doing to
be registering over temp table.

Okay, so there's a mistake here.

So if you notice here it is 1

but here we are doing
a show of all right,

so there is a mistake.

I'm just correcting it.

So here also it should be
1 I'm just updating

in the sheet itself so
that it will start working now.

So here I have just made it one.

So now after that it
will start working.

Okay, wherever it is going
to be all I have to make it one.

So that is the change
which I need to do here also.

And you will notice
it will start working.

So here also you
need to make it one.

So all those places

where ever it was so just
kind of a good point to make

so wherever you are working
on this we need to always ensure

that all these values
what you are putting up here.

Okay, so I could have also
done it like this one second.

In fact in this place.

I need not do all
this step one second.

Let me explain you also
why no in this place.

It's So see from here
this error started opening why

because my data frame

what I have created
here most one.

Let's execute it.

Now, you will notice
this Quest artwork.

See this is working.

Now.

After that.

I am creating a temp table
that temp table.

What we are creating is
let's say company on okay.

So this is the temp table
which we have created.

You can see this company
now in this case

if I am keeping this company
on itself it is going to work.

Because here anyway,

I'm going to use
the whatever temporary table

we have created, right?

So now let's execute.

So you can see now
it started book.

No further to that now,

we want to create
a correlation between them

so we can do that.

See this is going to give
me the correlation

between the two column names
and so that we can see here.

So this is the correlation the
more it is closer to 1 means the

better it is it means definitely
it is near to 1 it is 0.9,

which is a bigger value.

So definitely it is going
to be much they both are

highly correlated means
definitely they are impacting

each other stock price.

So this is all about the project

but Welcome to this interesting
session of spots remaining

from and Erica.

What is pathogenic?

Is it like really important?

Definitely?

Yes.

Is it really hot?

Definitely?

Yes.

That's the reason we
are learning this technology.

And this is one of the very
sort things in the market

when it's a hot thing means

in terms of job market
I'm talking about.

So let's see what will be
our agenda for today.

So we are going to Gus
about spark ecosystem

where we are going
to see that okay,

what is pop

how smarts the main threats
in the West Park ecosystem

wise path streaming we
are going to have overview

of stock streaming kind of
getting into the basics of that.

We will learn about these cream.

We will learn also about
these theme Transformations.

We will be
learning about caching

and persistence accumulators
broadcast variables checkpoints.

These are like Advanced
concept of paths.

And then in the end,

we will walk through a use case
of Twitter sentiment analysis.

Now, what is streaming
let's understand that.

So let me start
by us example to you.

So let's see if there is
a bank and in Bank.

Definitely.

I'm pretty sure all

of you must have views credit
card debit card all those karts

what dance provide now,

let's say you
have done a transaction.

From India just now
and within an art

and edit your card
is getting swept in u.s.

Is it even possible

for your car to vision
and arduous definitely know now

how that bank will realize

that it is a fraud connection

because Bank cannot let
that transition happen.

They need to stop it

at the time of when it
is getting swiped either.

You can block it.

Give a call to you ask you

whether It is a genuine
transaction or not.

Do something of that sort.

Now.

Do you think they will put
some manual person

behind the scene

that will be looking
at all the transaction

and you will block it manually.

No, so they require
something of the sort

where the data will
be getting stream.

And at the real time

they should be able to catch
with the help of some pattern.

They will do some processing

and they will get
some pattern out of it with

if it is not sounding
like a genuine transition.

They will immediately add
a block it I'll give you a call

maybe send me an OTP to confirm

whether it's a genuine
connection dot they

will not wait

till the next day to kind of
complete that transaction.

Otherwise if what happened
nobody is going to touch

that that right.

So that is the how we
work on stomach.

Now someone have mentioned

that without stream processing
of data is not even possible.

In fact, we can see

that there is no And big data
which is possible.

We cannot even talk
about internet of things.

Right and this this is
a very famous statement

from Donna Saint

do from C equals
3 lot of companies

like YouTube Netflix Facebook
Twitter iTunes topped Pandora.

All these companies
are using spark screaming.

Now.

What is this?

We have just seen with an
example to kind of got an idea.

Idea about steaming pack.

Now as I said with the time
growing with the internet doing

these three main Technologies
are becoming popular day by day.

It's a technique
to transfer the data

so that it can be processed
as a steady and continuous

drip means immediately

as and when the data is coming

you are continuously
processing it as well.

In fact,

this real-time streaming is
what is driving to this big data

and also internet of things now,

they will be lot of things
like fundamental unit

of streaming media streams.

We will also be
Transforming Our screen.

We will be doing it.

In fact, the companies

are using it with
their business intelligence.

We will see more details
in further of the slides.

But before that we will be
talking about spark ecosystem

when we talk about Spark mmm,

there are multiple libraries

which are present in a first one
is pop frequent now

in spark SQL is like

when you can SQL Developer
can write the query in SQL way

and it is going to get converted
into a spark way

and then going to give you
output kind of analogous to hide

but it is going to be faster
in comparison to hide

when we talk about sports clinic

that is what we are going
to learn it is going

to enable all the analytical
and Practical applications

for your live
streaming data M11.

Ml it is mostly
for machine learning.

And in fact,

the interesting part
about MLA is

that it is completely replacing
mom invited are almost replaced.

Now all the core contributors

of Mahal have moved
in two words the

towards the MLF thing

because of the faster response
performance is really good.

In MLA Graphics Graphics.

Okay.

Let me give you example
everybody must have used

Google Maps right now.

What you doing Google Map
you search for the path.

You put your Source you
put your destination.

Now when you just
search for the part,

it's certainly different paths

and then provide you
an optimal path right now

how it providing
the optimal party.

These things can be done
with the help of Graphics.

So wherever you can create
a kind of a graphical stuff.

Up, we will say

that we can use
Graphics spark up.

Now.

This is the kind
of a package provided for art.

So R is of Open Source,

which is mostly used by analysts

and now spark committee
won't infect all

the analysts kind of to move
towards the sparkling water.

And that's the reason

they have recently
stopped supporting spark

on we are all the analysts

can now execute the query
using spark environment

that's getting better
performance and we

can also work on Big Data.

That's that's all
about the ecosystem point

below this we are going to have
a core engine for engine

is the one which defines all
the basics of the participants

all the RGV related stuff

and not is going to be defined

in your staff for Engine
moving further now,

so as we have just
discussed this part we

are going to now discuss
past screaming indicate

which is going to enable
analytical and Interactive.

For live streaming data
know Y is positive

if I talk about bias
past him indefinitely.

We have just gotten after
different is very important.

That's the reason
we are learning it

but this is so powerful
that it is used now

for the by lot of companies
to perform their marketing they

kind of getting an idea

that what a customer
is looking for.

In fact, we are going to learn
a use case of similar to that

where we are going
to to use pasta me now

where we are going to use
a Twitter sentimental analysis,

which can be used
for your crisis management.

Maybe you want to check
all your products

on our behave service.

I just think target marketing

by all the companies
around the world.

This is getting used
in this way.

And that's the reason
spark steaming is gaining

the popularity and because
of its performance as well.

It is beeping
on other platforms.

At the moment
now moving further.

Let's eat Sparks training
features when we talk

about Sparks training teachers.

It's very easy to scale.

You can scale
to even multiple nodes

which can even run till hundreds
of most speed is going

to be very quick means
in a very short time.

You can scream as well as
processor data soil tolerant,

even it made sure

that even you're not losing
your data integration.

You with your bash time and
real-time processing is possible

and it can also be used
for your business analytics

which is used to track
the behavior of your customer.

So as you can see this
is super polite and it's

like we are kind of getting to
know so many interesting things

about this pasta me now next
quickly have an overview

so that we can get
some basics of spots.

Don't know let's understand.

Which box?

So as we have just discussed it
is for real-time streaming data.

It is useful addition
in your spark for API.

So we have already seen
at the base level.

We have that spark

or in our ecosystem on top
of that we have passed we

will impact Sparks claiming
is kind of adding a lot

of advantage to spark Community

because a lot of people are only
joining spark Community to kind

of use this pasta me.

It's so powerful.

Everyone wants to come

and want to use it
because all the other Frameworks

which we already have

which are existing are
not as good in terms

of performance in all

and and it's the easiness

of moving Sparks
coming is also great

if you compare your program
for let's say two orbits

from which is used
for real-time processing.

You will notice
that it is much easier

in terms of from
a developer point of your ass

that that's the reason a lot
of regular showing interest

in this domain now,

it will also enable Table
of high throughput

and fault-tolerant

so that you to stream your data
to process all the things up

and the fundamental unit
Force past dreaming is going

to be District.

What is this thing?

Let me explain it.

So this dream is
basically a series

of bodies to process
the real-time data.

What we generally do is

if you look
at this light inside you

when you get the data,

It is a continuous data you
divide it in two batches

of input data.

We are going to call it
as micro batch and then

we are going to get that is
of processed data though.

It is real time.

But still how come it is back

because definitely you
are doing processing

on some part of the data, right?

Even if it is coming
at real time.

And that is what we are going
to call it as micro batch.

Moving further now.

Let's see few more
details on it.

Now from where you
can get all your data.

What can be your
data sources here.

So if we talk about data sources
here now we can steal the data

from multiple sources
like Market of the past events.

You have statuses
like at based mongodb,

which are you know,
SQL babies elasticsearch post

Vis equal pocket file format you
can Get all the data from here.

Now after that you can also
don't do processing

with the help
of machine learning.

You can do the processing
with the help of your spark SQL

and then give the output.

So this is a very strong thing

that you are bringing
the data using spot screaming

but processing you can do

by using some other
Frameworks as well.

Right like machine learning
you can apply on the data

what you're getting
fatter years time.

You can also apply
your spots equal on the data,

which you're getting at.

the real time Moving further.

So this is a single thing now
in Sparks giving you

what you can just get the data
from multiple sources

like from cough cough prove
sefs kinases Twitter bringing it

to this path screaming
doing the processing

and storing it back
to your hdfs.

Maybe you can bring it to
your DB you can also publish

to your UI dashboard.

Next Tableau angularjs lot
of UI dashboards are there

in which you can publish
your output now.

Holly quotes,

let us just break down
into more fine-grained gutters.

Now we are going to get
our input data stream.

We are going to put it inside

of a spot screaming going to get
the batches of input data.

Once it executes
to his path engine.

We are going to get that chest
of processed data.

We have just seen
the same diagram before so

the same explanation for it.

Now again breaking it down
into more glamour part.

We are getting a d
string B string was

what Vulnerabilities of data
multiple set of Harmony,

so we are getting a d string.

So let's say we are getting
an rdd and the rate of time but

because now we are getting
real steam data, right?

So let's say in today right now.

I got one second.

Maybe now I got some one second
in one second.

I got more data now I got
more data in the next not Frank.

So that is what
we're talking about.

So we are creating data.

We are getting from time
0 to time what we get say

that we have an RGB at the rate

of Timbre similarly
it is this proceeding

with the time that He's
getting proceeded here.

Now in the next thing
we extracting the words

from an input Stream So

if you can notice

what we are doing here
from where let's say,

we started applying
doing our operations

as we started doing
our any sort of processing.

So as in when we get the data
in this timeframe,

we started being subversive.

It can be a flat map operation.

It can be any sort of operation
you're doing it can be even

a machine-learning opposite
of whatever you are doing

and then you are generating
the words in that kind of thing.

So this is how we
as we're seeing

that how gravity we can kind
of see all these part

at a very high level this work.

We again went into
detail then again,

we went into more detail.

And finally we have seen

that how we can even process
the data along the time

when we are screaming
our data as well.

Now one important point is just
like spark context is

mean entry point for
any spark application similar.

Need to work on streaming a spot

screaming you require
a streaming context.

What is that when you're passing
your input data stream you

when you are working
on the Spark engine

when you're walking
on this path screaming engine,

you have to use your system

in context of its using
screaming context only

you are going to get the batches

of your input data now
so streaming context

is going to consume a stream
of data in In Apache spark,

it is registers

and input D string to produce
or receiver object.

Now it is the main entry point
as we discussed

that like spark context is
the main entry point

for the spark application.

Similarly.

Your streaming context
is an entry point

for yourself Paxton.

Now does that mean
now Spa context is

not an entry point know

when you creates pastrini
it is dependent.

On your spots community.

So when you create
this thing in context

it is going to be dependent
on your spark of context only

because you will not be able
to create swimming contest

without spot Pockets.

So that's the reason it
is definitely required spark

also provide a number of default
implementations of sources,

like looking in the data
from Critter a factor 0 mq

which are accessible
from the context.

So it is supporting
so many things, right?

now If you notice this

what we are doing
in streaming contact,

this is just to give
you an idea about

how we can initialize
our system in context.

So we will be importing
these two libraries after that.

Can you see I'm passing
spot context SE right son

passing it every second.

We are collecting the data
means collect the data

for every 1 second.

You can increase this number
if you want and then this

is your SSC means
in every one second

what ever gonna happen?

I'm going to process it.

And what we're doing
in this place,

let's go to the D string topic
now now in these three

it is the full form
is discretized stream.

It's a basic abstraction

provided by your spa
streaming framework.

It's appointing a stream of data
and it is going to be received

from your source

and from processed
steaming context is related

to your response living
Fun Spot context is belonging.

To your spark or

if you remember the ecosystem
radical in the ecosystem,

we have that spark context right
now streaming context is built

with the help of spark context.

And in fact using
streaming context only

you will be able to perform
your sponsoring just like

without spark context you will

not able to execute anything

in spark application
just park application

will not be able
to do anything similarly

without streaming content.

You're streaming application
will not be able to do anything.

It just that screaming
context is built on top

of spark context.

Okay, so it now it's
a continuous stream

of data we can talk
about these three.

It is received from source
of on the processed data speed

generated by the
transformation of interesting.

If you look at this part
internally a these thing

can be represented by
a continuous series of I

need these this is important.

Now what we're doing is
every second remember last time

we have just seen an example

of like every second
whatever going to happen.

We are going to do processing.

So in that every second
whatever data you

are collecting and you're
performing your operation.

So the data

what you're getting here is
will be your District means

it's a Content you can say

that all these things
will be your D string point.

It's our Representation
by a continuous series

of kinetic energy so
many hundred is getting more

because let's say right
knocking one second.

What data I got collected.

I executed it.

I in the second second
this data is happening here.

Okay?

Okay.

Sorry for that.

Now in the second time
also the it is happening

a third second.

Also it is happening here.

No problem.

No, I'm not going
to do it now fine.

So in the third second Auto

if I did something
I'm processing it here.

Right.

So if you see
that this diagram itself,

so it is every second whatever
data is getting collected.

We are doing the processing

on top of it and the whole
countenance series of RDV

what we are seeing here
will be called as the strip.

Okay.

So this is what your distinct
moving further now

we are going to understand
the operation on these three.

So let's say you are doing

this operation on this dream
that you are getting.

The data from 0 to 1 again,

you are applying some operation

on that then whatever output
you get you're going to call

it as words the state
means this is the thing

what you're doing you're doing
a pack of operation.

That's the reason
we're calling it is at

what these three now similarly
whatever thing you're doing.

So you're going
to get accordingly

and output be screen
for it as well.

So this is what is happening
in this particular example now.

Flat map flatmap is API.

It is very similar to mac.

Its kind of platen
of your value.

Okay.

So let me explain you
with an example.

What is flat back?

So let's say
if I say that hi,

this is a doulica.

Welcome.

Okay, let's say listen later.

Now.

I want to apply a flatworm.

So let's say this is
a form of rdd.

Also now on this rdd,

let's say I apply flat back
to let's say our DB this is

the already flat map.

It's not map
Captain black pepper.

And then let's say you want
to define something for it.

So let's say you say that okay,

you are defining
a variable sale.

So let's say a a DOT now

after that you are defining
your thoughts split split.

We're splitting with respect
to visit now in this case

what is going to happen now?

I'm not saying the exacting here
just to give extremely flat back

just to kind of give
you an idea about box.

It is going to flatten
up this fight

with respect to the split

what you are mentioned here
means what it is going to now

create each element as one word.

It is going to create
like this high as one

what l 1 element this
as one One element

is ask another what

a one-element adwaita as
one water in the limit.

Bentham has one
vote for example.

So this is

how your platinum Works kind
of flatten up your whole file.

So this is what we are doing
in our stream effort.

We are our so this is
how this will work.

Now so we have just
understood this part.

Now, let's understand input
the stream and receivers.

Okay, what are these things?

Let's understand this fight.

Okay.

So what are the input
based impossible?

They can be basic Source
advances in basic Source

we can have filesystems
sockets Connections

in advance Source we
can have Kafka no Genesis.

Okay.

So your input these things are

under these things
representing the stream

of input data received
from streaming sources.

This is again the same thing.

Okay.

So this is there are
two type of things

which we just discussed.

Is your basic and second
is your advance?

Let's move brother.

Now what we are going
to see each other.

So if you notice let's see here.

There are some events often it
is going to your receiver

and then energy stream now I
will bees are getting created

and we are performing
some steps on it.

So the receiver sends
the data into the D string

where each back is going
to contain the RTD.

So this is what you're
this thing is doing receiver.

Is doing here now

moving further Transformations
on the D string.

Let's understand that.

What are the
Transformations available?

There are multiple
Transformations, which are

possibly the most popular.

Let's talk about that.

We have map flatmap filter
reduce Group by so there

are multiple Transformations
available via now.

It is like you are getting
your input data now you

will be applying any
of these operations.

Means any Transformations
that is going to happen.

And then on you this thing
is going to be created.

Okay, so that is
what's going to happen.

So let's explore it one by one.

So let's start with now

if I start with map
what happens with Mac

it is going to create
that judges of data.

Okay.

So let's say it is going
to create a map value

of it like this.

So let's say X is not to be
my is giving the output Z

that is giving
the output X, right.

So in this similar format,
this is going to get mad.

That is going to whatever
you're performing.

It is just going to create
batches of input data,

which you can execute it.

So it returns a new DC
by fasting each element

of the source D string
through a function,

which you have defined.

Let's discuss this lapis

that we have just
discussed it is going

to flatten up the things.

So in this case, also,

if you notice we are just
kind of flat inner it

is very similar to Mac.

But each input item
can be mapped to zero

or more outputs in items here.

Okay, and it is going to return
a new these three bypassing

each Source element
to a function for this fight.

So we have just seen an example
of that crap anyway,

so that seems awfully
can remember 70 more easy

for you to kind of
see the difference

between with markets has
no moving further filter

as the name States you
can now filter out the values.

So let's say you have

a huge data you are kind of we
want to filter out some values.

You just want to kind of walk
with some filter data.

Maybe you want to remove
some part of it.

Maybe you are trying
to put some Logic on it.

Does this line contains
this right under this line?

Is that so

in that case extreme only
with that particular criteria?

So this is what we do here,

but definitely most of the times
to Output is going to be smaller

in comparison to your input
reduce reduce is it's just

like it's going to do kind
of aggregation on the wall.

Let's say in the end you want
to sum up all the data

what you have

that is going to be done
with the help of reduce.

Now after that group

by group back is like it's going
to combine all the common values

that is what group
by is going to do.

So as you can see
in this example all the things

which are starting
with Seagal broom back

all the things we're starting

with J. Boardroom back
all the names starting

with C got goodbye.

Not.

So again, what is

this screen window now to give
you an example of this window?

Everybody must be
knowing Twitter, right?

So now what happens in total?

Let me go to my paint.

So insert in this example,

let's understand how
this windowing of Asians of so,

let's say in initials

per second in the initial
per second 10 seconds.

Let's say the tweets
are happening in this way.

Let's say cash
a hash a hashtag now,

which is the trading Twitter
definitely is right is

my training good maybe
in the next 10 seconds.

In the next 10 seconds
now again Hash A. Ashby.

Ashby is open

which is the trending
with be happening here.

Now.

Let's say in another 10 seconds.

Now this time let's say

hash be hash be so actually I
should be Hashmi zapping now,

which is trendy be lonely.

But now I want to find out

which is the trending
one in last 30.

Ashley right because
if I combine I can do it easily.

Now this is your been doing
operation example means you

are not only looking
at your current window,

but you're also looking

at your previous window
Vanessa current window.

I'm talking about let's say
10 seconds of slot

in this 10 seconds lat
let's say you are doing

this operation on has be has
to be has to be has to be

so this is a current
window now you are

not fully Computing with respect
to your current window.

But you are also considering
your previous window.

Now, let's say in this case.

If I ask you,

can you give me the output
of which is trending

in last 17 seconds?

Will you be able
to answer know why

because you don't have partial
information for 7 Seconds

you have information

for your 10 20 30 mins
multiple of them,

but not intermediate one.

So keep this in mind.

Okay, so you will be able
to perform in doing

operation only with respect
to your window size.

It's not like you can create

any partial value in can do
the window efficient now,

let's get back to the sides.

Now it's a similar thing.

So now it is shown here

that we are not only considering
the current window,

but we are also considering
the previous window

now next understand

the output operators are
operations of the business

when we talk
about output operations.

The output operations
are going to allow

the D string data to be pushed
out to your external system.

If you notice here means

whenever whatever processing
you have done with respect to

what What data you are doing
here now your output you

can store in multiple base
against original file system.

You can keep in your database.

You can keep it even
in your external systems

so you can keep
in multiple places.

So that is
what being reflected here.

Now, so if I talk
about output operation,

these are the one

which are supported we can print
out the value we can use save

as text file menu save
as take five.

It saves it into your chest.

If you want you can
also use it to save it

in the local pack system.

You can save it as
an object file.

Also, you can save
it as a Hadoop file

or you can also apply
for these are daily function.

Now what are for
each argument function?

Let's see this example.

So the mill Levy Spin on this
part in detail Banks we teach

you or in advocacy sessions,

but just to give
you an idea now.

This is a very
powerful primitive

that is going to allow
your data to be sent out

to your external systems.

So using this you
can send it across

to your web server system.

We have just seen
our external system

that we can give file system.

It can be anything.

So using this you
will be able to transfer it.

You can view will be
able to send it out

to your external systems.

Now, let's understand the cash
in and persistence now

when we talk
about caching and persistence,

so these 3 Ms. Also annoying
the developers to cash

or to persist the streams data

in the moral means you
can keep your data in memory.

You can cash your data
in the morning for longer time.

Even after your
action is complete.

It is not going to delete it

so you can just Use
this as many times

as you want

so you can simply use
the first method to do that.

So for your input streams

which are receiving the data
over the network may be using

taskbar Loom sockets.

The default persistence level
is set to the replicate

the data to two loads
for the for tolerance

like it is also going
to be replicating the data

into two parts

so you can see the same thing
in this diagram.

Let's understand this
accumulators broadcast

variables and checkpoints.

Now, these are mostly
for your performance.

But so this is going to help you
to kind of perform to help you

in the performance partner.

So it is accumulators is nothing

but environment that are only
added through and associative

and commutative operation.

Usually if you're coming
from Purdue background

if you have done let's say be

mapreduce programming you
must have seen something.

Counters like that,

they'll be used
for other counters

which kind of helps us to debug
the program as well and you

can perform some analysis
in the console itself.

Now this is similar
to you can do

with the accumulators as well.

So you can Implement
your contest with X

open this part you can
also some of the things

with this fact now you can

if you want to track
through UI you can also do

that as you can see
in this UI itself.

You can see all your excavators

as well now similarly
we have broadcast.

Erebus now broadcast Parables
allows the programmer to keep

your meat only bearable cast
on all the machines

which are available.

Now it is going
to be kind of cashing it

on all the machines now,

they can be used to give
every note of copy

of a large input data set

in an efficient manner so you
can just use that sparkle.

Also attempt to distribute the
distributed broadcast variable

using efficient bra strap.

I will do nothing to reduce
the communication process.

So as you can see here,

we are passing
this broadcast value

it is going to spark contest
and then it is broadcasting

to this places.

So this is what how it
is working in this application.

Generally when we teach
in this class has and also

since things are
Advanced concept,

we kind of we kind
of try to expand you

with the practicals
are not right now.

I just want to give you an idea
about what are these things?

So when you go

with the practicals
of all these things

that how activator see

how this is happening out
is getting broadcasted Things

become more and more fear
at that time right now.

I just want

that everybody at these data
high level overview of things.

Now moving further sub
what is checkpoints

so checkpoints are similar

to your checkpoints
in the gaming now,

hold on they can they make
it run 24/7 make it resilient

to the failure and related
to the application project.

So if you can see this diagram,

we are just
creating the checkpoint.

So as in the
metadata checkpoint,

you can see it is the saving
of the information

which is defining the streaming
computation if we talk

about data from check.

It is saving of the generated
a DD to the reliable storage.

So this is this
both are generating

the checkpoint now
now moving forward.

We are going to move
towards our project

where we are going to perform
our Twitter sentiment analysis.

Let's discuss a very
important Force case

of Twitter sentiment analysis.

This is going to
be very interesting

because we will just
do a real-time.

This on Twitter sentiment
analysis and they can be

lot of possibility
of this sentiment analysis

will be but we will
be taking something

for the turtle and it's going
to be very interesting.

So generally when we do
all this in know course,

it is more detailed

because right now in women
are definitely going in deep is

not very much possible,

but during the training
of a director,

you will learn all these things
within the trust awesome,

right that's there something

which we learned
during the session.

It's No, we talked
about some use cases of Twitter.

As I said there can be
multiple use cases

which are possible

because there are solutions

behind whatever the continue
doing it so much

of social media right now

in these days are
very active has been right.

It must be noticing

that even politicians
have started using Twitter

and their did all
the treats are being shown

in the news channel in cystic
of a heart-rending to it

because they are talking
about positive negative

in any politician
use Something right?

And if we talk
about anything is even

if we talk about let's

any Sports FIFA World Cup
is going on then you will notice

always return will be filled up
with lot of treatment.

So how we can make use of it
how we can do some analysis

on top of it that first we
are going to learn in this

so they can be multiple sort
of our sentiment analysis

think it can be done for
your crisis Management Service.

I just think target marketing
we can keep on talking about

when a new release release now

even the moviemakers
kind of glowing eyes.

Okay, hold this movie
is going to perform

so they can easily make
out of it beforehand.

Okay, this movie is going to go
in this kind of range of profit

or not interesting day.

I let us explore
not to Impossible even

in the political campaign
in 50 must have heard

that in u.s.

When the president
election was happening.

They have used in fact role

of social media of all
this analysis at all and then

that have ever played
a major role in winning

that election similarly,

how weather investors
want to predict

whether they should invest
in a particular company or not,

whether they want to check

that whether like we
should Target which customers

for advertisement

because we cannot Target
everyone problem with targeting

everyone is and if we try
to Target element,

it will be very costly
so we want to kind

of set it a little bit

because maybe my set
of people whom I

should send this advertisement
to be more effective

and Wells as well as a queen
is going to be cost effective

as well if you wanted
to do the products

and services also include
I guess we can also do this.

Now.

Let's see some use cases
like the him terms of use case.

I will show you a practical
how it comes.

So first of all,

we will be importing all
the required packages

because we are going
to not perform

or Twitter sentiment analysis.

So we will be requiring
some packages for that.

So we will be doing that as
a first step then we need

to SEC Oliver authentication
without or indication.

We cannot do anything
of now here the challenges

we cannot directly
put your username

and they don't you think

it will get Candidate put
your username and password.

So Peter came up with something.

Very smart thing.

What they did is they came
up with something

on his fourth indication tokens.

So you have to go
to death brought

twitter.com login from there

and you will find kind of all
this authentication tokens

available to you

for will be the recruit take
that and put it here then

as we have learned
the D string transformation,

you will be doing
all that computation

you so you will be having
my distinct honor of France.

Action, then you will be
generating your Tweet data.

I'm going to save it
in this particular directory.

Once you are done with this.

Then you are going
to extract your sentiment

once you extract it.

And you're done.

Let me show you quickly
how it works in our fear.

Now one more interesting thing
about a greater would be

that you will be getting all
this consideration machines.

So you need not worry

about from where I
will be getting all this.

Is it like very difficult
to install when I was waiting.

This open source location.

It was not working for me
in my operating system.

It was not working.

So many things we
have generally seen

people face issues to resolve

everything up be we kind

of provide all this fear
question from Rockville.

This pm has priest but yes,

that's what it has
everything pre-installed.

Whichever will be required
for your training.

So that's the best part
what we also provide.

So in this case your Eclipse
will already be there.

You need to just go
to your Eclipse location.

Let me show you how you can.

So cold that if you want

because it gives you it gives
you just need to go inside it

and double-click on it at that.

You need not go and kind
of installed eclipse

and not even the spot will
already be installed for you.

Let us go in our project.

So this is our project
which is in front of you.

This is my project
which we are going to war.

Now you can see

that we have first
imported all the libraries

that we have set
or more indication system

and then we have moved
and kind of ecstatic.

The D string transformation
extractor that we write

and then save
the output final effect.

So these are the things
which we have done

in this program has now let's
execute it to run this program.

It's very simple go
to run as and from run

as click on still application.

You will notice in the end.

It is releasing
that great good to see that

so it is executing the program.

Let us execute.

I did bring a taxi for Trump.

So use these for Trump any way
that we surveyed to be negative.

Right?

It's an achievement

because anything you do for Tom
will be to be negative Trump is

anyway the hot topic for us.

Maybe make it a little bigger.

You will notice a lot
of negative tweets coming up on.

Yes, now, I'm just stopping it

so that I can
show you something.

Yes.

It's filtering that we thought

so we have actually been written
back in the program itself.

You have given
at one location from using

that we were kind of asking

for a treetop Tom now
here we are doing analysis

and it is also going to tell us

whether it's a positive to a
negative resistance is situated.

It is giving up Faith

because term for Transit even
will not quit positive rate.

So that's something

which is so that's
the reason you're finding.

This is a negative.

Similarly if there
will be any other

that we should
be getting a static.

So right now if I keep on
moving ahead we will see

multiple negative traits
which will come up.

So that's how this program runs.

So this is how our program

we will be executing
we can distract it.

Even the output results will be
getting through at a location

as you can see in this
if I go to my location here,

this is my actual project
where it is running

so you can just come

to this location here
are on your output.

All your output
is Getting through there

so you can just take a look as

but yes, so it's
everything is done

by using space thing apart.

Okay.

That's what we've
seen right reverse

that we were seeing
it with respect

to these three transformations

in a so we have done all that
with have both passed anybody.

So that is one
of those awesome part about this

that you can do such
a powerful things with respect

to your with respect
to you this way.

Now, let's analyze the results.

So as we have just seen

that it is showing
the president's a positive

to a negative tweets.

So this is where your output
is getting Stone

as it shown you a doubt
will appear like this.

Okay.

This is just broke
your output to explicitly

principal also tell

whether it's a neutral
one positive one

negative one everything.

We have done it
with the help of Sparks.

I mean only now we
have done it for Trump

as I just explained you
that we have put

in our program itself from

like we have put
everything up here

and based on that only we
are getting all the software now

we can apply all
the sentiment analysis

and like this.

Like we have learned.

So I hope you have found
all this this specially

this use case very much useful

for you kind of
getting you that yes,

it is getting done by half.

But right now we
have put from here,

but if you want you can keep
on putting the hashtag as

well because that's
how we are doing it.

You can keep on
changing the tax.

Maybe you can kind of code
for let's say four people

for stuff is going

on a cricket match will be going
on we can just put the tweets

according to that just take the
in that case instead of trump.

You can put any player named

or maybe a Team name
and you will see all

that friendly becoming a father.

Okay, so that's
how you can play with this.

Now.

This is there are
multiple examples with it,

which we can play

and this new skills can be even
evolved multiple other type

of those cases.

You can just keep
on transforming it

according to your own use cases.

So that's it about Sparks coming
which I wanted to discuss.

So I hope you must
have found it useful.

So in classification generally

what happens just
to give you an example.

You must have notice
the spam email box.

I hope everybody
must be having have seen

that sparkle in your spam
email box Energy Mix.

Now when any new email comes up
how Google decide

whether it's a spam email
or unknown stamped image

that is done as an example
of classification plus 3,

let's say My ghost
in the Google news,

when you type
something it group.

All the news together

that is called your electric
regression equation is also one

of the very important
fact it is not here.

The regression is let's say
you have a house

and you want to sell that house

and you have no idea.

What is the optimal price?

You should keep for your house.

Now this regression
will help you too.

To achieve that collaborative
filtering you might have see

when you go
to your Amazon web page

that they show you
a recommendation, right?

You can buy this

because you are buying this
but this is done with the help

of colaborative filtering.

Before I move to the project,

I want to show you
some practical find how we

will be executing spark things.

So let me take you
to the VM machine

which will be provided
by a Dorita.

So this machines are also
provided by the Rekha.

So you need not worry
about from where I

will be getting the software.

What I will be doing
recite It Roll there.

Everything is taken care back
into they come now.

Once you will be coming

to this you will see
a machine like Like this,

let me close this.

So what will happen you will see
a blank machine like this.

Let me show you this.

So this is how your machine
will look like.

Now what you are going to do
in order to start working.

You will be opening
this permanent by clicking

on this black option.

Now after that,

what you can do is you
can now go to your spot now

how I can work with funds
in order to execute any program

in sparked by using
Funeral program you

will be entering it as fast -

Chanel if you type fast - gel

it will take you
to the scale of Ron

where you can write
your path program,

but by using scale
of programming language,

you can notice this.

Now, can you see the fact it
is also giving me 1.5.2 version.

So that is the version
of your spot.

Now you can see here.

You can also see this part of
our context available as a see

when you get connected
to your spark sure.

You can just see this will be
my default available to you.

Let us get connected.

It is sometime.

No, we got anything.

So we got connected
to this Kayla prom now

if I want to come out of it,

I will just type exit
it will just let me come

out of this product now.

Secondly, I can also write
my programs with my python.

So what I can do

if I want to do
programming and Spark,

but with provide
Python programming language,

I will be connecting
with by Sparks.

So I just need to type ice pack
in order to get connected.

Your fighter.

Okay.

I'm not getting connected now

because I'm not
going to require.

I think I will be explaining
everything that scalar item.

But if you want to get connected
you can type icebox.

So let's again get connected

to my staff -
sure now meanwhile,

this is getting connected.

Let us create a small pipe.

So let us create
a file so currently

if you notice I
don't have any file.

Okay.

I already have a DOT txt.

So let's say sake at a DOT txt.

So I have some data one.

Two three four five.

This is my data,
which is with me.

Now what I'm going to do,

let me push this file
and do select the effective

if it is already available

in my system as that means
SDK system Hadoop DFS -

ooh, Jack a dot txt just
to quickly check

if it is already available.

There is no sex by so let
me first put this file

to my system to put a dot txt.

So this will put it
in the default location

of x g of X.

Now if I want to read it,
I can see the specs.

So again, I'm assuming

that you're aware of this
as big as commands so you

can see now this one two,

three four Pilots coming
from a Hadoop file system.

Now what I want to do,

I want to use this file
in my in my system of spa now

how I can do that select
we come here.

So in skaila in skaila,

we do not have any Your float
and on like in Java

we use the Define
like this right integer

K is equal to 10

like this is used
to define buttons Kayla.

We do not use this data type.

In fact, what we do
is we call it as back.

So if I use
that a is equal to 10,

it will automatically identify

that it is
a integer value notice.

It will tell me that
a is of my integer type now

if I want to Update
this value to 20.

I can do that.

Now.

Let's say if I want to update
this to ABC like this.

This will smoke an error by

because a is already
defined as in danger

and you're trying to assign
some PVC string back.

So that is the reason
you got this error.

Similarly.

There is one more thing
called as value.

Well B is equal to 10.

Let's say if I do it works
exactly a similar to that.

But I have one difference
now in this case.

If I do basic want
to 20 you will see an error

and why does Sarah because when
you define something as well,

it is a constant.

It is not going
to be variable anymore.

It will be a constant

and that is the reason
if you define something as well,

it will be not updatable.

You will be should not be able
to update that value.

So this is how in Fela you
will be doing your program

so back for bearable part
of that for your constant,

but now so you will be
doing like this now,

let's use it for the example
what we have learned now.

Let's say if I want
to create and cut the V.

So Bal number is equal
to SC dot txt file.

Remember this API we
have learned the CPI

already St. Dot Txt file now.

Let me give this file a DOT txt.

If I give this file a dot txt.

It will be creating
an ID will see this file.

It is telling

that I created an rdd
of string type.

Now.

If I want to read this data,
I will call number dot connect.

This will print be the value
what was available.

Can you say now this line
what you are seeing here?

Is going to be from your memory.

This is your from my body.

It is reading a and that is
the reason it is showing up

in this particular manner.

So this is how you
will be performing your step.

No second thing.

I told you that sparked and walk
on Standalone systems as well.

Right?

So right now
what was happening was

that we have executed this part
in our history of this now

if I want to execute this Us
on our local file system.

Can I do that?

Yes, it can still do that.

What you need to do for that.

So is in that case
the difference will come here.

Now what the file you are giving

here would be instead
of giving like that.

You will be denoting
this file keyword before that.

And after that you need
to give you a local file.

For example, what is
this part slash home slash.

Advocacy.

This is a local park
not as deep as possible.

So you will be
writing / foam.

/schedule Erica a DOT PSD.

Now if you give this

this will be loading
the file into memory,

but not from your hdfs instead.

What does that is this loaded it

from your just loaded it
formula looks like this

so that is the defensive.

So as you can see
in the second case,

I am not even using my hdfs.

Which means what now?

Can you tell me why this
Sarah this is interesting.

Why do Sarah input path
does not exist

because I have given
a typo here.

Okay.

Now if you notice

by I did not get this error here
why I did not get this Elijah

this file do not exist.

But still I did not got

any error because of
lazy evaluation link

the evaluation kind
of made sure that even

if you have given
the wrong part in creating

And beyond ready but it
has not executed anything.

So all the output

or the error mistake
you are able to receive

when you hit that action
of Collective Now

in order to correct this value.

I need to connect this adorable
and this time if I execute it,

it will work.

Okay, you can see
this output 1 2 3 4 5.

So this time it works
by so now we should be

more clear about the lazy
evaluation of the even

if you are giving
the wrong file name

doesn't matter suppose.

I want to use Park
in production unit,

but not on top of Hadoop.

Is it possible?

Yes, you can do that.

You can do that Sonny,

but usually that's
not what you do.

But yes, if you
want to can do that,

there are a lot of things

which you can view
can also deploy it

on your Amazon clusters as that
lot of things you can do that.

How will it provided
distribute in that case?

We'll be using
some other distribution system.

So in that case you
are not using this fact,

you can deploy it
will be just death.

He will not be able
to kind of go across

and distribute in that Master.

You will not be able to lift
weight that redundancy,

but you can use them in Amazon
is the enough for that.

Okay, so that is

how you will be using this now
you're going to get so this is

how you will be performing
your practice as a sec

how you will be working
on this part.

I will be a training you
as I told you.

So this is how things work.

Now, let us see
an interesting use case.

So for that let us go back.

Back to our visiting this
is going to be very interesting.

So let's see this use case.

Look at this.

This is very interested.

Now this use case is for
earthquake detection using Spa.

So in Japan you
might have already seen

that there are so many
up to access coming you

might have thought about it.

I definitely you
might have not seen it

but you must have heard about it

that there are
so many earthquake

which happens in Japan now
how to solve that problem with

about I'm just going
to give you a glimpse

of what kind of problems

in solving the sessions

definitely we are not going to
walk through in detail of this

but you will get an idea
House of Prince fastest.

Okay, just to give you
a little bit of brief here.

But all these products
will learn at the time

of sessions now.

So let's see this part
how we will be using this bill.

So as everybody must be knowing
what is asked website.

So our crack is

like a shaking of your surface
of the Earth your own country.

Ignore all those events
that happen in tector.

If you're from India,

you might have seen recently
there was an earthquake incident

which came from Nepal
by even recently two days back.

Also there was upset incident.

So these are techniques
on coming now,

very important part is let's say

if the earthquake is
on major earthquake like arguing

or maybe tsunami
maybe forest fires,

maybe a volcano now,

it's very important
for them to kind of SC.

That black is going to come
they should be able to kind

of predicted beforehand.

It's not happen
that as a last moment.

They got to the that okay

Dirtbag is comes
after I came up cracking No,

it should not happen like that.

They should be able to estimate
all these things beforehand.

They should be able
to predict beforehand.

So this is the system
with Japan's is using already.

So this is a real-time kind of
use case what I am presenting.

It's so Japan is already
using this path finger

in order to solve
this earthquake problem.

We are going to see
that how they're using it.

Okay.

Now let's say what happens
in Japan earthquake model.

So whenever there is
an earthquake coming

for example at 2:46 p.m.

On March 4 2011 now

Japan earthquake early
warning was detected.

Now the thing was as
soon as it detected immediately,

they start sending
Not those fools to the lift

to the factories every station
through TV stations.

They immediately kind
of told everyone

so that all the students
were there in school.

They got the time to go
under the desk bullet trains,

which were running.

They stop.

Otherwise the capabilities
of us will start shaking now

the bullet trains are already
running at the very high speed.

They want to ensure

that there should be no sort
of casualty because of that

so all the bullet train Stop
all the elevators the lift

which were running.

They stop otherwise
some incident can happen

in 60 seconds 60 seconds

before this number they

were able to inform
almost every month.

They have send the message.

They have a broadcast

on TV all those things
they have done immediately

to all the people

so that they can send
at least this message

whoever can receive it

and that have saved millions

of So powerful they
were able to achieve

that they have done all this
with the help of Apache spark.

That is the most important job

how they've got you
can select everything

what they are doing there.

They are doing it
on the real time system, right?

They cannot just
collect the data

and then later the processes
they did everything as

a real-time system.

So they collected the data
immediately process it

and as soon has the detected

that has quick they
immediately inform the

in fact this happened in 2011.

Now they they start
using it very frequently

because Japan is one of the area

which is very frequently
of kind of affected by all this.

So as I said,

the main thing is we should be
able to process the data

and we are finding

that the bigger thing you
should be able to handle

the data from multiple sources

because data may be coming

from multiple sources may be
different different sources.

They might be suggesting some
of the other events.

It's because Which we
are predicting that okay,

this earthquake can happen.

It should be very
easy to use because

if it is very complicated
then in that case

for a user to use it

if you'd be very good
become competitive service.

You will not be able
to solve the problem.

Now even in the end

how to send the alert
message is important.

Okay.

So all those things
are taken care by your spark.

Now there are two kinds
of layer in your earthquake.

The number one layer
is a prime the way

and second is fake.

And we'll wait.

There are two kinds of wave

in an earthquake
Prime Z Wave is like

when the earthquake is
just about to start it start

to the city center

and it's vendor or Quake

is going to start secondary wave
is more severe than

which sparked after producing.

Now what happens
in secondary wheel is

when it's that start it
can do maximum damage

because primary ways you
can see the initial wave

but the second we
will be on top of that

so they will be some details
with respect to I 'm not going

in detail of that.

But yeah, there
will be some details

with respect to that.

Now what we are going
to do using Sparks.

We will be creating our arms.

So let's go and see
that in our machine

how we will be sick
calculating our Roc which using

which we will be solving
this problem later

and we will be calculating
this Roc with the help

of Apache spark.

Let us again come back
to this machine now

in order to walk on that.

Let's first exit
from this console.

Once you exit from this console
now what you're going to do.

I have already created
this project in kept it here

because we just want to give
you an overview of this.

Let me go to
my downloads section.

There is a project called
as Earth to so this is

your project initially

what all things you
will be having you

will not be having all
the things initial part.

So what will happen.

So let's say if I go
to my downloads from here,

I have worked too.

project Okay.

Now initially I
will not be having

this target directory project
directory bin directory.

We will be using
our SBT framework.

If you do not know SBP this
is the skill of Bill tooth

which takes care of all
your dependencies takes care

of all your dependencies are not
so it is very similar to Melvin

if you already know Megan you
this is because very similar

but at the same time
I prefer this BTW

because as BT is
more easier to write income.

I've been doing yoga never

so you will be writing
this bill taught as begins.

So this finally will provide you
build dot SBT now in this file,

you will be giving the name

of your project your
what's a version of is

because using version of scale
of what you are using.

What are the dependencies
you have with

what versions dependencies you
have like 4 stock 4 and using

1.5.2 version of stock.

So you are telling
that whatever in my program,

I am writing.

So if I require anything related
to spawn quote go and get it

from this website of dot Apache
dot box download it install it.

If I require any dependency

for spark streaming program for
this particular version 1.5.2.

Go to this website or this link

and executed similar theme
for Amanda password.

So you just telling them now

once you have done this you will
be creating a Folder structure.

Your folder structure
would be you need

to create a sassy folder.

After that.

You will be creating
a main folder from Main folder.

You will be creating
again a folder called

as Kayla now inside

that you will be
keeping your program.

So now here you will
be writing a program.

So you are writing you.

Can you see this screaming
to a scalar Network on scale

of our DOT Stella.

So let's keep it as
a black box for them.

So you will be writing
the code to achieve

this problem statement.

Now what we are going to do

that come out of this What
do you mean project folder

and from here?

We will be writing SBT packaged.

It will start downloading

with respect to your is beating
it will check your program.

Whatever dependency you require

for stock course starts
screaming stuck in the lift.

It will download

and install it it
will just download

and install it so we
are not going to execute it

because I've already
done it before

and it also takes some time.

So that's the reason
I'm not doing it now.

You have been this packet,

you will find all
this directly Target directly

toward project directed.

These got created
later on the now

what is going to happen.

Once you have created this
you will go to your Eclipse.

So you are a pure c will open.

So let me open my Eclipse.

So this is how you're
equipped to protect.

Now.

I already have this program
in front of me,

but let me tell you how you
will be bringing this program.

You will be going
to your import option

with We import you

will be selecting your existing
projects into workspace.

Next once you do

that you need to select
your main project.

For example, you need
to select this Earth to project

what you have created
and click on OK

once you do

that they will be
a project directory coming

from this Earth
to will come here.

Now what we need to do go
to your s RC / Main

and not ignore all this program.

I require only just are jocular
because this is

where I've written
my main function.

Important now after that

once you reach
to this you need to go

to your run as Kayla application

and your spot code
will start to execute now,

this will return me a row 0.

Okay.

Let's see this output.

Now if I see this,

this will show me
once it's finished executing.

See this our area
under carosi is this

so this is all computed
with the elbows path program.

Similarly.

There are other programs
also met will help you

to spin the data or not.

I'm not walking over all that.

Now, let's come back
to my wedding and see

that what is the next step

what we will be doing so you
can see this way will be next.

Is she getting created now,
I'm keeping my Roc here.

Now after you have created
your RZ you will be Our graph

now in Japan there is
one important thing.

Japan is already
of affected area of your organs.

And now the trouble here is

that whatever it's not the even
for a minor earthquake.

I should start sending
the alert right?

I don't want to do all that
for the minor minor affection.

In fact, the buildings
and the infrastructure.

What is created is
the point is in such a way

if any odd quack

below six magnitude
comes there there.

The phones are designed in a way
that they will be no damage.

They will be no damage them.

So this is the major thing

when you work with your Japan
free book now in Japan,

so that means with six
they are not even worried

but about six they

are worried now for that day
will be a graph simulation

what you can do you can do it
with Park as well.

Once you generate this graph you
will be seeing that anything

which is going above 6

if anything which
is going above 6,

Should immediately start
the vendor now ignore all

this programming side

because that is what we
have just created and showing

you this execution fact now

if you have to visualize
the same result,

this is what is happening.

This is showing my Roc but

if my artwork is going
to be greater than 6 then only

weighs those alert then only
send the alert to all the paper.

Otherwise take come

that is what the project
what we generally show.

Oh in our space program sent now
it is not the only project

we also kind of create
multiple other products as well.

For example, I kind
of create a model just

like how Walmart to it

how Walmart maybe creating

a whatever sales is happening
with respect to that.

They're using Apache spark

and at the end they are kind of
making you visualize the output

of doing whatever
analytics they're doing.

So that is ordering the spark.

So all those things
we walking through

when we do the per session all
the things you learn quick.

I feel that all these projects
are using right now,

since you do not know the topic

you are not able to get
hundred percent of the project.

But at that time

once you know each
and every topics of deadly

you will have a clearer picture
of how spark is handling.

All these use cases graphs
are very attractive

when it comes to modeling
real world data

because they are
intuitive flexible

and the theory supporting
them has Been maturing

for centuries welcome everyone

in today's session
on Spa Graphics.

So without any further delay,
let's look at the agenda first.

We start by understanding
the basics of craft Theory

and different types of craft.

Then we'll look
at the features of Graphics

further will understand

what is property graph and look
at various crafts operations.

Moving ahead.

We'll look at different graph
processing algorithms at last.

We'll look at a demo

where we will try
to analyze Ford's go by

data using pagerank algorithm.

Let's move to the first topic.

So we'll start
with basics of graph.

So graphs are I basically
made up of two sets called

vertices and edges.

The vertices are drawn
from some underlying type

and the set can be
finite or infinite.

Now each element

of the edge set is a pair
consisting of two elements

from the vertices set.

So your vertex is V1.

Then your vertex is V3.

Then your vertex is V2 and V4.

And your edges are V
1 comma V 3 then next

is V 1 comma V 2 Then
you have B2 comma V 3

and then you have V

2 comma V fo so basically
we represent vertices set

as closed in curly braces
all the name of vertices.

So we have V 1 we have V 2

we have V 3 and then
we have before

and we'll close the curly braces
and to represent the edge set.

We use curly braces again
and then in curly braces,

we specify those two vertex
which are joined by the edge.

So for this Edge,

we will use a viven comma V
3 and then for this Edge

will use we one comma V
2 and then for this Edge again,

we'll use V 2 comma V 4.

And then at last
for this Edge will use

we do comma V 3 and At Last I
will close the curly braces.

So this is your vertices set.

And this is your headset.

Now one, very
important thing that is

if headset is containing
U comma V or you can say

that are instead
is containing V 1 comma V 3.

So V1 is basically
a adjacent to V 3.

Similarly your V
1 is adjacent to V 2.

Then V2 is adjacent
to V for and looking at this

as you can say V2
is adjacent to V 3.

Now, let's quickly move
ahead and we'll look

at different types of craft.

So first we have
undirected graphs.

So basically in
an undirected graph,

we use straight lines
to represent the edges.

Now the order of the vertices
in the edge set does not matter

in undirected graph.

So the undirected graph usually
are drawn using straight lines

between the vertices.

Now it is almost
similar to the graph

which we have seen
in the last slide.

Similarly.

We can again represent
the vertices set as 5 comma

6 comma 7 comma 8 and the edge

set as 5 comma 6 then
5 comma 7 now talking

about directed graphs.

So basically in a directed graph
the order of vertices

in the edge set matters.

So we use Arrow
to represent the edges

as you can see in the image

as It was not the case
with the undirected graph

where we were using
the straight lines.

So in directed graph,

we use Arrow to denote the edges
and the important thing is

The Edge set should be similar.

It will contain
the source vertex

that is five in this case
and the destination vertex,

which is 6 in this case and this
is never similar to six comma

five you cannot represent
this Edge as 6 comma 5

because the direction always
Does indeed directed graph

similarly you can see

that 5 is adjacent to 6,

but you cannot say
that 6 is adjacent to 5.

So for this graph the vertices
said would be similar as 5 comma

6 comma 7 comma 8
which was similar

in undirected graph,

but in directed graph your Edge
set should be your first opal.

This one will be 5 comma
6 then you second Edge,

which is this one would be
five comma Mama seven,

and at last your this set
would be 7 comma 8 but in case

of undirected graph
you can write this as

8 comma 7 or in case

of undirected graph you can
write this one as seven comma

5 but this is not the case
with the directed graph.

You have to follow
the source vertex

and the destination vertex
to represent the edge.

So I hope you guys are clear
with the undirected

and directed graph.

Now.

Let's talk about
vertex label graph now.

A Vertex liberal graph
each vertex is labeled

with some data
in addition to the data

that identifies the vertex.

So basically we say this X
or this v as the vertex ID.

So there will be data

that would be added
to this vertex.

So let's say this vertex
would be 6 comma

and then we are adding the color

so it would be purple next.

This vertex would be 8 comma

and the color
would be green next.

We'll say See this as 7 comma
read and then this one is as

five comma blue now
the six or this five

or seven or eight.

These are vertex ID
and the additional data,

which is attached is the color
like blue purple green or red.

But only the identifying data
is present in the pair of edges

or you can say only the ID
of the vertex is present

in the edge set.

So here the Edsel.

Again similar to
your directed graph

that is your Source ID this

which is 5 and
then destination ID,

which is 6 in this case
then for this case.

It's similar as five comma
7 then in for this case.

It's similar as 7 comma 8

so we are not specifying
this additional data,

which is attached
to the vertices.

That is the color.

If you only specify
the identifiers of the vertex

that is the number

but your vertex set
would be something

like so this vertex

would be 5 comma blue
then your next vertex

will become 6 comma purple

then your next vertex
will become 8 comma green

and at last your last
vertex will be written

as 7 comma read.

So basically when you
are specifying the vertices set

in the vertex label
graph you attach

the additional information
in the vertices are set

but while representing

the edge set it is represented
similarly as A directed graph

where you have to just specify
the source vertex identifier

and then you have

to specify the destination
vertex identifier now.

I hope that you guys are clear
with underrated directed

and vertex label graph.

So let's quickly move forward
next we have cyclic graph.

So a cyclic graph
is a directed graph

with at least one cycle

and the cycle is the path
along with the directed edges

from a Vertex to itself.

So so once you see over here,

you can see

that from this vertex
V. It's moving toward x

7 then it's moving

to vertex Aid then with arrows
moving to vertex six.

And then again,
it's moving to vertex V.

So there should be at least
one cycle in a cyclic graph.

There might be a new component.

It's a Vertex 9 which is
attached over here again,

so it would be a cyclic graph

because it has
one complete cycle over here

and the important
thing to notice is

That the arrow should make
the cycle like from 5 to 7

and then from 7 to 8
and then 8 to 6

and 6 to 5 and let's say

that there is an arrow

from 5 to 6 and then there
is an arrow from 6 to 8.

So we have flipped the arrows.

So in that situation,

this is not a cyclic graph
because the arrows

are not completing the cycle.

So once you move from 5 to 7
and then from 7 to 8,

you cannot move from 8:00
to 6:00 and similarly

once you move from 5 to 6
and then 6 to 8.

You cannot move from 8 to 7.

So in that situation,
it's not a cyclic graph.

So let's clear all this thing.

So will represent this cycle

as five then using
double arrows will go to 7

and then we'll move to 8
and then we'll move to 6

and at last we'll
come back to 5 now.

We have Edge liberal graph.

So basically as label
graph is a graph.

The edges are
associated with labels.

So one can basically indicate
this by making the edge set

as be a set of triplets.

So for example,

let's say this H

in this Edge label graph
will be denoted as the source

which is 6 then the destination

which is 7 and then the label
of the edge which is blue.

So this Edge would
be defined something

like 6 comma 7 comma blue
and then for this

and Hurley The Source vertex

that is 7 the
destination vertex,

which is 8 then
the label of the edge,

which is white like
similarly for this Edge.

It's five comma 7 and
then blue comma red.

And it lasts for this Edge.

It's five comma six and then it
would be yellow common green,

which is the label of the edge.

So all these four edges
will become the headset

for this graph and the vertices
set is almost similar

that is 5 comma
6 comma 7 comma 8 now

to generalize this I would say x

comma y so X here is

the source vertex then why
here is the destination vertex?

X and then a here is
the label of the edge

then Edge label graph
are usually drawn

with the labels written
adjacent to the Earth

specifying the edges

as you can see.

We have mentioned blue white

and all those label
addition to the edges.

So I hope you guys a player
with the edge label graph,

which is nothing

but labels attached
to each and every Edge now,

let's talk about weighted graph.

So we did graph is
an edge label draft.

Where the labels
can be operated on by

usually automatic operators
or comparison operators,

like less than or greater
than symbol usually

these are integers
or floats and the idea is

that some edges
may be more expensive

and this cost is represented
by the edge labels

or weights now in short weighted
graphs are a special kind

of Edgley build rafts

where your Edge
is attached to a weight.

Generally, which is
a integer or a float

so that we can perform
some addition or subtraction

or different kind
of automatic operations

or it can be some kind

of conditional operations
like less than or greater

than so we'll again

represent this Edge as 5 comma
6 and then the weight as 3

and similarly will represent
this Edge as 6 comma

7 and the weight is again
6 so similarly we represent

these two edges as well.

So I hope

that you guys are clear
with the weighted graphs.

Now let's quickly
move ahead and look

at this directed acyclic graph.

So this is
a directed acyclic graph,

which is basically
without Cycles.

So as we just discussed
in cyclic graphs here,

you can see

that it is not completing
the graph from the directions

or you can say the direction
of the edges, right?

We can move from 5 to 7,

then seven to eight

but we cannot move from 8 to 6
and similarly we can move

from 5:00 to 6:00
then 6:00 to 8:00,

but we cannot move from 8 to 7.

So this is Not forming
a cycle and these kind

of crafts are known as
directed acyclic graph.

Now, they appear as special
cases in CS application all

the time and the vertices set

and the edge set
are represented similarly

as we have seen
earlier not talking

about the disconnected graph.

So vertices in a graph
do not need to be connected

to other vertices.

It is basically legal

for a graph to have
disconnected components

or even loan vertices
without a single connection.

So basically this disconnected
graph which has four vertices

but no edges.

Now.

Let me tell you something
important that is

what our sources and sinks.

So let's say we have
one Arrow from five to six

and one Arrow from 5 to 7
now word is with only

in arrows are called sink.

So the 7 and 6 are known
as sinks and the vertices

with only out arrows
are called sources.

So as you can see in the image
this Five only have out arrows

to six and seven.

So these are called sources now.

We'll talk about this
in a while guys.

Once we are going
through the pagerank algorithm.

So I hope that you guys know
what our vertices what our edges

how vertices and edges
represents the graph then

what are different
kinds of graph?

Let's move to the next topic.

So next let's know.

What is Park Graphics.

So talking about
Graphics Graphics is

a new component in spark.

For graphs and crafts
parallel computation now

at a high level graphic extends

The Spark rdd by introducing
a new graph abstraction

that is directed multigraph

that is properties
attached to each vertex

and Edge now to support
craft computation Graphics

basically exposes a set
of fundamental operators,

like finding sub graph
for joining vertices

or aggregating messages as well
as it also exposes and optimize.

This variant of the pregnant
a pi in addition Graphics also

provides you a collection
of graph algorithms

and Builders to simplify
your spark analytics tasks.

So basically your graphics
is extending your spark rdd.

Then you have Graphics
is providing an abstraction

that is directed multigraph

with properties attached
to each vertex and Edge.

So we'll look at this
property graph in a while.

Then again Graphics gives you
some fundamental operators

and Then it also

provides you some graph
algorithms and Builders

which makes your analytics
easier now to get started

you first need to import spark
and Graphics into your project.

So as you can see,

we are importing first Park
and then we are importing

spark Graphics to get
those graphics functionalities.

And at last we are importing

spark rdd to use those already
functionalities in our program.

But let me tell you
that if you are not using

spark shell then you
will need a spark.

Context in your program.

So I hope

that you guys are clear
with the features of graphics

and the libraries

which you need to import
in order to use Graphics.

So let us quickly move ahead
and look at the property graph.

Now property graph is something

as the name suggests property
graph have properties attached

to each vertex and Edge.

So the property graph

is a directed multigraph with
user-defined objects attached

to each vertex and Edge.

Now you might be wondering
what is undirected multigraph.

So a directed multi graph is a
directed graph with potentially

multiple parallel edges
sharing same source

and same destination vertex.

So as you can see in the image

that from San Francisco
to Los Angeles,

we have two edges and similarly
from Los Angeles to Chicago.

There are two edges.

So basically in
a directed multigraph,

the first thing is
the directed graph,

so it should have a Direction.

Ian attached to the edges
and then talking

about multigraph so
between Source vertex

and a destination vertex,

there could be two edges.

So the ability to
support parallel edges

basically simplifies
the modeling scenarios

where there can be
multiple relationships

between the same vertices
for an example.

Let's say these are two persons

so they can be friends
as well as they

can be co-workers, right?

So these kind of scenarios
can be Easily modeled using

directed multigraph now.

Each vertex is keyed by
a unique 64-bit long identifier,

which is basically the vertex ID
and it helps an indexing.

So each of your vertex
contains a Vertex ID,

which is a unique
64-bit long identifier

and similarly edges

have corresponding source and
destination vertex identifiers.

So this Edge would have

this vertex identifier as
well as This vertex identifier

or you can say Source vertex ID
and the destination vertex ID.

So as we discuss
this property graph

is basically parameterised
over the vertex and Edge types,

and these are the types
of objects associated

with each vertex and Edge.

So your graphics basically
optimizes the representation

of vertex and Edge types

and it reduces the in
memory footprint by storing

the primitive data types
in a specialized array.

In some cases it might be
desirable to have vertices

with different property types
in the same graph.

Now this can be accomplished
through inheritance.

So for an example to model
a user and product

in a bipartite graph,

or you can see
that we have user property

and we have product property.

Okay.

So let me first tell you
what is a bipartite graph.

So a bipartite graph
is also called a by graph

which is a set
of graph vertices.

Opposed into two disjoint sets
such that no two graph vertices

within the same
set are adjacent.

So as you can see over here,

we have user property and then
we have product property

but no to user property
can be adjacent or you

can say there should be no edges

that is joining any
of the to user property or

there should be no Edge

that should be joining
product property.

So in this scenario
we use inheritance.

So as you can see here,

we have class vertex
property now basically

what we are doing we
are creating another class

with user property.

And here we have name,

which is again a string
and we are extending

or you can say we are inheriting
the vertex property class.

Now again, in the case
of product property.

We have name that is
name of the product

which is again string and then
we have price of the product

which is double

and we are again extending
this vertex property graph

and at last You're grading a
graph with this vertex property

and then string.

So this is how we
can basically model user

and product as
a bipartite graph.

So we have created user property

as well as we have created
this product property

and we are extending
this vertex property class.

No talking about
this property graph.

It's similar to your rdd.

So like your rdd property graph
are immutable distributed

and fault tolerant.

So changes to the values
or structure of the graph.

Basically accomplished

by producing a new graph
with the desired changes

and the substantial part
of the original graph

which can be your structure
of the graph

or attributes or indices.

These are basically reused
in the new graph reducing

the cost of inherent
functional data structure.

So basically your property graph

once you're trying to change
values of structure.

So it creates a new graph
with changed structure

or changed values

and zero substantial part
of original graph.

Re used multiple times
to improve the performance

and it can be
your structure of the graph

which is getting reuse
or it can be your attributes

or indices of the graph
which is getting reused.

So this is how your property
graph provides efficiency.

Now, the graph is partitioned

across the executors
using a range

of vertex partitioning rules,

which are basically
Loosely defined

and similar to our DD
each partition of the graph

can be recreated

on different machines
in the event of Failure.

So this is how your property
graph provides fault tolerance.

So as we already
discussed logically

the property graph corresponds
to a pair of type collections,

including the properties
for each vertex and Edge

and as a consequence

the graph class contains
members to access the vertices

and the edges.

So as you can see we

have graphed class then you
can see we have vertices

and we have edges.

Now this vertex Rd DVD
is extending your rdd,

which is your body
D and then your vertex ID

and then your vertex property.

Similarly.

Your Edge rdd is extending

your Oddity with your Edge
property so the classes

that is vertex rdd

and HR DD extends under
optimized version of your rdd,

which includes vertex idn
vertex property and your rdd

which includes your Edge
property and Booth

this vertex rdd

and hrd provides additional
functionality build on top

of graph computation

and leverages internal
optimizations as well.

So this is the reason we use
this Vertex rdd or Edge already

because it already extends your
already containing your word.

X ID and vertex property

or your Edge property
it also provides you

additional functionalities built
on top of craft computation.

And again, it gives you some
internal optimizations as well.

Now, let me clear
this and let's take an example

of property graph

where the vertex property

might contain the user
name and occupation.

So as you can see in this table
that we have ID of the vertex

and then we have property
attached to each vertex.

That is the username
as well as the Station

of the user or you can see
the profession of the user

and we can annotate
the edges with the string

describing the relationship
between the users.

So so as you can see
first is Thomas

who is a professor
then second is Frank

who is also a professor then

as you can see third is Jenny.

She's a student and forth is Bob

who is a doctor now Thomas is
a colleague of Frank.

Then you can see

that Thomas is academic
advisor of Jenny again.

Frank is also a Make advisor

of Jenny and then the doctor
is the health advisor of Jenny.

So the resulting graph
would have a signature

of something like this.

So I'll explain this in a while.

So there are numerous ways
to construct the property graph

from raw files

or RDS or even synthetic
generators and we'll discuss it

in graph Builders,

but the very probable

and most General method
is to use graph object.

So let's take a look
at the code first.

And so first over here,

we are assuming
that Parker context

has already been constructed.

Then we are giving
the SES power context next.

We are creating an rdd
for the vertices.

So as you can see for users,

we have specified idd
and then vertex ID

and then these are two strings.

So first one would be

your username and the second one
will be your profession.

Then we are using SC paralyzed
and we are creating an array

where we are specifying
all the vertices so

And that is this one

and you are getting
the name as Thomas

and the profession

is Professor similarly
for to well Frank Professor.

Then 3L Jenny cheese student
and 4L Bob doctors.

So here we have created
the vertex next.

We are creating
an rdd for edges.

So first we are giving
the values relationship.

Then we are creating
an rdd with Edge string

and then we're using SC
paralyzed to create the edge

and in the array we are
specifying the A source vertex,

then we are specifying
the destination vertex.

And then we are
giving the relation

that is colleague similarly
for next Edge resources

when this nation is one

and then the profession
is academic advisor

and then it goes so on.

So then this line we
are defining a default user

in case there is a relationship
between missing users.

Now we have given
the name as default user

and the profession is missing.

Nature trying to build
an initial graph.

So for that we are using
this graph object.

So we have specified users
that is your vertices.

Then we are specifying the
relations that is your edges.

And then we are giving
the default user

which is basically
for any missing user.

So now as you can see over here,

we are using Edge case class
and edges have a source ID

and a destination ID,

which is basically
corresponding to your source

and destination vertex.

And in addition
to the Edge class.

We have an attribute member

which stores The Edge property
which is the relation over here

that is colleague

or it is academic advisor or it
is Health advisor and so on.

So, I hope

that you guys are clear
about creating a property graph

how to specify the vertices
how to specify edges and then

how to create a graph Now
we can deconstruct a graph

into respective vertex

and Edge views by using
a graph toward vertices

and graph edges members.

So as you can see
we are using craft

or vertices over here
and crafts dot edges over here.

Now what we are trying to do.

So first over here the graph
which we have created earlier.

So we have graphed

vertices dot filter Now
using this case class.

We have this vertex ID.

We have the name and then
we have the position.

And we are specifying
the position as doctor.

So first we are trying
to filter the profession

of the user as doctor.

And then we are trying to count.

It.

Next.

We are specifying
graph edges filter

and we are basically
trying to filter the edges

where the source ID is greater
than your destination ID.

And then we are trying
to count those edges.

We are using
a Scala case expression

as you can see to
deconstruct the temple.

You can say to deconstruct

the result on the other hand
craft edges returns a edge rdd,

which is containing
Edge string object.

So we could also have used
the case Class Type Constructor

as you can see here.

So again over here we
are using graph dot s

dot filter and over here.

We have given case h and then
we are specifying the property

that is Source destination
and then property of the edge

which is attached.

And then we are filtering it and
then we are trying to count it.

So this is how using Edge class
either you can see with edges

or you can see with vertices.

This is how you can go ahead
and deconstruct them.

Right because you're
grounded vertices

or your s dot vertices returns
a Vertex rdd or Edge rdd.

So to deconstruct them,

we basically use
this case class.

So I hope you

guys are clear about
transforming property graph.

And how do you use this case?

Us to deconstruct
the protects our DD or HR DD.

So now let's quickly move ahead.

Now in addition to the vertex

and Edge views
of the property graph

Graphics also exposes
a triplet view now,

you might be wondering
what is a triplet view.

So the triplet view
logically joins the vertex

and Edge properties
yielding an rdd edge triplet

with vertex property
and your Edge property.

So as you can see
it gives an rdd.

D with s triplet and then it

has vertex property as well as
H property associated with it

and it contains an instance
of each triplet class.

Now.

I am taking example of a join.

So in this joint we are trying
to select Source ID destination

ID Source attribute then

this is your Edge attribute

and then at last you
have destination attribute.

So basically your edges has
Alias e then your vertices

has Alias as source.

And again your vertices
has Alias as Nation so we

are trying to select
Source ID destination ID,

then Source, attribute
and destination attribute,

and we also selecting
The Edge attribute

and we are performing left join.

The edge Source ID should
be equal to Source ID

and the h destination ID should
be equal to destination ID.

And now your Edge
triplet class basically

extends your Edge class
by adding your Source attribute

and destination
attribute members

which contains the source
and destination properties

and we can use

the triplet view of a graph
to render a collection

of strings describing
relationship between users.

This is vertex 1 which is again
denoting your user one.

That is Thomas and
who is a professor

and is vertex 3,

which is denoting you Jenny
and she's a student.

And this is your Edge,

which is defining
the relationship between them.

So this is a h triplet

which is denoting
the both vertex as well

as the edge which denote
the relation between them.

So now looking at this code
first we have already

created the graph then we
are taking this graph.

We are finding the triplets

and then we are
mapping each triplet.

We are trying to find out
the triplet dot Source attribute

in which we are picking
up the username.

Then over here.

We are trying to pick up
the triplet attribute,

which is nothing
but the edge attribute

which is your academic advisor.

Then we are trying

to pick up the triplet
destination attribute.

It will again pick
up the username

of destination attribute,

which is username
of this vertex 3.

So for an example
in this situation,

it will print Thomas is
the academic advisor of Jenny.

So then we are trying
to take this facts.

We are collecting the facts

using this forage we have
Painting each of the triplet

that is present in this graph.

So I hope

that you guys are clear
with the concepts of triplet.

So now let's quickly take
a look at graph Builders.

So as I already told you

that Graphics provides
several ways of building a graph

from a collection of vertices
and edges either.

It can be stored in our DD
or it can be stored on disk.

So in this graph object first,
we have this apply method.

So basically this apply
method allows creating a graph

from rdd of vertices

and edges and duplicate vertices
are picked up our by Tralee

and the vertices

which are found in the Edge rdd
and are not present

in the vertices rdd are assigned
a default attribute.

So in this apply method first,

we are providing
the vertex rdd then

we are providing the edge rdd

and then we are providing
the default vertex attribute.

So it will create the vertex
which we have specified.

Then it will create the edges

which are specified and
if there is a vertex

which is being referred
by The Edge,

but it is not present
in this vertex rdd.

So So what it does it
creates that vertex

and assigns them the value of
this default vertex attribute.

Next we have from edges.

So graph Dot from edges
allows creating a graph only

from the rdd of edges

which automatically creates
any vertices mentioned

in the edges and assigns
them the default value.

So what happens over here
you provide the edge rdd

and all the vertices

that are present in the hrd
are automatically created

and Default value is assigned
to each of those vertices.

So graphed out

from adjustables basically
allows creating a graph

from only the rdd of vegetables

and it assigns the edges as
value 1 and again the vertices

which are specified by the edges
are automatically created

and the default value which

we are specifying over here
will be allocated to them.

So basically you're

from has double supports
deduplicating of edges,

which means you can remove
the duplicate edges,

but for that you have
to provide a partition strategy

in the unique edges parameter
as it is necessary to co-locate

The Identical edges

on the same partition duplicate
edges can be removed.

So moving ahead men of the graph
Builders re partitions,

the graph edges by default
instead edges are left

in their default partitions.

So as you can see,
we have a graph loader object,

which is basically used to load.

Crafts from the file system

so graft or group edges requires
the graph to be re-partition

because it assumes

that identical edges
will be co-located

on the same partition.

And so you must call
graph dot Partition by

before calling group edges.

So so now you can see the edge
list file method over here

which provides a way to load
a graph from the list of edges

which is present
on the disk and it

It passes the adjacency list
that is your Source vertex ID

and the destination vertex ID
Pairs and it creates a graph.

So now for an example,

let's say we have two and one
which is one Edge then you have

for one which is another Edge

and then you have 1/2
which is another Edge.

So it will load these edges

and then it will
create the graph.

So it will create 2,

then it will create
for and then it will create one.

And for to one it

will create the edge and then
for one it will create the edge

and at last we create
an edge for one and two.

So do you create a graph
something like this?

It creates a graph
from specified edges

where automatically
vertices are created

which are mentioned by the edges
and all the vertex

and Edge attribute
are set by default one

and as well as one
will be associated

with all the vertices.

So it will be 4 comma
1 then again for this.

It would be 1 comma
1 and similarly it would be

2 comma 1 for this vertex.

Now, let's go back to the code.

So then we have
this canonical orientation.

So this argument
allows reorienting edges

in the positive direction

that is from the lower Source ID

to the higher
destination ID now,

which is basically required
by your connected components

algorithm will talk

about this algorithm
in a while you guys

but before this
this basically helps

in view orienting your edges,

which means your Source vertex,

Tex should always be less
than your destination vertex.

So in that situation it
might reorient this Edge.

So it will reorient this Edge
and basically to reverse

direction of the edge
similarly over here.

So with the vertex

which is coming from 2 to 1
will be reoriented

and will be again reversed.

Now the talking

about the minimum Edge partition
this minimum Edge partition

basically specifies
the minimum number

of edge partitions
to generate There might be

more Edge partitions
than a specified.

So let's say the hdfs
file has more blocks.

So obviously more partitions
will be created

but this will give you
the minimum Edge partitions

that should be created.

So I hope

that you guys are clear
with this graph loader

how this graph loader Works

how you can go ahead
and provide the edge list file

and how it will create the craft

from this Edge list file and
then this canonical orientation

where we are again going
and reorienting the graph

and then we have
Minimum Edge partition

which is giving the minimum
number of edge partitions

that should be created.

So now I guess you guys are
clear with the graph Builder.

So how to go ahead and use
this graph object

and how to create graph
using apply from edges

and from vegetables method

and then I guess
you might be clear

with the graph loader object

and where you can go ahead and
create a graph from Edge list.

Now.

Let's move ahead and talk
about vertex and Edge rdd.

So as I already told you

that Graphics exposes
our DD views of the vertices

and edges stored
within the graph at however,

because Graphics again
maintains the vertices and edges

in optimize data structure

and these data structure

provide additional
functionalities as well.

Now, let us see some of
the additional functionalities

which are provided by them.

So let's first talk
about vertex rdd.

So I already told
you that vertex rdd.

He is basically extending
this rdd with vertex ID

and the vertex property and it
adds an additional constraint

that each vertex ID occurs only
words now moreover vertex rdd

a represents a set of vertices
each with an attribute

of type A now internally

what happens this is achieved
by storing the vertex attribute

in an reusable,

hash map data structure.

So suppose, this is
our hash map data structure.

So suppose if to vertex rdd

are derived from the same
base vertex rdd suppose.

These are two vertex rdd

which are basically derived
from this vertex rdd

so they can be joined
in constant time

without hash evaluations.

So you don't have to go ahead
and evaluate the properties

of both the vertices
you can easily go ahead

and you can join them
without the Yes,

and this is one of the way
in which this vertex

already provides you
the optimization now

to leverage this
indexed data structure

the vertex rdd exposes multiple
additional functionalities.

So it gives you
all these functions

as you can see here.

It gives you filter
map values then -

difference left join

in a joint and aggregate
using index functions.

So let us first discuss
about these functions.

So basically filter a function
filters the vertex set

but preserves the internal index
So based on some condition.

It filters the vertices

that are present
then in map values.

It is basically used
to transform the values

without changing the IDS

and which again preserves
your internal index.

So it does not change the idea
of the vertices and it helps

in transforming those values
now talking about the -

method it shows What is unique

in the said based
on their vertex IDs?

So what happens

if you are providing to set
of vertices first contains V1 V2

and V3 and second
one contains V3,

so it will return V1 and V2

because they are unique
in both the sets

and it is basically done
with the help of vertex ID.

So next we have dysfunction.

So it basically removes
the vertices from this set

that appears in another set Then
we have left join an inner join.

So join operators
basically take advantage

of the internal indexing
to accelerate join.

So you can go ahead
and you can perform left join

or you can perform inner join.

Next you have
aggregate using index.

So basically is aggregate
using index is nothing

by reduced by key,

but it uses index

on this rdd to accelerate
the Reduce by key function

or you can say reduced
by key operation.

So again filter is actually
Using bit set and there

by reusing the index

and preserving the ability to do

fast joints with other
vertex rdd now similarly

the map values operator as well.

Do not allow the map function
to change the vertex ID

and this again helps

in reusing the same
hash map data structure now both

of your left join as

well as your inner join
is able to identify

that whether the two vertex rdd

which are joining
are derived from the same.

Hash map or not.

And for this they basically use
linear scan did again don't have

to go ahead and search
for costly Point lookups.

So this is the benefit
of using vertex rdd.

So to summarize

your vertex audit abuses
hash map data structure,

which is again reusable.

They try to
preserve your indexes

so that it would be easier
to create a new vertex already

derive a new vertex already
from them then again

while performing some
joining or Relations,

it is pretty much easy to go
ahead perform a linear scan

and then you can go ahead
and join those two vertex rdd.

So it actually helps
in optimizing your performance.

Now moving ahead.

Let's talk about
HR DD now again,

as you can see your Edge
already is extending your rdd

with property Edge.

Now it organizes the edge
in Block partition using

one of the various
partitioning strategies,

which is again defined in Your
partition strategies attribute

or you can say partition
strategy parameter within

each partition each attribute

and a decency structure
are stored separately

which enables the maximum reuse

when changing the
attribute values.

So basically what it does while
storing your Edge attributes

and your Source vertex
and destination vertex,

they are stored separately so

that changing the values
of the attributes

either of the source
Vertex or Nation Vertex

or Edge attribute

so that it can be
reused as many times

as we need by changing
the attribute values itself.

So that once the vertex ID
is changed of an edge.

It could be easily changed

and the earlier part
can be reused now

as you can see,

we have three additional
functions over here

that is map values
reverse an inner join.

So in hrd basically map

values is to transform
the edge attributes

while preserving the structure.

ER it is helpful in transforming

so you can use map values and
map the values of Courage rdd.

Then you can go ahead and use
this reverse function

which rivers The Edge reusing
both attribute and structure.

So the source
becomes destination.

The destination becomes
Source not talking

about this inner join.

So it basically joins

to Edge rdds partitioned using
same partitioning strategy.

Now as we already discuss
that same partition strategies,

Tired because again
to co-locate you need

to use same partition strategy

and your identical
vertex should reside

in same partition to perform
join operation over them.

Now.

Let me quickly give you an idea
about optimization performed

in this Graphics.

So Graphics basically

adopts a Vertex cut approach to
distribute graph partitioning.

So suppose you have five vertex
and then they are connected.

Let's not worry
about the arrows, right?

Now or let's not worry
about Direction right now.

So either it can be divided
from the edges,

which is one approach or again.

It can be divided
from the vertex.

So in that situation,

it would be divided
something like this.

So rather than splitting crafts

along edges Graphics partition
is the graph along vertices,

which can again
reduce the communication

and storage overhead.

So logically what happens

that your edges
are assigned to machines

and allowing your vertices
to span multiple machines.

So what this is is basically
divided into multiple machines

and your edges is assigned
to a single machine right

then the exact method
of assigning edges.

Depends on the
partition strategy.

So the partition strategy is
the one which basically decides

how to assign the edges

to different machines or you
can send different partitions.

So user can choose

between different strategies
by partitioning the graph

with the help of this graft
Partition by operator.

Now as we discussed

that this craft or Partition

by operator three partitions
and then it divides

or relocates the edges

and basically we try
to put the identical edges.

On a single partition

so that different
operations like join

can be performed on them.

So once the edges have been
partitioned the mean challenge

is efficiently joining
the vertex attributes

with the edges right now

because real world graphs

typically have more
edges than vertices.

So we move vertex attributes
to the edges and because not all

the partitions will contain
edges adjacent to all vertices.

We internally maintain a row.

Routing table.

So the routing table is the one
who will broadcast the vertices

and 10 will implement the join
required for the operations.

So, I hope

that you guys are clear
how vertex rdd and hrd

works and then

how the optimizations take place

and how vertex cut optimizes
the operations in graphics.

Now, let's talk
about graph operators.

So just as already
have basic operations

like map filter reduced by

key property graph also have
Election of basic operators

that take user-defined functions
and produce new graphs

the transform properties and
structure Now The Co-operators

that have optimized
implementation are basically

defined in crafts class
and convenient operators

that are expressed

as a composition
of The Co-operators

are basically defined
in your graphs class.

But in Scala it
implicit the operators

in graph Ops class,

they are automatically available
as a member of graft class

so you can use them.

M using the graph
class as well now

as you can see we have
list of operators

like property operator,

then you have
structural operator.

Then you have join operator

and then you have something
called neighborhood operator.

So let's talk about them one
by one now talking

about property operators,

like rdd has map operator
the property graph contains

map vertices map edges and map
triplets operators right now.

Each of this operator basically
eels a new graph with the vertex

or Edge property.

Modified by the user-defined
map function based

on the user-defined map function
it basically transforms

or modifies the vertices

if it's map vertices

or it transform
or modify the edges

if it is map edges method

or map is operator and so
on format repeats as well.

Now the important thing
to note is that in each case.

The graph structure
is unaffected and this

is a key feature
of these operators.

Basically which allows
the resulting graph

to reuse the structural indices.

Of the original graph

each and every time you
apply a transformation,

so it creates a new graph

and the original
graph is unaffected

so that it can be used

so you can see it can be reused
in creating new graphs.

Right?

So your structure indices

can be used from the original
graph not talking

about this map vertices.

Let me use the highlighter.

So first we have map vertices.

So be it Maps the vertices

or you can still
transform the vertices.

So you provide vertex ID
and then vertex.

And you apply some of the
transformation function using

which so it will give you
a graph with newer text property

as you can see now same is
the case with map edges.

So again you provide the edges
then you transform the edges.

So initially it was Ed and then
you transform it to Edie to

and then the graph

which is given or you
can see the graph

which is returned is the graph
for the changed each attribute.

So you can see here
the attribute is ed2.

Same is the case
with Mark triplets.

So using Mark triplets,

you can use the edge triplet
where you can go ahead

and Target the vertex Properties
or you can say vertex attributes

or to be more specific
Source vertex attribute as well

as destination vertex attribute

and the edge attribute and then
you can apply transformation

over those Source attributes

or destination attributes
or the edge attributes

so you can change them and then
it will again return a graph

with the transformed values now,

I guess you guys are clear
the property operator.

So let's move Next operator

that is structural operator So
currently Graphics supports only

a simple set of commonly
use structural operators.

And we expect more
to be added in future.

Now you can see
in structural operator.

We have reversed operator.

Then we have subgraph operator.

Then we have masks operator

and then we have
group edges operator.

So let's talk about them one by
one so first reverse operator,

so as the name suggests,

it returns a new graph with all
the edge directions reversed.

So basically it will change
your Source vertex

into destination vertex,

and then it will change
your destination vertex

into Source vertex.

So it will reverse
the direction of your edges.

And the reverse operation
does not modify Vertex

or Edge Properties or change.

The number of edges.

It can be implemented

efficiently without
data movement or duplication.

So next we have
subgraph operator.

So basically subgraph
operator takes the vertex

and Edge predicates

or you can say Vertex
or edge condition

and Returns the Of
containing only the vertex

that satisfy those vertex
predicates and then it Returns

the edges that satisfy
the edge predicates.

So basically will give
a condition about edges and

vertices and those predicates

which are fulfilled
or those vertex

which are fulfilling the
predicates will be only returned

and again seems the case
with your edges

and then your graph
will be connected.

Now, the subgraph operator
can be used in a number

of situations to restrict
the graph to the vertices

and edges of interest

and eliminate the Rest
of the components,

right so you can see
this is The Edge predicate.

This is the vertex predicate.

Then we are providing
the extra plate with the vertex

and Edge attributes

and we are waiting

for the Boolean value then
same is the case with vertex.

We're providing the vertex
properties over here

or you can say vertex
attribute over here.

And then again,

it will yield a graph
which is a sub graph

of the original graph which will
fulfill those predicates now,

the next operator
is mask operator.

So mask operator Constructors.

Graph by returning a graph
that contains the vertices

and edges that are also found
in the input graph.

Basically, you can treat

this mask operator as
a comparison between two graphs.

So suppose.

We are comparing

graph 1 and graph 2 and it
will return this sub graph

which is common in both
the graphs again.

This can be used in conjunction
with the subgraph operator.

Basically to restrict

a graph based on properties
in another related graph, right.

And so I guess you guys are
clear with the mask operator.

So we're here.

We're providing a graph

and then we are providing
the input graph as well.

And then it will return a graph

which is basically a subset
of both of these graph

not talking about group edges.

So the group edges operator
merges the parallel edges

in the multigraph, right?

So what it does it,

the duplicate edges between pair
of vertices are merged

or you can say are
at can be aggregated

or perform some action

and in many numerical
applications I just can be added

and their weights can be
combined into a single edge,

right which will again
reduce the size of the graph.

So for an example,

you have to vertex V1 and V2
and there are two edges

with weight 10 and 15.

So actually what you can do is
you can merge those two edges

if they have same direction and
you can represent the way to 25.

So this will actually
reduce the size

of the graph now looking
at the next operator,

which is join operator.

So in many cases
it is necessary.

To join data from external
collection with graphs, right?

For example.

We might have
an extra user property

that we want to merge
with the existing graph

or we might want
to pull vertex property

from one graph to another right.

So these are some
of the situations

where you go ahead and use
this join operators.

So now as you can see over here,

the first operator
is joined vertices.

So the joint vertices operator
joins the vertices

with the input rdd

and returns a new graph
with the vertex properties.

Dean after applying
the user-defined map function

now the vertices
without a matching value

in the rdd basically retains
their original value not talking

about outer join vertices.

So it behaves similar
to join vertices except that

which user-defined map function
is applied to all the vertices

and can change
the vertex property type.

So suppose that you have
a old graph which has

a Vertex attribute as old price

and then you created
a new a graph from it

and then it has the vertex
attribute as new rice.

So you can go ahead
and join two of these graphs

and you can perform
an aggregation of both

the Old and New prices
in the new graph.

So in this kind of situation
join vertices are used

now moving ahead.

Let's talk about neighborhood
aggregation now key step

in many graph analytics
is aggregating the information

about the neighborhood
of each vertex for an example.

We might want to know the number
of followers each user has

Or the average age

of the follower of each user now
many iterative graph algorithms,

like pagerank shortest path,

then connected components
repeatedly aggregate

the properties of
neighboring vertices.

Now, it has four operators
in neighborhood aggregation.

So the first one is
your aggregate messages.

So the core aggregation
operation in graphics

is aggregate messages.

Now this operator

applies a user-defined
send message function

as you can see over here
to Each of the edge triplet

in the graph

and then it uses
merge message function

to aggregate those messages
at the destination vertex.

Now the user-defined

send message function
takes an edge context

as you can see and
which exposes the source

and destination address Buttes
along with the edge attribute

and functions like send
to Source or send

to destination is used
to send messages to source

and destination attributes.

Now you can think
of send message as the map.

Function in mapreduce and
the user-defined merge function

which actually takes
the two messages

which are present
on the same Vertex

or you can see
the same destination vertex

and it again combines

or aggregate those messages
and produces a single message.

Now, you can think
of the merge message

as reduce function
the mapreduce now,

the aggregate messages operator
returns a Vertex rdd.

Basically, it contains
the aggregated messages at each

of the destination vertex.

It's and vertices

that did not receive
a message are not included

in the returned vertex rdd.

So only those vertex

are returned which actually
have received the message

and then those messages
have been merged.

If any vertex
which haven't received.

The message will not be included
in the returned rdd

or you can say a return
vertex rdd now in addition

as you can see we have
a triplets Fields.

So aggregate messages takes
an optional triplet fields,

which indicates what data is.

Accessed in the edge content.

So the possible options

for the triplet fields
are defined interpret fields

to default value
of triplet Fields is triplet

Fields oil as you can see over
here this basically indicates

that user-defined send
message function May access

any of the fields
in the edge content.

So this triplet field argument
can be used to notify Graphics

that only these part of
the edge content will be needed

which basically allows Graphics
to select the optimize joining.

Strategy, so I hope

that you guys are clear
with the aggregate messages.

Let's quickly move ahead
and look at the second operator.

So the second operator is
mapreduce triplet transition.

Now in earlier versions

of Graphics neighborhood
aggregation was accomplished

using the mapreduce
triplets operator.

This mapreduce triplet
operator is used

in older versions of Graphics.

This operator takes
the user-defined map function,

which is applied to each triplet
and can yield messages

which are Aggregating using the
user-defined reduce functions.

This one is the reason
I defined malfunction.

And this one is your user
defined reduce function.

So it basically applies
the map function

to all the triplets

and then the aggregate
those messages using this user

defined reduce function.

Now the newer version of this
map produced triplets operator

is the aggregate messages
now moving ahead.

Let's talk about Computing
degree information operator.

So one of the common
aggregation task is Computing

the degree of each vertex.

That is the number of edges
adjacent to each vertex.

Now in the context
of directed graph.

It is often necessary to know
the in degree out degree.

Then the total degree of vertex.

These kind of things are
pretty much important

and the graph Ops class
contain a collection

of operators to compute
the degrees of each vertex.

So as you can see,

we have maximum input degree
than maximum output degree,

then maximum degrees
maximum degree will tell

us the number of Maximum
incoming edges then Max.

Degree will tell us
maximum number of output edges

and this Max degree

with actually tell us the number
of input as well as

output edges now moving
ahead to next operator

that is collecting
Neighbors in some cases.

It may be easier to express

the computation by collecting
neighboring vertices

and their attribute
at each vertex.

Now, this can be easily
accomplished using

the collect neighbors ID and
the collect neighbors operator.

So basically your collect
neighbor ID takes

The Edge direction
as the parameter

and it returns a Vertex rdd

that contains the array
of vertex ID

that is neighboring
to the particular vertex

now similarly The Collection
neighbors again takes

the edge directions as the input

and it will return you the array

with the vertex ID and
the vertex attribute both now,

let me quickly open

my VM and let us go through
the spark directory first.

Let me first open
my terminal so first

I'll start the Do demons so
for that I will go

to her do phone directory
genocide has been start

or lot asset script file.

So let me check

if the Hadoop demons
are running or not.

So as you can see that name,

no data node
secondary name node,

the node manager
and resource manager.

All the Demons
of Hadoop are up now.

I will navigate to spark home.

Let me first start
this park demons.

I See Spark demons are running

alko first minimize this and let
me take you to this park home.

And this is my spot directories.

I'll go inside now.

Let me first show you the data

which is by default present
with your spark.

So we'll open this in a new tab.

So you can see
we have two files

in this Graphics data directory.

Meanwhile, let me take you
to the example code.

So this is example
and inside so main scalar.

You can find

the graphics directory and
inside this Graphics directory

you Some of the sample codes
which are present over here.

So I will take you
to this aggregate

messages example dots
Kayla now meanwhile,

let me open the data as well.

So you'll be able to understand.

Now this is
followers dot txt file.

So basically you can imagine

these are the edges which
are representing the vertex.

So this is what x 2
and this is vertex 1 then

this is Vertex 4 and this
is vertex 1 and similarly.

So on these are representing
those vertex and

if you can remember I
have already told you

that inside graph loader class.

There is a function
called Edge list file

which takes the edges

from a file and then it
construct the graph based.

That now second you
have this user dot txt.

So these are basically the edges
with the vertex ID.

So vertex ID for this vertex
is 1 then for this is 2

and so on and then
this is the data

which is attached or you can say
the attribute of the edges.

So these are the vertex ID

which is 1 2 3 respectively
and this is the data

which is associated
with your each vertex.

So this is username and this
might be the name of your user.

Zur and so on now
you can also see

that in some of the cases
the name of the user is missing.

So as in this case the name

of the user is missing
these are the vertices

or you can see the vertex ID
and vertex attributes.

Now, let me take you through
this aggregate messages example,

so as you can see,
we are giving the name

of the packages over G Apache
spark examples dot Graphics,

then we are importing Graphics
in that very important.

Off class as well as

this vertex rdd next we
are using this graph generator.

I'll tell you why we
are using this graph generator

and then we are using
the spark session over here.

So this is an example

where we are using the aggregate
messages operator to compute

the average age of the more
senior followers of each user.

Okay.

So this is the object
of aggregate messages example.

Now, this is the main function
where we are first.

Realizing this box session then
the name of the application.

So you have to provide the name
of the application

and this is get

or create method now
next you are initializing

the spark context as SC
now coming to the code.

So we are specifying
a graph then this graph

is containing double and N now.

I just told you that we
are importing craft generator.

So this graph generator is

to generate a random
graph for Simplicity.

So you would have multiple
number of edges and vertices.

Says then you are using
this log normal graph.

You're passing the spark context

and you're specifying the number
of vertices as hundred.

So it will generate
hundred vertices for you.

Then what you are doing.

You are specifying
the map vertices

and you're trying
to map ID to double so

what this would do

this will basically map
your ID to double then

in next year trying
to calculate the older followers

where you have given
it as vertex rdd

and then put is nth and Also,

your vertex already
has sent as your vertex ID

and your data is double

which is associated
with each of the vertex

or you can say
the vertex attribute.

So you have this graph

which is basically
generated randomly

and then you are performing
aggregate messages.

So this is the aggregate
messages operator now,

if you can remember we first
have the send messages, right?

So inside this triplet,

we are specifying a function
that if the source attribute

of the triplet is board.

Destination attribute
of the triplet.

So basically it will return

if the followers age
is greater than the age

of person whom he

is following this tells
the followers is is greater

than the age of whom
he is following.

So in that situation,

it will send message
to the destination

with vertex containing counter

that is 1 and the age
of the source attribute

that is the age
of the follower so first

so you can see the age
of the destination on is less

than the age
of source attribute.

So it will tell you

if the follower is older
than the user or not.

So in that situation will send
one to the destination

and we'll send the age
of the source

or you can see the edge
of the follower then second.

I have told you
that we have merged messages.

So here we are adding
the counter and the H

in this reduce function.

So now what we are doing we
are dividing the total age

of the number

of older followers
to Write an average age

of older followers.

So this is the reason why
we have passed the attribute

of source vertex firstly

if we are specifying
this variable that is

average age of older followers.

And then we are specifying
the vertex rdd.

So this will be double
and then this older followers

that is the graph

which we are picking up
from here and then we

are trying to map the value.

So in the vertex,

we have ID and we have value so
in this situation We

are using this case class
about count and total age.

So what we are doing we
are taking this total age

and we are dividing it by count
which we have gathered from this

send message.

And then we have aggregated
using this reduce function.

We are again taking the total
age of the older followers.

And then we are trying
to divide it by count

to get the average age

when at last we are trying
to display the result and then

we are stopping this park.

So let me quickly open
the terminal so I

will go to examples

so I'd examples I took you
through the source directory

where the code is
present inside skaila.

And then inside there
is a spark directory

where you will find
the code but to execute

the example you need to go
to the jars territory.

Now, this is
the scale example jar

which you need to execute.

But before this,
let me take you to the hdfs.

So the URL is localhost.

Colon 5 0 0 7 0

And we'll go to utilities then

we'll go to browse
the file system.

So as you can see,

I have created a user
directory in which I

have specified the username.

That is Ed Eureka
and inside Ed Eureka.

I have placed my data directory

where we have this graphics
and inside the graphics.

We have both the file
that is followers Dot txt

and users dot txt.

So in this program,

we are not referring
to these files

but incoming examples will
be referring to these files.

So I would request you to first
move it to this hdfs directory.

So that spark can refer
the files in data Graphics.

Now, let me quickly minimize
this and the command

to execute is Spock -

submit and then I'll pass
this charge parameter

and I'll provide
the spark example jar.

So this is the jar then
I'll specify the class name.

So to get the class name.

I will go to the code.

I'll first take
the package name from here.

And then I'll take

the class name which is
aggregated messages example,

so this is my class.

And as I told you have
to provide the name

of the application.

So let me keep it as example
and I'll hit enter.

So now you can see the result.

So this is the followers

and this is the average
age of followers.

So it is 34 Den.

We have 52 which is
the count of follower.

And the average age is
seventy six point eight

that is it has
96 senior followers.

And then the average age

of the followers is
ninety nine point zero,

then it has
four senior followers

and the average age is 51.

Then this vertex has
16 senior followers

with the average age
of 57 point five.

5 and so on you can see
the result over here.

So I hope now you guys are clear
with aggregate messages

how to use aggregate messages

how to specify
the send message then

how to write the merge message.

So let's quickly go back
to the presentation.

Now, let us quickly move ahead

and look at some
of the graph algorithms.

So the first one is Page rank.

So page rank measures
the importance of each vertex

in a graph assuming

that an edge from U
to V represents.

And recommendation

or support of Vis importance
by you for an example.

Let's say if a Twitter user
is followed by many others user

will obviously rank
high graphics comes

with the static and dynamic
implementation of pagerank as

methods on page rank object

and static page rank runs
a fixed number of iterations,

which can be specified by you
while the dynamic page rank runs

until the ranks converge

what we mean by that is
it Stop changing by more

than a specified tolerance.

So it runs

until it have optimized
the page rank of each

of the vertices now graphs class
allows calling these algorithms

directly as methods
on crafts class.

Now, let's quickly go
back to the VM.

So this is the pagerank example.

Let me open this file.

So first we are specifying
this Graphics package,

then we are importing
the graph loader.

So as you can Remember
inside this graph

loader class we have
that edge list file operator,

which will basically create
the graph using the edges

and we have those edges
in our followers

dot txt file now coming back
to pagerank example now,

we're importing the spark
SQL Sparks session.

Now, this is Page
rank example object

and inside which we
have created a main class

and we have similarly created
this park session then Builders

and we're specifying
the app name

which Is to be provided then
we have get our grid method.

So this is

where we are initializing
the spark context

as you can remember.

I told you that using
this Edge list file method.

We are basically
creating the graph

from the followers dot txt file.

Now, we are running
the page rank over here.

So in rank it will give you all
the page rank of the vertices

that is inside this graph

which we have just
to reducing graph loader class.

So if you're passing
an integer as an an argument

to the page rank,

it will run
that number iterations.

Otherwise, if you're
passing a double value,

it will run
until the convergence.

So we are running
page rank on this graph

and we have passed the vertices.

Now after this we are trying
to load the users dot txt file

and then we are trying to play

the line by comma then
the field zero too long

and we are storing
the field one.

So basically field zero.

In your user txt is
your vertex ID or you

can see the ID of the user
and field one is your username.

So we are trying to load
these two Fields now.

We are trying
to rank by username.

So we are taking the users
and we are joining the ranks.

So this is where we
are using the join operation.

So Frank's by username.

We are trying to
attach those username

or put those username
with the page rank value.

So we are taking the users

then we are joining
the ranks it is again,

we are getting
from this page Rank

and then we are mapping
the ID user name and rank.

Second week sometime run
some iterations over the craft

and will try to converge it.

So after converging you
can see the user and the rank.

So the maximum rank is
with Barack Obama,

which is 1.45 then
with Lady Gaga.

It's 1.39 and then with
order ski and so on.

Let's go back to the slide.

So now after page rank,

let's quickly move ahead

to Connected components
the connected components

algorithm labels each
connected component

of the graph with the ID
of its lowest numbered vertex.

So let us quickly go
back to the VM.

Now let's go inside
the graphics directory

and now we'll open
this connect components example.

So again, it's the same very
important graph load

and Spark session.

Now, this is the connect
components example object makes

this is the main function
and inside the main function.

We are again specifying all

those Sparks session
then app name,

then we have spark context.

So it's similar.

So again using
this graph loader class

and using this Edge.

To file we are loading
the followers dot txt file.

Now in this graph.

We are using this connected
components algorithm.

And then we are trying to find
the connected components now

at last we are trying
to again load this user file

that is users Dot txt.

And we are trying to join
the connected components

with the username so over
here it is also the same thing

which we have discussed
in page rank,

which is taking the field 0

and field one
of your user dot txt file

and a at last we
are joining this users

and at last year trying to join
this users to connect component

that is from here.

Now.

We are printing the CC
by username collect.

So let us quickly go ahead and
execute this example as well.

So let me first copy
this object name.

that's name this
as example to so

as you can see Justin Bieber has
one connected component,

then you can see this has
three connected component.

Then this has
one connected component

than Barack Obama has one
connected component and so on.

So this basically
gives you an idea

about the connected components.

Now, let's quickly move back

to the slide will discuss
about the third algorithm

that is triangle counting.

So basically a Vertex is a part
of a triangle when it has

two adjacent vertices
with an edge between them.

So it will form
a triangle, right?

And then that vertex
is a part of a triangle

now Graphics implements
a triangle counting algorithm

in the Triangle count object.

Now that determines the number
of triangles passing

through each vertex providing
a measure of clustering

so we can compute
the triangle count

of the social network data set

from the pagerank section
1 mode thing to note is

that triangle count
requires the edges.

To be in
a canonical orientation.

That is your Source ID
should always be less

than your destination ID

and the graph will be
partition using craft

or Partition by Method now,

let's quickly go back.

So let me open
the graphics directory again,

and we'll see
the triangle counting example.

So again, it's the same

and the object is
triangle counting example,

then the main function
is same as well.

Now we are again using
this graph load of class

and we are loading
the followers dot txt

which contains the edges

as you can see here.

We are using this Partition

by argument and we are passing
the random vertex cut,

which is the partition strategy.

So this is how you can go ahead

and you can Implement
a partition strategy.

He is loading the edges
in canonical order

and partitioning the graph
for triangle count.

Now.

We are trying to find
out the triangle count

for each vertex.

So we have this try count

variable and then we are using
this triangle count algorithm

and then we are
specifying the vertices

so it will execute
triangle count over this graph

which we have just loaded
from follows dot txt file.

And again, we are basically
joining usernames.

So first we are Being
the usernames again here.

We are performing the join
between users and try counts.

So try counts is from here.

And then we are again
printing the value from here.

So again, this is the same.

Let us quickly go
ahead and execute

this triangle counting example.

So let me copy this.

I'll go back to the terminal.

I'll limit as example
3 and change the class name.

And I hit enter.

So now you can see
the triangle associated

with Justin Bieber 0 then
Barack Obama is one

with odors kids one

and with Jerry sick.

It's fun.

So for better understanding I
would recommend you to go ahead

and take this followers or txt.

And you can create
a graph by yourself.

And then you can attach
these users names with them

and then you will get an idea

about why it is giving
the number as 1 or 0.

So again the graph
which is connecting.

In two and four

is disconnect and it
is not completing any triangles.

So the value of these 3 are 0
and next year's second graph

which is connecting

your vertex 3 6 & 7
is completing one triangle.

So this is the reason why

these three vertices
have values one now.

Let me quickly go back.

So now I hope

that you guys are clear
with all the concepts

of graph operators
then graph algorithms.

Eames so now is the right
time and let us look

at a spa Graphics demo

where we'll go ahead

and we'll try to analyze
the force go by data.

So let me quickly go
back to my VM.

So let me first show
you the website

where you can go ahead and
download the Fords go by data.

So over here you can go

to download the fort
bike strip history data.

So you can go ahead and download
this 2017 Ford's trip data.

So I have already downloaded it.

So to avoid the typos,

I have already written
all the commands so

first let me go ahead and start
the spark shell So I'm inside

these Park shell now.

Let me first import graphics
and Spa body.

So I've successfully
imported graphics and Spark rdd.

Now, let me create
a spark SQL context as well.

So I have successfully
created this park SQL context.

So this is basically
for running SQL queries

over the data frames.

Now, let me go ahead
and import the data.

So I'm loading the data
in data frame.

So the format of file is CSV,

then an option the header
is already added.

So that's why it's true.

Then it will automatically
infer this schema

and then in the load parameter,

I have specified
the path of the file.

So I'll quickly hit enter.

So the data is loaded
in the data frame to check.

I'll use d f dot count
so it will give me the count.

So you can see it has
5 lakhs 19 2007 Red Rose now.

Let me click go back
and I'll print the schema.

So this is the schema
the duration in second,

then we have
the start time end time.

Then you have start station ID.

Then you have
start station name.

Then you have start
station latitude longitude

then end station ID

and station name then
end station latitude

and station longitude.

Then your bike ID user type then
the birth year of the member

and the gender
of the member now,

I'm trying to create
a data frame

that is Gas stations

so it will only create
the station ID and station name

which I'll be using as vertex.

So here I am trying
to create a data frame

with the name of just stations

where I am just selecting
the start station ID

and I'm casting it as float

and then I'm selecting
the start station name

and then I'm using
the distinct function to only

keep the unique values.

So I quickly go
ahead and hit enter.

So again, let me go

ahead and use this just stations
and I will print the schema.

So you can see
there is station ID,

and then there is
start station name.

It contains the unique values

of stations in this just
station data frame.

So now again,

I am taking this stations
where I'm selecting

these thought station ID
and and station ID.

Then I am using re distinct

which will again give
me the unique values

and I'm using this flat map

where I am specifying

the iterables where we
are taking the x0

that is your start station ID,

and I am taking x 1
which is your ends.

An ID and then again,

I'm applying this
distinct function

that it will keep only
the unique values and then

at last we have to d f function

which will convert
it to data frame.

So let me quickly go ahead
and execute this.

So I am printing this schema.

So as you can see
it has one column

that is value and it
has data type long.

So I have taken all
the start and end station ID

and using this flat map.

I have retreated
over all the start.

And and station ID and then
using the distinct function

and taking the unique values
and converting it to data frames

so I can use the stations
and using the station.

I will basically keep each
of the stations in a Vertex.

So this is the reason why
I'm taking the stations

or you can say I am taking
the unique stations

from the start station ID
and station ID

so that I can go ahead

and I can define
vertex as the stations.

So now we are creating
our set of vertices

and attaching a bit
of metadata to each one of them

which in our case is
the name of the station.

So as you can see we are
creating this station vertices,

which is again an rdd
with vertex ID and strength.

So we are using the station's
which we have just created.

We are joining it
with just stations

at the station value
should be equal

to just station station ID.

So as we have created stations,

And just station
so we are joining it.

And then selecting
the station ID

and start station name
then we are mapping row 0.

And Row 1 so your row
0 will basically be

your vertex ID and Row
1 will be the string.

That is the name of your station
to let me quickly go ahead

and execute this.

So let us quickly print this
using collect forage println.

So over here,

we are basically attaching
the edges or you can see we

are creating the trip edges
to all our individual rights

and then we'll get
the station values

and then we'll add
a dummy value of one.

So as you can see

that I am selecting
the start station and

and station from the DF

which is the first data frame
which we have loaded

and then I am mapping
it to row 0 + Row 1,

which is your source
and destination.

And then and then I'm attaching
a value one to each one of them.

So I'll hit enter.

Now, let me quickly go ahead
and print this station edges.

So just taking the source
ID of the vertex

and destination ID of the vertex

or you can say so station ID
or vertex station ID

and it is attaching value
one to each one of them.

So now you can go ahead
and build your graph.

But again as we discuss
that we need a default station

so you can have some situations

where your edges might be
indicating some vertices,

but that vertices
might not be present

in your vertex re D.

So for that situation,

we need to create
a default station.

So I created a default station
as missing station.

So now we are all set.

We can go ahead
and create the graph.

So the name of the graph
is station graph.

Then the vertices
are stationed vertices

which we have created

which basically contains
the station ID and station name

and then we have station edges

and at last we
have default station.

So let me quickly go ahead
and execute this.

So now I need to cash this graph
for faster access.

So I'll use cash function.

So let us quickly go ahead and
check the number of vertices.

So these are the number
of vertices again,

we can check the number
of edges as well.

So these are
the number of edges.

And to get a sanity check.

So let's go ahead
and check the number of records

that are present
in the data frame.

So as you can see
that the number of edges

in our graph and the count
in our data frame is similar,

or you can see the same.

So now let's go ahead and run
page rank on our data

so we can either run
a set number of iterations

or we can run it
until the convergence.

So in my case,
I'll run it till convergence.

So it's rank then
station graph then page rank.

So has specified
the double value

so it will Tell convergence
so let's wait for some time.

So now that we have executed
the pagerank algorithm.

So we got the ranks

which are attached
to each vertices.

So now let us quickly go ahead
and look at the ranks.

So we are joining ranks
with station vertices

and then we have sorting it
in descending values

and we are taking
the first 10 rows

and then we are printing them.

So let's quickly go
ahead and hit enter.

So you can see these are
the top 10 stations which have

the most pagerank values

so you can say it has
more number of incoming trips.

Now one question would be

what are the most common
destinations in the data set

from location to location

so we can do this by performing
a grouping operator and adding

The Edge counts together.

So basically this will give
a new graph except each Edge

will now be the sum of all
the semantically same edges.

So again, we are taking
the station graph.

We are performing Group
by edges H1 and H2.

So we are basically
grouping edges H1 and H2.

So we are aggregating them.

Then we are using triplet

and then we are sorting them
in descending order again.

And then we are
printing the triplets

from The Source vertex
and the number of trips

and then we are taking
the destination attribute

or you can see destination

Vertex or you can see
destination station.

So you can see there are

1933 trips from San
Francisco Ferry Building

to the station then again,

you can see there are
fourteen hundred and eleven

trips from San Francisco
to this location.

Then there are 1 0 to 5 trips

from this station
to San Francisco

and it goes so on so now we
have got a directed graph

that mean our
trip are directional

from one location to another

so now we can go ahead
and find the number of Trades

that Went to a specific station

and then leave
from a specific station.

So basically we are trying

to find the inbound
and outbound values

or you can say we are trying
to find in degree and out degree

of the stations.

So let us first calculate the in
degrees from using station graph

and I am using
n degree operator.

Then I'm joining it
with the station vertices

and then I'm sorting it again
in descending order

and then I'm taking
the top 10 values.

So let's quickly go
ahead and hit enter.

So these are the top 10 station
and you can see the in degrees.

So there are these many trips

which are coming
into these stations.

Not similarly.

We can find the out degree.

Now again, you can see
the out degrees as well.

So these are the stations
and these are the out degrees.

So again, you can go ahead
and perform some more operations

over this graph.

So you can go ahead
and find the station

which has most number
of trips things

that is most number of people
coming into that station,

but less people are
leaving that station

and again on the contrary
you can find out the stations

where there are
more number of edges

or you can set trip
leaving those stations.

But there are less number

of trips coming
into those stations.

So I guess you guys are
now clear with Spa Graphics.

Then we discuss
the different types

of crops then moving ahead.

We discuss the
features of grafx.

They'll be discuss something
about property graph.

We understood what
is property graph

how you can create vertex
how you can create edges

how to use Vertex or DD H Rd D.

Then we looked at some of
the important vertex operations

and at last we understood some
of the graph algorithms.

So I guess now you
guys are clear about

how to work with Bob Graphics.

Today's video is
on Hadoop versus park.

Now as we know organizations
from different domains

are investing in big
data analytics today.

They're analyzing large
data sets to uncover

all hidden patterns

unknown correlations market
trends customer preferences

and other useful
business information.

Analogy of findings
are helping organizations

and more effective marketing
new Revenue opportunities

and better customer service

and they're trying
to get competitive advantages

over rival organizations

and other business benefits
and Apache spark

and Hadoop are the two of most
prominent Big Data Frameworks

and I see people often comparing
these two technologies

and that is what exactly
we're going to do in this video.

Now, we'll compare these two big
data Frame Works based

on on different parameters,

but first it is important
to get an overview

about what is Hadoop.

And what is Apache spark?

So let me just tell you a little
bit about Hadoop Hadoop is

a framework to store

and process large sets of data
across computer clusters

and Hadoop can scale
from single computer system

up to thousands
of commodity systems

that offer local storage

and compute power and Hadoop
is composed of modules

that work together to create
the entire Hadoop framework.

These are some of the components

that we have in the
entire Hadoop framework

or the Hadoop ecosystem.

For example, let
me tell you about hdfs,

which is the storage unit
of Hadoop yarn, which is

for resource management.

There are different
than a little tools

like Apache Hive Pig nosql
databases like Apache hbase.

Even Apache spark

and Apache Stone fits
in the Hadoop ecosystem

for processing big data

in real-time for ingesting data
we have Tools like Flume

and scoop flumist used
to ingest unstructured data

or semi-structured data

where scoop is used to ingest
structured data into hdfs.

If you want to learn more
about these tools,

you can go to Eddie rei'kas

YouTube channel and look
for Hadoop tutorial

where everything has
been explained in detail.

Now, let's move to spark

Apache spark is a lightning-fast
cluster Computing technology

that is designed
for fast computation.

The main feature of spark
is it's in memory clusters.

Esther Computing

that increases the processing
of speed of an application fog

perform similar operations
to that of Hadoop modules,

but it uses an in-memory
processing and optimizes

the steps the primary
difference between mapreduce

and Hadoop and Spark is

that mapreduce users
persistent storage

and Spark uses resilient
distributed data sets,

which is known as
rdds which resides

in memory the different
components and Sparkle.

The spark origin the spark
or is the base engine

for large-scale parallel

and distributed data processing
further additional libraries

which are built on top of
the core allow diverse workloads

for streaming SQL

and machine learning spark
or is also responsible

for memory management
and fault recovery scheduling

and distributed and monitoring
jobs and a cluster

and interacting with
the storage systems as well.

Next up.

We have spark streaming.

Spark streaming is
the component of spark

which is used to process
real-time streaming data.

It enables high throughput

and fault-tolerant stream
processing of live data streams.

We have Sparks equal spark
SQL is a new module in spark

which integrates relational
processing with Sparks

functional programming API.

It supports querying
data either via SQL

or via the hive query language.

For those of you
familiar with rdbms.

Spark sequel will be an easy.

Transition from your earlier
tools where you can extend

the boundaries of traditional
relational data processing.

Next up is Graphics Ralph X is
the spark API for graphs

and graph parallel computation

and thus it extends
the spark resilient

distributed data sets with a
resilient distributed property.

Graph.

Next is Park Emma lip
for machine learning

Emma lip stands for machine
learning library spark.

Emma live is used
to perform machine.

In learning in Apache spark now

since you've got an overview
of both these two Frameworks,

I believe that the ground

is all set to compare
Apache spark and Hadoop.

Let's move ahead and compare
Apache spark with Hadoop

on different parameters
to understand their strengths.

We will be comparing
these two Frameworks

based on these parameters.

Let's start with performance
first Spark is fast

because it has

in-memory processing it
can also use For data,

that doesn't fit
into memory Sparks

in-memory processing delivers
near real-time analytics

and this makes Park suitable

for credit card
processing system machine

learning security analysis

and processing data
for iot sensors.

Now, let's talk
about hadoop's performance.

Now Hadoop has originally
designed to continuously

gather data from multiple
sources without worrying

about the type of data

and storing it

across distributed
environment and mapreduce.

Use uses batch processing
mapreduce was never built for

real-time processing main idea

behind yarn is parallel
processing over distributed data

set the problem
with comparing the two is

that they have different
way of processing

and the idea behind the
development is also Divergent

next ease-of-use spark comes

with a user-friendly apis
for Scala Java Python

and Sparks equal spark SQL
is very similar to SQL.

So it becomes easier

for a sequel developers
to learn it spark also

provides an interactive shell

for developers to query
and perform other actions

and have immediate feedback.

Now, let's talk about Hadoop.

You can ingest data in Hadoop
easily either by using shell

or integrating it
with multiple tools,

like scoop and Flume

and yarn is just
a processing framework

that can be integrated
with multiple tools

like Hive and pig for Analytics.

I visit data
warehousing component

which performs Reading Writing

and managing large data set
in a distributed environment

using sql-like interface
to conclude here.

Both of them have
their own ways to make

themselves user-friendly.

Now, let's come
to the cost Hadoop

and Spark are both Apache
open source projects.

So there's no cost for the
software cost is only associated

with the infrastructure both
the products are designed

in such a way

that Can run
on commodity Hardware

with low TCO or total
cost of ownership.

Well now you might
be wondering the ways

in which they are different.

They're all the same storage
and processing in Hadoop is

disc-based and Hadoop uses
standard amounts of memory.

So with Hadoop,

we need a lot

of disk space as well as
faster transfer speed Hadoop

also requires multiple
systems to distribute

the disk input output,

but in case of Apache spark

due to its in-memory processing
it requires a lot of memory,

but it can deal
with the standard.

Speed and amount of disk as
disk space is a relatively

inexpensive commodity

and since Park does not use
disk input output

for processing instead.

It requires large amounts

of RAM for executing
everything in memory.

So spark systems
incurs more cost

but yes one important thing
to keep in mind is

that Sparks technology reduces
the number of required systems,

it needs significantly
fewer systems that cost more

so there will be a point
at which spark reduces

the cost per unit

of the computation even with
the additional RAM requirement.

There are two types of
data processing batch processing

and stream processing batch
processing has been crucial

to the Big Data World

in simplest term batch
processing is working

with high data volumes
collected over a period

in batch processing data is
first collected then processed

and then the results

are produced at a later
stage and batch.

Is it efficient way

of processing large
static data sets?

Generally we perform

batch processing for archived
data sets for example,

calculating average income
of a country

or evaluating the change
in e-commerce in the last decade

now stream processing stream
processing is the current Trend

in the Big Data World need
of the hour is speed

and real-time information,

which is what stream processing

does batch processing
does not allow.

Businesses to quickly react
to changing business needs

and real-time stream processing
has seen a rapid growth

in that demand now coming
back to Apache Spark

versus Hadoop yarn is basically
a batch processing framework

when we submit a job to yarn.

It reads data from
the cluster performs operation

and write the results
back to the cluster

and then it again reads

the updated data performs
the next operation

and write the results back
to the cluster and Off

on the other hand spark is
designed to cover a wide range

of workloads such as

batch application iterative
algorithms interactive queries

and streaming as well.

Now, let's come to fault
tolerance Hadoop and Spark

both provides fault tolerance,

but have different
approaches for hdfs

and yarn both Master demons.

That is the name node in hdfs

and resource manager
in the arm checks the heartbeat

of the slave demons.

The slave demons are data nodes
and node managers.

So if any slave demon fails,

the master demons reschedules
all pending an in-progress

operations to another slave
now this method is effective

but it can significantly
increase the completion time

for operations with
single failure also

and as Hadoop uses
commodity hardware

and another way in which hdfs
ensures fault tolerance is

by replicating data.

Now let's talk about spark

as we discussed earlier rdds are
resilient distributed data sets

are building blocks
of Apache spark

and rdds are the one

which provide fault
tolerant to spark.

They can refer

to any data set present
and external storage system

like hdfs Edge base
shared file system Etc.

They can also be operated

parallely rdds can
persist a data set

and memory across operations.

It's which makes future actions
10 times much faster

if rdd is lost
it will automatically

get recomputed by using
the original Transformations.

And this is how spark provides
fault tolerance and at the end.

Let us talk about security.

Well Hadoop has
multiple ways of providing

security Hadoop supports
Kerberos for authentication,

but it is difficult
to handle nevertheless.

It also supports
third-party vendors like ldap.

For authentication,

they also offer
encryption hdfs supports

traditional file permissions as
well as Access Control lists,

Hadoop provides service level
authorization which guarantees

that clients have
the right permissions for

job submission spark currently
supports authentication

via a shared secret spark
can integrate with hdfs

and it can use hdfs ACLS

or Access Control lists
and file level permissions

sparking also run.

Yarn, leveraging the
capability of Kerberos.

Now.

This was the comparison
of these two Frameworks based

on these following parameters.

Now, let us understand use cases

where these Technologies
fit best use cases were

Hadoop fits best.

For example,

when you're analyzing
archive data yarn

allows parallel processing

over huge amounts of data parts
of data is processed parallely

and separately on
different data nodes

and gathers result
from each node manager in cases

when instant results
are not required now

Hadoop mapreduce is a good
and economical solution

for batch processing.

However, it is incapable

of processing data
in real-time use cases

where Spark fits best
in real-time Big Data analysis,

real-time data analysis
means processing data

that is getting generated by
the real-time event streams

coming in at the rate
of Billions of events

per second the strength

of spark lies in its abilities
to support streaming

of data along with
distributed processing

and Spark claims to process
data hundred times faster

than mapreduce while 10 times
faster with the discs.

It is used in graph
processing spark contains

a graph computation
Library called Graphics

which simplifies our life
in memory computation

along with inbuilt graph support
improves the performance.

Performance of algorithm
by a magnitude

of one or two degrees over
traditional mapreduce programs.

It is also used in iterative
machine learning algorithms

almost all machine learning
algorithms work iteratively

as we have seen earlier
iterative algorithms

involve input/output bottlenecks

in the mapreduce
implementations mapreduce

uses coarse-grained tasks

that are too heavy

for iterative algorithms spark
caches the intermediate data.

I said after each iteration

and runs multiple iterations
on the cache data set

which eventually reduces
the input output overhead

and executes the algorithm

faster in a fault-tolerant
manner sad the end which one is

the best the answer
to this is Hadoop

and Apache spark are
not competing with one another.

In fact, they complement
each other quite well,

how do brings huge
data sets under control

by commodity systems?

Systems and Spark provides
a real-time in-memory processing

for those data sets.

When we combine
Apache Sparks ability.

That is the high processing
speed and advanced analytics

and multiple integration support
with Hadoop slow cost operation

on commodity Hardware.

It gives the best results

Hadoop compliments Apache
spark capabilities spark

not completely replace a do
but the good news is

that the demand of spark is
currently at an all-time.

Hi, if you want to learn more
about the Hadoop ecosystem tools

and Apache spark,

don't forget to take
a look at the editor

Acres YouTube channel
and check out the big data

and Hadoop playlist.

Welcome everyone in
today's session on

kafka's Park streaming.

So without any further delay,
let's look at the agenda first.

We will start by understanding.

What is Apache Kafka?

Then we will discuss
about different components

of Apache Kafka
and it's architecture.

Further we will look
at different Kafka commands.

After that.

We'll take a brief overview
of Apache spark

and will understand
different spark components.

Finally.

We'll look at the demo

where we will use spark
streaming with Apache caf-pow.

Let's move to our first slide.

So in a real time scenario,

we have different
systems of services,

which will be communicating

with each other and
the data pipelines are the ones

which are establishing
connection between two servers

or two systems.

Now, let's take
an example of e-commerce.

Except site where it can have
multiple servers at front end

like Weber application server
for hosting application.

It can have a chat server

for the customers
to provide chart facilities.

Then it can have a separate
server for payment Etc.

Similarly organization can also
have multiple server

at the back end

which will be receiving messages
from different front end servers

based on the requirements.

Now they can have
a database server

which will be storing
the records then they

can have security systems
for user authentication

and authorization then
they can have

Real-time monitoring server,

which is basically
used for recommendations.

So all these data
pipelines becomes complex

with the increase
in number of systems

and adding a new system

or server requires
more data pipelines,

which will again
make the data flow

more complicated and complex.

Now managing.

These data pipelines also
become very difficult

as each data pipeline has
their own set of requirements

for example data pipelines,

which handles transaction
should be more fault tolerant

and robust on the other hand.

Clickstream data pipeline
can be more fragile.

So adding some pipelines

or removing some pipelines
becomes more difficult

from the complex system.

So now I hope that you would
have understood the problem

due to which misting
systems was originated.

Let's move to the next slide
and we'll understand

how Kafka solves this problem
now measuring system reduces

the complexity of data pipelines

and makes the communication
between systems more

simpler and manageable
using messaging system.

Now, you can easily
stablish remote Education

and send your data
easily across Netbook.

Now a different systems

may use different
platforms and languages

and messaging system

provides you a common
Paradigm independent

of any platformer language.

So basically it
decouples the platform

on which a front end server as
well as your back-end server

is running you can also stablish
a no synchronous communication

and send messages

so that the sender
does not have to wait

for the receiver
to process the messages.

Now one of the benefit
of messaging system is

that you can
Reliable communication.

So even when the receiver and
network is not working properly.

Your messages wouldn't
get lost not talking

about cough cough cough cough
decouples the data pipelines

and solves the complexity
problem the applications

which are producing messages
to Kafka are called producers

and the applications

which are consuming
those messages from Kafka

are called consumers.

Now, as you can see in the image
the front end server,

then your application server
will burn application server

to and chat server.

I using messages to Kafka
and these are called producers

and your database server

security systems real-time
monitoring server

than other services
and data warehouse.

These are basically
consuming the messages

and are called consumers.

So your producer sends
the message to Kafka

and then cough cash
to those messages and consumers

who want those
messages can subscribe

and receive them now
you can also have

multiple subscribers to
a single category of messages.

So you Database server

and your security system can
be consuming the same messages

which is produced
by application server

1 and again adding
a new consumer is very easy.

You can go ahead and
add a new consumer

and just subscribe
to the message categories

that is required.

So again, you can add
a new consumer say consumer one

and you can again
go ahead and subscribe

to the category of messages

which is produced by
application server one.

So, let's quickly move ahead.

Let's talk about
a Bocce Kafka so party.

Kafka is a distributed
publish/subscribe messaging

system messaging traditionally
has two models queuing

and publish/subscribe in a queue
a pool of consumers.

May read from a server

and each record only
goes to one of them

whereas in publish/subscribe.

The record is broadcasted
to all consumers.

So multiple consumers
can get the record

the Kafka cluster is distributed

and have multiple machines
running in parallel.

And this is the reason
why calf pies fast

scalable and fault.

Now let me tell you

that Kafka is developed
at LinkedIn and later.

It became a part
of Apache project.

Now, let us look at some
of the important terminologies.

So we'll first start with topic.

So topic is a category

or feed name to which
records are published

and Topic in Kafka are
always multi subscriber.

That is a topic can have
zero one or multiple consumers

that can subscribe the topic

and consume the data written
to it for an example.

You can have serious record
getting published in sales, too.

Topic you can
have product records

which is getting published
in product topic

and so on this will actually
segregate your messages

and consumer will only
subscribe the topic

that they need and again you
consumer can also subscribe

to two or more topics.

Now, let's talk
about partitions.

So Kafka topics are divided
into a number of partitions

and partitions allow
you to paralyze a topic

by splitting the data

in a particular
topic across multiple.

Brokers which means
each partition can be placed

on separate machine to allow
multiple consumers to read

from a topic parallelly.

So in case of serious
topic you can have

three partition partition
0 partition 1 and partition

to from where three consumers
can read data parallel.

Now moving ahead.

Let's talk about producers.

So producers are the one

who publishes the data
to topics of the choice.

Then you have consumers

so consumers can subscribe
to one or more topic.

And consume data

from that topic now consumers
basically label themselves

with a consumer group name

and each record publish
to a topic is delivered

to one consumer instance within
each subscribing consumer group.

So suppose you have
a consumer group.

Let's say consumer Group
1 and then you have

three consumers residing in it.

That is consumer a consumer be
an consumer see now

from the seals topic.

Each record can be read once
by consumer group Fun and it

And either be read by consumer a
or consumer be or consumer see

but it can only be consumed once
by the single consumer group

that is consumer group one.

But again, you can have
multiple consumer groups

which can subscribe to a topic

where one record can be consumed
by multiple consumers.

That is one consumer
from each consumer group.

So now let's say
you have a consumer one

and consumer group
to in consumer Group

1 we have to consumer

that is consumer a a
and consumer be

and consumer group to we

have to Consumers consumer key
and consumer to be so

if consumer Group
1 and consumer group

2 are consuming messages
from topic sales.

So the single record will be
consumed by consumer group one

as well as consumer group
2 and a single consumer

from both the consumer group
will consume the record once so,

I guess you are clear
with the concept of consumer

and consumer group Now
consumer instances can be

a separate process
or separate machines.

No talking about Brokers Brokers
are nothing but a single machine

in the CAF per cluster

and zookeeper is another Apache
open source project.

It's Tuesday metadata
information related

to Kafka cluster.

Like Brokers information
topics details Etc.

Zookeeper is basically the one

who is managing
the whole Kafka cluster.

Now, let's quickly go
to the next slide.

So suppose you have a topic.

Let's assume this is topic sales
and you have for partition

so you have Partition
0 partition 1 partition

to and partition three now you
have five Brokers over here.

Now, let's take the case
of partition 1 so

if the replication factor
is 3 it will have 3 copies

which will reside
on different Brokers.

So when the replica is
on broker to next is

on broker 3 and next is
on brokered 5 and

as you can see repl 5,

so this 5 is from this broker 5.

So the ID of the replica
is same as the ID of The broker

that hosts it now moving ahead.

One of the replica

of partition one will serve
as the leader replica.

So now the leader

of partition one is replica
five and any consumer coming

and consuming messages

from partition one will
be solved by this replica.

And these two replicas is
basically for fault tolerance.

So that once you're
broken five goes off

or your disc becomes corrupt,

so your replica 3 or replica

to to one of them
will again serve as a leader

and this is basically
decided on the basis

of most in sync replica.

So the replica
which will be most

in sync with this replica
will become the next leader.

So similarly this
partition 0 may decide

on broker one broker to
and broker three again

your partition to May
reside on broke of for group

of five and say broker one

and then your third
partition might reside

on these three brokers.

So suppose that this is
the leader for partition

to this is the leader

for partition 0 this is
the leader for partition 3.

This is the leader
for partition 1 right

so you can see

that for consumers can consume
data pad Ali from these Brokers

so it can consume
data from partition

to this consumer can consume
data from partition 0

and similarly for partition
3 and partition fun

now by maintaining
the replica basically helps.

Sin fault tolerance and keeping
different partition leaders

on different Brokers basically
helps in parallel execution

or you can say baddeley
consuming those messages.

So I hope that you
guys are clear

about topics partitions
and replicas now,

let's move to our next slide.

So this is how the whole
Kafka cluster looks like you

have multiple producers,

which is again producing
messages to Kafka.

Then this whole is
the Kafka cluster

where you have two nodes node
one has to broker.

Joker one and broker to
and the Note II has two Brokers

which is broker three and broke
of for again consumers

will be consuming data
from these Brokers

and zookeeper is the one

who is managing
this whole calf cluster.

Now, let's look

at some basic commands of Kafka
and understand how Kafka Works

how to go ahead
and start zookeeper

how to go ahead
and start Kafka server

and how to again go ahead

and produce some messages
to Kafka and then consume

some messages to Kafka.

So let me quickly.

on my VM So let me
quickly open the terminal.

Let me quickly go ahead
and execute sudo GPS

so that I can check
all the demons

that are running in my system.

So you can see I have named

no data node resource manager
node manager job is to server.

So now as all the hdfs demons

are burning let us
quickly go ahead

and start Kafka services.

So first I will go
to Kafka home.

So let me show
you the directory.

So my Kafka is in user lib.

Now.

Let me quickly go ahead
and start zookeeper service.

But before that,

let me show you
zookeeper dot properties file.

So decline Port is 2 1 8 1 so
my zookeeper will be running

on Port to 181

and the data directory
in which my zookeeper

will store all the metadata
is slash temp / zookeeper.

So let us quickly go ahead
and start zookeeper

and the command is bins
zookeeper server start.

So this is the script file

and then I'll pass
the properties file

which is inside config directory
and a little Meanwhile,

let me open another tab.

So here I will be starting
my first Kafka broker.

But before that let me show
you the properties file.

So we'll go
in config directory again,

and I have
server dot properties.

So this is the properties
of my first Kafka broker.

So first we have server Basics.

So here the broker idea
of my first broker is 0 then

the port is 9:09 to on which
my first broker will be running.

So it contains all
the socket server settings

then moving ahead.

We have log base X.

So in that log Basics,
this is log directory,

which is / them / Kafka -

logs so over here
my Kafka will store

all those messages or records,

which will be produced
by The Producers.

So all the records

which belongs to broker 0
will be stored at this location.

Now, the next section is
internal topic settings

in which the offset topical.

application factor is 1

then transaction State log
replication factor is 1 Next

we have log retention policy.

So the log retention
ours is 168.

So your records will be stored
for 168 hours by default

and then it will be deleted.

Then you have
zookeeper properties

where we have specified
zookeeper connect and

as we have seen
in Zookeeper dot properties file

that are zookeeper
will be running on Port 2 1 8 1

so we are giving
the address of Zookeeper

that is localized
to one eight one

and at last we have group.

Coordinator setting so
let us quickly go ahead

and start the first broker.

So the script file is
Kafka server started sh

and then we have to give
the properties file,

which is server dot properties
for the first broker.

I'll hit enter and meanwhile,
let me open another tab.

now I'll show you
the next properties file,

which is Server 1.

Properties.

So the things

which you have to change
for creating a new broker

is first you have
to change the broker ID.

So my earlier book ID was 0
the new broker ID is 1 again,

you can replicate this file
and for a new server,

you have to change
the broker idea to to then

you have to change the port
because on 9:09 to already.

My first broker is running
that is broker 0

so my broker.

Should connect to
a different port

and here I have specified
nine zero nine three.

Next thing what you have
to change is the log directory.

So here I have added a -
1 to the default log directory.

So all these records

which is stored to my broker
one will be going

to this particular directory

that is slashed
and slashed cough call logs -

1 And rest of the
things are similar,

so let me quickly go ahead
and start second broker as well.

And let me open
one more terminal.

And I'll start
broker to as well.

So the Zookeeper started then
procurve one is also started

and this is broker
to which is also started

and this is proof of 3.

So now let me
quickly minimize this

and I'll open a new terminal.

Now first, let us look
at some commands later

to Kafka topics.

So I'll quickly go ahead
and create a topic.

So again, let me first go
to my Kafka home directory.

Then the script file
is Kafka top it dot sh,

then the first parameter

is create then we have to give
the address of zoo keeper

because zookeeper is the one

who is actually containing
all the details related

to your topic.

So the address of my zookeeper
is localized to one eight one

then we'll give the topic name.

So let me name the topic
as Kafka -

spark next we have to specify
the replication factor

of the topic.

So it will replicate all
the partitions inside the topic

that many times.

So replication -

Factor as we
have three Brokers,

so let me keep it as 3
and then we have partitions.

So I will keep it as

three because we have
three Brokers running

and our consumer can go ahead

and consume messages parallely

from three Brokers and
let me press enter.

So now you can see
the topic is created.

Now, let us quickly go ahead
and list all the topics.

So the command

for listing all the topics
is dot slash bin again.

We'll open cough car
topic script file then -

- list and again will provide
the address of Zookeeper.

So do again list the topic

we have to first go to
the CAF core topic script file.

Then we have to give -

- list parameter and next we
have to give the zookeepers.

Which is localhost
181 I'll hit enter.

And you can see
I have this Kafka -

spark the kafka's
park topic has been created.

Now.

Let me show you
one more thing again.

We'll go to when cuff
card topics not sh

and we'll describe this topic.

I will pass the address
of zoo keeper,

which is localhost

to one eight one and then
I'll pause the topic name,

which is Kafka - Spark

So now you can see here.

The topic is cough by spark.

The partition count is
3 the replication factor is 3

and the config is as follows.

So here you can see all the
three partitions of the topic

that is partition 0 partition 1
and partition 2 then the leader

for partition 0 is
broker to the leader

for partition one is broker 0

and leader for partition
to is broker one

so you can see we have different
partition leaders residing on

And Brokers, so this is
basically for load balancing.

So that different partition
could be served

from different Brokers

and it could be
consumed parallely again,

you can see the replica

of this partition is residing
in all the three Brokers same

with Partition 1 and same
with Partition to

and it's showing you
the insync replica.

So in synch replica,

the first is to then you have 0
and then you have 1

and similarly with
Partition 1 and 2.

So now let us quickly.

Go ahead.

I'll reduce this to 1/2.

Wake me up in one more terminal.

The reason why I'm doing this is

that we can actually produce
message from One console

and then we can receive
the message in another console.

So for that I'll start cough
cough console producer first.

So the command is

dot slash bin cough cough
console producer dot sh

and then in case

of producer you have to give
the parameter as broker - list,

which is Localhost 9:09 to you
can provide any of the Brokers

that is running

and it will again take the rest
of the Brokers from there.

So you just have to provide
the address of one broker.

You can also provide
a set of Brokers

so you can give it
as localhost colon.

9:09 2 comma Lu closed:
9 0 9 3 and similarly.

So here I am passing the address

of the first broker now next
I have to mention the topic.

So topic is Kafka Spark.

And I'll hit enter.

So my console
producer is started.

Let me produce
a message saying hi.

Now in the second terminal
I will go ahead

and start the console consumer.

So again, the command is
Kafka console consumer not sh

and then in case of consumer,

you have to give the parameter
as bootstrap server.

So this is the thing
to notice guys that in case

of producer you have to give
the broker list by in.

So of consumer,

you have to give bootstrap
server and it is again the same

that is localhost 9:09 to which
the address of my broker 0

and then I will give the topic

which is cuff cost park
now adding this parameter

that is from -

beginning will basically
give me messages stored

in that topic from beginning.

Otherwise, if I'm not giving
this parameter - -

from beginning I'll only

I'm the recent messages

that has been produced after
starting this console consumer.

So let me hit enter

and you can see I'll get
a message saying hi first.

Well, I'm sorry guys.

The topic name I
have given is not correct.

Sorry for my typo.

Let me quickly corrected.

And let me hit enter.

So as you can see,
I am receiving the messages.

I received High then let
me produce some more messages.

So now you can see
all the messages

that I am producing

from console producer is getting
consumed by console consumer.

Now this console producer
as well as console consumer

is basically used by

the developers to actually
test the Kafka cluster.

So what happens if you are

if there is a producer

which is running and
which is producing

those messages to Kafka
then you can go ahead

and you can start console
consumer and check

whether the producer
is producing.

Messages or not
or you can again go ahead

and check the format

in which your message are
getting produced to the topic.

Those kind of testing part
is done using console consumer

and similarly using
console producer.

You do something
like you are creating a consumer

so you can go ahead you can
produce a message to Kafka topic

and then you can check

whether your consumer is
consuming that message or not.

This is basically used
for testing now,

let us quickly go ahead
and close this.

Now let us get back
to our slides now.

I have briefly covered Kafka

and the concepts of Kafka so
here basically I'm giving

you a small brief idea
about what Kafka is

and how Kafka works now

as we have understood why
we need misting systems.

What is cough cough?

What are different
terminologies and Kafka

how Kafka architecture works

and we have seen some
of the basic cuff Pokemons.

So let us now understand.

What is Apache spark.

So basically Apache spark

is an Source cluster
Computing framework

for near real-time processing
now spark provides

an interface for programming
the entire cluster

with implicit data parallelism

and fault tolerance will talk
about how spark provides

fault tolerance but talking
about implicit data parallelism.

That means you do not need
any special directives operators

or functions to enable
parallel execution.

It sparked by default provides
the data parallelism spark

is designed to cover
a wide range of workloads such.

As batch applications

iterative algorithms interactive
queries machine learning

algorithms and streaming.

So basically the main feature

of spark is it's
in memory cluster Computing

that increases the processing
speed of the application.

So what spark does spark does
not store the data in discs,

but it does it
transforms the data

and keep the data in memory.

So that quickly

multiple operations can
be applied over the data

and the final result
is only stored in the disk

now a On-site Spa can also do
batch processing hundred times

faster than mapreduce.

And this is the reason why
a patches Park is to go

to tool for big data processing
in the industry.

Now, let's quickly move
ahead and understand

how spark does this

so the answer is rdd

that is resilient distributed
data sets now an rdd is

a read-only partitioned
collection of records and you

can see it is a fundamental
data structure of spa.

So basically, ERD is

an immutable distributed
collection of objects.

So each data set
in rdd is divided

into logical partitions,

which may be computed
on different nodes

of the cluster now already
can contain any type

of python Java or scale objects.

Now talking about
the fault tolerance rdd

is a fault-tolerant collection
of elements that can be operated

on in parallel.

Now, how are ready does

that if rdd is lost
it will automatically

be recomputed by using original.

Nations and this is how spot
provides fault tolerance.

So I hope that you
guys are clear

that house Park
provides fault tolerance.

Now let's talk about
how we can create rdds.

So there are two ways to create
rdds first is paralyzing

an existing collection
in your driver program,

or you can refer a data set

in an external storage systems
such as shared file system.

It can be hdfs Edge base

or any other data source
offering a Hadoop input format

now spark makes use

of the concept of rdd to achieve
fast and efficient operations.

Now, let's quickly move ahead

and look how already So
first we create an rdd

which you can create
either by referring

to an external storage system.

And then once you create
an rdd you can go ahead

and you can apply
multiple Transformations

over that are ready.

Like will perform
filter map Union Etc.

And then again,

it gives you a new rdd or you
can see the transformed rdd

and at last you apply
some action and get

the result now this action
can be Count first

a can collect all those kind

of functions.

So now this is a brief idea
about what is rdd

and how rdd works.

So now let's quickly
move ahead and look

at the different workloads

that can be handled
by Apache spark.

So we have interactive
streaming analytics.

Then we have machine learning.

We have data integration.

We have spark
streaming and processing.

So let us talk about them one

by one first is spark
streaming and processing.

So now basically,

you know data arrives
at a steady rate.

Are you can say
at a continuous streams, right?

And then what you can do
you can again go ahead

and store the data set in disk

and then you can actually go
ahead and apply some processing

over it some analytics over it

and then get
some results out of it,

but this is not the scenario
with each and every case.

Let's take an example
of financial transactions

where you have to go
ahead and identify

and refuse potential
fraudulent transactions.

Now if you will go ahead

and store the data stream
and then you will go ahead

and apply some Assessing
it would be too late

and someone would have got
away with the money.

So in that scenario
what you need to do.

So you need to quickly take
that input data stream.

You need to apply
some Transformations over it

and then you have
to take actions accordingly.

Like you can send
some notification

or you can actually reject

that fraudulent transaction
something like that.

And then you can go ahead

and if you want you
can store those results

or data set in some
of the database

or you can see some
of the file system.

So we have some scenarios.

Very we have to actually
process the stream of data

and then we have to go ahead

and store the data

or perform some analysis on it
or take some necessary actions.

So this is where Spark
streaming comes into picture

and Spark is a best fit

for processing those continuous
input data streams.

Now moving to next
that is machine learning now,

as you know,

that first we create

a machine learning model
then we continuously feed

those incoming data
streams to the model.

And we get some
continuous output based

on the input values.

Now, we reuse
intermediate results

across multiple computation
in multi-stage applications,

which basically includes
substantial overhead due to

data replication disk
I/O and sterilization

which makes the system slow.

Now what Spock does spark rdd
will store intermediate result

in a distributed memory
instead of a stable storage

and make the system faster.

So as we saw in spark rdd
all the Transformations

will be applied over there

and all the transformed
rdds will be stored

in the memory itself
so we can quickly go ahead

and apply some more
iterative algorithms over there

and it does not take
much time in functions

like data replication or disk

I/O so all those overheads
will be reduced now

you might be wondering
that memories always very less.

So what if the memory
gets over so

if the distributed memory
is not sufficient

to store intermediate results,

then it will
store those results.

On the desk.

So I hope that you guys are
clear how sparks perform

this iterative machine
learning algorithms

and why spark is fast,

let's look at the next workload.

So next workload is
interactive streaming analytics.

Now as we already discussed
about streaming data

so user runs ad hoc queries
on the same subset of data

and each query will do a disk
I/O on the stable storage

which can dominate
applications execution time.

So, let me take an example.

Data scientist.

So basically you have
continuous streams of data,

which is coming in.

So what your data
scientists would do.

So do your data scientists
will either ask

some questions execute

some queries over the data
then view the result

and then he might alter
the initial question slightly

by seeing the output

or he might also drill
deeper into results

and execute some more queries
over the gathered result.

So there are multiple scenarios
in which your data scientist

would be running
some interactive queries.

On the streaming analytics
now house path helps

in this interactive
streaming analytics.

So each transformed our DD
may be recomputed each time.

You run an action on it, right?

And when you persist an rdd
in memory in which case

Park will keep all
the elements around

on the cluster for faster access

and whenever you will execute
the query next time

over the data,

then the query will
be executed quickly

and it will give you
a instant result, right?

So I hope that you
guys are clear

how spark helps in
interactive streaming analytics.

Now, let's talk
about data integration.

So basically as you know,

that in large organizations data
is basically produced

from different systems
across the business

and basically you
need a framework

which can actually integrate
different data sources.

So Spock is the one

which actually integrate
different data sources

so you can go ahead

and you can take the data
from Kafka Cassandra flu.

Umm hbase then Amazon S3.

Then you can perform some real
time analytics over there

or even say some near
real-time analytics over there.

You can apply some machine
learning algorithms

and then you can go ahead

and store the process
result in Apache hbase.

Then msql hdfs.

It could be your Kafka.

So spark basically gives
you a multiple options

where you can go ahead

and pick the data
from and again,

you can go ahead
and write the data into now.

Let's quickly move ahead
and we'll talk.

About different spark components
so you can see here.

I have a spark or engine.

So basically this
is the core engine

and on top of this core engine.

You have spark SQL spark
streaming then MLA,

then you have graphics
and the newest Parker.

Let's talk about them one
by one and we'll start

with spark core engine.

So spark or engine
is the base engine

for large-scale parallel

and distributed data processing
additional libraries,

which are built on top
of the core allows

divers workloads Force.

Streaming SQL machine learning
then you can go ahead

and execute our on spark

or you can go ahead
and execute python on spark

those kind of workloads.

You can easily go
ahead and execute.

So basically your spark
or engine is the one

who is managing all your memory,

then all your fault
recovery your scheduling

your Distributing of jobs

and monitoring jobs

on a cluster and interacting
with the storage system.

So in in short we
can see the spark

or engine is the heart of Spock

and on top of this all

of these libraries
are there so first,

let's talk about
spark streaming.

So spot streaming is
the component of Spas

which is used to process
real-time streaming data

as we just discussed

and it is a useful addition
to spark core API.

Now it enables high
throughput and fault

tolerance stream processing
for live data streams.

So you can go ahead

and you can perform all
the streaming data analytics

using this spark streaming then
You have Spock SQL over here.

So basically spark SQL is
a new module in spark

which integrates relational
processing of Sparks functional

programming API and it supports
querying data either via SQL

or SQL that is -

query language.

So basically for those of you

who are familiar with rdbms
Spock SQL is an easy transition

from your earlier tool

where you can go ahead
and extend the boundaries

of traditional relational
data processing now

talking about graphics.

So Graphics is

the spaag API for graphs
and crafts parallel computation.

It extends the spark rdd

with a resilient distributed
property graph a talking

at high level.

Basically Graphics extend
the graph already abstraction

by introducing the resilient
distributed property graph,

which is nothing

but a directed multigraph
with properties attached

to each vertex and Edge
next we have spark are so

basically it provides you
packages for our language

and then you can go ahead and

Leverage Park power

with our shell next
you have spark MLA.

So ml is basically stands
for machine learning library.

So spark MLM is used
to perform machine learning

in Apache spark.

Now many common machine learning

and statical algorithms
have been implemented

and are shipped with ML live

which simplifies large scale
machine learning pipelines,

which basically includes

summary statistics
correlations classification

and regression collaborative
filtering techniques.

New cluster analysis methods

then you have dimensionality
reduction techniques.

You have feature extraction
and transformation functions.

When you have
optimization algorithms,

it is basically a MLM package

or you can see a machine
learning package on top of spa.

Then you also have
something called by spark,

which is python package
for spark there.

You can go ahead
and leverage python over spark.

So I hope

that you guys are clear
with different spark components.

So before moving
to cough gasp,

ah, Exclaiming demo.

So I have just given you
a brief intro to Apache spark.

If you want a detailed tutorial
on Apache spark

or different components

of Apache spark like Apache
spark SQL spark data frames

or spark streaming
Spa Graphics Spock MLA,

so you can go to editor
Acres YouTube channel again.

So now we are here guys.

I know that you guys are waiting
for this demo from a while.

So now let's go ahead and look
at calf by spark streaming demo.

So let me quickly go
ahead and open.

my virtual machine
and I'll open a terminal.

So let me first check
all the demons

that are running in my system.

So my zookeeper is running

name node is running
data node is running.

The my resource manager

is running all the three cough
cough Brokers are running then

node manager is running

and job is to server is running.

So now I have to start
my spark demons.

So let me first go
to the spark home

and start this part demon.

The command is
a spin start or not.

Sh.

So let me quickly go ahead

and execute sudo JPS
to check my spark demons.

So you can see master
and vocal demons are running.

So let me close this terminal.

Let me go to
the project directory.

So basically, I
have two projects.

This is cough card
transaction producer.

And the next one is the spark
streaming Kafka master.

So first we will
be producing messages

from Kafka transaction producer

and then we'll be
streaming those records

which is basically produced by
this producer using the spark

streaming Kafka master.

So first, let me take you

through this cough
card transaction producer.

So this is
our cornbread XML file.

Let me open it with G edit.

So basically this is a me.

Project and and I have used
spring boot server.

So I have given Java version

as a you can see
cough cough client over here

and the version of Kafka client,

then you can see I'm putting
Jackson data bind.

Then ji-sun and then I
am packaging it as a war file

that is web archive file.

And here I am again specifying
the spring boot Maven plugins,

which is to be downloaded.

So let me quickly go ahead

and close this and we'll go
to this Source directory

and then we'll go inside main.

So basically this is the file
that is sales Jan 2009 file.

So let me show you
the file first.

So these are the records

which I'll be producing
to the Kafka.

So the fields
are transaction date

than product price payment

type the name city state
country account created

then last login latitude

and longitude.

So let me close this file
and then the application dot.

Yml is the main property file.

So in this application
dot yml am specifying

the bootstrap server,

which is localhost 9:09 to
than am specifying the Pause

which again resides
on localhost 9:09 to so here.

I have specified the broker list
now next I have product topic.

So the topic of the
product is transaction.

Then the partition count is 1

so basically you're a cks
config controls the criteria

under which requests
are considered complete

and the all setting we
have specified will result

in blocking on the full
Committee of the record.

It is the slowest burn

the most durable setting
not talking about retries.

So it will retry Thrice
then we have mempool size

and we have maximum pool size,

which is basically
for implementing Java threads

and at last we
have the file path.

So this is the path of the file,

which I have shown you just now
so messages will be consumed

from this file.

Let me quickly close this file
and we'll look at application

but properties so here we
have specified the properties

for Springboard server.

So we have server context path.

That is /n Eureka.

Then we have
spring application name

that is Kafka producer.

We have server Port

that is double line W8 and
the spring events timeout is 20.

So let me close this as well.

Let's go back.

Let's go inside Java calm
and Eureka Kafka.

So we'll explore
the important files one by one.

So let me first take you
through this dito directory.

And over here,
we have transaction dot Java.

So basically here we
are storing the model.

So basically you can see these
are the fields from the file,

which I have shown you.

So we have transaction date.

We have product price payment
type name city state country

and so on so we have created
variable for each field.

So what we are doing we
are basically creating a getter

and Setter function for
all these variables.

So we have get transaction ID,

which will basically
returned Transaction ID then

we have sent transaction ID,

which will basically
send the transaction ID.

Similarly.

We have get transaction date for
getting the transaction date.

Then we have set
transaction date and it

will set the transaction date
using this variable.

Then we have get products
and product get price set price

and all the getter

and Setter methods
for each of the variable.

This is the Constructor.

So here we are taking

all the parameters like
transaction date product price.

And then we are setting
the value of each

of the variables
using this operator.

So we are setting the value for
transaction date product price

payment and all of the fields
that is present over there.

Next.

We are also creating
a default Constructor

and then over here.

We are overriding
the tostring method

and what we are doing
we are basically

The transaction details

and we are
returning transaction date

and then the value
of transaction date product

then body of product price
then value of price

and so on for all the fields.

So basically this is the model
of the transaction

so we can go ahead

and we can create object
of this transaction

and then we can easily go ahead

and send the transaction
object as the value.

So this is the main
reason of creating

this transaction model, LOL.

Me quickly, go ahead
and close this file.

Let's go back and let's first
take a look at this config.

So this is Kafka
properties dot Java.

So what we did again

as I have shown you
the application dot yml file.

So we have taken all

the parameters that we
have specified over there.

That is your bootstrap
product topic partition count

then Brokers filename
and thread count.

So all these properties

then you have file path
then all these Days,

we have taken we have created

a variable and then
what we are doing again,

we are doing the same thing

as we did with
our transaction model.

We are creating a getter

and Setter method for each
of these variables.

So you can see we
have get file path

and we are returning
the file path.

Then we have set file path

where we are setting the file
path using this operator.

Similarly.

We have get product topics

at product topic then we
have greater incentive

for third count.

We have greater incentive.

for bootstrap and all
those properties No,

we can again go ahead

and call this cough
cough properties anywhere

and then we can easily

extract those values
using getter methods.

So let me quickly close
this file and I'll take you

to the configurations.

So in this configuration

what we are doing we
are creating the object

of Kafka properties
as you can see,

so what we are doing then we
are again creating a property's

object and then we
are setting the properties

so you can see

that we are Setting
the bootstrap server config

and then we are retrieving

the value using the cough
cough properties object.

And this is the get
bootstrap server function.

Then you can see we are setting
the acknowledgement config

and we are getting

the acknowledgement from this
get acknowledgement function.

And then we are using
this get rate rise method.

So from all these
Kafka properties object.

We are calling
those getter methods

and retrieving those values

and setting those values
in this property object.

So We have partitioner class.

So we are basically implementing
this default partitioner

which is present in

over G. Apache car park client
producer internals package.

Then we are creating
a producer over here

and we are passing this props

object which will set
the properties so over here.

We are passing
the key serializer,

which is the
string T serializer.

And then this is

the value serializer in which
we are creating new customer.

Distance Eliezer and then
we are passing transaction

over here and then it
will return the producer

and then we are implementing
thread we are again getting

the get minimum pool size

from Kafka properties and get
maximum pool size

from Kafka property.

So we're here.

We are implementing
Java threads now.

Let me quickly close this cough
pop producer configuration

where we are configuring
our Kafka producer.

Let's go back.

Let's quickly go to this API

which have event producer
EPA dot Java file.

So here we are basically
creating an event producer API

which has this
dispatch function.

So we'll use this dispatch
function to send the records.

So let me quickly
close this file.

Let's go back.

We have already seen this config

and configurations

in which we are basically
retrieving those values

from application dot yml file

and then we are Setting
the producer configurations,

then we have constants.

So in Kafka constants or Java,

we have created this Kafka
constant interface

where we have specified

the batch size account limit
check some limit then read

batch size minimum
balance maximum balance

minimum account maximum account.

Then we are also implementing
daytime for matter.

So we are specifying all
the constants over here.

Let me close this file.

Let's go back then this is
Manso will not look

at these two files,

but let me tell you what
does these two files

to these two files are
basically to record the metrics

of your Kafka like time in which

your thousand records have
been produced in cough power.

You can say time

in which records
are getting published to Kafka.

It will be monitored and then
you can record those starts.

So basically it helps

in optimizing the performance
of your Kafka producer, right?

You can actually know
how to do Recon.

How to add just
those configuration factors

and then you can
see the difference

or you can actually
monitor the stats

and then understand

or how you can actually make
your producer more efficient.

So these are basically

for those factors but let's
not worry about this right now.

Let's go back next.

Let me quickly take you
through this file utility.

So you have file
you treated or Java.

So basically what we
are doing over here,

we are reading each record

from the file we using
For reader so over here,

you can see we have this list
and then we have bufferedreader.

Then we have file reader.

So first we are reading the file

and then we are trying
to split each of the fields

present in the record.

And then we are setting the
value of those fields over here.

Then we are specifying
some of the exceptions

that may occur like
number format exception

or pass exception all
those kind of exception

we have specified over here
and then we are Closing this

so in this file.

We are basically
reading the records now.

Let me close this.

Let's go back.

Now.

Let's take a quick look
at the seal lizer.

So this is custom
Jason serializer.

So in serializer,

we have created
a custom decency réaliser.

Now, this is basically
to write the values as bites.

So the data which you will be
passing will be written in bytes

because as we know

that data is sent to Kafka
and form of pie.

And this is the reason
why we have created

this custom Jason serializer.

So now let me quickly close
this let's go back.

This file is basically for
my spring boot web application.

So let's not get into this.

Let's look at events
Red Dot Java.

So basically over here we
have event producer API.

So now we are trying to dispatch
those events and to show you

how dispatch function works.

Let me go back.

Let me open services

and even producer
I MPL is implementation.

So let me show you
how this dispatch works.

So basically over here
what we are doing first.

We are initializing.

So using the file utility.

We are basically reading
the files and read the file.

We are getting the path using
this Kafka properties object

and we are calling
this getter method of file path.

Then what we are doing
we are basically taking

the product list

and then we are trying
to dispatch it so

in dispatch Are basically
using Kafka producer

and then we are creating the
object of the producer record.

Then we are using the get topic
from this calf pad properties.

We are getting
this transaction ID

from the transaction

and then we are using event
producer send to send the data.

And finally we are trying

to monitor this but let's
not worry about the monitoring

and cash the monitoring
and start spot

so we can ignore this part Nets.

Let's quickly go back
and look at the last file

which is producer.

So let me show you
this event producer.

So what we are doing here,

we are actually
creating a logger.

So in this on completion method,

we are basically passing
the record metadata.

And if your e-except shin is
not null then it will basically

throw an error saying this
and recorded metadata else.

It will give you the send
message to topic partition.

All set and then
the record metadata

and topic and then it will give

you all the details regarding
topic partitions and offsets.

So I hope that you
guys have understood

how this cough cough producer
is working now is the time we

need to go ahead and we need
to quickly execute this.

So let me open
a terminal over here.

No first build this project.

We need to execute
mvn clean install.

This will install
all the dependencies.

So as you can see
our build is successful.

So let me minimize this and
this target directory is created

after you build
a wave in project.

So let me quickly go

inside this target directory and
this is the root dot bar file

that is root dot
web archive file

which we need to execute.

So let's quickly go ahead
and execute this file.

But before this to verify

whether the data
is getting produced in our car

for topics so for testing

as I already told you
We need to go ahead

and we need to open
a console consumer

so that we can check

that whether data
is getting published or not.

So let me quickly minimize this.

So let's quickly go to
Kafka directory and the command

is dot slash bin Kafka
console consumer and then -

- bootstrap server.

nine zero nine two Okay,
I'll let me check the topic.

What's the topic?

Let's go to our
application dot yml file.

So the topic
name is transaction.

Let me quickly minimize
this specify the topic name

and I'll hit enter.

So now let me place
this console aside.

And now let's quickly go ahead
and execute our project.

So for that
the command is Java -

jar and then we'll provide
the path of the file

that is inside.

Great, and the file is
rude dot war and here we go.

So now you can see
in the console consumer.

The records are
getting published.

Right?

So there are multiple records

which have been published
in our transaction topic

and you can verify this
using the console consumer.

So this is

where the developers use
the console consumer.

So now we have successfully
verified our producer.

So let me quickly go ahead
and stop the producer.

Lat, let me stop
consumer as well.

Let's quickly minimize this

and now let's go
to the second project.

That is Park
streaming Kafka Master.

Again.

We have specified
all the dependencies

that is required.

Let me quickly show
you those dependencies.

Now again, you
can see were here.

We have specified
Java version then we

have specified the artifacts
or you can see the dependencies.

So we have Scala compiler.

Then we have
spark streaming Kafka.

Then we have
cough cough clients.

Then Json data binding then we
have Maven compiler plug-in.

So all those dependencies
which are required.

We are specified over here.

So let me quickly go
ahead and close it.

Let's quickly move to the source
directory main then let's look

at the resources again.

So this is application
dot yml file.

So we have put

eight zero eight zero then we
have bootstrap server over here.

Then we have proven over here.

Then we have topic
is as transaction.

The group is transaction
partition count is one

and then the file name

so we won't be using
this file name then.

Let me quickly go ahead
and close this.

Let's go back.

Let's go back to Java
directory comms Park demo,

then this is the model.

So it's same

so these are all the fields
that are there

in the transaction
you have transaction.

Eight product price payment type

the name city state country
account created and so on.

And again, we have
specified all the getter

and Setter methods over here
and similarly again,

we have created
this transaction dto Constructor

where we have taken
all the parameters

and then we have setting
the values using this operator.

Next.

We are again over adding
this tostring function

and over here.

We are again returning the
details like transaction date

and then vario
transaction date product

and then value of product
and similarly all the fields.

So let me close this model.

Let's go back.

Let's look at cough covers,
then we are see realizer.

So this is the Jason serializer

which was there in our producer
and this is transaction decoder.

Let's take a look.

Now you have decoder

which is again implementing
decoder and we're passing

this transaction dto then again,

you can see we This problem
by its method

which we are overriding

and we are reading
the values using this bites

and then transaction
DDO class again,

if it is failing to pass we are
giving Json processing failed

for object this

and you can see we have

this transaction decoder
construct over here.

So let me quickly
again close this file.

Let's quickly go back.

And now let's take a look
at spot streaming app

where basically the data

which the producer project
will be producing to cough cough

will be actually consumed by
spark streaming application.

So spark streaming will stream
the data in real time

and then will display the data.

So in this park
streaming application,

we are creating conf object
and then we are setting

the application name
as cough by sandbox.

The master is local star
then we have Java.

Fog contest so here we
are specifying the spark context

and then next we are specifying
the Java streaming context.

So this object will basically

we used to take
the streaming data.

So we are passing this Java Spa
context over here as a parameter

and then we are specifying
the duration that is 2000.

Next.

We have Kafka parameters
should to connect

to Kafka you need
to specify this parameters.

So in Kafka parameters,

we are specifying
The Meta broken.

Why's that is localized 9:09 to
then we have Auto offset resent

that is smallest.

Then in topics the name
of the topic from which we

will be consuming messages
is transaction next Java.

We're creating a Java
pair input D streams.

So basically this D stream
is discrete stream,

which is the basic abstraction
of spark streaming

and is a continuous sequence

of rdds representing
a continuous stream

of data now the stream can I
The created from live data

from Kafka hdfs of Flume

or it can be generated

from transforming existing be
streams using operation

to over here.

We are again creating
a Java input D stream.

We are passing string
and transaction DTS parameters

and we are creating
direct Kafka stream object.

Then we're using
this Kafka you tails

and we are calling
the method create direct stream

where we are passing
the parameters as SSC

that is your spark
streaming context then

you have String dot class

which is basically
your key serializer.

Then transaction video
does not class

that is basically

your value serializer
then string decoder

that is to decode your key
and then transaction

decoded basically to
decode your transaction.

Then you have Kafka parameters,

which you have created here

where you have specified
broken list and auto

offset reset and then you
are specifying the topics

which is your transaction so
next using this Cordy stream,

you're actually continuously
iterating over the rdd

and then you are trying
to print your new rdd

with then already partition

and size then rdd count

and the record so already
for each record.

So you are printing the record

and then you are starting
these Park streaming context

and then you are waiting
for the termination.

So this is the spark
streaming application.

So let's first quickly go ahead
and execute this application.

They've been close this file.

Let's go to the source.

Now, let me quickly go ahead and
delete this target directory.

So now let me quickly open the
terminal MV and clean install.

So now as you can see the target
directory is again created

and this park streaming Kafka
snapshot jar is created.

So we need to execute this jar.

So let me quickly go ahead
and minimize it.

Let me close this terminal.

So now first I'll start
this pop streaming job.

So the command is Java -
jar inside the target directory.

We have this spark streaming of
college are so let's hit enter.

So let me know quickly go ahead
and start producing messages.

So I will minimize this and I
will wait for the messages.

So let me quickly close
this pot streaming job

and then I will show
you the consumed records

so you can see the record

that is consumed
from spark streaming.

So here you have got record
and transaction dto

and then transaction date
products all the details,

which we are specified.

You can see it over here.

So this is how spark
streaming works with Kafka now,

it's just a basic job again.

You can go ahead and you
can take Those transaction you

can perform some real-time
analytics over there

and then you can go ahead and
write those results so over here

we have just given
you a basic demo

in which we are producing
the records to Kafka

and then using spark streaming.

We are streaming those records
from Kafka again.

You can go ahead

and you can perform
multiple Transformations

over the data multiple actions

and produce some real-time
results using this data.

So this is just a basic demo
where we have shown you

how to basically
produce recalls to Kafka

and then consume those records
using spark streaming.

So let's quickly go
back to our slide.

Now as this was a basic project.

Let me explain you

one of the cough
by spark streaming project,

which is a Ted Eureka.

So basically there is a company
called Tech review.com.

So this take review.com
basically provide reviews

for your recent
and different Technologies,

like a smart watches phones
different operating systems

and anything new
that is coming into Market.

So what happens is the company
decided to include a new feature

which will basically allow
users to compare the popularity

or trend of multiple
Technologies based

on the Twitter feeds
and second for the USP.

They are basically

trying this comparison
to happen in real time.

So basically they
have assigned you this task

so that you have to go
ahead you have to take

the real-time Twitter feeds

then you have to show
the real time comparison

of various Technologies.

So again, the company is
is asking you to to identify

the minute literate

between different Technologies
by consuming Twitter streams

and writing aggregated minute
Li count to Cassandra

from where again -

boarding team will come
into picture and then they

will try to dashboard that data
and it can show you a graph

where you can see

how the trend of two different

or you can see various
Technologies are going ahead now

the solution strategy
which is there

so you have to continuously
stream the data from Twitter.

Then you will be storing

that those tweets
inside a cop car topic

then second again.

You have to
perform spark streaming.

So you will be continuously
streaming the data

and then you will be
applying some Transformations

which will basically
give you the minute trend

of the two technologies.

And again, you'll write it back
to a car for topic and at last

you'll write a consumer

that will be consuming messages
from the Casbah topic

and that will write the data
in your Cassandra database.

So First you have
to write a program

that will be consuming
data from Twitter

and I did to cough or topic.

Then you have to write
a spark streaming job,

which will be continuously
streaming the data from Kafka

and perform analytics
to identify the military Trend

and then it will write the data
back to a cuff for topic

and then you have
to write the third job

which will be
basically a consumer

that will consume data
from the table for topic

and write the data
to a Cassandra database.

But a spark is
a powerful framework,

which has been heavily
used in the industry

for real-time analytics
and machine learning purposes.

So before I proceed
with the session,

let's have a quick
look at the topics

which will be covering today.

So I'm starting
off by explaining

what exactly is by spot
and how it works.

When we go ahead.

We'll find out the various
advantages provided by spark.

Then I will be showing you

how to install
by sparking a systems.

Once we are done
with the installation.

I will talk about the
fundamental concepts of by spark

like this spark context.

Data frames MLA Oddities
and much more and finally,

I'll close of the session with
the demo in which I'll show you

how to implement by spark
to solve real life use cases.

So without any further Ado,

let's quickly embark
on our journey to pie spot now

before I start off
with by spark.

Let me first brief you
about the by spark ecosystem

as you can see from the diagram
the spark ecosystem is composed

of various components like
Sparks equals Park streaming.

Ml Abe graphics and the core
API component the spark.

Equal component is used
to Leverage The Power

of decorative queries

and optimize storage
by executing sql-like queries

on spark data,

which is presented in rdds

and other external sources
spark streaming component

allows developers
to perform batch processing

and streaming of data with ease
in the same application.

The machine learning library
eases the development

and deployment of
scalable machine learning

pipelines Graphics component.

Let's the data scientists work
with graph and non graph sources

to achieve flexibility
and resilience in graph.

Struction and Transformations

and finally the
spark core component.

It is the most vital component
of spark ecosystem,

which is responsible

for basic input output
functions scheduling

and monitoring the entire
spark ecosystem is built on top

of this code execution engine

which has extensible apis
in different languages

like Scala Python and Java
and in today's session,

I will specifically discuss

about the spark API
in Python programming languages,

which is more popularly
known as the pie Spa.

You might be wondering
why pie spot well to get

a better Insight.

Let me give you a brief
into pie spot.

Now as you already know
by spec is the collaboration

of two powerful Technologies,

which are spark which is

an open-source clustering
Computing framework built

around speed ease of use
and streaming analytics.

And the other one is python,
of course python,

which is a general purpose
high-level programming language.

It provides wide range
of libraries and is majorly used

for machine learning
and real-time analytics now,

Now which gives us by spark

which is a python
a pay for spark

that lets you harness
the Simplicity of Python

and The Power of Apache spark.

In order to tame
pick data up ice pack.

Also lets you use
the rdds and come

with a default integration
of Pi Forge a library.

We learn about rdds later
in this video now that you know,

what is pi spark.

Let's now see the advantages
of using spark with python

as we all know python
itself is very simple and easy.

So when Spock is written
in Python it To participate

quite easy to learn
and use moreover.

It's a dynamically type language

which means Oddities can hold
objects of multiple data types.

Not only does it also
makes the EPA simple

and comprehensive and talking

about the readability
of code maintenance

and familiarity with
the python API

for purchase Park is far better

than other programming
languages python also

provides various options
for visualization,

which is not possible using
Scala or Java moreover.

You can conveniently call
are directly from python

on above this python comes

with a wide range of libraries

like numpy pandas
Caitlin Seaborn matplotlib

and these debris is
in data analysis

and also provide mature

and time test statistics
with all these feature.

You can effortlessly program

and spice Park in case
you get stuck somewhere

or habit out.

There is a huge price

but Community out there whom you
can reach out and put your query

and that is very actor.

So I will make good use
of this opportunity to show you

how to install Pi spark

in a system now here
I'm using Red Hat Linux

based sent to a system
the same steps can be applied

for using Linux systems as well.

So in order to install
Pi spark first,

make sure that you have
Hadoop installed in your system.

So if you want to know more
about how to install Ado,

please check out
our new playlist on YouTube

or you can check out our blog on
a direct our website the first

of all you need to go to the
Apache spark official website,

which is parked
at a party Dot o-- r--

g-- and the download section you
can download the latest version

of spark release

which supports It's
the latest version of Hadoop

or Hadoop version
2.7 or above now.

Once you have downloaded it,

all you need to do is
extract it or add say

under the file contents.

And after that you
need to put in the path

where the spark is installed
in the bash RC file.

Now, you also need
to install pip

and jupyter notebook using
the pipe command and make sure

that the version
of piston or above so

as you can see here,

this is what our bash RC file
looks like here you can see

that we have put in the path

for Hadoop spark and as
well as Spunk driver python,

which is The jupyter Notebook.

What we'll do.

Is that the moment you
run the pie Spock shell

it will automatically open
a jupyter notebook for you.

Now.

I find jupyter notebook
very easy to work

with rather than the shell
is supposed to search choice now

that we are done
with the installation path.

Let's now dive deeper
into pie Sparkle on few

of its fundamentals,

which you need to know
in order to work with by Spar.

Now this timeline shows
the various topics,

which we will be covering under
the pie spark fundamentals.

So let's start off.

With the very first
Topic in our list.

That is the spark context.

The spark context is the heart
of any spark application.

It sets up internal services
and establishes a connection

to a spark execution environment
through a spark context object.

You can create rdds accumulators

and broadcast variable
access Park service's run jobs

and much more
the spark context allows

the spark driver application
to access the cluster

through a resource manager,

which can be yarn

or Sparks cluster manager
the driver program then runs.

Operations inside the executors
on the worker nodes

and Spark context uses the pie
for Jay to launch a jvm

which in turn creates
a Java spark context.

Now there are
various parameters,

which can be used
with a spark context object

like the Master app name

spark home the pie
files the environment

in which has set the path size
serializer configuration Gateway

and much more
among these parameters

the master and app name
are the most commonly used now

to give you a basic Insight
on how Spark program works.

I have listed down
its basic lifecycle phases

the typical life cycle
of a spark program includes

creating rdds from
external data sources

or paralyzed a collection
in your driver program.

Then we have the lazy
transformation in a lazily

transforming the base rdds

into new Oddities using
transformation then caching few

of those rdds for future reuse

and finally performing action
to execute parallel computation

and to produce the results.

The next Topic
in our list is added.

And I'm sure people

who have already worked with
spark a familiar with this term,

but for people
who are new to it,

let me just explain it.

No Artie T stands for
resilient distributed data set.

It is considered to be
the building block

of any spark application.

The reason behind this
is these elements run

and operate on multiple nodes
to do parallel processing

on a cluster.

And once you create an RTD,

it becomes immutable
and by imitable,

I mean that it is an object
whose State cannot be modified

after its created,

but we can transform
its values by up.

Applying certain transformation.

They have good
fault tolerance ability

and can automatically recover
for almost any failures.

This adds an added Advantage
not to achieve a certain task

multiple operations can
be applied on these IDs

which are categorized
in two ways the first

in the transformation

and the second one is
the actions the Transformations

are the operations

which are applied on an oddity
to create a new rdd.

Now these transformation work

on the principle
of lazy evaluation

and transformation
are lazy in nature.

Meaning when we call
some operation in our dirty.

It does not execute
immediately spark maintains,

the record of the operations
it is being called

through with the help
of direct acyclic graphs,

which is also known as the DHS

and since the Transformations
are lazy in nature.

So when we execute operation

any time by calling
an action on the data,

the lazy evaluation
data is not loaded

until it's necessary

and the moment we call out
the action all the computations

are performed parallely to give
you the desired output.

Put now a few important
Transformations are

the map flatmap filter

this thing reduced by

key map partition sort by
actions are the operations

which are applied on an rdd
to instruct a party spark

to apply computation

and pass the result back
to the driver few

of these actions include
collect the collectors mapreduce

take first now,

let me Implement few of these
for your better understanding.

So first of all,
let me show you the bash

as if I'll which I
was talking about.

So here you can see
in the bash RC file.

We provide the path
for all the Frameworks

which we have installed
in the system.

So for example,
you can see here.

We have installed
Hadoop the moment we

install and unzip it
or rather see entire it

I have shifted all my Frameworks
to one particular location

as you can see is
the US are the user

and inside this we have
the library and inside

that I have installed the Hadoop
and also the spa now

as you can see here,

we have two lines.

I'll highlight this one for
you the pie spark driver.

Titan which is the Jupiter

and we have given it as
a notebook the option available

as know to what we'll do
is at the moment.

I start spark will
automatically redirect me

to The jupyter Notebook.

So let me just rename
this notebook as rdd tutorial.

So let's get started.

So here to load any file
into an rdd suppose.

I'm loading a text file
you need to use the S

if it is a spark context
as C dot txt file

and you need to provide
the path of the data

which you are going to load.

So one thing to keep
in mind is that the default path

which the artery takes
or the jupyter.

Notebook takes is the hdfs path.

So in order to use
the local file system,

you need to mention
the file colon

and double forward slashes.

So once our sample data is

inside the ret not to
have a look at it.

We need to invoke
using it the action.

So let's go ahead and take
a look at the first five objects

or rather say the first five
elements of this particular rdt.

The sample it I have taken
here is about blockchain

as you can see.

We have one two,

three four and
five elements here.

Suppose I need to convert
all the data into a lowercase

and split it according
to word by word.

So for that I will
create a function

and in the function
I'll pass on this Oddity.

So I'm creating
as you can see here.

I'm creating rdd one

that is a new ID
and using the map function

or rather say the transformation
and passing on the function,

which I just created to lower
and to split it.

So if we have a look
at the output of our D1

As you can see here,

all the words are
in the lower case

and all of them are separated
with the help of a space bar.

Now this another transformation,

which is known as the flat map
to give you a flat and output

and I am passing
the same function

which I created earlier.

So let's go ahead

and have a look
at the output for this one.

So as you can see here,

we got the first five elements

which are the save one as we got
here the contrast transactions

and and the records.

So just one thing
to keep in mind.

Is at the flat map
is a transformation

where as take is the action now,

as you can see
that the contents

of the sample data
contains stop words.

So if I want to remove
all the stop was all you

need to do is start

and create a list of stop words
in which I have mentioned here

as you can see.

We have a all the as is

and now these are
not all the stop words.

So I've chosen only a few
of them just to show you

what exactly the output will be

and now we are using here
the filter transformation

and with the help of Lambda.

Function and which we have
X specified as X naught

in stock quotes and we
have created another rdd

which is added III

which will take the input

from our DD to so
let's go ahead and see

whether and and the
are removed or not.

This is you can see contracts
transaction records of them.

If you look at the output 5,

we have contracts transaction
and and the and in the

are not in this list,

but suppose I want
to group the data

according to the first
three characters of any element.

So for that I'll use

the group by and I'll use
the Lambda function again.

So let's have a look
at the output

so you can see we
have EDG and edges.

So the first three letters of
both words are same similarly.

We can find it using
the first two letters.

Also, let me just change it
to two so you can see we are gu

and guid just a guide

not these are
the basic Transformations

and actions but suppose.

I want to find out the sum
of the first thousand numbers.

Others have first
10,000 numbers.

All I need to do
is initialize another Oddity,

which is the
number underscore ID.

And we use the AC Dot
parallelized and the range

we have given is one to 10,000

and we'll use the reduce action

here to see the output
you can see here.

We have the sum
of the numbers ranging

from one to ten thousand.

Now this was all about rdd.

The next topic

that we have
on a list is broadcast

and accumulators now in spark
we perform parallel processing

through the Help
of shared variables

or when the driver sends
any tasks with the executor

present on the cluster a copy of
the shared variable is also sent

to the each node

of the cluster thus
maintaining High availability

and fault tolerance.

Now, this is done in order
to accomplish the task

and Apache spark supposed
to type of shared variables.

One of them is broadcast.

And the other one is
the accumulator now

broadcast variables are used
to save the copy of data

on all the nodes in a cluster.

Whereas the accumulator is
the variable that is used

for aggregating the incoming.

Information we are
different associative

and commutative operations now
moving on to our next topic

which is a spark configuration
the spark configuration class

provides a set
of configurations and parameters

that are needed to execute
a spark application

on the local system
or any cluster.

Now when you use
spark configuration object

to set the values
to these parameters,

they automatically take priority
over the system properties.

Now this class
contains various Getters

and Setters methods some
of which are Set method

which is used to set
a configuration property.

We have the set master

which is used for setting
the master URL.

Yeah the set app name,

which is used to set
an application name and we

have the get method to retrieve
a configuration value of a key.

And finally we
have set spark home

which is used for setting
the spark installation path

on worker nodes.

Now coming to the next
topic on our list

which is a spark files
the spark file class

contains only the class methods

so that the user cannot create
any spark files instance.

Now this helps in Dissolving
the path of the files

that are added using
the spark context add

file method the class Park files
contain to class methods

which are the get method and
the get root directory method.

Now, the get is used
to retrieve the absolute path

of a file added through
spark context to add file

and the get root directory

is used to retrieve
the root directory

that contains the files
that are added.

So this park context
dot add file.

Now, these are smart topics
and the next topic

that we will covering

in our list are the data frames
now data frames in a party.

Spark is a distributed
collection of rows

under named columns,

which is similar to
the relational database tables

or Excel sheets.

It also shares common attributes

with the rdds few
characteristics of data frames

are immutable in nature.

That is the same
as you can create a data frame,

but you cannot change it.

It allows lazy evaluation.

That is the task not executed

unless and until
an action is triggered

and moreover data frames
are distributed in nature,

which are designed
for processing large

collection of structure
or semi-structured data.

Can be created using
different data formats,

like loading the data

from source files
such as Json or CSV,

or you can load it
from an existing re

you can use databases
like hi Cassandra.

You can use pocket files.

You can use CSV XML files.

There are many sources

through which you can create
a particular R DT now,

let me show you how to create
a data frame in pie spark

and perform various actions
and Transformations on it.

So let's continue this
in the same notebook

which we have here now
here we have taken

In the NYC Flight data,

and I'm creating a data frame
which is the NYC flights

on the score

TF now to load the data.

We are using the spark dot
RI dot CSV method and you

to provide the path

which is the local path
of by default.

It takes the hdfs same as our GD

and one thing
to note down here is

that I've provided
two parameters extra here,

which is the info schema
and the header

if we do not provide
this as true of a skip it

what will happen.

Is that if your data set Is
the name of the columns

on the first row it will take
those as data as well.

It will not infer
the schema now.

Once we have loaded the data
in our data frame we need to use

the show action to have
a look at the output.

So as you can see here,

we have the output
which is exactly it

gives us the top 20 rows
or the particular data set.

We have the year month day
departure time deposit delay

arrival time arrival delay
and so many more attributes.

To print the schema

of the particular data frame
you need the transformation

or as say the action
of print schema.

So let's have a look
at the schema.

As you can see here we have here
which is integer month integer.

Almost half of them are integer.

We have the carrier as
string the tail number

a string the origin
string destination string

and so on now suppose.

I want to know

how many records are
there in my database

or the data frame rather say

so you need the count
function for this one.

I will provide but the results
so as you can see here,

we have three point
three million records

here three million

thirty six thousand
seven hundred seventy six

to be exact now suppose.

I want to have a look
at the flight name the origin

and the destination

of just these three columns
from the particular data frame.

We need to use
the select option.

So as you can see here,
we have the top 20 rows.

Now, what we saw
was the select query

on this particular data frame,

but if I wanted
to see or rather,

I want to check the summary.

Of any particular
column suppose.

I want to check the
what is the lowest count

or the highest count in
the particular distance column.

I need to use
the describe function here.

So I'll show you what
the summer it looks like.

So the distance the count

is the number of rows
total number of rows.

We have the mean the standard
deviation via the minimum value,

which is 17
and the maximum value,

which is 4983.

Now this gives you a summary
of the particular column

if you want to So
that we know

that the minimum distance is 70.

Let's go ahead and filter
out our data using

the filter function
in which the distance is 17.

So you can see here.

We have one data

in which in the 2013 year
the minimum distance here is 17

but similarly suppose I want
to have a look at the flash

which are originating from EWR.

Similarly.

We use the filter
function here as well.

Now the another Clause here,

which is the where
Clause is also used

for filtering the suppose.

I want to have a look

at the flight data
and filter it out to see

if the day at work.

Which the flight took off was
the second of any month suppose.

So here instead of filter.

We can also use a where clause

which will give us
the same output.

Now, we can also pass
on multiple parameters

and rather say
the multiple conditions.

So suppose I want the day
of the flight should be seventh

and the origin should be JFK

and the arrival delay
should be less than 0 I mean

that is for none
of the postponed fly.

So just to have a look

at these numbers
will use the way clause

and separate all the conditions
using the + symbol

so you can see
here all the data.

The day is 7 the origin is JFK

and the arrival delay
is less than 0 now.

These were the basic
Transformations and actions

on the particular data frame.

Now one thing we can also do
is create a temporary table

for SQL queries

if someone is

not good or is not Wanted
to all these transformation

and action add would rather
use SQL queries on the data.

They can use this register dot
temp table to create a table

for their particular data frame.

What we'll do is
convert the NYC flights

and a Squatty of data frame
into NYC endoscope flight table,

which can be used later
and SQL queries can be performed

on this particular table.

So you remember in the beginning
we use the NYC flies and score d

f dot show now we can use
the select asterisk from I

am just go flights to get
the same output now suppose

we want to look at the minimum
a time of any flights.

We use the select minimum
air time from NYC flights.

That is the SQL query.

We pass all the SQL query
in the sequel context

or SQL function.

So you can see here.

We have the minimum air time
as 20 now to have a look

at the Wreckers in which
the air time is minimum 20.

Now we can also use
nested SQL queries a suppose

if I want to check

which all flights have
the Minimum air time as 20 now

that cannot be done

in a simple SQL query we need
nested query for that one.

So selecting aspects
from New York flights

where the airtime
is in and inside

that we have another query,

which is Select minimum air time
from NYC flights.

Let's see if this works or not.

CS as you can see here,

we have two Flats which have
the minimum air time as 20.

So guys this is it
for data frames.

So let's get back
to our presentation

and have a look at the list
which we were following.

We completed data frames.

Next we have stories levels
now Storage level

in pie spark is a class
which helps in deciding

how the rdds should be stored

now based on this rdds
are either stored

in this or in memory or in

both the class Storage
level also decides

whether the RADS
should be serialized

or replicate its partition
for the final

and the last topic
for the today's list

is MLM blog MLM is
the machine learning APA

which is provided by spark,

which is also present in Python.

And this library
is heavily used in Python

for machine learning as

well as real-time streaming
analytics Aurelius algorithm

supported by this libraries
are first of all,

we have the spark dot m l live

now recently the spice
Park MN lips supports model

based collaborative filtering
by a small set of latent factors

and here all the users
and the products are described

which we can use
to predict them.

Missing entries however
to learn these latent factors

Park dot ml abuses
the alternatingly square

which is the ALS algorithm.

Next we have the MLF clustering

and are supervised

learning problem is clustering
now here we try to group subsets

of entities with one
another on the basis

of some notion of similarity.

Next.

We have the frequent
pattern matching,

which is the fpm now frequent
pattern matching is mining

frequent items item set
subsequences or other Lectures

that are usually

among the first steps to analyze
a large-scale data set.

This has been an active research
topic in data mining for years.

We have the linear algebra.

Now this algorithm
support spice Park,

I mean live utilities
for linear algebra.

We have collaborative filtering.

We have classification

for binary classification
various methods are available

in sparked MLA package such as
multi-class classification as

well as regression analysis
in classification some

of the most popular Terms
used are Nave by a strand

of forest decision tree

and so much and finally we
have the linear regression

now basically lead integration
comes from the family

of recreation algorithms
to find relationships

and dependencies between
variables is the main goal

of regression all the pie spark

MLA package also covers
other algorithm classes

and functions.

Let's now try to implement
all the concepts

which we have learned
in pie spark tutorial session

now here we are going to use
a heart disease prediction model

and we are going to predict
Using the decision tree

with the help of classification
as well as regression.

Now.

These all are part
of the ml Live library here.

Let's see how we
can perform these types

of functions and queries.

The first of all

what we need to do
is initialize the spark context.

Next we are going
to read the UCI data set

of the heart disease prediction

and we are going
to clean the data.

So let's import the pandas
and the numpy library here.

Let's create a data frame
as heart disease TF and

as mentioned earlier,

we are going to use
the read CSV method here

and here we don't have a header.

So we have provided
header as none.

Now the original data set
contains 300 3 rows

and 14 columns.

Now the categories
of diagnosis of heart disease

that we are projecting

if the value 0 is for 50% less
than narrowing and for the value

1 which we are giving
is for the values

which have 50% more
diameter of naren.

So here we are using
the numpy library.

These are particularly
old methods which is showing

the deprecated warning
but no issues it will work fine.

So as you can see here,

we have the categories
of diagnosis of heart disease

that we are predicting
the value 0 is 4 less than 50

and value 1 is greater than 50.

So what we did here
was clear the row

which have the question mark
or which have the empty spaces.

Now to get a look
at the data set here.

Now, you can see here.

We have zero at many places
instead of the question mark

which we had earlier

and now we are saving
it to a txt file.

And you can see her
after dropping the rose

with any empty values.

We have two ninety seven rows
and 14 columns.

But this is what the new
clear data set looks

like now we are importing
the ml lived library

and the regression here now here

what we are going to do
is create a label point,

which is a local Vector
associated with a label

or a response.

So for that we need to import
the MLF dot regression.

So for that we are
taking the text file

which we just created now
without the missing values.

Now next.

What we are going to do is
pass the MLA data line by line

into the MLM label Point object

and we are going
to convert the -

one labels to the 0 now.

Let's have a look after passing
the number of fishing lines.

Okay, we have to label .01.

That's cool.

Now next what we are going to do
is perform classification using

the decision tree.

So for that we need to import
the pie spark the ml 8.3.

So next what we have to do is
split the data into the training

and testing data

and we split here the data
into 70s 233 standard ratio,

70 being the training data set

and the 30% being the testing
data set next what we do is

that we train the model.

Which we are created here
using the training set.

We have created
a training model decision trees

or trained classifier.

We have used
a training data number

of classes is file
the categorical feature,

which we have given

maximum depth to which
we are classifying.

It is 3 the next

what we are going to do is
evaluate the model based

on the test data set now
and evaluate the error.

So here we are creating

predictions and we
are using the test data

to get the predictions
through the model

which we Do and we
are also going to find

the test errors here.

So as you can see here,

the test error is
zero point 2 2 9 7 we

have created a classification
decision tree model

in which the feature
less than 12 is 3 the value

of the features
distance 0 is 54.

So as you can see
our model is pretty good.

So now next we'll use regression
for the same purposes.

So let's perform the regression
using decision tree.

So as you can see
we have the train model

and we are using
the decision tree, too.

Trine request using
the training data the same

which we created using the
decision tree model up there.

We use the classification

now we are using
regression now similarly.

We are going to evaluate

our model using our test data
set and find that test errors

which is the mean squared error
here for aggression.

So let's have a look
at the mean square error here.

The mean square error is 0.168.

That is good.

Now finally if we have

a look at the Learned
regression tree model.

You can see we have created
the regression tree model

till the depth
of 3 with 15 notes.

And here we have
all the features

and classification of the tree.

Hello folks.

Welcome to spawn
interview questions.

The session has been planned
collectively to have commonly

asked interview questions later
to the smart technology

and the general answer
and the expectation

is already you are aware
of this particular technology.

To some extent and in general
the common questions being asked

as well as I will give
interaction with the technology

as so let's get this started.

So the agenda for
this particular session is

the basic questions
are going to cover

and questions later
to the spark core Technologies.

That's when I say spark

or that's going to be
the base and top

of spark or we have
four important components

which work that
is streaming Graphics.

Ml Abe and SQL
all these components

have been created to satisfy a
The government again interaction

with these Technologies and get

into the commonly
asked interview questions

and the questions also
framed such a way.

It covers the spectrum
of the doubts as well

as the features available
within that specific technology.

So let's take the first question

and look into the answer like
how commonly this covered.

What is Apache spark and Spark
It's with Apache Foundation now,

it's open source.

It's a cluster
Computing framework

for real-time processing.

So three main keywords over.

Here a purchase markets
are open source project.

It's used for cluster Computing.

And for a memory processing
along with real-time processing.

It's going to support
in memory Computing.

So the lots of project

which supports cluster Computing

along with that spark
differentiate Itself by doing

the in-memory Computing.

It's very active
community and out

of the Hadoop ecosystem
technology is Apache spark is

very active multiple releases.

We got last year.

It's a very inactive project
among the about your Basically,

it's a framework kind support
in memory Computing

and cluster Computing and you
may face this specific question

how spark is different

than mapreduce on
how you can compare it

with the mapreduce mapreduce
is the processing pathology

within the Hadoop ecosystem

and within Hadoop ecosystem.

We have hdfs Hadoop distributed
file system mapreduce going

to support distributed computing
and how spark is different.

So how we can compare
smart with them.

Mapreduce in a way
this comparison going

to help us to understand
the technology better.

But definitely

like we cannot compare these two
or two different methodologies

by which it's going to work
spark is very simple to program

but mapreduce there
is no abstraction

or the sense like all

the implementations we have
to provide and interactivity.

It's has an interactive mode to
work with inspark a mapreduce.

That is no interactive mode.

There are some
components like Apache.

Big and high

which facilitates has to do
the interactive Computing

or interactive programming

and smog supports
real-time stream processing

and to precisely
say with inspark

the stream processing is called
a near real-time processing.

There's nothing in the world
is Real Time processing.

It's near real-time processing.

It's going to do the processing
and micro batches.

I'll cover in detail

when we are moving
onto the streaming concept

and you're going to do

the batch processing on
the historical data in Matrix.

Zeus when I say stream
processing I will get the data

that is getting processed
in real time and do

the processing and get
the result either store it

on publish to publish Community.

We will be doing it let and see

wise mapreduce will have
very high latency

because it has to read
the data from hard disk,

but spark it will have
very low latency

because it can reprocess

are used the data
already cased in memory,

but there is a small catch
over here in spark first time

when the data gets loaded it
has Tool to read it

from the hard disk
same as mapreduce.

So once it is red it
will be there in the memory.

So spark is good.

Whenever we need to do I treat

a Computing so spark whenever
you do I treat a Computing again

and again to the processing
on the same data,

especially in machine learning
deep learning all we will be

using the iterative Computing
his Fox performs much better.

You will see
the rock performance

Improvement hundred times
faster than mapreduce.

But if it is one time processing
and fire-and-forget,

Get the type
of processing spark lately,

maybe the same latency,

you will be getting
a tan mapreduce maybe

like some improvements because
of the building block or spark.

That's the ID you may get
some additional Advantage.

So that's the key feature are
the key comparison factor

of sparkin mapreduce.

Now, let's get on to the key
features xnk features of spark.

We discussed over
the Speed and Performance.

It's going to use
the in-memory Computing

so Speed and Performance.

Place it's going to much better.

When we do actually to Computing
and Somali got the sense

the programming language
to be used with a spark.

It can be any of these languages
can be python.

Java are our scale.

Mm.

We can do programming
with any of these languages

and data formats
to give us a input.

We can give any data formats
like Jason back

with a data formats began

if there is a input
and the key selling point

with the spark is it's
lazy evaluation the

since it's going

To calculate the DAC cycle
directed acyclic graph

d a g because that is a th e
it's going to calculate

what all steps needs
to be executed to achieve

the final result.

So we need to give all
the steps as well as

what final result I want.

It's going to calculate
the optimal cycle

on optimal calculation.

What else tips needs
to be calculated

or what else tips needs
to be executed only those steps

it will be executing it.

So basically it's
a lazy execution only

if the results needs
to be processed,

it will be processing that.

Because of it and it's
about real-time Computing.

It's through spark streaming

that is a component
called spark streaming

which supports real-time
Computing and it gels

with Hadoop ecosystem variable.

It can run on top of Hadoop Ian

or it can Leverage The hdfs
to do the processing.

So when it leverages the hdfs
the Hadoop cluster container

can be used to do
the distributed computing

as well as it can leverage
the resource manager to manage

the resources so spot.

I can gel with the hdfs very
well as well as it can leverage

the resource manager

to share the resources
as well as data locality.

You can give each data locality.

It can do the processing we have

to the database data is located
within the hdfs and has a fleet

of machine learning
algorithms already implemented

right from clustering
classification regression.

All this logic
already implemented

and machine learning.

It's achieved using
MLA be within spark

and there is a component
called a graphics

which supports Maybe we
can solve the problems using

graph Theory using the component
Graphics within this park.

So these are the things
we can consider as

the key features of spark.

So when you discuss
with the installation

of the spark,

you may come across this year
on what is he on do you

need to install spark
on all nodes of young cluster?

So yarn is nothing
but another is US negotiator.

That's the resource manager
within the Hadoop ecosystem.

So that's going to provide the
resource management platform.

Ian going to provide
the resource management platform

across all the Clusters

and Spark It's going
to provide the data processing.

So wherever there is
a horse being used

that location response will be
used to do the data processing.

And of course, yes,

we need to have spark
installed on all the nodes.

It's Parker stores are located.

That's basically we need
those libraries an additional

to the installation of spark
and all the worker nodes.

We need to increase
the ram capacity

on the VOC emissions

as well as far going
to consume huge amounts.

Memory to do the processing it
will not do the mapreduce way

of working internally.

It's going to generate
the next cycle and do

the processing on top of yeah,

so Ian and the high level it's
like resource manager

or like an operating system
for the distributed computing.

It's going to coordinate
all the resource management

across the fleet
of servers on top of it.

I can have multiple components

like spark these giraffe
this park especially it's going

to help Just watch it
in memory Computing.

So sparkly on is nothing
but it's a resource manager

to manage the resource
across the cluster on top of it.

We can have spunk and yes,

we need to have spark installed

and all the notes on where
the spark yarn cluster is used

and also additional to that.

We need to have
the memory increased

in all the worker robots.

The next question goes

like this what file
system response support.

What is the file system then
we work in individual system.

We will be having
a file system to work

within that particular
operating system Mary

redistributed cluster or in
the distributed architecture.

We need a file system with which

where we can store the data
in a distribute mechanism.

How do comes with
the file system called hdfs.

It's called Hadoop
distributed file system

by data gets distributed
across multiple systems

and it will be coordinated by 2.

Different type of components
called name node and data node

and Spark it can use
this hdfs directly.

So you can have any files
in hdfs and start using it

within the spark ecosystem
and it gives another advantage

of data locality

when it does the distributed
processing wherever

the data is distributed.

The processing could be done
locally to that particular

Mission way data is located

and to start with as
a standalone mode.

You can use the local
file system aspect.

So this could be used especially

when we are doing
the development or any

of you see you can use
the local file system

and Amazon Cloud provides
another file system called.

Yes, three simple
storage service we call

that is the S3.

It's a block storage service.

This can also be leveraged

or used within spa
for the storage

and lot other file system.

Also, it supports there are
some file systems like Alex,

oh which provides
in memory storage

so we can leverage that
particular file system as well.

So we have seen
all the features.

What are the functionalities
available with inspark?

We're going to look
at the limitations

of using spark.

Of course every component

when it comes with
a huge power and Advantage.

It will have its own
limitations as well.

The equation illustrates
some limitations of using

spark spark utilizes
more storage space

compared to Hadoop

and it comes
to the installation.

It's going to consume more space
but in the Big Data world,

that's not a
very huge constraint

because storage cons is
not Great are very high

and our big data space and
developer needs to be careful

while running the apps
and Spark the reason

because it uses
in-memory Computing.

Of course, it handles
the memory very well.

But if you try to load
a huge amount of data

and the distributed environment
and if you try to do is join

when you try to do join

within the distributed world the
data going to get transferred

over the network network

is really a costly
resource So the plan

or design should be such
a way to reduce or minimize.

As the data transferred
over the network

and however the way
possible with all possible means

we should facilitate
distribution of theta

over multiple missions the more

we distribute the more
parallelism we can achieve

and the more results we can get
and cost efficiency.

If you try to compare the cost

how much cost involved

to do a particular
processing take any unit

in terms of processing
1 GB of data with say

like II Treaty processing

if you come Cost-wise in-memory
Computing always it's considered

because memory It's
relatively come costlier

than the storage

so that may act
like a bottleneck

and we cannot increase

the memory capacity of
the mission Beyond supplement.

So we have to grow horizontally.

So when we have
the data distributor

in memory across the cluster,

of course the network transfer

all those bottlenecks
will come into picture.

So we have to strike
the right balance

which will help us to achieve
the in-memory computing.

Whatever, they memory
computer repair it

will help us to achieve

and it consumes huge amount

of data processing
compared to Hadoop

and Spark it performs

better than use it as
a creative Computing

because it likes for both spark
and the other Technologies.

It has to read data

for the first time
from the hottest car

from other data source and Spark
performance is really better

when it reads the data
onto does the processing

when the data is available
in the cache,

of course is the DAC cycle.

It's going to give
us a lot of advantage

while doing the processing

but the in-memory
Computing processing

that's going to give
us lots of Leverage.

The next question
list some use cases

where Spark outperforms
Hadoop in processing.

The first thing is
the real time processing.

How do you cannot handle
real time processing

but spark and handle
real time processing.

So any data that's coming in
in the land architecture.

You will have three layers.

The most of the Big
Data projects will be

in the Lambda architecture.

You will have speed layer
by layer and sighs Leo

and the speed layer
whenever the river comes

in that needs to be processed
stored and handled.

So in those type

of real-time processing stock
is the best fit.

Of course, we can
Hadoop ecosystem.

We have other components

which does the real-time
processing like storm.

But when you want to Leverage
The Machine learning

along with the Sparks dreaming

on such computation spark
will be much better.

So that's why I like

when you have architecture

like a Lambda architecture
you want to have

all three layers bachelier
speed layer and service.

A spark and gel the speed layer
and service layer far better

and it's going to provide
better performance.

And whenever you do
the edge processing

especially like doing
a machine learning processing,

we will leverage
nitrate in Computing

and can perform a hundred times

faster than Hadoop
the more diversity processing

that we do the more data
will be read from the memory

and it's going to get as
much faster performance

than I did with mapreduce.

So again, remember whenever you
do the processing only buns,

so you're going to to do
the processing finally bonds

read process it and deliver.

The result spark
may not be the best fit

that can be done
with a mapreduce itself.

And there is another component
called akka it's

a messaging system
our message quantity

in system Sparkle
internally uses account

for scheduling our any task

that needs to be assigned
by the master to the worker

and the follow-up
of that particular task

by the master basically
asynchronous coordination system

and that's achieved using akka

I call programming internally
it's used by this monk

as such for the developers.

We don't need to worry
about a couple of growing up.

Of course we can leverage it

but the car is used internally
by the spawn for scheduling

and coordination between master
and the burqa and with inspark.

We have few major components.

Let's see, what are
the major components

of a possessed man.

The lay the components
of spot ecosystem start comes

with a core engine.

So that has the core.

Realities of what is required
from by the spark

of all this Punk Oddities
are the building blocks

of the spark core engine
on top of spark

or the basic functionalities are

file interaction file system
coordination all that's done

by the spark core engine

on top of spark core engine.

We have a number
of other offerings

to do machine learning to do
graph Computing to do streaming.

We have n number
of other components.

So the major use the components

of these components
like Sparks equal.

Spock streaming.

I'm a little graphics
and Spark our other high level.

We will see what are
these components Sparks

equal especially it's designed
to do the processing

against a structure data

so we can write SQL queries
and we can handle

or we can do the processing.

So it's going to give us
the interface to interact

with the data,

especially structure data
and other language

that we can use
it's more similar to

what we use within the SQL.

Well, I can say
99 percentage is seen

and most of the commonly used
functionalities within the SQL

have been implemented
within smocks equal

and Spark streaming is going to
support the stream processing.

That's the offering
available to handle

the stream processing and MLA

based the offering
to handle machine learning.

So the component name
is called ml in and has a list

of components a list

of machine learning
algorithms already defined

we can leverage and use any
of those machine learning.

Graphics again, it's
a graph processing offerings

within the spark.

It's going to support us
to achieve graph Computing

against the data that we have
like pagerank calculation.

How many connector identities

how many triangles all those
going to provide us a meaning

to that particular data

and Spark are is the component
is going to interact

or helpers to leverage.

The language are
within the spark environment

are is a statistical
programming language.

Each where we can do
statistical Computing,

which is Park environment

and we can leverage our language
by using this parka to get

that executed within the spark
a environment addition to that.

There are other components
as well like approximative is

it's called blink DB all other
things I can be test each.

So these are the major Lee used
components within spark.

So next question.

How can start be used
alongside her too?

So when we see a spark
performance much better it's

not a replacement to handle it.

Going to coexist

with the Hadoop right
Square leveraging the spark

and Hadoop together.

It's going to help us
to achieve the best result.

Yes.

Mark can do in memory Computing
or can handle the speed layer

and Hadoop comes
with the resource manager

so we can leverage
the resource manager

of Hadoop to make smart to work

and few processing be
don't need to Leverage

The in-memory Computing.

For example, one time processing
to the processing and forget.

I just store it we
can use mapreduce.

He's so the processing cost
Computing cost will be much less

compared to Spa

so we can amalgam eyes and get
strike the right balance

between the batch processing
and stream processing

when we have spark
along with Adam.

Let's have some detail question
later to spark core

with inspark or

as I mentioned earlier
the core building block

of spark or is our DD resilient
distributed data set.

It's a virtual.

It's not a physical entity.

It's a logical entity.

You will not See
this audit is existing.

The existence of hundred
will come into picture

when you take some action.

So this is our Unity
will be used are referred

to create the DAC cycle

and arteries will be optimized
to transform from one form

to another form to make a plan

how the data set needs
to be transformed

from one structure
to another structure.

And finally when you take some
against an RTD that existence

of the data structure

that resulted in data
will come into picture

and that can be stored
in any file system

whether it's GFS is 3

or any other file system
can be stored and

that it is can exist
in a partition form the sense.

It can get distributed
across multiple systems

and it's fault tolerant
and it's a fault tolerant.

If any of the artery
is lost any partition

of the RTD is lost.

It can regenerate only
that specific partition

it can regenerate

so that's a huge
advantage of our GD.

So it's a mass like first
the huge advantage of added.

It's a fault-tolerant

where it can regenerate
the last rdds.

And it can exist
in a distributed fashion

and it is immutable the

since once the RTD is defined on
like it it cannot be changed.

The next question is
how do we create rdds

in spark the two ways we
can create The Oddities one

as isn't the spark context we
can use any of the collections

that's available within this
scalar or in the Java and using

the paralyzed function.

We can create the RTD
and it's going to use

the underlying file
systems distribution mechanism

if The data is located
in distributed file system,

like hdfs.

It will leverage
that and it will make

those arteries available
in a number of systems.

So it's going to leverage
and follow the same distribution

and already Aspen

or we can create the rdt
by loading the data

from external sources
as well like its peace

and hdfs be may not consider
as an external Source.

It will be consider as
a file system of Hadoop.

So when Spock is working

with Hadoop mostly
the file system,

we will be using will be Hdfs,

if you can read
from it each piece

or even we can do
from other sources,

like Parkwood file or has
three different sources a roux.

You can read and create the RTD.

Next question is

what is executed memory
in spark application.

Every Spark application
will have fixed.

It keeps eyes and fixed number,

of course for the spark
executor executor is nothing

but the execution unit
available in every machine

and that's going to facilitate
to do the processing to do

the tasks in the Water machine,

so irrespective of whether you
use yarn resource manager

or any other measures

like resource manager
every worker Mission.

We will have an Executor

and within the executor
the task will be handled

and the memory to be allocated
for that particular executor is

what we Define as the hip size
and we can Define

how much amount

of memory should be used
for that particular executor

within the worker
machine as well.

As number of cores
can be used within the exit.

Our by the executor
with this path application

and that can be controlled

through the configuration
files of spark.

Next questions different
partitions in Apache spark.

So any data irrespective of

whether it is a small
data a large data,

we can divide those data sets

across multiple systems
the process of dividing the data

into multiple pieces

and making it to store

across multiple systems as
a different logical units.

It's called partitioning.

So in simple terms partitioning
is nothing but the process

of Dividing the data

and storing in multiple systems
is called partitions

and by default the conversion

of the data into R. TD
will happen in the system

where the partition is existing.

So the more the partition
the more parallelism

they are going to get
at the same time.

We have to be careful
not to trigger huge amount

of network data transfer as well

and every a DD can
be partitioned with inspark

and the panel
is the partitioning

going to help us to achieve
parallelism more the partition

that we have more.

Solutions can be done

and that the key thing
about the success

of the spark program is
minimizing the network traffic

while doing the parallel
processing and minimizing

the data transfer
within the systems of spark.

What operations does already
support so I can operate

multiple operations
against our GD.

So there are two type of things
we can do we can group it

into two one is transformations

in Transformations are did he
will get transformed

from one form to another form.

Select filtering grouping all

that like it's going
to get transformed

from one form to another form
one small example,

like reduced by key filter all
that will be Transformations.

The resultant of
the transformation will be

another rdd the same time.

We can take some actions
against the rdd

that's going to give
us the final result.

I can say count

how many records
or they are store

that result into the hdfs.

They all our actions so
multiple actions can be taken

against the RTD.

The existence of the data
will come into picture only

if I take some action
against not ready.

Okay.

Next question.

What do you understand
by transformations in spark?

So Transformations are
nothing but functions

mostly it will be higher
order functions within scale

and we have something
like a higher order functions

which will be applied
against the tardy.

Mostly against the list

of elements that we
have within the rdd

that function will get
applied by the existence

of the arditi will Come
into picture one lie

if we take some action against
it in this particular example,

I am reading the file

and having it within the rdd
Control Data then I am doing

some transformation using a map.

So it's going
to apply a function

so we can map I have

some function which will split
each record using the tab.

So the spit with the app
will be applied

against each record
within the raw data

and the resultant movies data
will again be another rdd,

but of course,
this will be a lazy operation.

The existence of movies data
will come into picture only

if I take some action
against it like count or print

or store only those actions
will generate the data.

So next question
Define functions of spark code.

So that's going to take care
of the memory management

and fault tolerance of rdds.

It's going to help us
to schedule distribute the task

and manage the jobs running
within the cluster

and so we're going to help
us to or store the rear

in the storage system as well
as reads data from the storage.

System that's to do the file
system level operations.

It's going to help us

and Spark core programming
can be done in any

of these languages
like Java scalar python

as well as using our so core is

that the horizontal level
on top of spark or we can have

a number of components

and there are different type
of rdds available one such

a special type is parody.

So next question.

What do you understand
by pay an rdd?

It's going to exist
in peace as a keys and values

so I can Some special functions

within the parodies
are special Transformations,

like connect all the values
corresponding to the same key

like solder Shuffle

what happens within
the shortened Shuffle of Hadoop

those type of operations

like you want

to consolidate our group
all the values corresponding

to the same key are
apply some functions

against all the values
corresponding to the same key.

Like I want to get the sum

of the value of all the keys
we can use the parody.

D and get that a cheat so
it's going to the data

within the re going to exist
in Pace keys and right.

Okay a question from Jason.

What are our Vector rdds

in machine learning you
will have huge amount

of processing handled by vectors

and matrices and we do lots
of operations Vector operations,

like effective actor

or transforming any data
into a vector form so vectors

like as the normal way
it will have a Direction.

And magnitude

so we can do some operations
like some two vectors

and what is the difference
between the vector A

and B as well as a and see

if the difference
between Vector A

and B is less compared to a

and C we can say the vector A

and B is somewhat similar
in terms of features.

So the vector R GD
will be used to represent

the vector directly and
that will be used extensively

while doing the
measuring and Jason.

Thank you other.

Is another question.

What is our GD lineage?

So here I any data
processing any Transformations

that we do it maintains
something called a lineage.

So what how data
is getting transformed

when the data is available
in the partition form

in multiple systems and
when we do the transformation,

it will undergo multiple steps
and in the distributed word.

It's very common to have
failures of machines

or machines going
out of the network

and the system our framework

as it should be

in a position to handle
small handles it through.

Did he leave eh it can restore
the last partition only assume

like out of ten machines
data is distributed

across five machines out of

that those five machines
One mission is lost.

So whatever the
latest transformation

that had the data

for that particular
partition the partition

in the last mission alone
can be regenerated and it knows

how to regenerate that data
on how to get that result

and data using the concept

of rdd lineage so
from which Each data source,

it got generated.

What was its previous step.

So the completely
is will be available

and it's maintained by
the spark framework internally.

We call that as Oddities in eh,

what is point driver to put
it simply for those

who are from her
do background yarn back room.

We can compare this
to at muster.

Every application will
have a spark driver

that will have a spot context

which is going to moderate
the complete execution

of the job that will connect
to the spark master.

Delivers the RTD graph

that is the lineage
for the master

and the coordinate the tasks.

What are the tasks

that gets executed
in the distributed environment?

It can do

the parallel processing
do the Transformations

and actions against the RTD.

So it's a single
point of contact

for that specific application.

So smart driver
is a short linked

and the spawn context
within this part driver

is going to be the coordinator
between the master and the tasks

that are running
and smart driver.

I can get started
in any of the executor

with inspark name types
of custom managers in spark.

So whenever you have
a group of machines,

you need a manager to manage

the resources the different type
of the store manager already.

We have seen the yarn
yet another assist ago.

She later which manages
the resources of Hadoop on top

of yarn we can make
Spock to book sometimes I

may want to have sparkle
own my organization

and not along with the Hadoop
or any other technology.

Then I can go

with the And alone spawn
has built-in cluster manager.

So only spawn can get
executed multiple systems.

But generally if we
have a cluster we

will try to leverage

various other Computing
platforms Computing Frameworks,

like graph processing
giraffe these on that.

We will try to
leverage that case.

We will go with yarn

or some generalized
resource manager,

like masseuse Ian.

It's very specific to Hadoop
and it comes along

with Hadoop measures is the
cluster level resource manager

and I have multiple clusters.

Within organization,
then you can use mrs.

Mrs. Is also a resource manager.

It's a separate table project
within Apache X question.

What do you understand

by worker node in a cluster
redistribute environment.

We will have n number
of workers we call

that is a worker node
or a slave node,

which does the actual
processing going to get

the data do the processing
and get us the result

and masternode going to assign

what has to be done by
one person own and it's going

to read the data available
in the specific work on.

Generally, the tasks assigned
to the worker node,

or the task will be assigned
to the output node data

is located in vigorous Pace.

Especially Hadoop always
it will try to achieve

the data locality.

That's what we can't is
the resource availability as

well as the availability

of the resource in terms
of CPU memory as well

will be considered

as you might have some data
in replicated in three missions.

All three machines are busy
doing the work and no CPU

or memory available
to start the other task.

It will not wait.

For those missions to complete
the job and get the resource

and do the processing it
will start the processing

and some other machine

which is going to be near

to that the missions having
the data and read the data

over the network.

So to answer straight
or commissions are nothing but

which does the actual work

and going to report

to the master in terms of what
is the resource utilization

and the tasks running

within the work emissions
will be doing the actual work

and what ways as past Vector
just few minutes back.

I was answering a question.

What is a vector
vector is nothing

but representing the data
in multi dimensional form?

The vector can
be multi-dimensional

Vector as well.

As you know, I am going
to represent a point in space.

I need three dimensions
the X y&z.

So the vector will
have three dimensions.

If I need to represent
a line in the species.

Then I need two points
to represent the starting point

of the line and the endpoint
of the line then I need a vector

which can hold

so it will have two Dimensions
the first First Dimension

will have one point
the second dimension

will have another Point

let us say point B

if I have to represent a plane
then I need another dimension

to represent two lines.

So each line

will be representing
two points same way.

I can represent any data
using a vector form

as you might have
huge number of feedback

or ratings of products
across an organization.

Let's take a simple example
Amazon Amazon have

millions of products.

Not every user not even
a single user would have

It was millions of all
the products within Amazon.

The only hardly
we would have used

like a point one percent
or like even less than that,

maybe like few hundred products.

We would have used
and rated the products

within amazing for
the complete lifetime.

If I have to represent
all ratings of the products

with director and see
the first position

of the rating it's going
to refer to the product

with ID 1 second position.

It's going to refer
to the product with ID 2.

So I will have million values
within that particular vector.

After out of million values,

I'll have only values
400 products where I

have provided the ratings.

So it may vary from number
1 to 5 for all others.

It will say 0 sparse
pins thinly distributed.

So to represent the huge amount
of data with the position

and saying this particular
position is having

a 0 value we can mention

that with a key and value.

So what position having

what value rather than storing
all Zero seconds told one lie

non-zeros the position of it and
that the corresponding value.

That means all others going
to be a zero value

so we can mention
this particular space

Vector mentioning it
to representa nonzero entities.

So to store only
the nonzero entities

this Mass Factor will be used

so that we don't need to based

on additional space was
during this past Vector.

Let's discuss some questions
on spark streaming.

How is streaming Dad
in sparking explained

with examples smart
streaming is used

for processing real-time

streaming data to precisely say
it's a micro batch processing.

So data will be collected
between every small interval say

maybe like .5 seconds
or every seconds

until you get processed.

So internally,

it's going to create
micro patches the data created

out of that micro batch we call
there is a d stream the stream

is like a and ready

so I can do
Transformations and actions.

Whatever that I do
with our DD I can do

With the stream as well

and Spark streaming can read
data from Flume hdfs are

other streaming services Aspen

and store the data
in the dashboard or in

any other database and it
provides very high throughput

as it can be processed with
a number of different systems

in a distributed
fashion again streaming.

This stream will be partitioned
internally and it has

the built-in feature
of fault tolerance,

even if any data is lost

and it's transformed already
is Lost it can regenerate

those rdds from the existing

or from the source data.

So these three is going
to be the building block

of streaming and it has
the fault tolerance mechanism

what we have within the RTD.

So this stream are specialized
on Didi specialized form

of our GD specifically to use it
within this box dreaming.

Okay.

Next question.

What is the significance
of sliding window operation?

That's a very interesting one
in the streaming data whenever

we do the Computing the data.

Density are the
business implications

of that specific data
May oscillate a lot.

For example within Twitter.

We used to say the trending
tweet hashtag just

because that hashtag
is very popular.

Maybe someone might have hacked
into the system

and used a number of tweets
maybe for that particular

our it might have appeared
millions of times just

because it appear billions
of times for that specific

and minute duration

or like say to three minute
duration each not getting

to the trending tank.

Trending hashtag for
that particular day

or for that particular month.

So what we will do we
will try to do an average.

So like a window
this current time frame

and T minus 1 T minus 2 all
the data we will consider

and we will try to find
the average or some

so the complete business logic
will be applied

against that particular window.

So any drastic changes
on to precisely say the spike

or deep very
drastic spinal cords

drastic deep in the pattern
of the data will be normalized.

So that's the

because significance of using
the sliding window operation

with inspark streaming

and smart can handle this
sliding window automatically.

It can store the prior data
the T minus 1 T minus 2 and

how big the window
needs to be maintained

or that can be handled easily
within the program

and it's at the abstract level.

Next question is

what is destroying the expansion
is discretized stream.

So that's the abstract form

or the which will form
of representation of the data.

For the spark
streaming the same way,

how are ready getting
transformed from one form

to another form?

We will have series

of oddities all put together
called as a d string

so this term is nothing

but it's another representation

of our GD are like
to group of oddities

because there is a stream

and I can apply
the streaming functions

or any of the functions
Transformations are actions

available within the streaming
against this D string

So within that
particular micro badge,

so I will Define What interval

the data should be collected
on should be processed

because there is a micro batch.

It could be every 1 second
or every hundred milliseconds

or every five seconds.

I can Define that page

particular period so
all the data is used

in that particular duration
will be considered

as a piece of data

and that will be called

as ADI string s question explain
casing in spark streaming.

Of course.

Yes Mark internally.

It uses in memory Computing.

So any data when it
is doing the Computing

that's killing generated
will be there in Mary but find

that if you do more and more
processing with other jobs

when there is a need
for more memory,

the least used on DDS will be
clear enough from the memory

or the least used data
available out of actions

from the arditi will be cleared
of from the memory.

Sometimes I may need
that data forever in memory,

very simple example,
like dictionary.

I want the dictionary words

should be always
available in memory

because I may do a spell check
against the Tweet comments

or feedback comments
and our of nines.

So what I can do I
can say KH those any data

that comes in we can cash it.

What possessed it in memory.

So even when there is a need
for memory by other applications

this specific data will
not be remote and especially

that will be used to do
the further processing

and the casing
also can be defined

whether it should be in memory
only I in memory and hard disk

that also we can Define it.

Let's discuss some questions
on spark graphics.

The next question is is there
an APA for implementing collapse

and Spark in graph Theory?

Everything will be represented

as a graph is a graph it
will have nodes and edges.

So all will be represented
using the arteries.

So it's going to extend
the RTD and there is

a component called graphics

and it exposes
the functionalities

to represent a graph we can have
H RG D buttocks rdd by creating.

During the edges and vertex.

I can create a graph

and this graph can exist
in a distributed environment.

So same way we will be
in a position to do

the parallel processing as well.

So Graphics, it's just
a form of representing

the data paragraphs with edges
and the traces and of course,

yes, it provides the APA
to implement out create

the graph do the processing
on the graph the APA

so divided what is Page rank?

Graphics we didn't have sex
once the graph is created.

We can calculate the page rank
for a particular note.

So that's very similar to
how we have the page rank

for the websites within Google
the higher the page rank.

That means it's more important
within that particular graph.

It's going to
show the importance

of that particular node

or Edge within that particular
graph is a graph is

a connected set of data.

All right, I will be connected
using the property

and How much important

that property makes we will have
a value Associated to it.

So within pagerank
we can calculate

like a static page rank.

It will run a number

of iterations or there
is another page

and code anomic page rank
that will get executed

till we reach
a particular saturation level

and the saturation level can be
defined with multiple criterias

and the APA is

because there is
a graph operations.

And be direct executed
against those graph

and they all are available
as a PA within the graphics.

What is lineage graph?

So the audit is very similar

to the graphics how the
graph representation every rtt.

Internally.

It will have the relation saying

how that particular
rdd got created.

And from where how
that got transformed argit is

how their got transformed.

So the complete lineage
or the complete history

or the complete path
will be recorded

within the lineage.

That will be used in case

if any particular partition
of the target is lost.

It can be regenerated.

Even if the complete
artery is lost.

We can regenerate

so it will have the complete
information on what are

the partitions where it is
existing water Transformations.

It had undergone.

What is the resultant and you

if anything is lost
in the middle,

it knows where to recalculate

from and what are essential
things needs to be recalculated.

It's going to save us a lot
of time and if that Audrey

is never being used it will now.

Ever get recalculated.

So they recalculation also
triggers based on the action

only on need basis.

It will recalculate

that's why it's going
to use the memory optimally

does Apache spark provide
checkpointing officially

like the example
like a streaming

and if any data is lost within
that particular sliding window,

we cannot get back the data are
like the data will be lost

because Jim I'm making
a window of say 24

asks to do some averaging.

Each I'm making a sliding window
of 24 hours every 24 hours.

It will keep on getting slider
and if you lose any system

as in there is a complete
failure of the cluster.

I may lose the data

because it's all available
in the memory.

So how to recalculate

if the data system is lost
it follows something

called a checkpointing

so we can check point
the data and directly.

It's provided by the spark APA.

We have to just
provide the location

where it should get checked
pointed and you can read

that particular data back
when you Not the system again,

whatever the state it was

in be can regenerate
that particular data.

So yes to answer the question

straight about this path
points check monitoring

and it will help us
to regenerate the state

what it was earlier.

Let's move on to the next
component spark ml it.

How is machine learning

implemented in spark
the machine learning again?

It's a very huge ocean by itself

and it's not a technology
specific to spark

which learning is
a common data science.

It's a Set of data science work
where we have different type

of algorithms different
categories of algorithm,

like clustering regression
dimensionality reduction

or that we have

and all these algorithms
are most of the algorithms

have been implemented
in spark and smart is

the preferred framework

or before preferred application
component to do the machine

learning algorithm nowadays

or machine learning
processing the reason

because most of the machine
learning algorithms needs

to be executed i3t real number.

Of times till we get
the optimal result maybe

like say twenty five
iterations are 58 iterations

or till we get
that specific accuracy.

You will keep on running
the processing again and again

and smog is very good fit
whenever you want to do

the processing again and again

because the data
will be available in memory.

I can read it faster store
the data back into the memory

again reach faster

and all this machine learning
algorithms have been provided

within the spark a separate
component called ml lip

and within mlsp We
have other components

like feature Association
to extract the features.

You may be wondering
how they can process

the images the core thing about
processing a image or audio

or video is about
extracting the feature

and comparing the future
how much they are related.

So that's where
vectors matrices all

that will come into picture
and we can have pipeline

of processing as well
to the processing

one then take the result
and do the processing

to and it has persistence
algorithm as well.

The result of it
the generator process

the result it can be persisted

and reloaded back

into the system to
continue the processing

from that particular Point
onwards next question.

What are categories
of machine learning machine

learning assets different
categories available supervised

or unsupervised and
reinforced learning supervised

and surprised it's very popular

where we will know some
I'll give an example.

I'll know well
in advance what category

that belongs to Z. Want
to do a character recognition

while training the data,

I can give information saying
this particular image belongs

to this particular
category character

or this particular number

and I can train sometimes I
will not know well in advance

assume like I may have
different type of images

like it may have
cars bikes cat dog all that.

I want to know
how many category available.

No, I will not know well
in advance so I want to group it

how many category available

and then I'll
realize saying okay,

they're all this belongs
to a particular category.

I'll identify the pattern
within the category

and I'll give
a category named say

like all these images
belongs to boot category

on looks like a boat.

So leaving it to the system
by providing this value or not.

Let's say the cat is different
type of machine learning comes

into picture and

as such machine learning is
not specific to It's going

to help us to achieve to run
this machine learning algorithms

what our spark ml lead
tools MLA business thing

but machine learning library

or machine learning offering

within this Mark and has a
number of algorithms implemented

and it provides very
good feature to persist

the result generally
in machine learning.

We will generate
a model the pattern

of the data recorder
is a model the model

will be persisted either in
different forms Like Pat.

Quit I have
Through different forms,

it can be stored opposite
district and has methodologies

to extract the features
from a set of data.

I may have million images.

I want to extract

the common features available
within those millions of images

and other utilities
available to process to define

or like to define the seed
the randomizing it so

different utilities are
available as well as pipelines.

That's very specific to spark

where I can Channel
Arrange the sequence

of steps to be undergone by
the machine learning submission

learning one algorithm first

and then the result
of it will be fed

into a machine learning
algorithm to like that.

We can have a sequence

of execution and
that will be defined using

the pipeline's is Honorable
features of spark Emily.

What are some popular algorithms
and Utilities in spark Emily.

So these are some popular
algorithms like regression

classification basic statistics
recommendation system.

It's a comedy system is
like well implemented.

All we have to provide
is give the data.

If you give the ratings and
products within an organization,

if you have the complete damp,

we can build the recommendation
system in no time.

And if you give any user you
can give a recommendation.

These are the products
the user may like

and those products

can be displayed in the search
result recommendation system

really works on the basis
of the feedback

that we are providing
for the earlier products

that we had bought.

Bustling dimensionality
reduction whenever

we do transitioning
with the huge amount of data,

it's very very compute-intensive

and we may have
to reduce the dimensions,

especially the matrix dimensions

within them early
without losing the features.

What are the features
available without losing it?

We should reduce
the dimensionality

and there are
some algorithms available to do

that dimensionality reduction
and feature extraction.

So what are the common features
are features available

within that particular image
and I can Compare

what are the common

across common features
available within those images?

That's how we
will group those images.

So get me

whether this particular image
the person looking

like this image available
in the database or not.

For example,
assume the organization

or the police department crime
Department maintaining a list

of persons committed crime
and if we get a new photo

when they do a search they
may not have the exact photo bit

by bit the photo

might have been taken
with a different background.

Front lighting's different
locations different time.

So a hundred percent the data
will be different on bits

and bytes will be different
but look nice.

Yes, they are going to be seeing
so I'm going to search

the photo looking similar

to this particular
photograph as the input.

I'll provide to achieve

that we will be extracting
the features in each

of those photos.

We will extract the features
and we will try to match

the feature rather than the bits

and bytes and optimization as
well in terms of processing

or doing the piping.

There are a number of algorithms
to do the optimization.

Let's move on to spark SQL.

Is there a module
to implement sequence Park?

How does it work so
directly not the sequel

may be very similar to high
whatever the structure data

that we have.

We can read the data

or extract the meaning
out of the data using SQL

and it exposes the APA

and we can use those API to read
the data or create data frames

and spunk SQL has four major.

Degrees data source
data Frame data frame is

like the representation
of X and Y data

or like Excel data
multi-dimensional structure data

and abstract form
on top of dataframe.

I can do the
query and internally,

it has interpreter
and Optimizer any query

I fire that will
get interpreted or optimized

and get executed using
the SQL services and get

the data from the data frame

or it An read the data
from the data source

and do the processing.

What is a package file?

It's a format of the file

where the data
in some structured form,

especially the result
of the Spock SQL can be stored

or returned in some persistence
and the packet again.

It is a open source from Apache
its data serialization technique

where we can serialize the data
using the pad could form

and to precisely say,

it's a columnar storage.

It's going to consume
less space it will use

the keys and values.

Store the data and also it helps
you to access a specific data

from that packaged form
using the query so backward.

It's another open source format
data serialization format

to store the data

on purses the data as well

as to retrieve the data list
the functions of Sparks equal.

You can be used
to load the varieties

of structured data, of course,

yes monks equal can work only
with the structure data.

It can be used to load varieties

of structured data
and you can use SQL

like it's to query
against the program

and it can be used

with external tools to connect
to this park as well.

It gives very good
the integration with the SQL

and using python
Java Scala code.

We can create an rdd
from the structure data

available directly using
this box equal.

I can generate the TD.

So it's going to
facilitate the people

from database background to make
the program faster and quicker.

Next question is

what do you understand
by lazy evaluation?

So whenever you do any operation
within the spark word,

it will not do the processing
immediately it look

for the final results
that we are asking for it.

If it doesn't ask
for the final result.

It doesn't need to do
the processing So based

on the final action
until we do the action.

There will not be
any Transformations.

I will there will not be
any actual processing happening.

It will just understand

what our Transformations
it has to do finally

if you ask The action
then in optimized way,

it's going to complete
the data processing and get

us the final result.

So to answer straight
lazy evaluation is doing

the processing one Leon need
of the resultant data.

The data is not required.

It's not going
to do the processing.

Can you use Funk to access

and analyze data stored
in Cassandra data piece?

Yes, it is possible.

Okay, not only Cassandra
any of the nosql database it

can very well do the processing

and Sandra also works
in a distributed architecture.

It's a nosql database

so it can leverage
the data locality.

The query can
be executed locally

where the Cassandra
notes are available.

It's going to make
the query execution faster

and reduce the network load
and Spark executors.

It will try to get started

or the spark executors
in the mission

where the Cassandra
notes are available

or data is available going
to do the processing locally.

So it's going to leverage
the data locality.

T next question,

how can you
minimize data transfers

when working with spark

if you ask the core
design the success

of the spark program depends on

how much you are reducing
the network transfer.

This network transfer
is very costly operation

and you cannot paralyzed

in case multiple ways are
especially two ways to avoid.

This one is called
broadcast variable

and at Co-operators
broadcast variable.

It will help us
to transfer any static data

or any informations
keep on publish.

To multiple systems.

So I'll see

if any data to be transferred
to multiple executors

to be used in common.

I can broadcast it

and I might want to consolidate
the values happening

in multiple workers in
a single centralized location.

I can use accumulator.

So this will help us to achieve
the data consolidation

of data distribution
in the distributed world.

The ap11 are not abstract level

where we don't need
to do the heavy lifting

that's taken care
by the spark for us.

What our broadcast
variables just now

as we discussed the value
of the common value

that we need.

I am a want that to be available
in multiple executors

multiple workers simple example
you want to do a spell check

on the Tweet
Commons the dictionary

which has the right
list of words.

I'll have the complete list.

I want that particular
dictionary to be available

in each executor

so that with a task with
that's running locally

in those Executives can refer
to that particular.

Task and get the processing
done by avoiding

the network data transfer.

So the process of Distributing
the data from the spark context

to the executors

where the task going
to run is achieved

using broadcast variables

and the built-in within the
spark APA using this parquet p--

we can create
the bronchus variable

and the process of Distributing
this data available

in all executors is taken care
by the spark framework explain

accumulators in spark.

The similar way how we
have broadcast variables.

We have accumulators
as well simple example,

you want to count how many
error codes are available

in the distributed environment

as your data is distributed

across multiple systems
multiple Executives.

Each executor will do
the process thing count

the records anatomically.

I may want the total count.

So what I will do I will ask
to maintain an accumulator,

of course, it will be maintained
in this more context.

In the driver program
the driver program going

to be one per application.

It will keep on
getting accumulated

and whenever I want I
can read those values

and take any appropriate action.

So it's like more or less the
accumulators and practice videos

looks opposite each other,

but the purpose
is totally different.

Why is there a need
for workers variable

when working with Apache Spark
It's read only variable

and it will be cached in memory
in a distributed fashion

and it eliminates the The work

of moving the data
from a centralized location

that is Spong driver

or from a particular program
to all the executors

within the cluster where
the transfer into get executed.

We don't need to worry about
where the task will get executed

within the cluster.

So when compared

with the accumulators
broadcast variables,

it's going to have
a read-only operation.

The executors cannot change

the value can only
read those values.

It cannot update so mostly
will be used like a quiche.

Have for the
identity next question,

how can you trigger
automatically naps in spark

to handle accumulated metadata.

So there is a parameter

that we can set TTL the
will get triggered along

with the running jobs
and intermediately.

It's going to write the data
result into the disc

or cleaned unnecessary data
or clean the rdds.

That's not being used.

The least used RTD.

It will get cleaned

and click keep the metadata as
well as the memory clean water.

The various levels
of persistence in Apache spark

when you say data
should be stored in memory.

It can be indifferent now
you can be possessed it

so it can be in memory of only
or memory and disk or disk only

and when it is getting stored
we can ask it to store it

in a civilized form.

So the reason why we may store
or possess dress,

I want this particular

on very this form
of body little back

for using so I can really

back maybe I may not need
it very immediate.

So I don't want that to keep
occupying my memory.

I'll write it to the hard disk

and I'll read it back
whenever there is a need.

I'll read it back
the next question.

What do you understand
by schema rdd,

so schema rdd will be used as
slave Within These Punk's equal.

So the RTD will have the meta
information built into it.

It will have the schema
also very similar to

what we have the database
schema the structure

of the particular data

and when I have a structure it
will be easy for me.

To handle the data

so data and the structure
will be existing together

and the schema are ready.

Now.

It's called as a data frame

but it's Mark and dataframe
term is very popular

in languages like our
as other languages.

It's very popular.

So it's going to have the data
and The Meta information

about that data saying
what column was structure it.

Is it explain the scenario

where you will be
using spark streaming

as you may want to do
a sentiment analysis

of Twitter's so there
I will be streamed

so we will Flume sort of a tool
to harvest the information

from Peter and fit it
into spark streaming.

It will extract or identify
the sentiment of each

and every tweet and Market

whether it is positive
or negative and accordingly

the data will be
the structure data

that we tidy

whether it is positive
or negative maybe

percentage of positive

and percentage of negative
sentiment store it

in some structured form.

Then you can leverage this park
Sequel and do grouping

or filtering Based
on the sentiment

and maybe I can use
a machine learning algorithm.

What drives that
particular tweet to be

in the negative side.

Is there any similarity between
all this negative sentiment

negative tweets may be specific

to a product a specific time
by when the Tweet was sweeter

or from a specific region

that we it was
Twitter those analysis

could be done by leveraging
the MLA above spark.

So Emily streaming core
all going to work together.

All these are like different.

Offerings available to
solve different problems.

So with this we are coming
to end of this interview

questions discussion of spark.

I hope you all enjoyed.

I hope it was constructive
and useful one.

The more information
about editor is available

in this website to record
at cou only best

and keep visiting the website
for blocks and latest updates.

Thank you folks.

I hope you have enjoyed
listening to this video.

Please be kind enough to like it

and you can comment any
of your doubts and queries

and we will reply them

at the earliest do look out
for more videos in our playlist

And subscribe to Edureka
channel to learn more.

Happy learning.