All language subtitles for Splunk Tutorial For Beginners _ Troubleshooting Splunk (1080p_30fps_H264-128kbit_AAC).anglais

[Music]

hello friends welcome back to my channel

and uh today we are back with another

exciting tutorial so this will be a

splunk tutorial for beginners and there

has been a lot of queries a lot of

requests on how to

troubleshoot splunk issues so like if we

had done a lot of tutorials like how to

set up splunk and writing queries and

all those things but there was a request

like if in a real productive scenarios

they may face some issues with

splunk environment how it can be

troubleshoot so in that cases you know i

just wanted to show you some of the tips

or you know what are the log files what

kind of indexes you know what kind of

queries you know you can use what kind

of uh extra tools like b2 uh which you

can use it for uh you know

finding out some of uh splunk issues how

you can analyze that okay so some of the

common splunk issues which you uh come

across or face would be like some data

quality issues or it can be a search

performance or capacity issues or some

license issues with the splunk or you

know it can be a server usage issues so

these are some common issues or splunk

crash issues there are a lot of more but

these are some common issues which you

may come across and the way you know how

you can analyze i will just show you

some tips like you know what kind of log

files you can go and check and in which

your log file you can easily find out

what so let's uh check out on those

things so before i get into that i would

request you like if you are new to my

channel or if you have not subscribed to

my channel click on the subscribe button

also like my videos share and comment

so let's get started with some of the

common splunk logs okay so that would be

the best place to start so you splunk

always write you know most of its

contents into some of the logs or some

of the internal indexes which we will

see so some of the common logs which you

can go and identify these issues are

like one is log file is called splunk

dot log and another one is called

metrics dot log and auric.log and

license usage.log and splunk access dot

log there are other locks as well but

these are some uh major logs which you

can be helpful and out of this splunk

dot log is the main one where it will

write all the contents what splunk d is

doing and let's go and check out this

where it will be saved

since i'm using windows machine and i

have installed splunk on this machine

this uh log files are located under your

splunk installation folder so if it's

under c program files splunk uh so that

would be your splunk home and that

should be underwater log and splunk so

that will be the place where the logs

will be installed available where in the

case of linux installation it would be

like your splunk home for example if

your splunk home is slash opt

slash splunk then you know the log file

will be under you know

log

splunk so it'll be slash opt slash

splunk or slash war slash log slash

splunk so that would be the place where

all these logs file will be there so

as you as you can see there are so much

of log file here right so mainly you

know what we want to do is we

we want to start with the splunk d dot

log that would be the main log where the

splunk will write all its uh you know

information and you have other logs

likes plan the access dot log you know

which i mentioned you have something

called audit.log you have metrics.log

license usage.log so all these locks

there are other locks as well but these

are some major lock now if you go to

this splunk dot log as i mentioned it

will start writing all these uh

activities what it doing like splunk d

starting what the system info what is

the you know uh system information you

know

all those details it's trying to put it

here you can see that right uh you can

read through it you know when you want

to find and you know it will also tell

you you know what it's trying to do with

each you know uh splunk uh

about you it's uh

indexing and all those things you can

see it's the stanza which is looking for

even log right and also it will be

talking about the uh i know what file

it's watching so if i search for

watch

so you can see if i search for watch it

will tell you like you know different

files which is watching so when splunk

is looking uh those files mean it will

watch those files and if there is any

changes it will uh get that into the

index as per our configurations right so

that's how it's be done for example i

have windows

update dot log

which i am uh having it so you can see

you know uh there is a log which is uh

you know added into this so it will be

watching that

file and it's also writing you know it's

different kind of buckets

all those things it's also look for crc

right so all those details will be

available in this log so it's not

mandatory that you need to open this

file and uh read through it there you

can also directly access this from your

splunk instance you can write a query

and you can find out those information

here

so when we talk about splunk dot log

which i mentioned splunk dot log

provides an overview of what splunk is

doing so all the events are logged uh

for all the splunk components in this uh

splunk dot log and as i mentioned it's

available in this folder like splunk

home which is you know in windows should

be under program file splunk whereas

linux it should be the place where you

install war log splunk right and this is

a query which you can run

okay and i'll show you what we get and

uh if you see yeah this is a sample

which i show like in the splunk d dialog

it's watches the file which is we want

to index so if you have a scenario like

you know your file is not getting

indexed uh the best thing is you go and

check in the log where you have the file

is getting watched and also if you have

a forwarder which is sending the file

you can also check the splunk d log in

the

forwarder client so that you know you

can see what is happening from there as

well

so if i run this uh query over here on

the splunk instance you can see i'm

running on the source

and if i run it

you can see it will give you the whole

components what we are having in inside

the splunk dot log so it's give you a

bunch of you know details what is done

so if you see the bucket roll over if i

check it it will tell you all the

components or the locks based on that so

you can see a roll to the hot db bucket

for you know uh

from this one right and you can see here

finished moving hot to one so all those

informations are written in this uh

log file like how the buckets are moved

from hot to warm and all those things so

you can see all the details over here

when i just search for that component

right so similar way you can check what

the components so it's just a simple way

like you just put a stats on the count

by component and you will be getting you

know the details so it's not mandatory

that you need to go to the splunk dot

log and then you search it there

so before we go into the other log files

i also want to show you like some of the

indexes which will be you know very

useful for analyzing so some of the

index like underscore internal and

underscore audit so these are two main

uh indexes other than the other indexes

which you will create these are system

indexes which will be very useful for

your analysis or troubleshooting a lot

of splunk issues so one of a simple

query like if you put the

underscore internal and if you skip the

stats by source type and source we can

get some information from the let's see

that

so if i go and put

the source and source type so it will

give us the details on what are the

contents we have it in the underscore

internal so you can see you know there

are different kind of source type and

where the source the file is coming for

example splunk d is the source type

where it's how multiple log files

getting returned so splunk d dot log is

one utility metrics license usage health

dot log so all these things are written

by splunk d right and we have ui

access which is the splunk ui access dot

log and we have config we have mongodb

scheduler right so there are web access

which will be web access dot log so the

different kind of log where it is

written by which component it give you

the detail using this query

if i want to simplify it a little bit

more for a splunk d uh case alone so if

i go i like index

internals source type is splunk d and

source with the splunk dot log

because there are different logs let's

see what we get from output from there

so this should be similar like what you

go and search in the splunk dot log

because we are finding uh all the

details for

splunk d dot log right so now uh

whatever we see in that log file we are

going to get it here for example you

know reading the file you know

all those in details we will have it

here right and you'll have different

filters over here like

let's see we have source type right we

have a file so you can see different

kind of files you can filter with that

right

so if i go and select

one of the file here

you can see uh it is it will show you

that file alone so for example if i go

and change this to

windows

update

dot log

so you can see if i run with this

windowsupdate.log we get some locks

based on that right so if you see the

file is watched uh you know at this time

right and

also parsing the content because it's

coming from this uh

you know uh stance of our you know

uh configuration right so that's what

it's trying to do

uh so you can change the value of from

24 hours to which or so then you'll get

more uh details as well so there are

different ways you can simply you know

find out which log is uh you know

in splunk d so it's not that you have to

open the splunk dot log

from the location and you read through

it so you can easily

filter it using your splunk search as

well

and the same way like you can do it for

a matrix logs as well so let's find what

we get there

so if i go and search for metrics

so you can see like we get a lot of

output here as well so metrics you can

see there are more metrics like

throughput you know instantaneous kb

right so different other parameters you

get it from metrics

so if you talk about metrics this

metrics log is where the splunk stores

all the information

of various group of splunk activities

such as like throughput skew sizes

response time job counts so this the

kind of whole information is getting

stored on the metrics dot log so if you

want to analyze some of those you need

to check these logs metrics dot log so

that's why i just want to show you now

there is also other way like all these

informations if you see it's getting

stored to the index underscore internal

right so if you don't want to search

whole information if you want to get

some like errors or warnings you can

even query with you know

that kind of query in the splunk uh

search you can also advance that you

know if you want to specifically do for

splunk the dot log so you can do that

and you can table it with some log

message and all those things so this is

a sample

query which you can run so i will just

run it and show you what we get

so if i run this query for

error or warning

so you can see we get uh some output so

where you know the

message will be based on that so

you can see the search has been

cancelled right so that's uh one of

error message you can see we have error

right

so it's giving a here if you see it's a

warning message right the message search

has been cancelled so based on that

those uh content also we can do some

search

so these are some samples i just want to

show you like you know what you can make

use of this uh kind of queries to find

some

output so you can see you know host

component the level warning level info

error you know this which server it's

happening so this kind of queries you

can write when you have some issues so

based on what is if you are want to find

some error messages or warning message

you can try to run this kind of queries

and you will get some kind of uh input

from this which you can analyze it

so similar like matrix logs or splunk

dot log we have something called

audi.log so if you want to find some

information about any user activity such

as logins or search

run and all those things the auric.log

is the best place so this audit.log is

not for uh web-based ui login uh

activities this audit is for other user

activities okay so we'll also talk about

there's another called audit access.log

which will be mainly used for ui login

things okay so license usage is also

another log which you can make use of it

to find out how much index the volume of

buys per license indexes source source

type those kind of things are happening

using site license usage dot log and as

i mentioned splunk the access dot log

this gives you the details of splunk d

through ui is logged in like uh web

splunk web cli or post get uh delete

search so when you do it through the

splunk ui login those things will get

started in the splunk the access dot log

okay so you can see you know there are

other scenarios also like uh you may

face some issues like you know you have

a splunk search which is getting skipped

or something right so if you want to get

uh the details like what kind of

searches are getting skipped you can use

some queries like this like underscore

internal and the source type should be

scheduler and the status like skip so if

you are having skip searches you can

find it with this let's see what we get

so let me run this search okay

so you can see i don't have any output

at all because i don't have any searches

which is skip so i'll just remove the

status so let's see what we get so i

have the details about the searches now

so i have no source type scheduler and

you can see

completed reading history so those kind

of things is there now if i go to the

status

i have only states called success so

that means the searches are success but

if you have you know some skip searches

you can use uh the status equal to

skipped then you will see what other

searches are skipped then based on that

you will be able to take your actions

based on that you may also want to

analyze some of like concurrent searches

and all those things so in this case you

can use this search like or you have

internal source type metric and the

group is like search concurrent source

so that will also give you some detail

like how much of concurrent searches we

can do those kind of things

so let me do that search over here okay

so i will do a

search for with this metrics.log

so

you can see we have uh search

concurrency right and uh the metrics so

minimum queue so those kind of

information is available in this uh you

know search and because what we are

trying to do is we are making use of the

underscore internal and the source types

right so we are using mainly splunk dot

lock or and the metrics dot log uh

i think we have also used scheduler so

different kind of source we have it and

using that we can make use of different

kind of events uh the log files over

here itself so it's not like you have to

open the files manually on the server

and do it so this is also an easiest way

so you can write down your queries and

you can filter it based on that

so other scenario will be like you may

face like your sprung splunk instance

get crashed right so you can use uh this

query like where source type will be

splunk d crash dot log and it will tell

you like how much time this splunk

instance has been crashed you may also

want to find out like when the splunk

instance has been restarting or spla you

know when the server was restarted last

time the splunk's instance so you can

use the splunk the dot lock with the

statement like splunk d starting so

let's see what output we get from this

so i will just search for uh splunk this

star search starting first because if

you want to check out check you know

when your splunk has been restarted so

if you see in last 24 hours uh this my

splunk instance has been restarted at

this time so it was

splunk this starting message right if i

change it for whole time let me see it

may give me more informations

so you can see it has done uh three to

four three times the restart right so

this is a new setup so it's not uh too

old so

you know so it has been only just

restarted uh three times so this is also

where you can find out when the splunk

was restarted

if you see this other search where we

talk about splunk the crash dot log

i don't think i will have any data here

because it has not been crashed

anywhere

so we don't have that source type at all

so but if you have this you know um

file because if your splunk has been

crashed you'll have this source type

then you should be able to do this

search okay so let's uh you know uh

uh find out some more better information

other than this uh

you know tools or like the search

queries which you are using so what we

understood is like we have different

kind of log files so mainly you know uh

when there is a data mismatch you have

to go to splunk dot log and you have to

find out what data is getting indexed

and also you need to check the forwarder

lock so you can see what forward is

sending and what uh splunk d on the

indexer is getting so if that matches

yeah fine if there is a description see

then the data is not reaching to the

indexer at all right and you know you

may also see there will be reason like

the deployment server is not getting

configured correctly that's why the

splunk uh

configurations are not getting reflected

there so that kind of scenario happen

sometimes you know the crc file you know

it's checking the crc it may not see any

changes so you'll have to tweak

something like you know there are other

commands available in the stand server

you can add where the crc

values so you can by default it comes

around like 256. you can change a little

bit so that you can search a little bit

more of the file so you have to be very

careful when you work on that so it can

re-index the whole data again so you

need to read through the documentation

and check you know with splunk support

you can raise splunk tickets if you have

any queries if you have a enterprise

license so you can register them and

they'll be able to support and also you

know you can have any license issues and

everything sometimes you have to go back

to them

so another interesting tool which splunk

is having is called splunk b tool which

will be

very helpful so you can run the command

like splunk help b tool

which is in the windows if it's in the

linux you can put dot slash in the

location of our splunk bin and then you

allow splunk help b2 and you know let's

see what we get when you run this in our

machine

so i'm inside the c program file splunk

bin where the splunk will be available

okay

so if you see if i run a splunk help b

tool it give you

quite some information like what

a different kind of uh you know

stanzas or you can do arguments with

this so you can see b tool config file

list b check right and you have uh other

optional parameters which you can use

right so different uh ways you can use

this tool so let me show you some of uh

simple examples

so if i want to list all the indexes

this is one of a way you can do it like

splunk b tool indexes dot list so it

will list you all the details about the

indexes you can then you can filter it

using uh graph in linux or in maybe in

windows like a find string something

like that so you can filter it let me

show you with this command

so if you see like you know when i run

this command it has given a complete

list of

uh you know the indexes for example it's

underscore audit which is uh index and

it's giving a huge bunch of information

about that

that specific index and then it will go

to the next index so if you see we have

any underscore introspection which is

another index and let me see you have

underscore internal so you have a

a whole list of indexes it's showing up

and you know it will tell you what the

complete information about a lot of

information is there what you require

you have to filter it so i just want to

show you like you know the b tool gives

a different kind of this is a once one

or two example i'm just showing you how

to read through the documentation for b

tool there are huge more options you

have it

so and one more other example with b2

like you can also use it to search uh

like searches list in that specific app

so if you see this we are using b tool

app under the search and we are using

save searches list so it will list you

all the saved searchers let's see what

we get

so if you see if i run uh this command b

tool app search search searches list it

will give you the uh different kind of

searches like errors in the last 24

hours in the last hour right

license usage so similar way like if you

have served uh some kind of searches it

will give you the complete uh details on

when using this search query so these

are some example okay so uh as i

mentioned you know the main way of

analyzing the splunk uh issues are using

the log files and also the indexes like

underscore internal or underscore audit

you can write down your own queries you

can get the you know performance issues

or skip searches or you know it can be

you know concurrent searches issues or

the file is not getting indexed or it

can be any kind of scenarios you just

need to put what kind of data you need

to analyze it then you should be able to

easily interpret interpreted

so that is all for this tutorial so uh

next tutorial maybe i will come up with

the specific scenarios but i have to

make the kind of scenario then i have to

show you how to fix that so that's a

little bit complicated so but i will try

to find out that kind of tutorials in

the upcoming tutorials but for now

i'm logging off so i would request you

like if you are new to my channel or if

you're not subscribed kindly subscribe

to my channel for more videos and also

like my videos share and comment so

thank you for watching

[Music]