Learning Elastic Stack 6.0 by Shukla, Pranav -- Read -- Imperial Library of Trantor

Index

Title Page Copyright

Learning Elastic Stack 6.0

Credits Disclaimer About the Authors About the Reviewer www.PacktPub.com

Why subscribe?

Customer Feedback Preface

What this book covers What you need for this book Who this book is for Conventions Reader feedback Customer support

Downloading the example code Downloading the color images of this book Errata Piracy Questions

Introducing Elastic Stack

What is Elasticsearch, and why use it?

Schemaless and document-oriented Searching Analytics Rich client library support and the REST API Easy to operate and easy to scale Near real time Lightning fast Fault tolerant

Exploring the components of Elastic Stack

Elasticsearch Logstash Beats Kibana X-Pack

Security Monitoring Reporting Alerting Graph

Elastic Cloud

Use cases of Elastic Stack

Log and security analytics Product search Metrics analytics Web search and website search

Downloading and installing

Installing Elasticsearch Installing Kibana

Summary

Getting Started with Elasticsearch

Using the Kibana Console UI Core concepts

Index Type Document Node Cluster Shards and replicas Mappings and data types

Data types

Core datatypes Complex datatypes Other datatypes

Mappings

Creating an index with the name catalog Defining the mappings for the type of product

Inverted index

CRUD operations

Index API

Indexing a document by providing an ID Indexing a document without providing an ID

Get API Update API Delete API

Creating indexes and taking control of mapping

Creating an index Creating type mapping in an existing index Updating a mapping

REST API overview

Common API conventions

Formatting the JSON response Dealing with multiple indices

Searching all documents in one index Searching all documents in multiple indexes Searching all documents of a particular type in all indices

Summary

Searching-What is Relevant

Basics of text analysis

Understanding Elasticsearch analyzers

Character filters Tokenizer

Standard Tokenizer

Token filters

Using built-in analyzers

Standard Analyzer

Implementing autocomplete with a custom analyzer

Searching from structured data

Range query

Range query on numeric types Range query with score boosting Range query on dates

Exists query Term query

Searching from full text

Match query

Operator minimum_should_match Fuzziness

Match phrase query Multi match query

Querying multiple fields with defaults Boosting one or more fields With types of multi match queries

Writing compound queries

Constant score query Bool query

Combining OR conditions Combining conditions AND and OR conditions Adding NOT conditions

Summary

Analytics with Elasticsearch

The basics of aggregations

Bucket aggregations Metric aggregations Matrix aggregations Pipeline aggregations

Preparing data for analysis

Understanding the structure of data Loading the data using Logstash

Metric aggregations

Sum, average, min, and max aggregations

Sum aggregation Average aggregation Min aggregation Max aggregation

Stats and extended stats aggregations

Stats aggregation Extended stats Aggregation

Cardinality aggregation

Bucket aggregations

Bucketing on string data

Terms aggregation

Bucketing on numeric data

Histogram aggregation Range aggregation

Aggregations on filtered data Nesting aggregations Bucketing on custom conditions

Filter aggregation Filters aggregation

Bucketing on date/time data

Date Histogram aggregation

Creating buckets across time Using a different time zone Computing other metrics within sliced time intervals Focusing on a specific day and changing intervals

Bucketing on geo-spatial data

Geo distance aggregation GeoHash grid aggregation

Pipeline aggregations

Calculating the cumulative sum of usage over time

Summary

Analyzing Log Data

Log analysis challenges

Logstash

Installation and configuration

Prerequisites

Downloading and installing Logstash

Installing on Windows Installing on Linux Running Logstash

Logstash architecture Overview of Logstash plugins

Installing or updating plugins

Input plugins Output plugins Filter plugins Codec plugins

Exploring plugins

Exploring Input plugins

File Beats JDBC IMAP

Output plugins

Elasticsearch CSV Kafka PagerDuty

Codec plugins

JSON Rubydebug Multiline

Filter plugins

Ingest node

Defining a pipeline Ingest APIs

Put pipeline API Get Pipeline API Delete pipeline API Simulate pipeline API

Summary

Building Data Pipelines with Logstash

Parsing and enriching logs using Logstash

Filter plugins

CSV filter Mutate filter Grok filter Date filter Geoip filter Useragent filter

Introducing Beats

Beats by Elastic.co

Filebeat Metricbeat Packetbeat Heartbeat Winlogbeat Auditbeat

Community Beats Logstash versus Beats

Filebeat

Downloading and installing Filebeat

Installing on Windows Installing on Linux

Architecture Configuring Filebeat

Filebeat prospectors Filebeat global options Filebeat general options Output configuration Filebeat modules

Summary

Visualizing data with Kibana

Downloading and installing Kibana

Installing on Windows Installing on Linux Configuring Kibana

Data preparation Kibana UI

User interaction Configuring the index pattern Discover

Elasticsearch query string Elasticsearch DSL query

Visualize

Kibana aggregations

Bucket aggregations Metric

Creating a visualization Visualization types

Line, area, and bar charts Data table MarkDown widget Metric Goal Gauge Pie charts Co-ordinate maps Region maps Tag cloud

Visualizations in action

Response codes over time Top 10 URLs requested Bandwidth usage of top five countries over time Web traffic originating from different countries Most used user agent

Dashboards

Creating a dashboard Saving the dashboard Cloning the dashboard Sharing the dashboard

Timelion

Timelion UI Timelion expressions

Using plugins

Installing plugins Removing plugins

Summary

Elastic X-Pack

Installing X-Pack

Installing X-Pack on Elasticsearch Installing X-Pack on Kibana Uninstalling X-Pack

Configuring X-Pack Security

User authentication User authorization Security in action

New user creation

Deleting a user Changing the password

New role creation

How to Delete/Edit a role

Document-level security or field-level security X-Pack security APIs

User management APIs Role management APIs

Monitoring Elasticsearch

Monitoring UI

Elasticsearch metrics

Overview tab Nodes tab The Indices tab

Alerting

Anatomy of a watch Alerting in action

Create a new alert

Threshold Alert Advanced Watch

How to Delete/Deactivate/Edit a Watch

Summary

Running Elastic Stack in Production

Hosting Elastic Stack on a managed cloud

Getting up and running on Elastic Cloud Using Kibana Overriding configuration Recovering from a snapshot

Hosting Elastic Stack on your own

Selecting hardware Selecting an operating system Configuring Elasticsearch nodes

JVM heap size Disable swapping File descriptors Thread pools and garbage collector

Managing and monitoring Elasticsearch Running in Docker containers Special considerations while deploying to a cloud

Choosing instance type Changing default ports; do not expose ports! Proxy requests Binding HTTP to local addresses Installing EC2 discovery plugin Installing S3 repository plugin Setting up periodic snapshots

Backing up and restoring

Setting up a repository for snapshots

Shared filesystem

Cloud or distributed filesystems Taking snapshots Restoring a specific snapshot

Setting up index aliases

Understanding index aliases How index aliases can help

Setting up index templates

Defining an index template Creating indexes on the fly

Modeling time series data

Scaling the index with unpredictable volume over time

Unit of parallelism in Elasticsearch

The effect of the number of shards on the relevance score The effect of the number of shards on the accuracy of aggregations

Changing the mapping over time

New fields get added Existing fields get removed

Automatically deleting older documents How index-per-timeframe solves these issues

Scaling with index-per-timeframe Changing the mapping over time Automatically deleting older documents

Summary

Building a Sensor Data Analytics Application

Introduction to the application

Understanding the sensor-generated data Understanding the sensor metadata Understanding the final stored data

Modeling data in Elasticsearch

Defining an index template Understanding the mapping

Setting up the metadata database Building the Logstash data pipeline

Accept JSON requests over the web Enrich the JSON with the metadata we have in the MySQL database

The jdbc_streaming plugin The mutate plugin

Move the looked-up fields that are under lookupResult directly in JSON Combine the latitude and longitude fields under lookupResult as a location field Remove the unnecessary fields

Store the resulting documents in Elasticsearch

Sending data to Logstash over HTTP Visualizing the data in Kibana

Set up an index pattern in Kibana Build visualizations

How does the average temperature change over time? How does the average humidity change over time? How do temperature and humidity change at each location over time? Can I visualize temperature and humidity over a map? How are the sensors distributed across departments?

Create a dashboard

Summary

Monitoring Server Infrastructure

Metricbeat

Downloading and installing Metricbeat

Installing on Windows Installing on Linux

Architecture

Event structure

Configuring Metricbeat

Module configuration

Enabling module configs in the modules.d directory Enabling module config in the metricbeat.yml file

General settings Output configuration Logging

Capturing system metrics

Running Metricbeat with the system module Specifying aliases Visualizing system metrics using Kibana

Deploymezs architecture Summary

← Prev
Back
Next →

← Prev
Back
Next →