Learning Elastic Stack 7.0 · 2nd Edition by Kumar, Sharath -- Read -- Imperial Library of Trantor

Index

Title Page Copyright and Credits

Learning Elastic Stack 7.0 Second Edition

About Packt

Why subscribe? Packt.com

Contributors

About the authors About the reviewer Packt is searching for authors like you

Preface

Who this book is for What this book covers To get the most out of this book

Download the example code files Download the color images Conventions used

Get in touch

Reviews

Section 1: Introduction to Elastic Stack and Elasticsearch Introducing Elastic Stack

What is Elasticsearch, and why use it?

Schemaless and document-oriented Searching capability Analytics Rich client library support and the REST API Easy to operate and easy to scale  Near real-time capable Lightning–fast Fault-tolerant

Exploring the components of the Elastic Stack

Elasticsearch Logstash Beats Kibana X-Pack

Security Monitoring Reporting Alerting Graph Machine learning

Elastic Cloud

Use cases of Elastic Stack

Log and security analytics Product search Metrics analytics Web search and website search

Downloading and installing

Installing Elasticsearch Installing Kibana

Summary

Getting Started with Elasticsearch

Using the Kibana Console UI Core concepts of Elasticsearch

Indexes Types Documents Nodes Clusters Shards and replicas Mappings and datatypes

Datatypes

Core datatypes Complex datatypes Other datatypes

Mappings

Creating an index with the name catalog Defining the mappings for the type of product

Inverted indexes

CRUD operations

Index API

Indexing a document by providing an ID Indexing a document without providing an ID

Get API Update API Delete API

Creating indexes and taking control of mapping

Creating an index Creating type mapping in an existing index Updating a mapping

REST API overview

Common API conventions

Formatting the JSON response Dealing with multiple indexes

Searching all documents in one index Searching all documents in multiple indexes Searching all the documents of a particular type in all indexes

Summary

Section 2: Analytics and Visualizing Data Searching - What is Relevant

The basics of text analysis

Understanding Elasticsearch analyzers

Character filters Tokenizer

Standard tokenizer

Token filters

Using built-in analyzers

Standard analyzer

Implementing autocomplete with a custom analyzer

Searching from structured data

Range query

Range query on numeric types Range query with score boosting Range query on dates

Exists query Term query

Searching from the full text

Match query

Operator Minimum should match Fuzziness

Match phrase query Multi match query

Querying multiple fields with defaults Boosting one or more fields With types of multi match queries

Writing compound queries

Constant score query Bool query

Combining OR conditions Combining AND and OR conditions Adding NOT conditions

Modeling relationships

has_child query has_parent query parent_id query

Summary

Analytics with Elasticsearch

The basics of aggregations

Bucket aggregations Metric aggregations Matrix aggregations Pipeline aggregations

Preparing data for analysis

Understanding the structure of the data Loading the data using Logstash

Metric aggregations

Sum, average, min, and max aggregations

Sum aggregation Average aggregation Min aggregation Max aggregation

Stats and extended stats aggregations

Stats aggregation Extended stats aggregation

Cardinality aggregation

Bucket aggregations

Bucketing on string data

Terms aggregation

Bucketing on numerical data

Histogram aggregation Range aggregation

Aggregations on filtered data Nesting aggregations Bucketing on custom conditions

Filter aggregation Filters aggregation

Bucketing on date/time data

Date Histogram aggregation

Creating buckets across time periods Using a different time zone Computing other metrics within sliced time intervals Focusing on a specific day and changing intervals

Bucketing on geospatial data

Geodistance aggregation GeoHash grid aggregation

Pipeline aggregations

Calculating the cumulative sum of usage over time

Summary

Analyzing Log Data

Log analysis challenges Using Logstash

Installation and configuration

Prerequisites Downloading and installing Logstash

Installing on Windows Installing on Linux

Running Logstash

The Logstash architecture Overview of Logstash plugins

Installing or updating plugins

Input plugins Output plugins Filter plugins Codec plugins

Exploring plugins

Exploring input plugins

File Beats JDBC IMAP

Output plugins

Elasticsearch CSV Kafka PagerDuty

Codec plugins

JSON Rubydebug  Multiline

Filter plugins

Ingest node

Defining a pipeline  Ingest APIs

Put pipeline API Get pipeline API Delete pipeline API Simulate pipeline API

Summary

Building Data Pipelines with Logstash

Parsing and enriching logs using Logstash

Filter plugins

CSV filter  Mutate filter Grok filter Date filter Geoip filter Useragent filter

Introducing Beats

Beats by Elastic.co

Filebeat Metricbeat Packetbeat Heartbeat Winlogbeat Auditbeat Journalbeat Functionbeat

Community Beats Logstash versus Beats

Filebeat

Downloading and installing Filebeat

Installing on Windows Installing on Linux

Architecture Configuring Filebeat

Filebeat inputs Filebeat general/global options Output configuration  Logging Filebeat modules

Summary

Visualizing Data with Kibana

Downloading and installing Kibana

Installing on Windows Installing on Linux Configuring Kibana

Preparing data Kibana UI

User interaction Configuring the index pattern Discover

Elasticsearch query string/Lucene query Elasticsearch DSL query KQL

Visualize

Kibana aggregations

Bucket aggregations Metric

Creating a visualization Visualization types

Line, area, and bar charts Data tables Markdown widgets Metrics Goals Gauges Pie charts Co-ordinate maps Region maps Tag clouds

Visualizations in action

Response codes over time Top 10 requested URLs Bandwidth usage of the top five countries over time Web traffic originating from different countries Most used user agent

Dashboards

Creating a dashboard Saving the dashboard  Cloning the dashboard Sharing the dashboard 

Timelion

Timelion  Timelion expressions

Using plugins

Installing plugins Removing plugins

Summary

Section 3: Elastic Stack Extensions Elastic X-Pack

Installing Elasticsearch and Kibana with X-Pack

Installation Activating X-Pack trial account

Generating passwords for default users

Configuring X-Pack Securing Elasticsearch and Kibana

User authentication User authorization Security in action

Creating a new user

Deleting a user Changing the password

Creating a new role

Deleting or editing a role

Document-level security or field-level security X-Pack security APIs

User Management APIs Role Management APIs

Monitoring Elasticsearch

Monitoring UI

Elasticsearch metrics

Overview tab Nodes tab The Indices tab

Alerting

Anatomy of a watch Alerting in action

Creating a new alert

Threshold Alert Advanced Watch

Deleting/deactivating/editing a watch

Summary

Section 4: Production and Server Infrastructure Running Elastic Stack in Production

Hosting Elastic Stack on a managed cloud

Getting up and running on Elastic Cloud Using Kibana Overriding configuration  Recovering from a snapshot

Hosting Elastic Stack on your own

Selecting hardware Selecting an operating system Configuring Elasticsearch nodes

JVM heap size Disable swapping File descriptors Thread pools and garbage collector

Managing and monitoring Elasticsearch Running in Docker containers Special considerations while deploying to a cloud

Choosing instance type Changing default ports; do not expose ports! Proxy requests Binding HTTP to local addresses Installing EC2 discovery plugin Installing the S3 repository plugin Setting up periodic snapshots

Backing up and restoring

Setting up a repository for snapshots

Shared filesystem

Cloud or distributed filesystems Taking snapshots Restoring a specific snapshot

Setting up index aliases

Understanding index aliases How index aliases can help

Setting up index templates

Defining an index template Creating indexes on the fly

Modeling time series data

Scaling the index with unpredictable volume over time

Unit of parallelism in Elasticsearch

The effect of the number of shards on the relevance score The effect of the number of shards on the accuracy of aggregations

Changing the mapping over time

New fields get added Existing fields get removed

Automatically deleting older documents How index-per-timeframe solves these issues

Scaling with index-per-timeframe Changing the mapping over time Automatically deleting older documents

Summary

Building a Sensor Data Analytics Application

Introduction to the application

Understanding the sensor-generated data Understanding the sensor metadata Understanding the final stored data

Modeling data in Elasticsearch

Defining an index template Understanding the mapping

Setting up the metadata database Building the Logstash data pipeline

Accepting JSON requests over the web Enriching the JSON with the metadata we have in the MySQL database

The jdbc_streaming plugin  The mutate plugin

Moving the looked-up fields that are under lookupResult directly in JSON Combining the latitude and longitude fields under lookupResult as a location field Removing the unnecessary fields

Store the resulting documents in Elasticsearch

Sending data to Logstash over HTTP Visualizing the data in Kibana

Setting up an index pattern in Kibana Building visualizations

How does the average temperature change over time? How does the average humidity change over time? How do temperature and humidity change at each location over time? Can I visualize temperature and humidity over a map? How are the sensors distributed across departments?

Creating a dashboard

Summary

Monitoring Server Infrastructure

Metricbeat

Downloading and installing Metricbeat

Installing on Windows Installing on Linux

Architecture

Event structure

Configuring Metricbeat

Module configuration

Enabling module configs in the modules.d directory Enabling module configs in the metricbeat.yml file

General settings Output configuration  Logging

Capturing system metrics

Running Metricbeat with the system module Specifying aliases Visualizing system metrics using Kibana

Deployment architecture Summary

Other Books You May Enjoy

Leave a review - let other readers know what you think

← Prev
Back
Next →

← Prev
Back
Next →