Monitoring With Graphite by Dixon, Jason -- Read -- Imperial Library of Trantor

Index

Monitoring with Graphite Preface

Who Should Read This Book Why I Wrote This Book A Word on Monitoring Today Navigating This Book Online Resources Conventions Used in This Book Safari® Books Online How to Contact Us Acknowledgments

1. What is Graphite?

What is time series data? Time-Series Databases

Storage Considerations Prioritizing Operations

What is the history of Graphite? What makes Graphite unique?

Simple Metrics Format Graphing API Rapid Prototyping Rich Statistical Library Chained Functions

Case Studies: Who uses Graphite in production?

Booking.com GitHub Etsy Electronic Arts

Why should I use Graphite?

2. Monitoring Conventions

Three Tenets of Monitoring

Fault Detection Alerting Capacity Planning

Rethinking the Poll/Pull Model

Pull Model Push Model

Where does Graphite Fit into the Picture? Composable Monitoring Systems

Telemetry

Collection Agents and Sensors Application Instrumentation Logging Business Telemetry

Metrics Router Aggregation State Engine Notification Routers Storage Engine Visualization

Conclusion

3. Graphite Components: The Moving Parts

Carbon

carbon-cache carbon-relay

Consistent-Hashing Rules-based Routing Aggregated-Consistent-Hashing

carbon-aggregator Filtering Metrics Internal Statistics Network Security Considerations

Whisper

How do Whisper files get created? Retention Policies and Archives The Laws of Whisper Archives Calculating Whisper file sizes Deconstructing a Whisper file Which archive handles my query? Aggregation Methods xFilesFactor Planning Your Namespaces Performance Considerations

Graphite-Web

Django Framework Webserver Database

SQLite PostgreSQL vs MySQL

Memcached Events Storage Backends

RRD Ceres Cyanite

Putting it all together

Basic Setup Vertical Scaling Horizontal Scaling Multi-Site Replication

A Final Thought

4. Building Your First Graphite Server

Installation

Are there containers or images already available?

Running Synthesize in Vagrant Running Synthesize Manually Next Steps with Synthesize

Where does Graphite store all my files? Are packages available for my distro? What installation methods are available? Should I use Virtualenv? Using sudo effectively Dependencies

Debian/Ubuntu RHEL/Fedora/CentOS

Installing from Source

Preparing your web database Configuring Carbon

carbon.conf storage-schemas.conf storage-aggregation.conf Some final preparations Starting your Carbon daemons

Configuring Graphite-Web

local_settings.py Setting up Apache

Verifying your Graphite installation

Carbon Statistics Feeding new data to Carbon Building your first graph

Loading the web interface Adding Metrics to a Graph

5. The Graphite User Interface

Finding Metrics

Navigating the Tree

Metrics MyGraphs and User Graphs

Using the Search feature Working smarter with the Auto-Completer Wildcards

The Graphite Composer window The Embedded Chart The Toolbar

Selecting Recent Data Refreshing the Graph Selecting a Date Range Exporting a Short URL Loading a Graph from URL Saving to My Graphs Deleting from My Graphs

The Graph Options menu

Adding a Graph Title Overriding the Graph Legend Toggling Axes and the Grid Applying a Graph Template Line Chart Modes

Slope Line Staircase Line Connected Line Connected Line Limit Draw Null as Zero

Area and Stacked Graphs

First Only Stacked All Filled Area Alpha Value

Tweaking the Y-Axis

A Not-So-Quick Example Y-Axis Labels Upper and Lower Boundaries

The Graph Data dialog

What are Targets anyways? Building a Carbon Performance graph

Adding metric targets Adding functions to targets Removing functions from targets

Sharing your work

6. Troubleshooting Graphite Performance

First, the Basics The Troubleshooting Toolbelt

Generating Metrics and Benchmarking CPU Utilization Disk Performance (I/O)

iostat iotop

Networking

ping tcpdump netstat iftop

Inspecting Metrics

whisper-info.py whisper-fetch.py whisper-dump.py API text formats

Configuration Settings

Carbon

Noisy Logging Cache and Whisper Performance Message Queueing and Forwarding

Graphite-Web

Logging

Carbon

console.log creates.log updates.log query.log listener.log clients.log aggregator.log

Graphite-Web

exception.log rendering.log cache.log info.log

Kernel messages

Failure Scenarios

The Full Disk

Orphaned Whisper files Corrupted Whisper files Logfile rotation

CPU Saturation Rendering Problems

Networking Failures Mismatched Schema Slow Rendering or Timeouts

Taking it to the next level

About the Author Copyright

← Prev
Back
Next →

← Prev
Back
Next →