Network Security Through Data Analysis · From Data to Action by Collins, Michael -- Read -- Imperial Library of Trantor

Index

Preface

Audience Contents of This Book Changes Between Editions Conventions Used in This Book Using Code Examples O’Reilly Safari How to Contact Us Acknowledgments

I. Data 1. Organizing Data: Vantage, Domain, Action, and Validity

Domain Vantage

Choosing Vantage

Actions: What a Sensor Does with Data Validity and Action

Internal Validity External Validity Construct Validity Statistical Validity Attacker and Attack Issues

Further Reading

2. Vantage: Understanding Sensor Placement in Networks

The Basics of Network Layering

Network Layers and Vantage

Network Layers and Addressing

MAC Addresses

MAC format and access

IPv4 Format and Addresses IPv6 Format and Addresses Validity Challenges from Middlebox Network Data

DHCP NAT Proxies Load balancing VPNs

Further Reading

3. Sensors in the Network Domain

Packet and Frame Formats

Rolling Buffers Limiting the Data Captured from Each Packet Filtering Specific Types of Packets What If It’s Not Ethernet?

NetFlow

NetFlow v5 Formats and Fields

NetFlow v9 and IPFIX

NetFlow Generation and Collection

Data Collection via IDS

Classifying IDSs IDS as Classifier

Improving IDS Performance

Enhancing IDS Detection Configuring Snort Enhancing IDS Response Prefetching Data

Middlebox Logs and Their Impact

VPN Logs Proxy Logs NAT Logs

Further Reading

4. Data in the Service Domain

What and Why Logfiles as the Basis for Service Data Accessing and Manipulating Logfiles The Contents of Logfiles

The Characteristics of a Good Log Message Existing Logfiles and How to Manipulate Them Stateful Logfiles

Further Reading

5. Sensors in the Service Domain

Representative Logfile Formats

HTTP: CLF and ELF

Simple Mail Transfer Protocol (SMTP)

Sendmail Microsoft Exchange: Message Tracking Logs

Additional Useful Logfiles

Staged Logging LDAP and Directory Services File Transfer, Storage, and Databases

Logfile Transport: Transfers, Syslog, and Message Queues

Transfer and Logfile Rotation Syslog

Further Reading

6. Data and Sensors in the Host Domain

A Host: From the Network’s View The Network Interfaces The Host: Tracking Identity Processes

Structure

PID and PPID UID Command and path Memory, CPU, terminal, and start time

Filesystem Historical Data: Commands and Logins Other Data and Sensors: HIPS and AV Further Reading

7. Data and Sensors in the Active Domain

Discovery, Assessment, and Maintenance Discovery: ping, traceroute, netcat, and Half of nmap

Checking Connectivity: Using ping to Connect to an Address Tracerouting Using nc as a Swiss Army Multitool nmap Scanning for Discovery

Assessment: nmap, a Bunch of Clients, and a Lot of Repositories

Basic Assessment with nmap

Using Active Vantage Data for Verification Further Reading

II. Tools 8. Getting Data in One Place

High-Level Architecture

The Sensor Network The Repository

Archive Annotation Knowledge base

Query Processing Real-Time Processing Source Control

Log Data and the CRUD Paradigm A Brief Introduction to NoSQL Systems Further Reading

9. The SiLK Suite

What Is SiLK and How Does It Work? Acquiring and Installing SiLK

The Datafiles

Choosing and Formatting Output Field Manipulation: rwcut Basic Field Manipulation: rwfilter

Ports and Protocols Size IP Addresses Time TCP Options Helper Options Miscellaneous Filtering Options and Some Hacks

rwfileinfo and Provenance Combining Information Flows: rwcount rwset and IP Sets rwuniq rwbag Advanced SiLK Facilities

PMAPs

Collecting SiLK Data

YAF rwptoflow rwtuc rwrandomizeip

Further Reading

10. Reference and Lookup: Tools for Figuring Out Who Someone Is

MAC and Hardware Addresses IP Addressing

IPv4 Addresses, Their Structure, and Significant Addresses IPv6 Addresses, Their Structure, and Significant Addresses IP Intelligence: Geolocation and Demographics

DNS

DNS Name Structure Forward DNS Querying Using dig The DNS Reverse Lookup Using whois to Find Ownership DNS Blackhole Lists

Search Engines

General Search Engines Scanning Repositories, Shodan et al

Further Reading

III. Analytics

An Overview of Attacker Behavior Further Reading

11. Exploratory Data Analysis and Visualization

The Goal of EDA: Applying Analysis EDA Workflow Variables and Visualization Univariate Visualization

Histograms Bar Plots (Not Pie Charts) The Five-Number Summary and the Boxplot Generating a Boxplot

Bivariate Description

Scatterplots

Multivariate Visualization

Other Visualizations and Their Role

Pairs plots and trellising Spider plots ROC curves

Operationalizing Security Visualization

Rule one: Bound and partition your visualization to manage disruptions Rule two: Label anomalies Rule three: Use trendlines, distinguish artifacts from observations Rule four: Be consistent across plots Rule five: Annotate with contextual information Rule six: Avoid flash in favor of expressiveness Rule seven: When performing long jobs, give the user some status feedback

Fitting and Estimation

Is It Normal? Simply Visualizing: Projected Values and QQ Plots Fit Tests: K-S and S-W

Further Reading

12. On Analyzing Text

Text Encoding

Unicode, UTF, and ASCII Encoding for Attackers

Base64 encoding Informal encoding/obfuscation Compression Encryption

Basic Skills

Finding a String Manipulating Delimiters Splitting Along Delimiters Regular Expressions

Techniques for Text Analysis

N-Gram Analysis Jaccard Distance Hamming Distance Levenshtein Distance Entropy and Compressibility Homoglyphs

Further Reading

13. On Fumbling

Fumbling: Misconfiguration, Automation, and Scanning

Lookup Failures Automation Scanning

Identifying Fumbling

IP Fumbling: Dark Addresses and Spread TCP Fumbling: Failed Sessions

Unidirectional flow filtering Dark ports and UDP fumbling

ICMP Messages and Fumbling

Fumbling at the Service Level

HTTP Fumbling SMTP Fumbling DNS Fumbling

Detecting and Analyzing Fumbling

Building Fumbling Alarms Forensic Analysis of Fumbling Engineering a Network to Take Advantage of Fumbling

14. On Volume and Time

The Workday and Its Impact on Network Traffic Volume Beaconing File Transfers/Raiding Locality

DDoS, Flash Crowds, and Resource Exhaustion DDoS and Routing Infrastructure

Applying Volume and Locality Analysis

Data Selection Using Volume as an Alarm Using Beaconing as an Alarm Using Locality as an Alarm Engineering Solutions

Further Reading

15. On Graphs

Graph Attributes: What Is a Graph? Labeling, Weight, and Paths Components and Connectivity Clustering Coefficient Analyzing Graphs

Using Component Analysis as an Alarm Using Centrality Analysis for Forensics Using Breadth-First Searches Forensically Using Centrality Analysis for Engineering

Further Reading

16. On Insider Threat

Insider Threat Versus Other Classes of Attacks Avoiding Toxicity Modes of Attack

Data Theft and Exfiltration Credential Theft Sabotage

Insider Threat Data: Logistics and Collection

Applying Sector-Based Workflow to Insider Threat Physical Data Sources Keeping Track of User Identity

Further Reading

17. On Threat Intelligence

Defining Threat Intelligence

Data Types

Types of threat intelligence data Maturity and format of threat intelligence data Provenance of threat intelligence data

Creating a Threat Intelligence Program

Identifying Goals Starting with Free Sources Determining Data Output Purchasing Sources

Brief Remarks on Creating Threat Intelligence Further Reading

18. Application Identification

Mechanisms for Application Identification

Port Number Application Identification by Banner Grabbing Application Identification by Behavior Application Identification by Subsidiary Site

Application Banners: Identifying and Classifying

Non-Web Banners Web Client Banners: The User-Agent String

Further Reading

19. On Network Mapping

Creating an Initial Network Inventory and Map

Creating an Inventory: Data, Coverage, and Files Phase I: The First Three Questions

The default network

Phase II: Examining the IP Space

Identifying asymmetric traffic Identifying dark space Finding network appliances

Phase III: Identifying Blind and Confusing Traffic

Identifying NATs Identifying proxies Identifying VPN traffic

Phase IV: Identifying Clients and Servers

Identifying servers

Identifying Sensing and Blocking Infrastructure

Updating the Inventory: Toward Continuous Audit Further Reading

20. On Working with Ops

Ops Environments: An Overview Operational Workflows

Escalation Workflow Sector Workflow Hunting Workflow Hardening Workflow

A hardening scenario

Forensic Workflow Switching Workflows