Table of Contents

1. Preface
   1. Who this book is for
   2. What this book covers
   3. To get the most out of this book
      1. Download the example code files
      2. Conventions used
   4. Get in touch
      1. Reviews
1. Introducing Web Scraping and Go
   1. What is web scraping?
   2. Why do you need a web scraper?
      1. Search engines
      2. Price comparison
      3. Building datasets
   3. What is Go?
   4. Why is Go a good fit for web scraping?
      1. Go is fast
      2. Go is safe
      3. Go is simple
   5. How to set up a Go development environment
      1. Go language and tools
      2. Git
      3. Editor
   6. Summary
2. The Request/Response Cycle
   1. What do HTTP requests look like?
      1. HTTP request methods
      2. HTTP headers
      3. Query parameters
      4. Request body
   2. What do HTTP responses look like?
      1. Status line
      2. Response headers
      3. Response body
   3. What are HTTP status codes?
      1. 100–199 range
      2. 200–299 range
      3. 300–399 range
      4. 400–499 range
      5. 500–599 range
   4. What do HTTP requests/responses look like in Go?
      1. A simple request example
   5. Summary
3. Web Scraping Etiquette
   1. What is a robots.txt file?
   2. What is a User-Agent string?
      1. Example
   3. How to throttle your scraper
   4. How to use caching
      1. Cache-Control
      2. Expires
      3. Etag
      4. Caching content in Go
   5. Summary
4. Parsing HTML
   1. What is the HTML format?
      1. Syntax
      2. Structure
   2. Searching using the strings package
      1. Example – Counting links
      2. Example – Doctype check
   3. Searching using the regexp package
      1. Example – Finding links
      2. Example – Finding prices
   4. Searching using XPath queries
      1. Example – Daily deals
      2. Example – Collecting products
   5. Searching using Cascading Style Sheets selectors
      1. Example – Daily deals
      2. Example – Collecting products
   6. Summary
5. Web Scraping Navigation
   1. Following links
      1. Example – Daily deals
   2. Submitting forms
      1. Example – Submitting searches
      2. Example – POST method
   3. Avoiding loops
   4. Breadth-first versus depth-first crawling
      1. Depth-first
      2. Breadth-first
   5. Navigating with JavaScript
      1. Example – Book reviews
   6. Summary
6. Protecting Your Web Scraper
   1. Virtual private servers
   2. Proxies
      1. Public and shared proxies
      2. Dedicated proxies
         1. Price
         2. Location
         3. Type
         4. Anonymity
      3. Proxies in Go
   3. Virtual private networks
   4. Boundaries
      1. Whitelists
      2. Blacklists
   5. Summary
7. Scraping with Concurrency
   1. What is concurrency
   2. Concurrency pitfalls
      1. Race conditions
      2. Deadlocks
   3. The Go concurrency model
      1. Goroutines
      2. Channels
   4. sync package helpers
      1. Conditions
      2. Atomic counters
   5. Summary
8. Scraping at 100x
   1. Components of a web scraping system
      1. Queue
      2. Cache
      3. Storage
      4. Logs
   2. Scraping HTML pages with colly
   3. Scraping JavaScript pages with chrome-protocol
      1. Example – Amazon Daily Deals
   4. Distributed scraping with dataflowkit
      1. The Fetch service
      2. The Parse service
   5. Summary
9. Other Books You May Enjoy
   1. Leave a review - let other readers know what you think