Hands-On Software Engineering with Golang by Achilleas Anagnostopoulos -- Read -- Imperial Library of Trantor

Index

Title Page Copyright and Credits

Hands-On Software Engineering with Golang

Dedication About Packt

Why subscribe?

Contributors

About the author About the reviewer Packt is searching for authors like you

Preface

Who this book is for What this book covers To get the most out of this book

Download the example code files Code in Action Download the color images Conventions used

Get in touch

Reviews

Section 1: Software Engineering and the Software Development Life Cycle A Bird's-Eye View of Software Engineering

What is software engineering? Types of software engineering roles

The role of the software engineer (SWE) The role of the software development engineer in test (SDET) The role of the site reliability engineer (SRE) The role of the release engineer (RE) The role of the system architect

A list of software development models that all engineers should know

Waterfall Iterative enhancement Spiral Agile

Lean

Eliminate waste Create knowledge Defer commitment Build in quality Deliver fast Respect and empower people See and optimize the whole

Scrum

Scrum roles Essential Scrum events

Kanban

DevOps

The CAMS model The three ways model

Summary Questions Further reading

Section 2: Best Practices for Maintainable and Testable Go Code Best Practices for Writing Clean and Maintainable Go Code

The SOLID principles of object-oriented design

Single responsibility Open/closed principle Liskov substitution Interface segregation Dependency inversion Applying the SOLID principles

Organizing code into packages

Naming conventions for Go packages Circular dependencies

Breaking circular dependencies via implicit interfaces Sometimes, code repetition is not a bad idea!

Tips and tools for writing lean and easy-to-maintain Go code

Optimizing function implementations for readability Variable naming conventions Using Go interfaces effectively Zero values are your friends Using tools to analyze and manipulate Go programs

Taking care of formatting and imports (gofmt, goimports) Refactoring code across packages (gorename, gomvpkg, fix) Improving code quality metrics with the help of linters

Summary Questions Further reading

Dependency Management

What's all the fuss about software versioning?

Semantic versioning

Comparing semantic versions Applying semantic versioning to Go packages

Managing the source code for multiple package versions

Single repository with versioned folders Single repository – multiple branches

Vendoring – the good, the bad, and the ugly

Benefits of vendoring dependencies Is vendoring always a good idea?

Strategies and tools for vendoring dependencies

The dep tool

The Gopkg.toml file The Gopkg.lock file

Go modules – the way forward Fork packages

Summary Questions Further reading

The Art of Testing

Technical requirements Unit testing

Mocks, stubs, fakes, and spies – commonalities and differences

Stubs and spies! Mocks

Introducing gomock Exploring the details of the project we want to write tests for Leveraging gomock to write a unit test for our application

Fake objects

Black-box versus white-box testing for Go packages – an example

The services behind the facade Writing black-box tests Boosting code coverage via white-box tests

Table-driven tests versus subtests

Table-driven tests Subtests The best of both worlds

Using third-party testing frameworks

Integration versus functional testing

Integration tests Functional tests Functional tests part deux – testing in production!

Smoke tests Chaos testing – breaking your systems in fun and interesting ways! Tips and tricks for writing tests

Using environment variables to set up or skip tests Speeding up testing for local development Excluding classes of tests via build flags This is not the output you are looking for – mocking calls to external binaries Testing timeouts is easy when you have all the time in the world!

Summary Questions Further reading

Section 3: Designing and Building a Multi-Tier System from Scratch The Links 'R'; Us Project

System overview – what are we going to be building? Selecting an SDLC model for our project

Iterating faster using an Agile framework Elephant carpaccio – how to iterate even faster!

Requirements analysis

Functional requirements

User story – link submission User story – search User story – crawl link graph User story – calculate PageRank scores User story – monitor Links 'R' Us health

Non-functional requirements

Service-level objectives Security considerations Being good netizens

System component modeling

The crawler

The link filter The link fetcher The content extractor The link extractor

The content indexer The link provider The link graph The PageRank calculator The metrics store The frontend

Monolith or microservices? The ultimate question Summary Questions Further reading

Building a Persistence Layer

Technical requirements

Running tests that require CockroachDB Running tests that require Elasticsearch

Exploring a taxonomy of database systems

Key-value stores Relational databases NoSQL databases Document databases

Understanding the need for a data layer abstraction Designing the data layer for the link graph component

Creating an ER diagram for the link graph store Listing the required set of operations for the data access layer Defining a Go interface for the link graph

Partitioning links and edges for processing the graph in parallel Iterating Links and Edges

Verifying graph implementations using a shared test suite Implementing an in-memory graph store

Upserting links Upserting edges Looking up links Iterating links/edges Removing stale edges Setting up a test suite for the graph implementation

Scaling across with a CockroachDB-backed graph implementation

Dealing with DB migrations An overview of the DB schema for the CockroachDB implementation Upserting links Upserting edges Looking up links Iterating links/edges Removing stale edges Setting up a test suite for the CockroachDB implementation

Designing the data layer for the text indexer component

A model for indexed documents Listing the set of operations that the text indexer needs to support

Defining the Indexer interface

Verifying indexer implementations using a shared test suite An in-memory Indexer implementation using bleve

Indexing documents Looking up documents and updating their PageRank score Searching the index Iterating the list of search results Setting up a test suite for the in-memory indexer

Scaling across an Elasticsearch indexer implementation

Creating a new Elasticsearch indexer instance Indexing and looking up documents Performing paginated searches Updating the PageRank score for a document Setting up a test suite for the Elasticsearch indexer

Summary Questions Further reading

Data-Processing Pipelines

Technical requirements Building a generic data-processing pipeline in Go

Design goals for the pipeline package Modeling pipeline payloads Multistage processing

Stageless pipelines – is that even possible?

Strategies for handling errors

Accumulating and returning all errors Using a dead-letter queue Terminating the pipeline's execution if an error occurs

Synchronous versus asynchronous pipelines

Synchronous pipelines Asynchronous pipelines

Implementing a stage worker for executing payload processors

FIFO Fixed and dynamic worker pools 1-to-N broadcasting

Implementing the input source worker Implementing the output sink worker Putting it all together – the pipeline API

Building a crawler pipeline for the Links 'R' Us project

Defining the payload for the crawler Implementing a source and a sink for the crawler Fetching the contents of graph links Extracting outgoing links from retrieved webpages Extracting the title and text from retrieved web pages Inserting discovered outgoing links to the graph Indexing the contents of retrieved web pages Assembling and running the pipeline

Summary Questions Further reading

Graph-Based Data Processing

Technical requirements Exploring the Bulk Synchronous Parallel model Building a graph processing system in Go

Queueing and delivering messages

The Message interface Queues and message iterators Implementing an in-memory, thread-safe queue

Modeling the vertices and edges of graphs

Defining the Vertex and Edge types Inserting vertices and edges into the graph

Sharing global graph state through data aggregation

Defining the Aggregator interface Registering and looking up aggregators Implementing a lock-free accumulator for float64 values

Sending and receiving messages Implementing graph-based algorithms using compute functions

Achieving vertical scaling by executing compute functions in parallel Orchestrating the execution of super-steps

Creating and managing Graph instances

Solving interesting graph problems

Searching graphs for the shortest path

The sequential Dijkstra algorithm Leveraging a gossip protocol to run Dijkstra in parallel

Graph coloring

A sequential greedy algorithm for coloring undirected graphs Exploiting parallelism for undirected graph coloring

Calculating PageRank scores

The model of the random surfer An iterative approach to PageRank score calculation Reaching convergence – when should we stop iterating? Web graphs in the real world – dealing with dead ends Defining an API for the PageRank calculator Implementing a compute function to calculate PageRank scores

Summary Further reading

Communicating with the Outside World

Technical requirements Designing robust, secure, and backward-compatible REST APIs

Using human-readable paths for RESTful resources Controlling access to API endpoints

Basic HTTP authentication Securing TLS connections from eavesdropping Authenticating to external service providers using OAuth2

Dealing with API versions

Including the API version as a route prefix Negotiating API versions via HTTP Accept headers

Building RESTful APIs in Go

Building RPC-based APIs with the help of gRPC

Comparing gRPC to REST Defining messages using protocol buffers

Defining messages Versioning message definitions Representing collections Modeling field unions The Any type

Implementing RPC services

Unary RPCs Server-streaming RPCs Client-streaming RPCs Bi-directional streaming RPCs

Security considerations for gRPC APIs

Decoupling Links 'R' Us components from the underlying data stores

Defining RPCs for accessing a remote link-graph instance Defining RPCs for accessing a text-indexer instance Creating high-level clients for accessing data stores over gRPC

Summary Questions Further reading

Building, Packaging, and Deploying Software

Technical requirements Building and packaging Go services using Docker

Benefits of containerization Best practices for dockerizing Go applications Selecting a suitable base container for your application

A gentle introduction to Kubernetes

Peeking under the hood Summarizing the most common Kubernetes resource types Running a Kubernetes cluster on your laptop!

Building and deploying a monolithic version of Links 'R' Us

Distributing computation across application instances

Carving the UUID space into non-overlapping partitions Assigning a partition range to each pod

Building wrappers for the application services The crawler service The PageRank calculator service Serving a fully functioning frontend to users

Specifying the endpoints for the frontend application Performing searches and paginating results Generating convincing summaries for search results Highlighting search keywords

Orchestrating the execution of individual services Putting everything together

Terminating the application in a clean way Dockerizing and starting a single instance of the monolith

Deploying and scaling the monolith on Kubernetes

Setting up the required namespaces Deploying CockroachDB and Elasticsearch using Helm Deploying Links 'R' Us

Summary Questions Further reading

Section 4: Scaling Out to Handle a Growing Number of Users Splitting Monoliths into Microservices

Technical requirements Monoliths versus service-oriented architectures

Is there something inherently wrong with monoliths? Microservice anti-patterns and how to deal with them

Monitoring the state of your microservices

Tracing requests through distributed systems

The OpenTracing project Stepping through a distributed tracing example

The provider service The aggregator service The gateway Putting it all together

Capturing and visualizing traces using Jaeger

Making logging your trusted ally

Logging best practices The devil is in the (logging) details Shipping and indexing logs inside Kubernetes

Running a log collector on each Kubernetes node Using a sidecar container to collect logs Shipping logs directly from the application

Introspecting live Go services

Building a microservice-based version of Links 'R' Us

Decoupling access to the data stores Breaking down the monolith into distinct services Deploying the microservices that comprise the Links 'R' Us project

Deploying the link-graph and text-indexer API services Deploying the web crawler Deploying the PageRank service Deploying the frontend service

Locking down access to our Kubernetes cluster using network policies

Summary Questions Further reading

Building Distributed Graph-Processing Systems

Technical requirements Introducing the master/worker model

Ensuring that masters are highly available

The leader-follower configuration The multi-master configuration

Strategies for discovering nodes Recovering from errors

Out-of-core distributed graph processing

Describing the system architecture, requirements, and limitations Modeling a state machine for executing graph computations Establishing a communication protocol between workers and masters

Defining a job queue RPC service Establishing protocol buffer definitions for worker payloads Establishing protocol buffer definitions for master payloads

Defining abstractions for working with bi-directional gRPC streams

Remote worker stream Remote master stream

Creating a distributed barrier for the graph execution steps

Implementing a step barrier for individual workers Implementing a step barrier for the master

Creating custom executor factories for wrapping existing graph instances

The workers' executor factory The master's executor factory

Coordinating the execution of a graph job

Simplifying end user interactions with the dbspgraph package The worker job coordinator

Running a new job Transitioning through the stages of the graph's state machine Handling incoming payloads from the master Using the master as an outgoing message relay

The master job coordinator

Running a new job Transitioning through the stages for the graph's state machine Handling incoming worker payloads Relaying messages between workers

Defining package-level APIs for working with master and worker nodes

Instantiating and operating worker nodes Instantiating and operating master nodes

Handling incoming gRPC connections Running a new job

Deploying a distributed version of the Links 'R' Us PageRank calculator

Retrofitting master and worker capabilities to the PageRank calculator service

Serializing PageRank messages and aggregator values Defining job runners for the master and the worker

Implementing the job runner for master nodes The worker job runner

Deploying the final Links 'R' Us version to Kubernetes

Summary Questions Further reading

Metrics Collection and Visualization

Technical requirements Monitoring from the perspective of a site reliability engineer

Service-level indicators (SLIs) Service-level objectives (SLOs) Service-level agreements (SLAs)

Exploring options for collecting and aggregating metrics

Comparing push versus pull systems Capturing metrics using Prometheus

Supported metric types Automating the detection of scrape targets

Static and file-based scrape target configuration Querying the underlying cloud provider Leveraging the API exposed by Kubernetes

Instrumenting Go code

Registering metrics with Prometheus Vector-based metrics Exporting metrics for scraping

Visualizing collected metrics using Grafana Using Prometheus as an end-to-end solution for alerting

Using Prometheus as a source for alert events Handling alert events

Grouping alerts together Selectively muting alerts Configuring alert receivers Routing alerts to receivers

Summary Questions Further reading

Epilogue Assessments

Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 6 Chapter 7 Chapter 8 Chapter 9 Chapter 10 Chapter 11 Chapter 12 Chapter 13

Other Books You May Enjoy

Leave a review - let other readers know what you think

← Prev
Back
Next →

← Prev
Back
Next →