"Inside every well-written large program is a well-written small program."
- Tony Hoare
Pipelines are a fairly standard and used way to segregate the processing of data into multiple stages. In this chapter, we will be exploring the basic principles behind data-processing pipelines and present a blueprint for implementing generic, concurrent-safe, and reusable pipelines using Go primitives, such as channels, contexts, and go-routines.
In this chapter, you will learn about the following:
- Designing a generic processing pipeline from scratch using Go primitives
- Approaches to modeling pipeline payloads in a generic way
- Strategies for dealing with errors that can occur while a pipeline is executing
- Pros and cons of synchronous and asynchronous pipeline design
- Applying pipeline design concepts to building the Links 'R' Us crawler component