Building Event-Driven Microservices by Adam Bellemare -- Read -- Imperial Library of Trantor

Index

Preface

Conventions Used in This Book O’Reilly Online Learning How to Contact Us Acknowledgments

1. Why Event-Driven Microservices

What Are Event-Driven Microservices? Introduction to Domain-Driven Design and Bounded Contexts

Leveraging Domain Models and Bounded Contexts Aligning Bounded Contexts with Business Requirements

Communication Structures

Business Communication Structures Implementation Communication Structures Data Communication Structures Conway’s Law and Communication Structures

Communication Structures in Traditional Computing

Option 1: Make a New Service Option 2: Add It to the Existing Service Pros and Cons of Each Option The Team Scenario, Continued Conflicting Pressures

Event-Driven Communication Structures

Events Are the Basis of Communication Event Streams Provide the Single Source of Truth Consumers Perform Their Own Modeling and Querying Data Communication Is Improved Across the Organization Accessible Data Supports Business Communication Changes

Asynchronous Event-Driven Microservices

Example Team Using Event-Driven Microservices

Synchronous Microservices

Drawbacks of Synchronous Microservices

Point-to-point couplings Dependent scaling Service failure handling API versioning and dependency management Data access tied to the implementation Distributed monoliths Testing

Benefits of Synchronous Microservices

Summary

2. Event-Driven Microservice Fundamentals

Building Topologies

Microservice Topology Business Topology

The Contents of an Event The Structure of an Event

Unkeyed Event Entity Event Keyed Event

Materializing State from Entity Events Event Data Definitions and Schemas Microservice Single Writer Principle Powering Microservices with the Event Broker

Event Storage and Serving Additional Factors to Consider

Support tooling Hosted services Client libraries and processing frameworks Community support Long-term and tiered storage

Event Brokers Versus Message Brokers

Consuming from the Immutable Log

Consuming as an event stream Consuming as a queue

Providing a Single Source of Truth

Managing Microservices at Scale

Putting Microservices into Containers Putting Microservices into Virtual Machines Managing Containers and Virtual Machines

Paying the Microservice Tax Summary

3. Communication and Data Contracts

Event-Driven Data Contracts

Using Explicit Schemas as Contracts Schema Definition Comments Full-Featured Schema Evolution Code Generator Support Breaking Schema Changes

Accommodating breaking schema changes for entities Accommodating breaking schema changes for events

Selecting an Event Format Designing Events

Tell the Truth, the Whole Truth, and Nothing but the Truth Use a Singular Event Definition per Stream Use the Narrowest Data Types Keep Events Single-Purpose

Example: Overloading event definitions

Minimize the Size of Events Involve Prospective Consumers in the Event Design Avoid Events as Semaphores or Signals

Summary

4. Integrating Event-Driven Architectures with Existing Systems

What Is Data Liberation?

Compromises for Data Liberation Converting Liberated Data to Events

Data Liberation Patterns Data Liberation Frameworks Liberating Data by Query

Bulk Loading Incremental Timestamp Loading Autoincrementing ID Loading Custom Querying Incremental Updating Benefits of Query-Based Updating Drawbacks of Query-Based Updating

Liberating Data Using Change-Data Capture Logs

Benefits of Using Data Store Logs Drawbacks of Using Data Base Logs

Liberating Data Using Outbox Tables

Performance Considerations Isolating Internal Data Models Ensuring Schema Compatibility

Benefits of event-production with outbox tables Drawbacks of event production with outbox tables

Capturing Change-Data Using Triggers

Benefits of using triggers Drawbacks of using triggers

Making Data Definition Changes to Data Sets Under Capture

Handling After-the-Fact Data Definition Changes for the Query and CDC Log Patterns Handling Data Definition Changes for Change-Data Table Capture Patterns

Sinking Event Data to Data Stores The Impacts of Sinking and Sourcing on a Business Summary

5. Event-Driven Processing Basics

Composing Stateless Topologies

Transformations Branching and Merging Streams

Repartitioning Event Streams

Example: Repartitioning an Event Stream

Copartitioning Event Streams

Example: Copartitioning an Event Stream

Assigning Partitions to a Consumer Instance

Assigning Partitions with the Partition Assignor Assigning Copartitioned Partitions Partition Assignment Strategies

Round-robin assignment Static assignment Custom assignment

Recovering from Stateless Processing Instance Failures Summary

6. Deterministic Stream Processing

Determinism with Event-Driven Workflows Timestamps

Synchronizing Distributed Timestamps Processing with Timestamped Events

Example: Selecting order of events when processing multiple partitions

Event Scheduling and Deterministic Processing

Custom Event Schedulers Processing Based on Event Time, Processing Time, and Ingestion Time Timestamp Extraction by the Consumer Request-Response Calls to External Systems

Watermarks

Watermarks in Parallel Processing

Stream Time

Stream Time in Parallel Processing

Out-of-Order and Late-Arriving Events

Late Events with Watermarks and Stream Time Causes and Impacts of Out-of-Order Events

Sourcing from out-of-order data Multiple producers to multiple partitions

Time-Sensitive Functions and Windowing

Tumbling windows Sliding windows Session windows

Handling Late Events Reprocessing Versus Processing in Near-Real Time Intermittent Failures and Late Events Producer/Event Broker Connectivity Issues Summary and Further Reading

7. Stateful Streaming

State Stores and Materializing State from an Event Stream Recording State to a Changelog Event Stream Materializing State to an Internal State Store

Materializing Global State Advantages of Using Internal State

Scalability requirements are offloaded from the developer High-performance disk-based options Flexibility to use network-attached disk

Disadvantages of Using Internal State

Limited to using runtime-defined disk Wasted disk space

Scaling and Recovery of Internal State

Using hot replicas Restoring and scaling from changelogs Restoring and scaling from input event streams

Materializing State to an External State Store

Advantages of External State

Full data locality Technology

Drawbacks of External State

Management of multiple technologies Performance loss due to network latency Financial cost of external state store services Full data locality

Scaling and Recovery with External State Stores

Using the source streams Using changelogs Using snapshots

Rebuilding Versus Migrating State Stores

Rebuilding Migrating

Transactions and Effectively Once Processing

Example: Stock Accounting Service Effectively Once Processing with Client-Broker Transactions Effectively Once Processing Without Client-Broker Transactions

Generating duplicate events Identifying duplicate events Guarding against duplicates Maintaining consistent state

Summary

8. Building Workflows with Microservices

The Choreography Pattern

A Simple Event-Driven Choreography Example Creating and Modifying a Choreographed Workflow Monitoring a Choreographed Workflow

The Orchestration Pattern

A Simple Event-Driven Orchestration Example A Simple Direct-Call Orchestration Example Comparing Direct-Call and Event-Driven Orchestration Creating and Modifying an Orchestration Workflow Monitoring the Orchestration Workflow

Distributed Transactions

Choreographed Transactions: The Saga Pattern

Choreography Example

Orchestrated Transactions

Compensation Workflows Summary

9. Microservices Using Function-as-a-Service

Designing Function-Based Solutions as Microservices

Ensure Strict Membership to a Bounded Context Commit Offsets Only After Processing Has Completed

When the function has completed its processing When the function has first started

Less Is More

Choosing a FaaS Provider Building Microservices Out of Functions Cold Start and Warm Starts Starting Functions with Triggers

Triggering Based on New Events: The Event-Stream Listener Triggering Based on Consumer Group Lag Triggering on a Schedule Triggering Using Webhooks Triggering on Resource Events

Performing Business Work with Functions Maintaining State Functions Calling Other Functions

Event-Driven Communication Pattern Direct-Call Pattern

Choreography and asychronous function calling Orchestration and synchronous function calling

Event stream—triggered processing Queue-triggered event processing

Termination and Shutdown Tuning Your Functions

Allocating Sufficient Resources Batch Event-Processing Parameters

Scaling Your FaaS Solutions Summary

10. Basic Producer and Consumer Microservices

Where Do BPCs Work Well?

Integration with Existing and Legacy Systems

Example: Sidecar pattern

Stateful Business Logic That Isn’t Reliant Upon Event Order

Example: Book publishing

When the Data Layer Does Much of the Work Independent Scaling of the Processing and Data Layer

Example: Perform aggregations on event data to build user engagement profiles

Hybrid BPC Applications with External Stream Processing

Example: Using an External Stream-Processing Framework to Join Event Streams

Summary

11. Heavyweight Framework Microservices

A Brief History of Heavyweight Frameworks The Inner Workings of Heavyweight Frameworks Benefits and Limitations Cluster Setup Options and Execution Modes

Use a Hosted Service Build Your Own Full Cluster Create Clusters with CMS Integration

Deploying and running the cluster using the CMS Specifying resources for a single job using the CMS

Application Submission Modes

Driver Mode Cluster Mode

Handling State and Using Checkpoints Scaling Applications and Handling Event Stream Partitions

Scaling an Application While It Is Running Scaling an Application by Restarting It Autoscaling Applications

Recovering from Failures Multitenancy Considerations Languages and Syntax Choosing a Framework Example: Session Windowing of Clicks and Views Summary

12. Lightweight Framework Microservices

Benefits and Limitations Lightweight Processing Handling State and Using Changelogs Scaling Applications and Recovering from Failures

Event Shuffling State Assignment State Replication and Hot Replicas

Choosing a Lightweight Framework

Apache Kafka Streams Apache Samza: Embedded Mode

Languages and Syntax Stream-Table-Table Join: Enrichment Pattern Summary

13. Integrating Event-Driven and Request-Response Microservices

Handling External Events

Autonomously Generated Events Reactively Generated Events

Handling Autonomously Generated Analytical Events Integrating with Third-Party Request-Response APIs Processing and Serving Stateful Data

Serving Real-Time Requests with Internal State Stores Serving Real-Time Requests with External State Stores

Serving requests via the materializing event-driven microservice Serving requests via a separate microservice

Handling Requests Within an Event-Driven Workflow

Processing Events for User Interfaces

Example: Newspaper publishing workflow (approval pattern) Separating the editor and advertiser approval services

Micro-Frontends in Request-Response Applications The Benefits of Microfrontends

Composition-Based Microservices Easy Alignment to Business Requirements

Drawbacks of Microfrontends

Potentially Inconsistent UI Elements and Styling Varying Microfrontend Performance Example: Experience Search and Review Application

Summary

14. Supportive Tooling

Microservice-to-Team Assignment System Event Stream Creation and Modification Event Stream Metadata Tagging Quotas Schema Registry Schema Creation and Modification Notifications Offset Management Permissions and Access Control Lists for Event Streams State Management and Application Reset Consumer Offset Lag Monitoring Streamlined Microservice Creation Process Container Management Controls Cluster Creation and Management

Programmatic Bringup of Event Brokers Programmatic Bringup of Compute Resources Cross-Cluster Event Data Replication Programmatic Bringup of Tooling

Dependency Tracking and Topology Visualization

Topology Example

Summary

15. Testing Event-Driven Microservices

General Testing Principles Unit-Testing Topology Functions

Stateless Functions Stateful Functions

Testing the Topology Testing Schema Evolution and Compatibility Integration Testing of Event-Driven Microservices Local Integration Testing

Create a Temporary Environment Within the Runtime of Your Test Code Create a Temporary Environment External to Your Test Code Integrate Hosted Services Using Mocking and Simulator Options Integrate Remote Services That Have No Local Options

Full Remote Integration Testing

Programmatically Create a Temporary Integration Testing Environment

Populating with events from production Populating with events from a curated testing source Creating mock events using schemas

Testing Using a Shared Environment Testing Using the Production Environment

Choosing Your Full-Remote Integration Testing Strategy Summary

16. Deploying Event-Driven Microservices

Principles of Microservice Deployment Architectural Components of Microservice Deployment

Continuous Integration, Delivery, and Deployment Systems Container Management Systems and Commodity Hardware

The Basic Full-Stop Deployment Pattern The Rolling Update Pattern The Breaking Schema Change Pattern

Eventual Migration via Two Event Streams Synchronized Migration to the New Event Stream

The Blue-Green Deployment Pattern Summary

17. Conclusion

Communication Layers Business Domains and Bounded Contexts Shareable Tools and Infrastructure Schematized Events Data Liberation and the Single Source of Truth Microservices Microservice Implementation Options Testing Deploying Final Words

Index

← Prev
Back
Next →

← Prev
Back
Next →