Most single-process implementations of event-driven programming are handled as soon as they appear in a serial fashion. Whether it is a callback-based style GUI application or full-fledged signaling in the style of the blinker library, an event-driven application usually maintains some kind of mapping between events and lists of handlers to execute whenever one of these events happens.
This style of information passing in distributed applications is usually realized through a request-response communication. A request-response is a bidirectional and obviously synchronous way of communication between services. It can definitely be a basis for simple event handling, but has many downsides that make it really inefficient in large-scale or complex systems. The biggest problem with request-response communication is that it introduces relatively high coupling between components:
- Every communicating component needs to be able to locate dependent services. In other words, event emitters need to know the network addresses of network handlers.
- A subscription happens directly in the service that emits the event. This means that, in order to create a new event connection, more than one service often has to be modified.
- Both sides of communication must agree on the communication protocol and message format. This makes potential changes more complex.
- A service that emits events must handle potential errors that are returned in responses from dependent services.
- Request-response communication often cannot be easily handled in an asynchronous way. This means that event-based architecture built on top of a request-response communication rarely benefits from concurrent processing flows.
Due to the preceding reasons, event-driven architectures are usually implemented using the concept of message queues, rather than request-response cycles. A message queue is a communication mechanism in the form of a dedicated service or library that is only concerned about the messages and their intended delivery mechanism. We've already mentioned a practical usage example of message queues in the Using task queues and delayed processing section of Chapter 14, Optimization – Some Powerful Techniques.
Message queues allow for the loose coupling of services because they isolate event emitters and handlers from each other. Event emitters publish messages directly to the queue, but don't need to care if any other service listens to its events. Similarly, event handlers consume events directly from the queue and don't need to worry about who produced the events (sometimes, information about the event emitter is important, but, in such situations, it is either in the contents of the delivered message or takes part in the message routing mechanism). In such a communication flow, there is never a direct synchronous connection between event emitters and event handlers, and all information passing happens through the queue.
In some circumstances, this decoupling can be taken to such an extreme that a single service can communicate with itself by an external queuing mechanism. This isn't so surprising, because message queues are already a great way of inter-thread communication that allows you to avoid locking (see Chapter 15, Concurrency).
Besides loose coupling, message queues (especially in the form of dedicated services) have many additional capabilities:
- Most message queues are able to provide message persistence. This means that, even if message queues service dies, no messages will be lost.
- Many message queues support message delivery/processing confirmations and allow you to define a retry mechanism for messages that fail to deliver. This, with the support of message persistency, guarantees that if a message was successfully submitted, it will eventually be processed, even in the case of transient network or service failures.
- Message queues are naturally concurrent. With various message distribution semantics (for example, fan-out and round-robin) it is a great basis of a highly scalable and distributed architecture.
When it comes to the actual implementation of the message queue, we can distinguish two major architectures:
- Brokered message queues: In this architecture, there is one service (or cluster of services) that is responsible for accepting and distributing events. The most common example of open source brokered message queue systems are RabbitMQ and Apache Kafka. A popular cloud-based service is Amazon SQS. These types of systems are most capable in terms of message persistence and built-in message delivery semantics.
- Brokerless message queues: These are implemented solely as a programming library. The leading and most popular brokerless messaging library is ZeroMQ (often spelled as ØMQ). The biggest advantage of brokerless messaging is elasticity. They trade operational simplicity (no additional centralized service or cluster of services to maintain) for feature completeness (things like persistence and complex message delivery needs to be implemented inside of services).
Both types of messaging approaches have advantages and disadvantages. In brokered message queues, there is always an additional service to maintain (in case of open source queues running on their own infrastructure) or additional entry on your cloud provider invoice (in case of cloud-based services). Such messaging systems quickly became a critical part of your architecture. If such service stops working, all your systems stop as well because of inter-service communication. What you get in return are usually systems where everything is available out-of-the-box and only a matter of proper configuration or a few API calls.
With brokerless messaging, your communication is often more distributed. What, in code, appears to be a simple event publication to some abstract channel is often just code-level abstraction for peer-to-peer communication that happens under the hood of the brokerless messaging library. This means that your system architecture does not depend on a single messaging service or cluster. Even if some services are dead, the rest of the system can still communicate with each other. The downside of this approach is that you're usually on your own when it comes to things like message persistency and delivery/processing confirmations or delivery retries. If you have such needs, you will either have to implement such capabilities directly in your services or build your own messaging broker using brokerless messaging libraries.