Defining a job queue RPC service

We will be taking a slightly unorthodox approach and start by defining our one and only RPC first. The reason for this is that the selection of the RPC type (unary versus stream) will greatly influence the way we define the various payloads.

For example, if we opt to use a streaming RPC, we will need to define a kind of envelope message that can represent the different types of messages exchanged between the master and the workers. On the other hand, if we decide in favor of unary RPCs, we can presumably define multiple methods and avoid the need for envelope messages.

Without further ado, let's take a look at the RPC definition for our job queue:

service JobQueue {
  rpc JobStream(stream WorkerPayload) returns (stream MasterPayload);
}

As you can see, we will actually be using a bi-directional streaming RPC! This comes with a cost; we need to define two envelope messages, one for workers and one for the master. So, what was the deciding factor that drove us to the ostensibly more complicated solution of bi-directional streaming?

The answer has to do with the way that gRPC schedules messages for delivery. If you carefully examine the gRPC specification, you will notice that only streaming RPCs guarantees that messages will be delivered in the order in which they were published.

This fact is of paramount importance for our particular use case, that is, if we are not able to enforce in-order message delivery, a worker waiting on a barrier could potentially handle a message before exiting the barrier. As a result, the worker would not only behave in a non-deterministic way (good luck debugging that!), but the algorithm would also produce the wrong results.

Another benefit of the stream-based approach is that we can exploit the heartbeat mechanism that is inherently built into gRPC and efficiently detect whether a worker's connection to the master gets severed.