Three strategies can be used for solving the discovery and load balancing problem:
Server proxying—your client sends its requests to a load balancer that knows the servers (either by querying a service registry or by being the service registry) and proxies the requests to your back-end services.
External load balancing—your client queries an external load-balancing service that knows the servers and tells the client which server to send the RPC.
Client-side balancing—your client queries a service registry to learn about the servers, picks the server to send its RPC, and sends its RPC directly to the server.
Using a server proxy is the most commonly used discovery and load-balancing pattern. Most servers don’t trust their clients enough to give them control over how load balancing works because these decisions might affect the service’s availability (for example, allowing a client to target a single server and call it until it’s unavailable). You can put a proxy between clients and servers to act as a trust boundary. The proxy lets you control how your system ingests requests, as all the networking behind the proxy is in your network, trusted, and under your control. The server proxy knows about the servers it proxies to by maintaining or calling a service registry. People often use AWS’s Elastic Load Balancer (ELB) to load balance external traffic from the internet. The ELB is an example of a service-side discovery router—incoming requests hit the ELB, and the ELB proxies that request to one instance registered with the ELB.
For complex and very accurate load balancing, you can run an external load balancer. The external load balancer knows all the servers and potentially all the clients, so it has all the data to decide the best server for the client to call. You pay for external load balancers with operational burden. I’ve never needed an external load balancer.
Alternatively, you can use client-side load balancing when you trust the clients. Client-side load balancing reduces latency and increases efficiency because requests go directly to their destination—there are no intermediates. This load balancing pattern is resilient because there isn’t a single point of failure. However, you need to work on your network and security to give clients direct access to your servers.
We’ll build client-side discovery and load balancing into our service because we control both the client and server and we designed our service for low-latency, high-throughput applications.