When building distributed services, you’re communicating between the services over a network. To send data (such as your structs) over a network, you need to encode the data in a format to transmit, and lots of programmers choose JSON. When you’re building public APIs or you’re creating a project where you don’t control the clients, JSON makes sense because it’s accessible—both for humans to read and computers to parse. But when you’re building private APIs or building projects where you do control the clients, you can make use of a mechanism for structuring and transmitting data that—compared to JSON—makes you more productive and helps you create services that are faster, have more features, and have fewer bugs.
So what is this mechanism? Protocol buffers (also known as protobuf), which is Google’s language and platform-neutral extensible mechanism for structuring and serializing data. The advantages of using protobuf are that it:
Protobuf lets you define how you want your data structured, compile your protobuf into code in potentially many languages, and then read and write your structured data to and from different data streams. Protocol buffers are good for communicating between two systems (such as microservices), which is why Google used protobuf when building gRPC to develop a high-performance remote procedure call (RPC) framework.
If you haven’t worked with protobuf before, you may have some of the same concerns I had—that protobuf seems like a lot of extra work. I promise you that, after working with it in this chapter and the rest of the book, you’ll see that it’s really not so bad. It offers many advantages over JSON, and it’ll end up saving you a lot of work.
Here’s a quick example that shows what protocol buffers look like and how they work. Imagine you work at Twitter and one of the object types you work with are Tweets. Tweets, at the very least, comprise the author’s message. If you defined this in protobuf, it would look like this:
| | syntax = "proto3"; |
| | |
| | package twitter; |
| | |
| | message Tweet { |
| | string message = 1; |
| | } |
You’d then compile this protobuf into code in the language of your choice. For example, the protobuf compiler would take this protobuf and generate the following Go code:
| | // Code generated by protoc-gen-go. DO NOT EDIT. |
| | // source: example.proto |
| | |
| | package twitter |
| | |
| | type Tweet struct { |
| | Message string `protobuf:"bytes,1,opt,name=message,proto3" |
| | json:"message,omitempty"` |
| | // Note: Protobuf generates internal fields and methods |
| | // I haven't included for brevity. |
| | } |
But why not just write that Go code yourself? Why use protobuf instead? I’m glad you asked.