gRPC is a concise, fast, and powerful remote procedure call protocol. It leverages Google’s ProtoBuf as the wire format and HTTP/2 as transport. Besides simple RPC invocations, gRPC supports additional metadata and client and/or server-side streaming of messages (providing support for server push messages). Getting started can be daunting. Understanding the internals is even more challenging. Assuming prior knowledge of HTTP/2, can we use HTTP/2 and ProtoBuf by hand to create a gRPC client? This article implements a rudimentary client for educational purposes based on the Python library httpx.

Protocol Buffer

HTTP/2 is used as the transport protocol of gRPC. However, before we can send our first gRPC message with a normal HTTP/2 client, we first need to understand its wire format: Protocol Buffers. This article uses the Hello World from the gRPC guides. The greeter service is deployed at grpc-helloworld.sauerburger.com. The full definition of the Greeter service is hosted on GitHub. The service is defined in ProtoBuf’s own syntax. Important for this article are the message structs. The service expects a HelloRequest message as input and returns a HelloReply. Let’s focus on the request. The message is defined as follows.

message HelloRequest {
  string name = 1;
}

Each request message has a single field, a name field of type string. The number assignment (=1) is ProtoBuf’s way of enumerating the field. The field id will be important in the following. We could, and in fact, you should always, use Protocol Buffer compiler protoc to generate serialization and parsing code for your programming language of choice and use that code to build request payloads. However, for this article, we will dissect the structure of the message by hand.

Let’s assume we want to send a request to the Greeter service with name = "World". We expect to get a “Hello World” greeting in response. How to encode this as a HelloRequest message?

One fundamental concept in ProtoBuf is the variable length integer or varint. How many bytes do you need to encode a 32-bit unsigned integer? Well, 4 bytes. If we know that most of the time we’ll be dealing with smaller numbers, is there a more efficient encoding? Yes: variable length integers. To encode an integer in that way, we need to chop it into 7-bit chunks (with padding to the left). Each chunk is prefixed by a single bit: 1 indicating that the next byte is also part of the same integer encoding, and 0 indicating that it’s the last byte. For example, the integer 150 encoded as a varint is \x96\x01 in Python byte string notation. Protocol Buffer makes heavy use of this technique. However, for our simple examples in this article, we don’t really need it.

With varints out of the way, let’s encode our name = "World" message. The first step requires encoding our string using UTF-8. Since our name consists solely of ASCII characters, the binary version of does not need further explanation.

Each field in a ProtoBuf message is encoded using the “tag-length-value” scheme. This means that it first specifies the field identifier, in our case 1, and its wire type, here a variable length string, then the string’s length as a varint, and finally the UTF-8-encoded string itself. The combination of field identifier and wire type is also encoded as a varint. However, in our example, the field information and the length are less than 128, so it looks like a normal 1-byte integer.

The HelloRequest message specifies only a single, mandatory, and non-repeatable field. The resulting bytes \x0a\x05World is exactly what ProtoBuf’s SerializeToString() function would return if we used protoc to generate the serializer code. You can verify this yourself by following this end-to-end example.

gRPC

The wire format of gRPC consists of more than just plain ProtoBuf messages. The first byte in a gRPC request indicates the compression algorithm for the whole body. We’ll stick to \x00 to disable compression. A gRPC body can contain a stream of ProtoBuf messages. Bidirectional message streaming is one of the strengths of gRPC. Each ProtoBuf message is prefixed by its length encoded as a big-endian 32-bit unsigned integer. This allows clients and servers to break the stream into individual ProtoBuf messages, but it also limits the size of each individual message. For our trivial service, we’ll not use message streams and content ourselves with just a single message.

The encoding procedure is illustrated in the following schema.

gRPC and proto buffer message structure

This was the fast introduction to Protocol Buffer’s wire format as we need it for a simple gRPC service. Feel free to dive into all the nitty gritty details.

The final message that we want to send as HTTP/2 body is \x00\x00\x00\x00\x07\x0a\x05World in Python byte string notation.

The last missing puzzle piece is the appropriate HTTP headers.

The client

It’s time to build our rudimentary client and send the message that we assembled in the previous section. The two key ingredients are

  • force HTTP/2 in the httpx client, and
  • the appropriate HTTP headers.

On a superficial level, gRPC is just HTTP with Protocol Buffer bodies. The first header that we need to take care of is the Content-Type. Set it to application/grpc. gRPC uses trailing headers, a HTTP feature, that’s currently not supported by httpx. Usually in HTTP, the header key-value are sent before the message body. With trailing headers, some header fields might be sent after the HTTP body. gRPC uses this for example to set the status code. A request might fail after an initial part of the body was already sent. The trailing status code header can indicate failure in these cases. The last header, the grpc-accept-encoding: identity instructs the server to not use any compression. It would complicate the decoding for use, and it probably wouldn’t make our message any smaller anyway. So, let’s put this together.

import httpx
client = httpx.Client(http1=False, http2=True)
response = client.post(
  "https://grpc-helloworld.sauerburger.io/helloworld.Greeter/SayHello",
  headers={
    "content-type": "application/grpc",
    "te": "trailers",
    "grpc-accept-encoding": "identity",
  },
  data=b"\x00\x00\x00\x00\x07\x0a\x05World"
)
print(response.content)

The script prints b'\x00\x00\x00\x00\r\n\x0bHello World'.

The response

To decode the response, we can follow the same steps as above, just in reverse.

  • 00 (hex), 1 byte, no compression,
  • 00 00 00 0d (hex), 32-bit integer, message length: 13 bytes,
  • 0000 1010 (bin), var int, field identifier 1 in wire type 2, i.e. string or byte
  • 0000 1011 (bin), var int, string length: 11 bytes,
  • Hello World (utf-8), string value of the message field.

In short, the server replied with “Hello World”.

Summary

Is it instructive, to reverse engineer the gRPC protocol with Protocol Buffer and httpx? Definitively.

Should you do this anywhere else? No, this is clearly only for educational purposes. For everything else, use the auto-generated gRPC client and server.