Wire protocol#

pw_rpc2: Next gen, zero-copy async RPC with end-to-end backpressure

Pigweed RPC2 uses a binary-framed wire protocol designed for low processing overhead. Each packet consists of a fixed-size header (dependent on the packet type) sharing a common prefix followed by type-specific fields. In packets with payloads, the payload is located immediately following the header.

All multi-byte numeric fields are encoded in little-endian byte order.

Connection state and establishment#

As RPC2 runs over a generic Socket interface, a newly established connection must first identify that the peer is a compatible RPC endpoint before regular data flow can begin.

To achieve this, RPC2 uses two distinct categories of packets:

Handshake packets are used during the initial establishment phase. They contain magic numbers which identify the sender as an RPC endpoint.
RPC packets are used for regular data exchange. After the handshake completes, all subsequent packets over the connection are one of these.

Three-way handshake#

Each connection initiated by an RPC client and accepted by a server begins with a three-way handshake loosely modeled after TCP. Its purpose is to identify the endpoints and ensure compatibility. In the future, it may be extended to support negotiation of connection parameters.

Handshake flow#

The handshake begins when the client opens the connection.

The client sends a SYN (type 1) with its protocol version (currently always 1).
The server responds with a SYN_ACK (type 2) containing the minimum of its version and the client’s (currently also 1).
The client sends an ACK (type 3), accepting the server’s version, to complete the handshake.

As there is currently only version 1, this handshake primarily establishes that both endpoints speak RPC. However, clients and servers should be written to accept newer versions from the peer, falling back if they aren’t supported.

If either endpoint receives an invalid or unexpected packet during the handshake, it should terminate the connection.

        sequenceDiagram
    autonumber
    actor C as Initiator (Client)
    actor S as Responder (Server)

    C->>S: SYN (version = 1)
    Note over S: Validate magic ('PRPC')
    S->>C: SYN_ACK (negotiated_version)
    Note over C: Validate response magic and version
    C->>S: ACK (negotiated_version)
    Note over C,S: Connection established, begin RPC phase
    C->>S: START (service_id, method_id, call_id)

Handshake packet#

The handshake uses a dedicated 8-byte packet.

Offset	Size	Name	Description
0	4	`magic` (`uint32_t`)	`0x43505250` (“PRPC”)
4	1	`version` (`uint8_t`)	Protocol version, currently `1`
5	1	`type` (`uint8_t`)	`1` (SYN), `2` (SYN_ACK), or `3` (ACK)
6	2	`reserved` (`uint16_t`)	Written as `0`, ignored on receipt

RPC packets#

Once the handshake is complete, RPC packets are sent over the connection.

RPC call flows#

The diagrams below show the lifecycles of each of the four RPC call types. Packets are labeled with their type flags; see RPC packet types. Note that the server may always terminate the RPC early by sending an OK_TERMINAL packet, and either side may abort it with an ERROR_TERMINAL packet.

Unary RPC#

        sequenceDiagram
    participant C as Client
    participant S as Server

    C->>S: START | HAS_PAYLOAD | STREAM_END (request)
    S->>C: SERVER | HAS_PAYLOAD | OK_TERMINAL (response)

Server streaming RPC#

        sequenceDiagram
    participant C as Client
    participant S as Server

    C->>S: START | HAS_PAYLOAD | STREAM_END (request)
    loop Zero or more
        S->>C: SERVER | HAS_PAYLOAD (response)
    end
    S->>C: SERVER | OK_TERMINAL

Client streaming RPC#

        sequenceDiagram
    participant C as Client
    participant S as Server

    C->>S: START
    loop Zero or more
        C->>S: HAS_PAYLOAD (request)
    end
    C->>S: STREAM_END
    S->>C: SERVER | HAS_PAYLOAD | OK_TERMINAL (response)

Bidirectional streaming RPC#

        sequenceDiagram
    participant C as Client
    participant S as Server

    C->>S: START
    par Client stream
        loop Zero or more
            C->>S: HAS_PAYLOAD (request)
        end
        C->>S: STREAM_END
    and Server stream
        loop Zero or more
            S->>C: SERVER | HAS_PAYLOAD (response)
        end
    end
    S->>C: SERVER | OK_TERMINAL

RPC packet header structure#

Every RPC packet begins with a common 5-byte header:

Offset	Size	Name	Description
0	4	`call_id` (`uint32_t`)	Client-assigned ID for this specific invocation. Selected when sending the initial request, and echoed afterwards.
4	1	`type` (`uint8_t`)	The type of packet. Determines further fields.

RPC packet types#

The type byte of an RPC packet is not an enumeration. It is a bitfield of independent properties, and each packet combines them:

Bit:    7   6   5     4   3        2          1           0
      +---+---+---+-----------+--------+-------------+---------+
      | 0 | 0 | 0 | CloseMode | START  | HAS_PAYLOAD | SERVER  |
      +---+---+---+-----------+--------+-------------+---------+

Bits	Mask	Name	Meaning
0	`00001`	`SERVER`	`0`: sent by the client. `1`: sent by the server.
1	`00010`	`HAS_PAYLOAD`	The packet carries a message, which may be empty, following its header. A packet without this bit ends with its header.
2	`00100`	`START`	The packet starts a new call. Its header includes the `service_id` and `method_id` to invoke.
4:3	`11000`	`CloseMode`	`00` (`OPEN`, `00000`): The sender’s stream stays open. `01` (`STREAM_END`, `01000`): The sender half-closes its stream. `10` (`OK_TERMINAL`, `10000`): The RPC completes successfully in both directions. `11` (`ERROR_TERMINAL`, `11000`): The RPC aborts in both directions with an error code following the header.
7:5	`0xE0`	Reserved	Must be `0`.

Only the following 14 combinations are valid. A packet of any other type is rejected as malformed.

Value	Combination	Description
`00010`	`HAS_PAYLOAD`	Client stream message.
`00011`	`SERVER \| HAS_PAYLOAD`	Server stream message.
`00100`	`START`	Opens a client or bidirectional streaming call without a message.
`00110`	`START \| HAS_PAYLOAD`	Opens a client or bidirectional streaming call with its first message.
`01000`	`STREAM_END`	Client half-closes its stream.
`01001`	`SERVER \| STREAM_END`	Server half-closes its stream. The RPC continues until the server ends it with `10001`, `10011`, or `11001`. See below.
`01010`	`HAS_PAYLOAD \| STREAM_END`	Client’s final stream message, closing its stream.
`01011`	`SERVER \| HAS_PAYLOAD \| STREAM_END`	Server’s final stream message, half-closing its stream. Like `01001`, the RPC continues until the server ends it.
`01100`	`START \| STREAM_END`	Opens a client or bidirectional streaming call and immediately closes the client’s stream without sending a message.
`01110`	`START \| HAS_PAYLOAD \| STREAM_END`	Starts a unary or server streaming call with its request, which may be empty. Also valid for a client streaming call with a single message.
`10001`	`SERVER \| OK_TERMINAL`	Server finishes its stream and completes the RPC.
`10011`	`SERVER \| HAS_PAYLOAD \| OK_TERMINAL`	Server’s response, completing a unary or client streaming RPC.
`11000`	`ERROR_TERMINAL`	Client aborts the RPC. Carries an 8-bit RPC error code.
`11001`	`SERVER \| ERROR_TERMINAL`	Server aborts the RPC. Carries an 8-bit RPC error code.

A packet without HAS_PAYLOAD must end with its header. A receiver rejects any packet without HAS_PAYLOAD that has trailing bytes as malformed.

When the server ends the RPC with 10001 or 10011, the client stops sending on any stream it still has open.

Server half-close#

A server may half-close its stream with 01001 or 01011 to stop sending messages while it continues to read the client’s stream. This does not end the RPC: the server still sends a terminal packet (10001, 10011, or 11001) later, which carries the call’s final status.

The protocol allows either the client or the server to half-close its stream. The C++ server currently always ends the RPC when it closes its stream, for consistency with gRPC. Server half-close packets are valid, however, so every client must handle them: after a server half-close, a client drops any further server messages and waits for the terminal packet before completing the call.

Method type mismatches#

Because the type byte states what a packet contains, an endpoint does not rely on its own view of a method’s type to parse a packet. If the client and server disagree about a method’s type, the endpoint that receives a packet its method cannot accept fails the call with METHOD_TYPE_MISMATCH instead of misreading or dropping a message:

A unary or server streaming method takes a single request, so the server accepts only 01110 (START | HAS_PAYLOAD | STREAM_END) to start it. It rejects any other START packet with the server error METHOD_TYPE_MISMATCH without invoking the method.
A unary or client streaming call takes a single response, so the client accepts only 10011 (SERVER | HAS_PAYLOAD | OK_TERMINAL) or an error (11001) from the server. For any other packet, it sends the client error METHOD_TYPE_MISMATCH, which aborts the call on the server, before delivering anything from the packet. If the packet already ended the RPC (10001), the client sends nothing.

Either way, the client’s call fails because of METHOD_TYPE_MISMATCH. The C++ client reports this as kUnimplemented; see Error handling.

A streaming endpoint accepts 01110 or 10011, which represent a stream of a single message. A mismatch that produces only these packets completes normally because no message is lost or misread:

Client’s method type	Server’s method type	Result
Unary	Client streaming	Works: the client sends `01110` and the server replies `10011`.
Unary	Server streaming	Fails, unless the server replies with a single `10011`.
Unary	Bidirectional streaming	Fails, unless the server replies with a single `10011`.
Server streaming	Unary	Works: the server’s `10011` is a stream of one message.
Server streaming	Client streaming	Works, as above.
Server streaming	Bidirectional streaming	Works: the client sends `01110` and reads the server’s stream.
Client streaming	Unary or server streaming	Fails: the server requires `01110`.
Client streaming	Bidirectional streaming	Fails, unless the server replies with a single `10011`.
Bidirectional streaming	Unary or server streaming	Fails: the server requires `01110`.
Bidirectional streaming	Client streaming	Works: the server’s `10011` is a stream of one message.

Unexpected packets#

An endpoint handles packets that it does not expect as follows. It never replies to a terminal packet (10001, 10011, 11000, or 11001): the sender has already finished with the call, and replying to an error with an error could bounce between the endpoints forever.

A packet for the wrong endpoint. The receiver drops a packet type that only its own role may send, such as a server message arriving at a server. If the packet is not terminal, the receiver replies with RECEIVED_PACKET_FOR_WRONG_ENDPOINT. If a call with the packet’s call ID is active, it ends with a protocol error.
A non-terminal packet for an unknown call. The receiver drops the packet and replies with RECEIVED_PACKET_FOR_UNKNOWN_CALL. When both endpoints behave correctly, this only happens in a race with the end of the call: the receiver has already finished with the call, and so has the peer by the time the reply arrives, so the reply has no effect. If an active call receives RECEIVED_PACKET_FOR_UNKNOWN_CALL, the endpoints are out of sync, and the call ends with a protocol error.
A terminal packet for an unknown call. The receiver drops the packet without replying.
A START packet for an active call. The client reused the ID of a call that is still active. The server drops the new START packet, ends the active call, and replies with RECEIVED_DUPLICATE_START, which ends the call on the client. Neither call proceeds.
A message after the peer ended its stream. The receiver drops the message without replying, and still applies any stream end or terminal status in the same packet.

Packet structures#

The fields of each RPC packet are listed below.

Payload framing

RPC2 runs over a datagram socket, so each packet from the peer is received in full. There is no length field in the protocol itself. Transport implementations used with RPC must ensure that buffers read from the peer over an RPC connection contain only, and exactly, the bytes of a single packet without any padding.

Start packets (`START` set)#

Starts an RPC. Its header size is 13 bytes.

Offset	Size	Name	Description
0	5	Header	Common RPC packet header with `START` set
5	4	`service_id` (`uint32_t`)	ID of the service to invoke
9	4	`method_id` (`uint32_t`)	ID of the method to invoke
13	Variable	Payload	Remainder of the packet, present only if `HAS_PAYLOAD` is set

Error packets (`ERROR_TERMINAL` set)#

Aborts an RPC call immediately. Its header size is 6 bytes.

Offset	Size	Name	Description
0	5	Header	Common RPC packet header with `type` `11000` or `11001`
5	1	`error` (`uint8_t`)	RPC error code

All other packets#

Every other packet has a 5-byte header consisting only of the common RPC packet header. If HAS_PAYLOAD is set, the payload is the remainder of the packet.

RPC error codes#

Error packets (11000 and 11001) carry an 8-bit RPC error code. Every code names exactly one condition, so that an endpoint’s logs say precisely why the peer ended a call.

The top two bits of a code identify which endpoint may send it:

Top bits	Range	Origin
`00`	`0x00`–`0x3F`	Either endpoint. Valid in both client (`11000`) and server (`11001`) error packets, except `NONE` (`0x00`), which is never sent.
`01`	`0x40`–`0x7F`	Server only. Only valid in server error packets (`11001`).
`10`	`0x80`–`0xBF`	Client only. Only valid in client error packets (`11000`). No client-only codes are currently defined.
`11`	`0xC0`–`0xFF`	Reserved for future protocol versions. Not valid in any error packet in protocol version 1.

Each protocol version defines a set of error codes. An endpoint must not send an error code that the negotiated protocol version does not define. A receiver treats any code it does not recognize, any code that the sender may not send, and zero as a protocol violation.

As a result, a well-behaved peer never sends a code that the receiver does not know. A later protocol version may add codes. When the negotiated version is older than the newest version that an endpoint supports, the endpoint sends the closest code that the negotiated version defines.

The tables below list the codes that protocol version 1 defines. Every other value is unassigned or reserved. The C++ implementation reports each code to the application as the action it calls for; see Error handling.

Either-endpoint error codes (0x00–0x3F)#
Value	Name	Description
`0x00`	`NONE`	No error. Never sent.
`0x01`	`INTERNAL`	A bug in the RPC implementation.
`0x02`	`CANCELLED`	Application code cancelled the call, either explicitly or, on the client, by dropping a handle to the call before the call finished.
`0x03`	`METHOD_TYPE_MISMATCH`	A packet does not match the method’s type. Unary and server streaming calls must be started with `START \| HAS_PAYLOAD \| STREAM_END` (`01110`); unary and client streaming calls must be answered with `SERVER \| HAS_PAYLOAD \| OK_TERMINAL` (`10011`).
`0x04`	`RECEIVED_PACKET_FOR_WRONG_ENDPOINT`	The endpoint received a packet type that may only be sent by its own role (a server received a server-to-client packet, or a client received a client-to-server packet).
`0x05`	`RECEIVED_PACKET_FOR_UNKNOWN_CALL`	The endpoint received a non-terminal packet for a call ID that it does not know about. See Unexpected packets.

The server-only codes are in the order of a call’s lifecycle: 0x40 to 0x44 reject a call before its method runs (0x42 also ends the active call that has the same ID), 0x45 and 0x46 end a call for a reason outside the method’s control after the method started, and 0x47 and 0x48 report that the server’s application code ended the call.

Server-only error codes (0x40–0x7F)#
Value	Name	Description
`0x40`	`UNKNOWN_SERVICE`	The requested service is not registered on the server.
`0x41`	`UNKNOWN_METHOD`	The requested method is not registered on the target service.
`0x42`	`RECEIVED_DUPLICATE_START`	The server received a `START` packet for a call ID that is still active. Both calls end. See Unexpected packets.
`0x43`	`INVALID_REQUEST_PAYLOAD`	The request payload could not be decoded.
`0x44`	`FAILED_TO_ALLOCATE_CALL`	The server could not allocate state for the call. The method never ran.
`0x45`	`FAILED_TO_ALLOCATE_CALL_RESOURCES_WHILE_RUNNING`	An allocation failed after the method started.
`0x46`	`SERVICE_UNREGISTERED`	The target service was unregistered from the server while the call was running. A server that does not support unregistering services never sends this code.
`0x47`	`DROPPED_WITHOUT_RESPONSE`	The handle that a unary or client streaming method uses to respond was destroyed without responding or cancelling.
`0x48`	`METHOD_RETURNED_WITHOUT_FINISHING`	The method’s implementation completed while a handle it handed out for writing still held the call open.