pw_protobuf

The protobuf module provides a lightweight interface for encoding and decoding the Protocol Buffer wire format.

Note

The protobuf module is a work in progress. Wire format encoding and decoding is supported, though the APIs are not final. C++ code generation exists for encoding, but not decoding.

Design

Unlike other protobuf libraries, which typically provide in-memory data structures to represent protobuf messages, pw_protobuf operates directly on the wire format and leaves data storage to the user. This has a few benefits. The primary one is that it allows the library to be incredibly small, with the encoder and decoder each having a code size of around 1.5K and negligible RAM usage. Users can choose the tradeoffs most suitable for their product on top of this core implementation.

pw_protobuf also provides zero-overhead C++ code generation which wraps its low-level wire format operations with a user-friendly API for processing specific protobuf messages. The code generation integrates with Pigweed’s GN build system.

Configuration

pw_protobuf supports the following configuration options.

  • PW_PROTOBUF_CFG_MAX_VARINT_SIZE: When encoding nested messages, the number of bytes to reserve for the varint submessage length. Nested messages are limited in size to the maximum value that can be varint-encoded into this reserved space.

    The values that can be set, and their corresponding maximum submessage lengths, are outlined below.

    MAX_VARINT_SIZE

    Maximum submessage length

    1 byte

    127

    2 bytes

    16,383 or < 16KiB

    3 bytes

    2,097,151 or < 2048KiB

    4 bytes (default)

    268,435,455 or < 256MiB

    5 bytes

    4,294,967,295 or < 4GiB (max uint32_t)

Usage

pw_protobuf splits wire format encoding and decoding operations. Links to the design and APIs of each are listed in below.

See also pw_protobuf_compiler for details on pw_protobuf’s build system integration.

pw_protobuf functionality

Comparison with other protobuf libraries

protobuf-lite

protobuf-lite is the official reduced-size C++ implementation of protobuf. It uses a restricted subset of the protobuf library’s features to minimize code size. However, is is still around 150K in size and requires dynamic memory allocation, making it unsuitable for many embedded systems.

nanopb

nanopb is a commonly used embedded protobuf library with very small code size and full code generation. It provides both encoding/decoding functionality and in-memory C structs representing protobuf messages.

nanopb works well for many embedded products; however, using its generated code can run into RAM usage issues when processing nontrivial protobuf messages due to the necessity of defining a struct capable of storing all configurations of the message, which can grow incredibly large. In one project, Pigweed developers encountered an 11K struct statically allocated for a single message—over twice the size of the final encoded output! (This was what prompted the development of pw_protobuf.)

To avoid this issue, it is possible to use nanopb’s low-level encode/decode functions to process individual message fields directly, but this loses all of the useful semantics of code generation. pw_protobuf is designed to optimize for this use case; it allows for efficient operations on the wire format with an intuitive user interface.

Depending on the requirements of a project, either of these libraries could be suitable.