pw_protobuf Encoding

Usage

Pigweed’s protobuf encoders encode directly to the wire format of a proto rather than staging information to a mutable datastructure. This means any writes of a value are final, and can’t be referenced or modified as a later step in the encode process.

MemoryEncoder

A MemoryEncoder directly encodes a proto to an in-memory buffer.

// Writes a proto response to the provided buffer, returning the encode
// status and number of bytes written.
StatusWithSize WriteProtoResponse(ByteSpan response) {
  // All proto writes are directly written to the `response` buffer.
  MemoryEncoder encoder(response);
  encoder.WriteUint32(kMagicNumberField, 0x1a1a2b2b);
  encoder.WriteString(kFavoriteFood, "cookies");
  return StatusWithSize(encoder.status(), encoder.size());
}

StreamingEncoder

pw_protobuf’s StreamingEncoder class operates on pw::stream::Writer objects to serialized proto data. This means you can directly encode a proto to something like pw::sys_io without needing to build the complete message in memory first.

#include "pw_protobuf/streaming_encoder.h"
#include "pw_stream/sys_io_stream.h"
#include "pw_bytes/span.h"

pw::stream::SysIoWriter sys_io_writer;
pw::protobuf::StreamingEncoder my_proto_encoder(sys_io_writer,
                                                pw::ByteSpan());

// Once this line returns, the field has been written to the Writer.
my_proto_encoder.WriteInt64(kTimestampFieldNumber, system::GetUnixEpoch());

// There's no intermediate buffering when writing a string directly to a
// StreamingEncoder.
my_proto_encoder.WriteString(kWelcomeMessageFieldNumber,
                             "Welcome to Pigweed!");
if (!my_proto_encoder.status().ok()) {
  PW_LOG_INFO("Failed to encode proto; %s", my_proto_encoder.status().str());
}

Nested submessages

Writing proto messages with nested submessages requires buffering due to limitations of the proto format. Every proto submessage must know the size of the submessage before its final serialization can begin. A streaming encoder can be passed a scratch buffer to use when constructing nested messages. All submessage data is buffered to this scratch buffer until the submessage is finalized. Note that the contents of this scratch buffer is not necessarily valid proto data, so don’t try to use it directly.

MemoryEncoder objects use the final destination buffer rather than relying on a scratch buffer. Note that this means your destination buffer might need additional space for overhead incurred by nesting submessages. The MaxScratchBufferSize() helper function can be useful in estimating how much space to allocate to account for nested submessage encoding overhead.

#include "pw_protobuf/streaming_encoder.h"
#include "pw_stream/sys_io_stream.h"
#include "pw_bytes/span.h"

pw::stream::SysIoWriter sys_io_writer;
// The scratch buffer should be at least as big as the largest nested
// submessage. It's a good idea to be a little generous.
std::byte submessage_scratch_buffer[64];

// Provide the scratch buffer to the proto encoder. The buffer's lifetime must
// match the lifetime of the encoder.
pw::protobuf::StreamingEncoder my_proto_encoder(sys_io_writer,
                                                submessage_scratch_buffer);

StreamingEncoder nested_encoder =
    my_proto_encoder.GetNestedEncoder(kPetsFieldNumber);

// There's no intermediate buffering when writing a string directly to a
// StreamingEncoder.
nested_encoder.WriteString(kNameFieldNumber, "Spot");
nested_encoder.WriteString(kPetTypeFieldNumber, "dog");
// Since this message is only nested one deep, the submessage is serialized to
// the writer as soon as we finalize the submessage.
PW_CHECK_OK(nested_encoder.Finalize());

{  // If a nested_encoder is destroyed it will silently Finalize().
  StreamingEncoder nested_encoder_2 =
      my_proto_encoder.GetNestedEncoder(kPetsFieldNumber);
  nested_encoder_2.WriteString(kNameFieldNumber, "Slippers");
  nested_encoder_2.WriteString(kPetTypeFieldNumber, "rabbit");
}  // When this scope ends, the nested encoder is serialized to the Writer.

// If an encode error occurs when encoding the nested messages, it will be
// reflected at the root encoder.
if (!my_proto_encoder.status().ok()) {
  PW_LOG_INFO("Failed to encode proto; %s", my_proto_encoder.status().str());
}

Warning

When a nested submessage is created, any writes to the parent encoder that created the nested encoder will trigger a crash. To resume writing to a parent encoder, Finalize() the submessage encoder first.

Error Handling

While individual write calls on a proto encoder return pw::Status objects, the encoder tracks all status returns and “latches” onto the first error encountered. This status can be accessed via StreamingEncoder::status().

Codegen

pw_protobuf encoder codegen integration is supported in GN, Bazel, and CMake. The codegen is just a light wrapper around the StreamEncoder and MemoryEncoder objects, providing named helper functions to write proto fields rather than requiring that field numbers are directly passed to an encoder. Namespaced proto enums are also generated, and used as the arguments when writing enum fields of a proto message.

All generated messages provide a Fields enum that can be used directly for out-of-band encoding, or with the pw::protobuf::Decoder.

This module’s codegen is available through the *.pwpb sub-target of a pw_proto_library in GN, CMake, and Bazel. See pw_protobuf_compiler’s documentation for more information on build system integration for pw_protobuf codegen.

Example BUILD.gn:

import("//build_overrides/pigweed.gni")

import("$dir_pw_build/target_types.gni")
import("$dir_pw_protobuf_compiler/proto.gni")

# This target controls where the *.pwpb.h headers end up on the include path.
# In this example, it's at "pet_daycare_protos/client.pwpb.h".
pw_proto_library("pet_daycare_protos") {
  sources = [
    "pet_daycare_protos/client.proto",
  ]
}

pw_source_set("example_client") {
  sources = [ "example_client.cc" ]
  deps = [
    ":pet_daycare_protos.pwpb",
    dir_pw_bytes,
    dir_pw_stream,
  ]
}

Example pet_daycare_protos/client.proto:

syntax = "proto3";
// The proto package controls the namespacing of the codegen. If this package
// were fuzzy.friends, the namespace for codegen would be fuzzy::friends::*.
package fuzzy_friends;

message Pet {
  string name = 1;
  string pet_type = 2;
}

message Client {
  repeated Pet pets = 1;
}

Example example_client.cc:

#include "pet_daycare_protos/client.pwpb.h"
#include "pw_protobuf/streaming_encoder.h"
#include "pw_stream/sys_io_stream.h"
#include "pw_bytes/span.h"

pw::stream::SysIoWriter sys_io_writer;
std::byte submessage_scratch_buffer[64];
// The constructor is the same as a pw::protobuf::StreamingEncoder.
fuzzy_friends::Client::StreamEncoder client(sys_io_writer,
                                            submessage_scratch_buffer);

fuzzy_friends::Pet::StreamEncoder pet1 = client.GetPetsEncoder();

pet1.WriteName("Spot");
pet1.WritePetType("dog");
PW_CHECK_OK(pet1.Finalize());

{  // Since pet2 is scoped, it will automatically Finalize() on destruction.
  fuzzy_friends::Pet::StreamEncoder pet2 = client.GetPetsEncoder();
  pet2.WriteName("Slippers");
  pet2.WritePetType("rabbit");
}

if (!client.status().ok()) {
  PW_LOG_INFO("Failed to encode proto; %s", client.status().str());
}