pw_enum#

pw_enum: Rich enum support

pw_enum supports automatic stringifying and tokenizing of C++ enums. It works by parsing C++ standard header files and generating versions of those headers with minimal additions needed to support these features. Enums can also be defined in language-neutral TOML or JSON files, from which pw_enum generates equivalent C++ and Python code.

Why use pw_enum?

Efficient string or tokenized logging: Stringifies or tokenizes logs automatically for seamless logging.
Automatic content-based versioning: Generates version hashes to prevent collisions as values change.
Cross-language consistency: Generates matching C++ and Python enums from a single TOML or JSON definition.

Automatic tokenized and stringified enums#

pw_enum works on enums declared in standard C++ header files. To use pw_enum:

Declare one or more enums in a header file.
Include the header file in a versioned enum target (e.g. pw_cc_enum_source_set for GN, pw_cc_enum_library for Bazel, pw_add_enum_library for CMake).
Include pw_enum/generate.h in the header file.
Register the enum using the PW_ENUM(MyEnum, …) macro at global scope. List the fully qualified enum name, followed by all of its enumerators. If an enumerator has multiple aliases, only include one of them.

Important

The PW_ENUM macro must be called at global scope (outside of any namespace blocks, class definitions, or functions).

If PW_ENUM is called inside a namespace block, class, or function, the C++ compiler will reject it with a compilation error indicating that the template specialization of _PW_ENUM_cannot_be_used_within_namespaces must occur at global scope.

pw_enum headers are parsed during the build to support versioned tokenization and stringification with pw::EnumToString().

Example#

Declare enums in a standard C++ header and call PW_ENUM(MyEnum, …) at the bottom of the file, outside of any namespace blocks (in the global namespace).

#pragma once

#include <cstdint>

#include "pw_enum/generate.h"

namespace my::nested::pkg {

// Declare the enum as normal.
enum class MyEnum : uint8_t {
  kAlpha = 0,
  kBeta,
  kAliasedBeta = kBeta,
};

}  // namespace my::nested::pkg

PW_ENUM(my::nested::pkg::MyEnum, kAlpha, kBeta, kAliasedBeta);

Use the enum normally. It is tokenized with PW_TOKENIZE_ENUM and works with tokenized logs and pw::EnumToString().

#include "enum_example/basic_enum.h"

#include "pw_enum/to_string.h"
#include "pw_log/log.h"
#include "pw_log/tokenized_args.h"

namespace my::nested::pkg {

const char* HandleEnum(MyEnum value) {
  // Log the enum as a string or token, depending on the logging backend.
  PW_LOG_INFO("The enum value is: " MY_NESTED_PKG_MY_ENUM_FMT,
              PW_LOG_ENUM(value));

  switch (value) {
    case MyEnum::kAlpha:
      // Handle case kAlpha
      break;
    case MyEnum::kBeta:
      // Handle case kBeta
      break;
  }

  // The const char* string version of the enum is always available.
  return pw::EnumToString(value);
}

}  // namespace my::nested::pkg

Enums can reference values from other enums, even if they reside in different files and namespaces.

#pragma once

#include <cstdint>

#include "enum_example/basic_enum.h"
#include "pw_enum/generate.h"

namespace my::nested::pkg {

enum class OtherEnum : uint8_t {
  kFirst = 0,
  kSecond = static_cast<uint8_t>(my::nested::pkg::MyEnum::kBeta),
};

enum class AnotherEnum {
  kX = 1,
  kY = 2,
};

}  // namespace my::nested::pkg

PW_ENUM(my::nested::pkg::OtherEnum, kFirst, kSecond);
PW_ENUM(my::nested::pkg::AnotherEnum, kX, kY);

#pragma once

#include "enum_example/basic_enum.h"
#include "pw_enum/generate.h"

namespace my::other::pkg {

enum class ReferencesOtherEnum {
  kFirstValue = 0,
  kFromOther = static_cast<int>(my::nested::pkg::MyEnum::kAlpha),
};

}  // namespace my::other::pkg

PW_ENUM(my::other::pkg::ReferencesOtherEnum, kFirstValue, kFromOther);

pw_cc_enum_library(
    name = "advanced_enum",
    enum_headers = ["enum_example/references_other_enum.h"],
    strip_include_prefix = ".",
    deps = [":basic_enum"],
)

Enumerator names#

By default, enumerator names that follow Google’s kEnumName style are converted to upper snake case, without the k prefix (ENUM_NAME). Names that do not follow Google style are used directly.

To override the default enumerator name, specify it in the PW_ENUM(name, …) macro with a string literal after =. For example:

PW_ENUM(my::Enum,                 // String name:
        kStandardStyle,           // "STANDARD_STYLE"
        kCustom = "custom_name",  // "custom_name"
        nonStandard,              // "nonStandard"
);

Enumerator aliases#

If multiple enumerator names share the same value (aliases), they can be registered together in the PW_ENUM macro. The generator groups registered aliases, sorting their display names alphabetically and joining them with | (e.g. "ALPHA|ALIAS_ALPHA"). To omit aliases, simply leave them out of PW_ENUM.

Status enums#

pw_enum integrates seamlessly with pw_status’s pw::StatusBase to declare status-like classes wrapping custom enums. Use PW_ENUM_STATUS in place of PW_ENUM. The first argument is the fully qualified name of the status class to generate. The second argument is the enumerator that represents success, including its enum type (e.g. I2cStatusCode::kOk); its namespace is inferred from the status class if omitted. Enums in other namespaces or nested in classes must be fully qualified. The remaining arguments are the enumerators to register, as with PW_ENUM.

Place PW_ENUM_STATUS at the bottom of the header, after any declarations that reference the generated status class; otherwise, those declarations require a forward declaration of the status class.

#include <cstdint>

#include "pw_enum/generate.h"

namespace my_driver {

enum class I2cStatusCode : uint8_t {
  kOk,
  kNack,
  kBusError,
  kTimeout,
};

}  // namespace my_driver

// Generates the my_driver::I2cStatus class. The second argument is the OK
// enumerator (its namespace is inferred from the status class).
PW_ENUM_STATUS(
    my_driver::I2cStatus, I2cStatusCode::kOk, kOk, kNack, kBusError, kTimeout);

pw_cc_enum_library(
    name = "i2c_status",
    enum_headers = ["private/enum_example/i2c_status.h"],
    strip_include_prefix = "private",
)

pw_enum generates a final, [[nodiscard]] class named I2cStatus derived from pw::StatusBase, with:

I2cStatus::Code aliased to the underlying enum (e.g., I2cStatus::Code::kNack).
Static factory functions (e.g., I2cStatus::Nack(), I2cStatus::Timeout()) generated for non-OK kEnumName-style enumerators. No factory is generated for the OK code; default construct (I2cStatus()) for OK.
Is*() query functions (e.g., status.IsNack(), status.IsTimeout()) generated for kEnumName-style enumerators, including OK codes such as kReady. IsOk() is not generated for codes named OK, since it would duplicate ok().
An explicit constructor from I2cStatus::Code (e.g., I2cStatus(I2cStatus::Code::kNack)).
Full integration with PW_TRY, ok(), code(), str(), and tokenized logging.

Enumerators that do not get a factory or Is*() function are noted with a // PW_ENUM_STATUS: No ... comment in the generated header. A code named OK (e.g. kOk) that is not the OK code is an error, since it would conflict with ok().

I2cStatus WriteByte(uint8_t address, uint8_t byte) {
  if (address == 0) {
    return I2cStatus::Nack();  // Factory for I2cStatusCode::kNack
  }
  if (byte == 0xFF) {
    return I2cStatus(I2cStatus::Code::kBusError);  // Explicit construction
  }
  return I2cStatus();  // Default constructed as OK
}

I2cStatus WritePacket(uint8_t address, uint8_t header, uint8_t payload) {
  PW_TRY(WriteByte(address, header));  // Returns early on error
  PW_TRY(WriteByte(address, payload));
  return I2cStatus();
}

bool ShouldRetry(I2cStatus status) {
  return status.IsNack() || status.IsTimeout();  // Query the code
}

Logging enums#

Enums generated by pw_enum natively support Pigweed’s tokenized logging infrastructure.

Versioned format macro: pw_enum generates a macro to use in the format string for the enum. The macro is named for the namespace and enum name (e.g. MY_NESTED_PKG_MY_ENUM_FMT). The macro evaluates to a string literal that can be concatenated into a format string.

The macro is versioned based on the enum’s contents. The version changes automatically when the enum changes, so tokenized logs of enums never have collisions.

A *_DOMAIN macro (e.g. MY_NESTED_PKG_MY_ENUM_DOMAIN) is also generated with the enum’s tokenization domain, for use with nested tokenization.
Argument macro: Include pw_log/tokenized_args.h and use PW_LOG_ENUM(value) as the argument to the log statement.

When using a tokenizing logging backend, the generated format macro evaluates to PW_TOKEN_FMT(::namespace::Enum), and PW_LOG_ENUM resolves to pw::tokenizer::EnumToToken(), logging the 32-bit token. When using a standard string-based logging backend, the format macro yields the string format specifier %s, and PW_LOG_ENUM resolves to pw::EnumToString(), which yields the string representation.

Example#

  PW_LOG_INFO("State " MY_NESTED_PKG_OTHER_ENUM_FMT ": received packet",
              PW_LOG_ENUM(state));

Logging versioned enums#

If you need to log versioned enums using nested tokenization (so that the enum’s unique domain and its value are logged together to prevent collisions and support reliable detokenization), you can use the versioned logging macro:

Versioned argument macro: Include pw_log/tokenized_args.h and use PW_LOG_ENUM_VERSIONED as the argument to the log statement, paired with the format specifier PW_LOG_ENUM_VERSIONED_FMT.

When using a tokenizing logging backend, the format macro evaluates to a nested token format specifier (e.g. "${$#%08" PRIx32 "}#%08" PRIx32), and PW_LOG_ENUM_VERSIONED expands to two arguments: the 32-bit versioned domain token and the 32-bit enum value token. On non-tokenizing backends, it yields the string format specifier %s::%s and resolves to the enum’s domain name string followed by its stringified value.

Example#

  PW_LOG_INFO("State " PW_LOG_ENUM_VERSIONED_FMT() ": received packet",
              PW_LOG_ENUM_VERSIONED(state));

Build integration#

pw_enum provides build integration for Bazel, GN, and CMake.

Bazel

Use the pw_cc_enum_library rule from //pw_enum:pw_cc_enum_library.bzl.

pw_cc_enum_library(
    name = "basic_enum",
    enum_headers = ["private/pw_enum_private/basic_enum.h"],
    strip_include_prefix = "private",
    deps = [":base_enum"],
)

GN

Use the pw_cc_enum_source_set template from “$dir_pw_enum/pw_cc_enum_source_set.gni”.

pw_cc_enum_source_set("basic_enum") {
  enum_headers = [ "private/pw_enum_private/basic_enum.h" ]
  public_deps = [ ":base_enum" ]
  include_dirs = [ "private" ]
}

CMake

Use the pw_add_enum_library function from pw_enum/pw_cc_enum_library.cmake.

pw_add_enum_library(pw_enum.basic_enum
  ENUM_HEADERS
    private/pw_enum_private/basic_enum.h
  PUBLIC_INCLUDES
    private
  PUBLIC_DEPS
    pw_enum.base_enum
)

Language-neutral enums#

Enums can be defined in language-neutral TOML (.toml) or JSON (.json) files, from which pw_enum generates C++ and Python code. Use TOML or JSON when an enum’s single source of truth must be shared across languages, such as an enum used by firmware and host tools. TOML is preferred for human-authored files because it supports comments, hex and binary integer literals, and a more concise syntax; JSON is also supported using the same structure. For enums that are only used in C++, prefer standard C++ headers.

TOML- and JSON-defined enums get the same C++ features as header-defined enums: pw::EnumToString(), pw::EnumTraits, versioned tokenization, *_FMT and *_DOMAIN macros, and optional status classes.

Example#

Define enums in a TOML file:

namespace = "enum_example"

[enums.SensorState]
type = "uint8"
doc = "The state of a sensor."
values = [
  { name = "OFF", doc = "The sensor is powered off." },
  { name = "IDLE" },
  { name = "SAMPLING" },
  { name = "FAULT", value = 0x80 },
]

[enums.SensorErrorCode]
type = "uint8"
status = { name = "SensorStatus", ok = "OK" }
values = [
  { name = "OK" },
  { name = "NOT_READY" },
  { name = "OVERFLOW" },
]

Declare a pw_enum target for the file, and generate code for it with pw_cc_enum_library and pw_py_enum_library:

pw_enum(
    name = "sensor_enums",
    srcs = ["enum_example/sensor.toml"],
)

pw_cc_enum_library(
    name = "sensor_enums_cc",
    enums = [":sensor_enums"],
)

pw_py_enum_library(
    name = "sensor_enums_py",
    enums = [":sensor_enums"],
)

The generated C++ header is included at the source file’s path, with a .h extension:

// Generated from pw_enum/examples/enum_example/sensor.toml.
#include "pw_enum/examples/enum_example/sensor.h"

SensorStatus Sample(SensorState state) {
  PW_LOG_INFO("Sampling in state " ENUM_EXAMPLE_SENSOR_STATE_FMT,
              PW_LOG_ENUM(state));
  if (state != SensorState::kIdle) {
    return SensorStatus::NotReady();
  }
  return SensorStatus();
}

The generated Python module is imported at the source file’s path. Each enum is a standard enum.IntEnum that only depends on the Python standard library and is fully understood by type checkers.

# Generated from pw_enum/examples/enum_example/sensor.toml.
from pw_enum.examples.enum_example.sensor import SensorState, SensorStatus

state = SensorState(0x80)
self.assertIs(state, SensorState.FAULT)
self.assertEqual(str(state), "FAULT")  # Matches pw::EnumToString
self.assertFalse(SensorState.is_valid(4))
self.assertEqual(SensorStatus.NOT_READY, 1)  # IntEnum: compares to int
self.assertFalse(SensorStatus.NOT_READY.ok())

File format#

Each TOML or JSON file defines one or more enums in a namespace. Both formats use the same schema. Unknown fields are errors.

Field	Description
`namespace`	Required. Dot-separated namespace for the file’s enums (e.g. `my.pkg`). Sets the C++ namespace (`my::pkg`) and qualifies references from other files.
`cc.namespace`	Optional. Overrides the C++ namespace (e.g. `my::other_pkg`).
`enums.<Name>`	Required. Table/object mapping each `PascalCase` enum name to its definition. The enum name is used in every language.
`enums.<Name>.type`	Optional. Underlying type: `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, or `uint64`. Defaults to `int32`. Values must fit in the type.
`enums.<Name>.doc`	Optional. Documentation, emitted as a C++ comment and Python docstring. Lines cannot end with a backslash, which would continue the C++ comment onto the next line.
`enums.<Name>.status`	Optional. Makes the enum a status. `name` is the `PascalCase` status name, which must differ from the enum name, and `ok` names the OK enumerator. See Statuses.
`enums.<Name>.allow_alias`	Optional. Allows enumerators to share values without explicitly aliasing each other. Defaults to `false`. See Aliases.
`enums.<Name>.values`	Required. List of enumerators (in TOML, `values = [...]` or `[[enums.<Name>.values]]`).
`values[].name`	Required. `UPPER_SNAKE_CASE` enumerator name. Used as the Python name and the display name, and converted to `kPascalCase` for C++.
`values[].value`	Optional. An integer or an expression string. Defaults to the previous enumerator’s value plus one, or `0` for the first enumerator.
`values[].display_name`	Optional. Overrides the stringified and tokenized name. Display names must be printable, cannot contain `\|`, and must be unique among enumerators with different values.
`values[].doc`	Optional. Documentation for the enumerator. Lines cannot end with a backslash.
`values[].cc.name`	Optional. Overrides the C++ enumerator name.

Value expressions support integer literals (10, 0x1F, 0b101), parentheses, unary - + ~, and binary + - * << >> | & ^. Expressions may reference enumerators by name:

ALPHA: an enumerator in the same enum.
Other.ALPHA: an enumerator in an enum in the same namespace.
my.pkg.Other.ALPHA: a fully qualified enumerator.

Referenced enums must be defined in the same pw_enum target or in one of its deps. Generated code always uses the resolved integer values, so every language sees identical values. TOML and JSON enums cannot reference C++ constants; use header-defined enums for that. Header-defined enums can reference TOML- or JSON-defined enums by including their generated headers.

Aliases#

Like Protocol Buffers, pw_enum rejects enumerators that share a value, since duplicate values are easy to create by accident (e.g. with implicit values). Aliases share a token, and their display names are joined with | (e.g. "ALIAS_SECOND|SECOND").

To define an alias, set its value to the name of another enumerator in the same enum:

[enums.MyEnum]
values = [
  { name = "SECOND", value = 2 },
  { name = "ALIAS_SECOND", value = "SECOND" },
]

To allow enumerators to share values in other ways, such as through expressions or implicit values, set allow_alias = true on the enum.

Statuses#

An enum with a status describes the outcome of an operation, with one enumerator for success:

[enums.SensorErrorCode]
status = { name = "SensorStatus", ok = "OK" }
values = [
  { name = "OK" },
  { name = "NOT_READY" },
]

Each language decides how to represent the status and its codes:

C++ generates the enum class SensorErrorCode and a SensorStatus class that wraps it, exactly like PW_ENUM_STATUS (see Status enums).
Python does not distinguish between a status and its code, so it only generates class SensorStatus(enum.IntEnum). There is no SensorErrorCode class. SensorStatus has an ok() method, which returns whether it is the OK enumerator.

An enumerator named OK must be the OK enumerator or an alias of it, since it would otherwise conflict with ok().

Names and paths#

A source file’s path determines where its generated code lives in every language. Its namespace only affects languages with namespaces inside files, such as C++.

Concept	TOML / JSON	C++	Python
File	`bar/baz/foo.toml` (or `.json`)	`#include "bar/baz/foo.h"`	`import bar.baz.foo`
Namespace	`"my.pkg"`	`namespace my::pkg`	(none; the module path is the namespace)
Enum	`[enums.MyEnum]`	`enum class MyEnum`	`class MyEnum(enum.IntEnum)`
Status	`status = { name = "MyStatus", ... }`	`class MyStatus` wrapping `enum class MyEnum`	`class MyStatus(enum.IntEnum)` (no `MyEnum`)
Enumerator	`IO_ERROR`	`kIoError`	`IO_ERROR`
Display name	`display_name` or `name`	pw::EnumToString()	`str()`
Token domain	(derived)	`EnumTraits<MyEnum>::kTokenDomain`	`MyEnum.__pw_enum_token_domain__`

By default, paths are relative to the repository root in Bazel, the GN root (//) in GN, and CMAKE_SOURCE_DIR in CMake. To change the path of the generated code, set the prefix options on the pw_enum target:

Bazel: strip_import_prefix and import_prefix, which work like the attributes of the same names on proto_library.
GN: strip_prefix and prefix.
CMake: STRIP_PREFIX and PREFIX.

For example, stripping public makes public/bar/foo.toml generate #include "bar/foo.h" and import bar.foo. In GN and CMake, the default paths depend on where the project is checked out, so enums that are used across projects or build systems should always strip a prefix.

If generated Python modules are part of a package that is also defined elsewhere (e.g. in a py/ directory), extend the package’s path in its __init__.py:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

A TOML- or JSON-defined enum has the same token domain as a header-defined enum with the same C++ name, enumerator values, and display names. Enums can move between TOML/JSON and C++ headers without changing their tokens.

Build integration#

Bazel

pw_enum from //pw_enum:pw_enum.bzl declares TOML or JSON enum sources. deps lists other pw_enum targets that the sources reference.
pw_cc_enum_library from //pw_enum:pw_cc_enum_library.bzl generates C++ headers for the pw_enum targets in enums. enums may be combined with enum_headers, hdrs, and srcs, which may include the generated headers.
pw_py_enum_library from //pw_enum:pw_py_enum_library.bzl generates Python modules for the pw_enum targets in enums.

Code is generated once per pw_enum target, no matter how many libraries reference it.

GN

pw_enum from “$dir_pw_enum/pw_enum.gni” declares TOML or JSON enum sources. It generates C++ code in the <name>.cc subtarget and Python code in the <name>.python subtarget, a pw_python_package. Building <name> validates the sources.
pw_cc_enum_source_set accepts pw_enum targets in enums, which may be combined with enum_headers.

pw_enum("toml_enums") {
  sources = [ "private/pw_enum_private/toml_enums.toml" ]
  strip_prefix = "private"
  deps = [ ":json_base_enum" ]
}

pw_cc_enum_source_set("toml_enums_cc") {
  enums = [
    ":json_base_enum",
    ":toml_enums",
  ]
}

CMake

pw_add_enum from pw_enum/pw_enum.cmake declares TOML or JSON enum sources and generates C++ code in the <name>.cc library.
pw_add_enum_library accepts pw_add_enum targets in ENUMS, which may be combined with ENUM_HEADERS.

Python code is not generated in CMake.

pw_add_enum(pw_enum._toml_enums
  SOURCES
    private/pw_enum_private/toml_enums.toml
  STRIP_PREFIX
    private
  DEPS
    pw_enum._json_base_enum
)

pw_add_enum_library(pw_enum._toml_enums_cc
  ENUMS
    pw_enum._json_base_enum
    pw_enum._toml_enums
)

Stringifying enums#

pw::EnumToString() returns a string version of an enum. It uses a FTADLE extension point PwEnumToString(enum). FTADLE is a pattern that enables customization by searching for a matching function via Argument-Dependent Lookup (ADL). For more information, see Designing Extension Points With FTADLE.

If you don’t use the build-time generation targets, you can manually use PW_TOKENIZE_ENUM in pw_tokenizer/enum.h to tokenize the enum and implement PwEnumToString.

Enum traits and validation#

Enums registered with PW_ENUM automatically specialize pw::EnumTraits (defined in pw_enum/traits.h). pw::EnumTraits<EnumType> provides compile-time metadata about the enum, including its names, values, and an IsValid function. See pw::EnumTraits for the full list of members.

// A fixed-size array indexed by any contiguous PW_ENUM enum. Aliased
// enumerators share an entry, and enums that do not start at 0 are offset by
// kMin.
template <typename Enum, typename T>
class EnumArray {
 public:
  using Traits = pw::EnumTraits<Enum>;
  static_assert(Traits::kIsContiguous, "EnumArray requires a contiguous enum");

  constexpr T& operator[](Enum value) { return values_[Index(value)]; }
  constexpr const T& operator[](Enum value) const {
    return values_[Index(value)];
  }

  constexpr size_t size() const { return Traits::kDistinctValueCount; }

 private:
  static constexpr size_t Index(Enum value) {
    return static_cast<size_t>(
        static_cast<typename Traits::underlying_type>(value) -
        static_cast<typename Traits::underlying_type>(Traits::kMin));
  }

  std::array<T, Traits::kDistinctValueCount> values_{};
};

Use pw::has_enum_traits_v to detect whether an enum has traits, and pw::IsValidEnum() to validate a value:

using my::nested::pkg::MyEnum;

static_assert(pw::has_enum_traits_v<MyEnum>);

// The enum type is deduced when validating an enumerator.
static_assert(pw::IsValidEnum(MyEnum::kAlpha));

// Name the enum explicitly to validate an integer, such as a value received
// over the wire. Validating before the cast keeps out-of-range values from
// becoming enumerators that no switch case handles.
pw::Result<MyEnum> DecodeMyEnum(uint32_t wire_value) {
  if (!pw::IsValidEnum<MyEnum>(wire_value)) {
    return pw::Status::DataLoss();
  }
  return static_cast<MyEnum>(wire_value);
}

Note

Do not specialize pw::EnumTraits by hand. Specializations are identified by an internal tag that only the code generator emits, so a hand-written one is not recognized by pw::has_enum_traits_v and is rejected by pw::IsValidEnum(). Register the enum with PW_ENUM instead.

IsValid accepts an enumerator or an integer of any type, but deliberately rejects bool, since nearly everything converts implicitly to bool and would silently validate as 0 or 1. This is unrelated to the enum’s underlying type: an enum class E : bool is supported like any other enum.

Enum domain names#

pw::PwEnumDomainName() returns a fallback string domain name of a given enum type, used by non-tokenizing logging backends to prefix versioned enum logs.

By default, this function returns "Enum". If an enum uses a custom tokenization domain and you want non-tokenizing logging backends to prefix the enum values with that custom domain name instead of "Enum", you can provide a custom template specialization for the enum type:

#include "pw_enum/to_string.h"

template <>
constexpr const char* pw::PwEnumDomainName<my::CustomEnum>() {
  return "CustomDomain";
}

Enum domain tokens#

pw::tokenizer::EnumDomainToken returns the 32-bit tokenization domain token of a given enum type. It relies on the primary template pw::tokenizer::PwEnumDomainToken<T>().

To get the domain token for an enum:

#include "pw_tokenizer/enum.h"

constexpr uint32_t domain = pw::tokenizer::EnumDomainToken<MyEnum>();

If the enum is versioned using pw_cc_enum_source_set, this returns its unique versioned domain token (hash). If the enum has not been tokenized or specialized, attempting to get its domain token will result in a compile-time error.

Cross language support#

Enums defined in TOML or JSON generate C++ and Python code (see Language-neutral enums). Header-defined enums currently only generate C++. The header parser extracts the full enum definition and resolves all enumerator values, so it could also be used to generate code for other languages.

Background#

pw_tokenizer is one of Pigweed’s most widely adopted features. It has supported nested tokenization—a tokenized message inside another tokenized message—since the early days. Initially, only Base64-encoded messages were supported, which is inefficient. Support for directly encoding nested messages as 32-bit integers was added later (see 0105: Nested Tokens and Tokenized Log Arguments).

With support for encoding tokens as integers, supporting rich enums was a clear next step. This culminated in the creation of pw_tokenizer/enum.h and its supporting macros. This approach uses the enum’s integral value as a nested token, discriminated by its namespace to avoid collisions between different enum types. The result is highly efficient enum logs that are still readable and user-friendly.

The need for versioning#

Real-world deployment of pw_tokenizer/enum.h soon revealed a critical flaw. When enum values were changed or reordered during development, the resulting tokens changed. When merging token databases from different builds, this led to collisions, where the same token mapped to different string representations. It became clear that enum tokenization required versioning.

Several alternatives were explored for automatic enum versioning. A key constraint was that enum values must be able to be set with expressions, which may reference constants or other enum values. Approaches considered included:

Tokenize names instead of values: Hash the enumerator names and generate a function with a switch statement to map values to tokens at runtime.
Version in the domain: Incorporate a hash of the enum’s contents (names and values) into the tokenization domain, requiring two arguments to log an enum (the version and the value).
Calculate tokens from a base: This approach used a hash of the enum’s contents as a base offset, adding the enum value to it at the call site.

Ultimately, these approaches were ruled out because they increased code size relative to the existing implementation, primarily due to the additional code required at the call site.

The code size penalty could be avoided if there were a constexpr way to insert the enum’s version into the log format string. Then, the existing token logging macros could be used (PW_LOG_FMT). For example:

// If there were a way to define this macro during compilation, versioned
// enums would have no code size cost relative to unversioned enums.
#define MY_ENUM_FMT PW_LOG_FMT("::my::Enum::version_1234")

PW_LOG_INFO("My enum: " MY_ENUM_FMT, PW_LOG_ENUM(my_enum))

Unfortunately, there is no way get the versioned enum domain into a concatenatable string literal. This is required for compatibility with pw_log’s C-style API. If pw_log offered a C++-only API, this would be feasible, but adding such an API was out of scope.

Generating enums#

Generating enums appeared to be the only way to get the enum’s version into a string literal at compile time. This led to the creation of pw_enum.

JSON definition#

The initial implementation of pw_enum generated C++ headers from JSON files. While this worked well technically, it proved too difficult for projects to adopt due to the friction of maintaining JSON definitions for standard C++ enums. Protocol Buffers were considered in place of JSON, but they are too limited for this use case. Protobufs do not support setting enumerator values based on other enums or external constants.

Language-neutral TOML and JSON definitions later returned as an opt-in alternative to C++ headers, for enums that must be shared across languages (see Language-neutral enums).

Parse C++ source#

Finally, multiple approaches for parsing enum definitions out of C++ source code were explored. These included:

Use libclang from Python to parse header files. This would be robust and even perform constexpr evaluation of enumerator values. Unfortunately, libclang is a large dependency, and is not readily available on all platforms.
Parse clang’s -ast-dump output. This would be fairly robust, but would involve parsing moderately complex, non-standard text output intended for human consumption. It also requires clang, which not all projects build with.
Use a custom Python parser. This approach would be utterly impractical and brittle.

Ultimately, parsing C++ source directly proved infeasible. The final design avoids parsing arbitrary C++ source with the PW_ENUM(name, …) macro.

Design#

The final design of pw_enum addresses the constraints identified during its evolution by combining standard C++ header files with a specialized build-time generator powered by compile-time template evaluation.

This architecture provides a seamless user experience with zero runtime overhead and robust protection against database collisions, while maintaining the full expressiveness of standard C++ enum definitions.

Source files#

Users define enums in standard C++ header files. To opt-in to pw_enum, the header includes pw_enum/generate.h and registers each enum by calling the PW_ENUM(…) macro with the enum name and names of all of its enumerators.

The macro’s primary purpose is to capture enum metadata in an easily parsable format. PW_ENUM() expands to the enum name and list of enumerators, surrounded by unique markers. A Python script searches a preprocessed source file for the markers and extracts the enum metadata.

The macro also serves to require that users list the file in a versioned enum target (see Build system integration), which is necessary for it to be processed. If the file is not processed by pw_enum machinery, the macro expands to static_assert(false), causing the build to fail with an informative error.

Enumerator evaluation#

Enumerator values can be defined by arbitrary C++ expressions. The values may change, even if the individual source file does not change.

The parse.py script evaluates enumerators generating a source file that references them. The source file instantiates a template with the enumerators as template arguments. Compilation fails, but includes the enumerator values in an easily parsable form.

This solution is far from ideal, but has proven to be robust. It evaluates enumerators with the same toolchain as the rest of the project. Printing compile-time constants with failed template instantiations is a common workaround to achieve compile-time “printf” functionality. The script searches for a unique template name and doesn’t depend on a particular compiler or version.

Build-time generation#

After parsing, the versioned enum target runs a Python script (pw_enum/py/pw_enum/generate.py) that generates a header.

Enum generation: The script generates a “shadowed” version of the header in the build directory. This generated header contains the original content, plus a footer with tokenization metadata. It also replaces the PW_ENUM(...) calls with _PW_ENUM_GENERATED(...). PW_ENUM(...) expands to static_assert(false) to require users to build headers with versioned enum targets.
Versioning: A unique version hash is calculated for each enum based on its fully qualified name and the names and values of all its enumerators. This hash is used to construct a unique tokenization domain (e.g., ::namespace::_pw_enum_HASH::EnumName). This ensures that if the enum changes, the domain changes, preventing collisions in merged token databases.
Tokenization: The generated footer includes a call to PW_TOKENIZE_ENUM_CUSTOM from pw_tokenizer, which registers the enum values and their string representations in the database.
Enum traits: The generated footer defines a specialization of pw::EnumTraits providing compile-time metadata and validation for the enum. Enumerators that share a value (aliases) count as a single distinct value. See Enum traits and validation.

Build system integration#

The versioned enum build rules automate the process of parsing C++ headers and generating versioned enum metadata. It invokes the pw_enum generator with the correct compilation flags and ensures the generated headers are prioritized during compilation.

Bazel (pw_cc_enum_library.bzl) creates an internal library target to collect the compilation flags (includes, defines) required to parse the header correctly and passes them to the Python script.
CMake (pw_cc_enum_library.cmake) creates an internal interface library to collect includes and defines from dependencies, and uses file(GENERATE) to produce a flags file for the generator. It uses -iquote to ensure that the build system prioritizes the generated shadowed header over the original source header.
GN (pw_cc_enum_source_set.gni) compiles a placeholder C++ file with the enum’s dependencies to generate a target Ninja file. The generator script parses the target Ninja file and the toolchain’s Ninja file to extract the compiler and its compilation flags (defines, includes, and flags) to run the evaluation step. This is similar to how pw_compilation_testing works. Like CMake, the GN build uses -iquote to ensure that the build system prioritizes the generated shadowed header over the original source header.

Language-neutral enums#

TOML- and JSON-defined enums follow the same architecture as Protocol Buffers: a language-neutral source target, and language-specific rules that generate code from it.

Resolution: pw_enum/py/pw_enum/json_source.py validates TOML and JSON sources and resolves every enumerator value, including references to other enums, into the same in-memory representation that the C++ header parser produces. Resolution is pure Python and does not require a compiler. Sources may only reference enums in their own target or its direct deps.
Generation: Language-specific generators (pw_enum/py/pw_enum/generate.py and pw_enum/py/pw_enum/generate_py.py) generate code from the resolved enums. Every language uses the same resolution code, so values, display names, and versions match across languages. The C++ generator emits the enum definition followed by the same footer that header-defined enums get.
Build integration: In Bazel, pw_cc_enum_library and pw_py_enum_library attach aspects to their enums, like cc_proto_library and py_proto_library. Each aspect resolves the enum sources and generates code in a single action, once per pw_enum target, which avoids duplicate headers when several libraries reference the same enums. pw_enum targets also run a validation action, so building one reports errors without generating code. In GN and CMake, which lack aspects, the pw_enum template or function itself declares the code generation targets. GN collects the enum sources of dependencies with metadata, and CMake with generator expressions.

Python enums are generated as static source rather than created dynamically from TOML or JSON at import time. Type checkers cannot see the members of dynamically created enums. Protobuf’s Python code faces the same problem and requires separate .pyi stubs.