The pw_log_tokenized module contains utilities for tokenized logging. It connects pw_log to pw_tokenizer.

C++ backend

pw_log_tokenized provides a backend for pw_log that tokenizes log messages with the pw_tokenizer module. By default, log messages are tokenized with the PW_TOKENIZE_TO_GLOBAL_HANDLER_WITH_PAYLOAD macro. The log level, 16-bit tokenized module name, and flags bits are passed through the payload argument. The macro eventually passes logs to the pw_tokenizer_HandleEncodedMessageWithPayload function, which must be implemented by the application.

Example implementation:

extern "C" void pw_tokenizer_HandleEncodedMessageWithPayload(
    pw_tokenizer_Payload payload, const uint8_t message[], size_t size) {
  // The metadata object provides the log level, module token, and flags.
  // These values can be recorded and used for runtime filtering.
  pw::log_tokenized::Metadata metadata(payload);

  if (metadata.level() < current_log_level) {

  if (metadata.flags() & HIGH_PRIORITY_LOG != 0) {
    EmitHighPriorityLog(metadata.module(), message, size);
  } else {
    EmitLowPriorityLog(metadata.module(), message, size);

See the documentation for pw_tokenizer for further details.

Using a custom macro

Applications may use their own macro instead of PW_TOKENIZE_TO_GLOBAL_HANDLER_WITH_PAYLOAD by setting the PW_LOG_TOKENIZED_ENCODE_MESSAGE config macro. This macro should take arguments equivalent to PW_TOKENIZE_TO_GLOBAL_HANDLER_WITH_PAYLOAD:

PW_LOG_TOKENIZED_ENCODE_MESSAGE(log_metadata, message, ...)
  • log_metadata (pw_tokenizer_Payload) – Packed metadata for the log message. See the Metadata class for how to unpack the details.

  • message (const char*) – The log message format string (untokenized)

For instructions on how to implement a custom tokenization macro, see Tokenize with a custom macro.

Build targets

The GN build for pw_log_tokenized has two targets: pw_log_tokenized and log_backend. The pw_log_tokenized target provides the pw_log_tokenized/log_tokenized.h header. The log_backend target implements the backend for the pw_log facade. pw_log_tokenized invokes the pw_tokenizer:global_handler_with_payload facade, which must be implemented by the user of pw_log_tokenized.

Python package

pw_log_tokenized includes a Python package for decoding tokenized logs.


Tools for working with tokenized logs.

class pw_log_tokenized.Metadata(_value: int, log_bits: int = 6, module_bits: int = 16, flag_bits: int = 10)

Parses the metadata payload sent by pw_log_tokenized.

__init__(_value: int, log_bits: int = 6, module_bits: int = 16, flag_bits: int = 10)None

Initialize self. See help(type(self)) for accurate signature.

flag_bits: int = 10
log_bits: int = 6
module_bits: int = 16