pw_log_tokenized#

Stable

The pw_log_tokenized module contains utilities for tokenized logging. It connects pw_log to pw_tokenizer and supports Tokenized log arguments.

C++ backend#

pw_log_tokenized provides two backends for pw_log that tokenize log messages with the pw_tokenizer module. One backend includes metadata, and the other omits it to save code size.

With metadata#

The default //pw_log_tokenized backend passes the log level, 16-bit tokenized module name, and flag bits through a metadata argument. The macro eventually passes logs to the pw_log_tokenized_HandleLog() function, which must be implemented by the application.

Example implementation:

extern "C" void pw_log_tokenized_HandleLog(uint32_t metadata,
                                           const uint8_t message[],
                                           size_t size) {
  // The metadata object provides the log level, module token, and flags.
  // These values can be recorded and used for runtime filtering.
  pw::log_tokenized::Metadata info(metadata);

  if (info.level() < current_log_level) {
    return;
  }

  if (info.flags() & HIGH_PRIORITY_LOG != 0) {
    EmitHighPriorityLog(info.module(), message, size);
  } else {
    EmitLowPriorityLog(info.module(), message, size);
  }
}

Without metadata#

The //pw_log_tokenized:light backend omits the metadata entirely to save code size. The macro eventually passes logs to the pw_log_tokenized_HandleLogWithoutMetadata() function, which must be implemented by the application.

Example implementation:

extern "C" void pw_log_tokenized_HandleLogWithoutMetadata(
    const uint8_t message[], size_t size) {
  // Since no metadata is provided, applications might route these logs
  // to a default stream without filtering.
  EmitLog(message, size);
}

See the documentation for pw_tokenizer for further details.

Metadata in the format string#

With tokenized logging, the log format string is converted to a 32-bit token. Regardless of how long the format string is, it’s always represented by a 32-bit token. Because of this, metadata can be packed into the tokenized string with no cost.

pw_log_tokenized uses a simple key-value format to encode metadata in a format string. Each field starts with the ■ (U+25A0 “Black Square”) character, followed by the key name, the ♦ (U+2666 “Black Diamond Suit”) character, and then the value. The string is encoded as UTF-8. Key names are comprised of alphanumeric ASCII characters and underscore and start with a letter.

"■key1♦contents1■key2♦contents2■key3♦contents3"

This format makes the message easily machine parseable and human readable. It is extremely unlikely to conflict with log message contents due to the characters used.

pw_log_tokenized uses three fields: msg, module, and file. Implementations may add other fields, but they will be ignored by the pw_log_tokenized tooling.

"■msg♦Hyperdrive %d set to %f■module♦engine■file♦propulsion/hyper.cc"

Using key-value pairs allows placing the fields in any order. pw_log_tokenized places the message first. This is prefered when tokenizing C code because the tokenizer only hashes a fixed number of characters. If the file were first, the long path might take most of the hashed characters, increasing the odds of a collision with other strings in that file. In C++, all characters in the string are hashed, so the order is not important.

The format string is created by the PW_LOG_TOKENIZED_FORMAT_STRING macro.

The metadata bit field#

pw_log_tokenized packs runtime-accessible metadata into a 32-bit integer which is passed as the metadata argument for pw_log_tokenizer’s global handler with metadata facade. Packing this metadata into a single word rather than separate arguments reduces the code size significantly.

Four items are packed into the metadata argument (when used):

  • Log level – Used for runtime log filtering by level.

  • Line number – Used to track where a log message originated.

  • Log flags – Implementation-defined log flags.

  • Tokenized PW_LOG_MODULE_NAME – Used for runtime log filtering by module.

For applications that do not need log metadata at runtime, pw_log_tokenized provides a macro that omits the metadata to save code size: PW_LOG_TOKENIZED_TO_GLOBAL_HANDLER (used by the //pw_log_tokenized:light backend). If this macro is used as the pw_log backend, the log is routed to the pw_log_tokenized_HandleLogWithoutMetadata() function, bypassing the typical log handler.

Configuring metadata bit fields#

The number of bits to use for each metadata field is configurable through macros in pw_log/config.h. The field widths must sum to 32 bits. A field with zero bits allocated is excluded from the log metadata.

Creating and reading Metadata#

pw_log_tokenized provides GenericMetadata to facilitate the creation and interpretation of packed log Metadata.

The following example shows that a Metadata object can be created from a uint32_t log metadata.

extern "C" void pw_log_tokenized_HandleLog(uint32_t metadata,
                                           const uint8_t message[],
                                           size_t size_bytes) {
  pw::log_tokenized::Metadata info = metadata;
  // Check the log level to see if this log is a crash.
  if (info.level() == PW_LOG_LEVEL_FATAL) {
    HandleCrash(info,
                pw::ConstByteSpan(reinterpret_cast<const std::byte*>(message),
                                  size_bytes));
    PW_UNREACHABLE;
  }
  // ...
}

It’s also possible to get a uint32_t representation of a Metadata object:

// Logs an explicitly created string token.
void LogToken(uint32_t token, int level, int line_number, int module) {
  const uint32_t metadata =
      log_tokenized::Metadata(level, module, PW_LOG_FLAGS, line_number).value();
  std::array<std::byte, sizeof(token)> token_buffer =
      pw::bytes::CopyInOrder(endian::little, token);

  pw_log_tokenized_HandleLog(
      metadata,
      reinterpret_cast<const uint8_t*>(token_buffer.data()),
      token_buffer.size());
}

The binary tokenized message may be encoded in the prefixed Base64 format with the pw::log_tokenized::PrefixedBase64Encode() function.

Parsing metadata fields#

The metadata fields packed into the format string can be parsed with the pw::log_tokenized::ParseFields() function. This function takes a string and a callback that is called for each key-value pair.

Build targets#

The build for pw_log_tokenized provides two backend targets for the pw_log facade: //pw_log_tokenized and //pw_log_tokenized:light. Both targets provide the pw_log_tokenized/log_tokenized.h header.

  • //pw_log_tokenized routes logs to the pw_log_tokenized:handler facade, which must be implemented by the user.

  • //pw_log_tokenized:light routes logs to the pw_log_tokenized:light_handler facade, which must also be implemented by the user.

GCC has a bug resulting in section attributes of templated functions being ignored. This in turn means that log tokenization cannot work for templated functions, because the token database entries are lost at build time. For more information see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70435. If you are using GCC, the gcc_partially_tokenized target can be used as a backend for the pw_log facade instead which tokenizes as much as possible and uses the pw_log_string:handler for the rest using string logging.

Python package#

pw_log_tokenized includes a Python package for decoding tokenized logs.

pw_log_tokenized#

Tools for working with tokenized logs.

class pw_log_tokenized.FormatStringWithMetadata(string: str)#

Parses metadata from a log format string with metadata fields.

__init__(string: str) → None#
property file: str#
property message: str#

Displays the msg field or the whole string if it is not present.

property module: str#
class pw_log_tokenized.Metadata(value: int, log_bits: int = 3, line_bits: int = 11, flag_bits: int = 2, module_bits: int = 16)#

Parses the metadata payload used by pw_log_tokenized.

__init__(value: int, log_bits: int = 3, line_bits: int = 11, flag_bits: int = 2, module_bits: int = 16) → None#
flag_bits: int = 2#
line_bits: int = 11#
log_bits: int = 3#
module_bits: int = 16#
value: int#
On this page