Get started with pw_tokenizer#
Compress strings to shrink logs by +75%
There are two sides to
pw_tokenizer, which we call tokenization and
Tokenization converts string literals in the source code to binary tokens at compile time. If the string has printf-style arguments, these are encoded to compact binary form at runtime.
Detokenization converts tokenized strings back to the original human-readable strings.
Here’s an overview of what happens when
pw_tokenizer is used:
During compilation, the
pw_tokenizermodule hashes string literals to generate stable 32-bit tokens.
The tokenization macro removes these strings by declaring them in an ELF section that is excluded from the final binary.
After compilation, strings are extracted from the ELF to build a database of tokenized strings for use by the detokenizer. The ELF file may also be used directly.
During operation, the device encodes the string token and its arguments, if any.
The encoded tokenized strings are sent off-device or stored.
Off-device, the detokenizer tools use the token database to decode the strings to human-readable form.
Integrating with Bazel / GN / CMake projects#
pw_tokenizer requires a few steps beyond building the code. This
section describes one way
pw_tokenizer might be integrated with a project.
These steps can be adapted as needed.
pw_tokenizerto your build. Build files for GN, CMake, and Bazel are provided. For Make or other build systems, add the files specified in the BUILD.gn’s
pw_tokenizertarget to the build.
Use the tokenization macros in your code. See Tokenization.
Add the contents of
pw_tokenizer_linker_sections.ldto your project’s linker script. In GN and CMake, this step is done automatically.
Compile your code to produce an ELF file.
database.py createon the ELF file to generate a CSV token database. See Managing token databases.
Commit the token database to your repository. See notes in Database management.
database.py addcommand to your build to automatically update the committed token database. In GN, use the
pw_tokenizer_databasetemplate to do this. See Update a database.
detokenize.pyor the C++ detokenization library with your tools to decode tokenized logs. See Detokenization.
Using with Zephyr#
pw_tokenizer with Zephyr, 3 Kconfigs can be used currently:
CONFIG_PIGWEED_TOKENIZERwill automatically link
pw_tokenizeras well as any dependencies.
CONFIG_PIGWEED_TOKENIZER_BASE64will automatically link
pw_tokenizer.base64as well as any dependencies.
CONFIG_PIGWEED_DETOKENIZERwill automatically link
pw_tokenizer.decoderas well as any dependencies.
Once enabled, the tokenizer headers can be included like any Zephyr headers:
Zephyr handles the additional linker sections via
pw_tokenizer_zephyr.ld which is added to the end of the linker file
via a call to