Macro pw_tokenizer::token

macro_rules! token {
    ($string:literal) => { ... };
}

Return the u32 token for the specified string and add it to the token database.

This is where the magic happens in pw_tokenizer! … and by magic we mean hiding information in a special linker section that ends up in the final elf binary but does not get flashed to the device.

Two things are accomplished here:

1. The string is hashed into its stable u32 token. This is the value that is returned from the macro.
2. A token database entry is generated, assigned to a unique static symbol, placed in a linker section named pw_tokenizer.entries.<TOKEN_HASH>. A linker script is responsible for picking these symbols up and aggregating them into a single .pw_tokenizer.entries section in the final binary.

Example

use pw_tokenizer::token;

let token = token!("hello, \"world\"");
assert_eq!(token, 3537412730);

Currently there is no support for encoding tokens to specific domains or with “fixed lengths” per pw_tokenizer_core::hash_bytes_fixed.