pw::tokenizer::Detokenizer Class Reference

Overview

Decodes and detokenizes from a token database. This class builds a hash table of tokens to give O(1) token lookups.

Public Member Functions
	Detokenizer (const TokenDatabase &database)
	Detokenizer (DomainTokenEntriesMap &&database)
	Constructs a detokenizer by directly passing the parsed database.
DetokenizedString	Detokenize (const span< const std::byte > &encoded, std::string_view domain=kDefaultDomain) const
DetokenizedString	Detokenize (const span< const uint8_t > &encoded, std::string_view domain=kDefaultDomain) const
	Overload of Detokenize for span<const uint8_t>.
DetokenizedString	Detokenize (std::string_view encoded, std::string_view domain=kDefaultDomain) const
	Overload of Detokenize for std::string_view.
DetokenizedString	Detokenize (const void *encoded, size_t size_bytes, std::string_view domain=kDefaultDomain) const
	Overload of Detokenize for a pointer and length.
DetokenizedString	RecursiveDetokenize (const span< const std::byte > &encoded, std::string_view domain=kDefaultDomain) const
DetokenizedString	RecursiveDetokenize (const span< const uint8_t > &encoded, std::string_view domain=kDefaultDomain) const
	Overload of Detokenize for span<const uint8_t>.
DetokenizedString	RecursiveDetokenize (std::string_view encoded, std::string_view domain=kDefaultDomain) const
	Overload of Detokenize for std::string_view.
DetokenizedString	RecursiveDetokenize (const void *encoded, size_t size_bytes, std::string_view domain=kDefaultDomain) const
	Overload of Detokenize for a pointer and length.
DetokenizedString	DetokenizeBase64Message (std::string_view text) const
std::string	DetokenizeText (std::string_view text) const
std::string	DecodeOptionallyTokenizedData (const span< const std::byte > &optionally_tokenized_data)
const DomainTokenEntriesMap &	database () const
span< const TokenizedStringEntry >	DatabaseLookup (uint32_t token, std::string_view domain) const

Static Public Member Functions
static Result< Detokenizer >	FromElfSection (span< const std::byte > elf_section)
static Result< Detokenizer >	FromElfSection (span< const uint8_t > elf_section)
	Overload of FromElfSection for a uint8_t span.
static Result< Detokenizer >	FromElfFile (stream::SeekableReader &stream)
static Result< Detokenizer >	FromCsv (std::string_view csv)
	Constructs a detokenizer from a CSV database.

Constructor & Destructor Documentation

Detokenizer()

pw::tokenizer::Detokenizer::Detokenizer ( const TokenDatabase &  database )

explicit

Constructs a detokenizer from a TokenDatabase. The TokenDatabase is not referenced by the Detokenizer after construction; its memory can be freed.

Member Function Documentation

DecodeOptionallyTokenizedData()

std::string pw::tokenizer::Detokenizer::DecodeOptionallyTokenizedData

(

const span< const std::byte > &

optionally_tokenized_data

)

Decodes data that may or may not be tokenized, such as proto fields marked as optionally tokenized.

This function currently only supports Base64 nested tokenized messages. Support for hexadecimal-encoded string literals will be added.

This function currently assumes when data is not tokenized it is printable ASCII. Otherwise, the returned string will be base64-encoded.

Parameters

[in]

optionally_tokenized_data

Data optionally tokenized.

Returns

The decoded text if successfully detokenized or if the data is printable, otherwise returns the data base64-encoded.

Detokenize()

DetokenizedString pw::tokenizer::Detokenizer::Detokenize ( const span< const std::byte > &  encoded, std::string_view  domain = kDefaultDomain  ) const

inline

Decodes and detokenizes the binary encoded message. Returns a DetokenizedString that stores all possible detokenized string results.

DetokenizeBase64Message()

DetokenizedString pw::tokenizer::Detokenizer::DetokenizeBase64Message

(

std::string_view

text

)

const

Decodes and detokenizes a Base64-encoded message. Returns a DetokenizedString that stores all possible detokenized string results.

DetokenizeText()

std::string pw::tokenizer::Detokenizer::DetokenizeText ( std::string_view  text ) const

inline

Decodes and detokenizes nested tokenized messages in a string.

This function currently only supports Base64 nested tokenized messages. Support for hexadecimal-encoded string literals will be added.

Parameters

[in]

text

Text potentially containing tokenized messages.

Returns

The original string with nested tokenized messages decoded in context. Messages that fail to decode are left as-is.

FromElfFile()

static Result< Detokenizer > pw::tokenizer::Detokenizer::FromElfFile ( stream::SeekableReader &  stream )

static

Constructs a detokenizer from the .pw_tokenizer.entries section of an ELF binary.

FromElfSection()

static Result< Detokenizer > pw::tokenizer::Detokenizer::FromElfSection ( span< const std::byte >  elf_section )

static

Constructs a detokenizer from the .pw_tokenizer.entries section of an ELF binary.

RecursiveDetokenize()

DetokenizedString pw::tokenizer::Detokenizer::RecursiveDetokenize ( const span< const std::byte > &  encoded, std::string_view  domain = kDefaultDomain  ) const

inline

Decodes and detokenizes the binary encoded message. Returns a DetokenizedString that stores all possible detokenized string results.

---

The documentation for this class was generated from the following file:

• pw_tokenizer/public/pw_tokenizer/detokenize.h