Pigweed
C/C++ API Reference
|
Public Member Functions | |
Detokenizer (const TokenDatabase &database) | |
Detokenizer (DomainTokenEntriesMap &&database) | |
Constructs a detokenizer by directly passing the parsed database. | |
DetokenizedString | Detokenize (const span< const std::byte > &encoded, std::string_view domain=kDefaultDomain) const |
DetokenizedString | Detokenize (const span< const uint8_t > &encoded, std::string_view domain=kDefaultDomain) const |
Overload of Detokenize for span<const uint8_t> . | |
DetokenizedString | Detokenize (std::string_view encoded, std::string_view domain=kDefaultDomain) const |
Overload of Detokenize for std::string_view . | |
DetokenizedString | Detokenize (const void *encoded, size_t size_bytes, std::string_view domain=kDefaultDomain) const |
Overload of Detokenize for a pointer and length. | |
DetokenizedString | RecursiveDetokenize (const span< const std::byte > &encoded, std::string_view domain=kDefaultDomain) const |
DetokenizedString | RecursiveDetokenize (const span< const uint8_t > &encoded, std::string_view domain=kDefaultDomain) const |
Overload of Detokenize for span<const uint8_t> . | |
DetokenizedString | RecursiveDetokenize (std::string_view encoded, std::string_view domain=kDefaultDomain) const |
Overload of Detokenize for std::string_view . | |
DetokenizedString | RecursiveDetokenize (const void *encoded, size_t size_bytes, std::string_view domain=kDefaultDomain) const |
Overload of Detokenize for a pointer and length. | |
DetokenizedString | DetokenizeBase64Message (std::string_view text) const |
std::string | DetokenizeText (std::string_view text) const |
std::string | DecodeOptionallyTokenizedData (const span< const std::byte > &optionally_tokenized_data) |
const DomainTokenEntriesMap & | database () const |
span< const TokenizedStringEntry > | DatabaseLookup (uint32_t token, std::string_view domain) const |
Static Public Member Functions | |
static Result< Detokenizer > | FromElfSection (span< const std::byte > elf_section) |
static Result< Detokenizer > | FromElfSection (span< const uint8_t > elf_section) |
Overload of FromElfSection for a uint8_t span. | |
static Result< Detokenizer > | FromElfFile (stream::SeekableReader &stream) |
static Result< Detokenizer > | FromCsv (std::string_view csv) |
Constructs a detokenizer from a CSV database. | |
Decodes and detokenizes from a token database. This class builds a hash table of tokens to give O(1)
token lookups.
|
explicit |
Constructs a detokenizer from a TokenDatabase
. The TokenDatabase
is not referenced by the Detokenizer
after construction; its memory can be freed.
std::string pw::tokenizer::Detokenizer::DecodeOptionallyTokenizedData | ( | const span< const std::byte > & | optionally_tokenized_data | ) |
Decodes data that may or may not be tokenized, such as proto fields marked as optionally tokenized.
This function currently only supports Base64 nested tokenized messages. Support for hexadecimal-encoded string literals will be added.
This function currently assumes when data is not tokenized it is printable ASCII. Otherwise, the returned string will be base64-encoded.
[in] | optionally_tokenized_data | Data optionally tokenized. |
|
inline |
Decodes and detokenizes the binary encoded message. Returns a DetokenizedString
that stores all possible detokenized string results.
DetokenizedString pw::tokenizer::Detokenizer::DetokenizeBase64Message | ( | std::string_view | text | ) | const |
Decodes and detokenizes a Base64-encoded message. Returns a DetokenizedString
that stores all possible detokenized string results.
|
inline |
Decodes and detokenizes nested tokenized messages in a string.
This function currently only supports Base64 nested tokenized messages. Support for hexadecimal-encoded string literals will be added.
[in] | text | Text potentially containing tokenized messages. |
|
static |
Constructs a detokenizer from the .pw_tokenizer.entries
section of an ELF binary.
|
static |
Constructs a detokenizer from the .pw_tokenizer.entries
section of an ELF binary.
|
inline |
Decodes and detokenizes the binary encoded message. Returns a DetokenizedString
that stores all possible detokenized string results.