Crate pw_tokenizer_core

pw_tokenizer_core

This crate provides the core functionality of calculating a token hash for a string or byte sequence. This is intended to provide a minimal core for both the main pw_tokenizer and pw_tokenizer_macro crates. Users should prefer depending pw_tokenizer instead of this crate.

Structs

• Hasher
  Hasher is used to calculate a token’s hash value.

Constants

• HASH_CONSTANT
• TOKENIZER_ENTRY_MAGIC

Functions

• hash_bytes
  Calculate the hash for a sequence of bytes.
• hash_bytes_fixed
  Calculate the hash for a sequence of bytes, examining at most len bytes.
• hash_string
  Calculate the hash for a string.