Crate pw_tokenizer_core

pw_tokenizer_core

This crate provides the core functionality of calculating a token hash for a string or byte sequence. This is intended to provide a minimal core for both the main pw_tokenizer and pw_tokenizer_macro crates. Users should prefer depending pw_tokenizer instead of this crate.

Structs

Hasher

Hasher is used to calculate a token’s hash value.

Constants

HASH_CONSTANT

TOKENIZER_ENTRY_MAGIC

Functions

hash_bytes

Calculate the hash for a sequence of bytes.

hash_bytes_fixed

Calculate the hash for a sequence of bytes, examining at most len bytes.

hash_string

Calculate the hash for a string.