pw_varint#

Functions for encoding and decoding variable length integers

Stable C C++ Rust

The pw_varint module provides functions for encoding and decoding variable length integers or varints. For smaller values, varints require less memory than a fixed-size encoding. For example, a 32-bit (4-byte) integer requires 1–5 bytes when varint-encoded.

pw_varint supports custom variable-length encodings with different terminator bit values and positions ( pw::varint::Format ). The basic encoding for unsigned integers is Little Endian Base 128 (LEB128). ZigZag encoding is also supported, which maps negative integers to positive integers to improve encoding density for LEB128.

Protocol Buffers and HDLC use variable-length integer encodings for integers.

Compatibility#

API Reference#

C#

PW_VARINT_MAX_INT32_SIZE_BYTES#

Maximum size of an LEB128-encoded uint32_t.

PW_VARINT_MAX_INT64_SIZE_BYTES#

Maximum size of an LEB128-encoded uint64_t.

size_t pw_varint_Encode32(uint32_t integer, void *output, size_t output_size_bytes)#

Encodes a 32-bit integer as LEB128.

Returns:

the number of bytes written

size_t pw_varint_Encode64(uint64_t integer, void *output, size_t output_size_bytes)#

Encodes a 64-bit integer as LEB128.

Returns:

the number of bytes written

size_t pw_varint_Decode32(const void *input, size_t input_size_bytes, uint32_t *output)#

Decodes an LEB128-encoded integer to a uint32_t.

Returns:

the number of bytes read; 0 if decoding failed

size_t pw_varint_Decode64(const void *input, size_t input_size_bytes, uint64_t *output)#

Decodes an LEB128-encoded integer to a uint64_t.

Returns:

the number of bytes read; 0 if decoding failed

static inline uint32_t pw_varint_ZigZagEncode32(int32_t n)#

Zig-zag encodes an int32_t, returning it as a uint32_t.

static inline uint64_t pw_varint_ZigZagEncode64(int64_t n)#

Zig-zag encodes an int64_t, returning it as a uint64_t.

static inline int32_t pw_varint_ZigZagDecode32(uint32_t n)#

Zig-zag decodes a uint64_t, returning it as an int64_t.

static inline int64_t pw_varint_ZigZagDecode64(uint64_t n)#

Zig-zag decodes a uint64_t, returning it as an int64_t.

PW_VARINT_ENCODED_SIZE_BYTES(value)#

Macro that returns the encoded size of up to a 64-bit integer. This is inefficient, but is a constant expression if the input is a constant. Use pw_varint_EncodedSizeBytes for runtime encoded size calculation.

size_t pw_varint_EncodedSizeBytes(uint64_t integer)#

Returns the size of a uint64_t when encoded as a varint (LEB128).

enum pw_varint_Format#

Describes a custom varint format.

Values:

enumerator PW_VARINT_ZERO_TERMINATED_LEAST_SIGNIFICANT#
enumerator PW_VARINT_ZERO_TERMINATED_MOST_SIGNIFICANT#
enumerator PW_VARINT_ONE_TERMINATED_LEAST_SIGNIFICANT#
enumerator PW_VARINT_ONE_TERMINATED_MOST_SIGNIFICANT#
size_t pw_varint_EncodeCustom(uint64_t integer, void *output, size_t output_size, pw_varint_Format format)#

Encodes a uint64_t using a custom varint format.

size_t pw_varint_DecodeCustom(const void *input, size_t input_size, uint64_t *output, pw_varint_Format format)#

Decodes a uint64_t using a custom varint format.

C++#

constexpr size_t pw::varint::kMaxVarint32SizeBytes = 5#

Maximum size of a varint (LEB128) encoded uint32_t.

constexpr size_t pw::varint::kMaxVarint64SizeBytes = 10#

Maximum size of a varint (LEB128) encoded uint64_t.

template<typename T>
constexpr std::make_unsigned_t<T> pw::varint::ZigZagEncode(T n)#

ZigZag encodes a signed integer. This maps small negative numbers to small, unsigned positive numbers, which improves their density for LEB128 encoding.

ZigZag encoding works by moving the sign bit from the most-significant bit to the least-significant bit. For the signed k-bit integer n, the formula is:

(n << 1) ^ (n >> (k - 1))

See the following for a description of ZigZag encoding: https://protobuf.dev/programming-guides/encoding/#signed-ints

template<typename T>
constexpr std::make_signed_t<T> pw::varint::ZigZagDecode(T n)#

ZigZag decodes a signed integer.

The calculation is done modulo std::numeric_limits<T>::max()+1, so the unsigned integer overflows are intentional.

template<typename T, typename = std::enable_if_t<std::is_integral<T>::value || std::is_convertible<T, uint64_t>::value>>
constexpr size_t pw::varint::EncodedSize(
T integer,
)#

Computes the size of an integer when encoded as a varint.

Parameters:

integer – The integer whose encoded size is to be computed. integer can be signed or unsigned.

Returns:

The size of integer when encoded as a varint.

inline size_t pw::varint::EncodeLittleEndianBase128(uint64_t integer, const span<std::byte> &output)#

Encodes a uint64_t with Little-Endian Base 128 (LEB128) encoding.

Returns:

the number of bytes written; 0 if the buffer is too small

template<typename T>
size_t pw::varint::Encode(T integer, const span<std::byte> &output)#

Encodes the provided integer using a variable-length encoding and returns the number of bytes written.

The encoding is the same as used in protocol buffers. Signed integers are ZigZag encoded to remove leading 1s from small negative numbers, then the resulting number is encoded as Little Endian Base 128 (LEB128). Unsigned integers are encoded directly as LEB128.

Returns the number of bytes written or 0 if the result didn’t fit in the encoding buffer.

inline size_t pw::varint::Decode(const span<const std::byte> &input, int64_t *output)#

Decodes a varint-encoded value. If reading into a signed integer, the value is ZigZag decoded.

Returns the number of bytes read from the input if successful. Returns zero if the result does not fit in a int64_t/ uint64_t or if the input is exhausted before the number terminates. Reads a maximum of 10 bytes.

The following example decodes multiple varints from a buffer:

while (!data.empty()) {
  int64_t value;
  size_t bytes = Decode(data, &value);

  if (bytes == 0u) {
    return Status::DataLoss();
  }
  results.push_back(value);
  data = data.subspan(bytes)
}
inline size_t pw::varint::Decode(const span<const std::byte> &input, uint64_t *value)#

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

constexpr uint64_t pw::varint::MaxValueInBytes(size_t bytes)#

Returns the maximum (max) integer value that can be encoded as a varint into the specified number of bytes.

The following table lists the max value for each byte size:

Bytes

Max value

1

127

2

16,383

3

2,097,151

4

268,435,455

5

34,359,738,367

6

4,398,046,511,103

7

562,949,953,421,311

8

72,057,594,037,927,935

9

9,223,372,036,854,775,807

10

(uint64 max value)

Parameters:

bytes – The size of the varint, in bytes. 5 bytes are needed for the max uint32 value. 10 bytes are needed for the max uint64 value.

Returns:

The max integer value for a varint of size bytes.

enum class pw::varint::Format#

Describes a custom varint format.

Values:

enumerator kZeroTerminatedLeastSignificant#
enumerator kZeroTerminatedMostSignificant#
enumerator kOneTerminatedLeastSignificant#
enumerator kOneTerminatedMostSignificant#
inline size_t pw::varint::Encode(uint64_t value, span<std::byte> output, Format format)#

Encodes a varint in a custom format.

inline size_t pw::varint::Decode(span<const std::byte> input, uint64_t *value, Format format)#

Decodes a varint from a custom format.

Stream API#

StatusWithSize pw::varint::Read(stream::Reader &reader, uint64_t *output, size_t max_size = std::numeric_limits<size_t>::max())#
StatusWithSize pw::varint::Read(stream::Reader &reader, int64_t *output, size_t max_size = std::numeric_limits<size_t>::max())#

Decodes a variable-length integer (varint) from the current position of a pw::stream. Reads a maximum of 10 bytes or max_size, whichever is smaller.

This method always returns the number of bytes read, even on error.

Parameters:

  • reader – The pw::stream to read from.

  • output – The integer to read into. If reading into a signed integer, the value is ZigZag-decoded.

  • max_size – The maximum number of bytes to read. The upper limit is 10 bytes.

Returns:

Code

Description

OK

The decoded value is placed in output.

OUT_OF_RANGE

No input is available, e.g. the stream is closed.

DATA_LOSS

The decoded value is too large for output or is incomplete, e.g. the input was exhausted after a partial varint was read.

Rust#

pw_varint’s Rust API is documented in our rustdoc API docs.

Zephyr#

To enable pw_varint for Zephyr add CONFIG_PIGWEED_VARINT=y to the project’s configuration.

On this page