pw_multibuf#

A buffer API optimized for zero-copy messaging

Unstable C++17

Sending or receiving messages via RPC, transfer, or sockets often requires a series of intermediate buffers, each requiring their own copy of the data. pw_multibuf allows data to be written once, eliminating the memory, CPU and latency overhead of copying.

How does it work?#

pw_multibuf uses several techniques to minimize copying of data:

  • Header and Footer Reservation: Lower-level components can reserve space within a buffer for headers and/or footers. This allows headers and footers to be added to user-provided data without moving users’ data.

  • Native Scatter/Gather and Fragmentation Support: Buffers can refer to multiple separate chunks of memory. Messages can be built up from discontiguous allocations, and users’ data can be fragmented across multiple packets.

  • Divisible Memory Regions: Incoming buffers can be divided without a copy, allowing incoming data to be freely demultiplexed.

What kinds of data is this for?#

pw_multibuf is best used in code that wants to read, write, or pass along data which are one of the following:

  • Large: pw_multibuf is designed to allow breaking up data into multiple chunks. It also supports asynchronous allocation for when there may not be sufficient space for incoming data.

  • Communications-Oriented: Data which is being received or sent across sockets, various packets, or shared-memory protocols can benefit from the fragmentation, multiplexing, and header/footer-reservation properties of pw_multibuf.

  • Copy-Averse: pw_multibuf is structured to allow users to pass around and mutate buffers without copying or moving data in-memory. This can be especially useful when working in systems that are latency-sensitive, need to pass large amounts of data, or when memory usage is constrained.

API Reference#

Most users of pw_multibuf will start by allocating a MultiBuf using a MultiBufAllocator class, such as the SimpleAllocator.

MultiBuf s consist of a number of Chunk s of contiguous memory. These Chunk s can be grown, shrunk, modified, or extracted from the MultiBuf. MultiBuf exposes an std::byte iterator interface as well as a Chunk iterator available through the Chunks() method.

An RAII-style OwnedChunk is also provided, and manages the lifetime of Chunk s which are not currently stored inside of a MultiBuf.

class Chunk#

A handle to a contiguous slice of data.

A Chunk is similar to a ByteSpan, but is aware of the underlying memory allocation, and is able to split, shrink, and grow into neighboring empty space.

This class is optimized to allow multiple owners to write into neighboring regions of the same allocation. One important usecase for this is communication protocols which want to reserve space at the front or rear of a buffer for headers or footers.

In order to support zero-copy DMA of communications buffers, allocators can create properly-aligned Chunk regions in appropriate memory. The driver can then DiscardPrefix in order to reserve bytes for headers, Truncate in order to reserve bytes for footers, and then pass the Chunk to the user to fill in. The header and footer space can then be reclaimed using the ClaimPrefix and ClaimSuffix methods.

Public Functions

bool CanMerge(const Chunk &next_chunk) const#

Returns if next_chunk is mergeable into the end of this Chunk.

This will only succeed when the two Chunk s are adjacent in memory and originated from the same allocation.

bool Merge(OwnedChunk &next_chunk)#

Attempts to merge next_chunk into the end of this Chunk.

If the chunks are successfully merged, this Chunk will be extended forwards to encompass the space of next_chunk, and next_chunk will be emptied and Released.

This will only succeed when the two Chunk s are adjacent in memory and originated from the same allocation.

If the chunks are not mergeable, neither Chunk will be modified.

bool ClaimPrefix(size_t bytes_to_claim)#

Attempts to add bytes_to_claim to the front of this buffer by advancing its range backwards in memory. Returns true if the operation succeeded.

This will only succeed if this Chunk points to a section of a region that has unreferenced bytes preceeding it. For example, a Chunk which has been shrunk using DiscardPrefix can be re-expanded using ClaimPrefix.

This method will acquire a mutex and is not IRQ safe.

bool ClaimSuffix(size_t bytes_to_claim)#

Attempts to add bytes_to_claim to the front of this buffer by advancing its range forwards in memory. Returns true if the operation succeeded.

This will only succeed if this Chunk points to a section of a region that has unreferenced bytes following it. For example, a Chunk which has been shrunk using Truncate can be re-expanded using

This method will acquire a mutex and is not IRQ safe.

void DiscardPrefix(size_t bytes_to_discard)#

Shrinks this handle to refer to the data beginning at offset bytes_to_discard.

Does not modify the underlying data.

This method will acquire a mutex and is not IRQ safe.

inline void DiscardFront(size_t bytes_to_discard)#

Deprecated alias for DiscardPrefix.

void Slice(size_t begin, size_t end)#

Shrinks this handle to refer to data in the range begin..<end.

Does not modify the underlying data.

This method will acquire a mutex and is not IRQ safe.

void Truncate(size_t len)#

Shrinks this handle to refer to only the first len bytes.

Does not modify the underlying data.

This method will acquire a mutex and is not IRQ safe.

std::optional<OwnedChunk> TakePrefix(size_t bytes_to_take)#

Attempts to shrink this handle to refer to the data beginning at offset bytes_to_take, returning the first bytes_to_take bytes as a new OwnedChunk.

If the inner call to AllocateChunkClass fails, this function will return `std::nullopt and this handle’s span will not change.

This method will acquire a mutex and is not IRQ safe.

std::optional<OwnedChunk> TakeSuffix(size_t bytes_to_take)#

Attempts to shrink this handle to refer only the first len - bytes_to_take bytes, returning the last bytes_to_take bytes as a new OwnedChunk.

If the inner call to AllocateChunkClass fails, this function will return std::nullopt and this handle’s span will not change.

This method will acquire a mutex and is not IRQ safe.

class OwnedChunk#

An RAII handle to a contiguous slice of data.

Note: OwnedChunk may acquire a pw::sync::Mutex during destruction, and so must not be destroyed within ISR contexts.

Public Functions

inline ~OwnedChunk()#

This method will acquire a mutex and is not IRQ safe.

void Release()#

Decrements the reference count on the underlying chunk of data and empties this handle so that span() now returns an empty (zero-sized) span.

Does not modify the underlying data, but may cause it to be deallocated if this was the only remaining Chunk referring to its region.

This method is equivalent to { Chunk _unused = std::move(chunk_ref); }

This method will acquire a mutex and is not IRQ safe.

inline Chunk *Take() &&#

Returns the contained Chunk* and empties this OwnedChunk without releasing the underlying Chunk.

class MultiBuf#

A buffer optimized for zero-copy data transfer.

A MultiBuf consists of multiple Chunk s of data.

Public Functions

void Release() noexcept#

Decrements the reference count on the underlying chunks of data and empties this MultiBuf so that size() == 0.

Does not modify the underlying data, but may cause it to be deallocated

This method is equivalent to { MultiBuf _unused = std::move(multibuf); }

This method will acquire a mutex and is not IRQ safe.

inline ~MultiBuf()#

This destructor will acquire a mutex and is not IRQ safe.

size_t size() const#

Returns the number of bytes in this container.

This method’s complexity is O(Chunks().size()).

bool empty() const#

Returns whether the container is empty (size() == 0).

This method’s complexity is O(Chunks().size()), but will be more efficient than size() == 0 in most cases.

inline iterator begin()#

Returns an iterator pointing to the first byte of this `MultiBuf.

inline const_iterator begin() const#

Returns a const iterator pointing to the first byte of this `MultiBuf.

inline const_iterator cbegin() const#

Returns a const iterator pointing to the first byte of this `MultiBuf.

inline iterator end()#

Returns an iterator pointing to the end of this MultiBuf.

inline const_iterator end() const#

Returns a const iterator pointing to the end of this MultiBuf.

inline const_iterator cend() const#

Returns a const iterator pointing to the end of this MultiBuf.

bool ClaimPrefix(size_t bytes_to_claim)#

Attempts to add bytes_to_claim to the front of this buffer by advancing its range backwards in memory. Returns true if the operation succeeded.

This will only succeed if the first Chunk in this buffer points to a section of a region that has unreferenced bytes preceeding it. See also Chunk::ClaimPrefix.

This method will acquire a mutex and is not IRQ safe.

bool ClaimSuffix(size_t bytes_to_claim)#

Attempts to add bytes_to_claim to the front of this buffer by advancing its range forwards in memory. Returns true if the operation succeeded.

This will only succeed if the last Chunk in this buffer points to a section of a region that has unreferenced bytes following it. See also Chunk::ClaimSuffix.

This method will acquire a mutex and is not IRQ safe.

void DiscardPrefix(size_t bytes_to_discard)#

Shrinks this handle to refer to the data beginning at offset bytes_to_discard.

Does not modify the underlying data.

This method will acquire a mutex and is not IRQ safe.

void Slice(size_t begin, size_t end)#

Shrinks this handle to refer to data in the range begin..<end.

Does not modify the underlying data.

This method will acquire a mutex and is not IRQ safe.

void Truncate(size_t len)#

Shrinks this handle to refer to only the first len bytes.

Does not modify the underlying data.

This method will acquire a mutex and is not IRQ safe.

std::optional<MultiBuf> TakePrefix(size_t bytes_to_take)#

Attempts to shrink this handle to refer to the data beginning at offset bytes_to_take, returning the first bytes_to_take bytes as a new MultiBuf.

If the inner call to AllocateChunkClass fails, this function will return `std::nullopt and this handle’s span will not change.

This method will acquire a mutex and is not IRQ safe.

std::optional<MultiBuf> TakeSuffix(size_t bytes_to_take)#

Attempts to shrink this handle to refer only the first len - bytes_to_take bytes, returning the last bytes_to_take bytes as a new MultiBuf.

If the inner call to AllocateChunkClass fails, this function will return std::nullopt and this handle’s span will not change.

This method will acquire a mutex and is not IRQ safe.

void PushPrefix(MultiBuf &&front)#

Pushes front onto the front of this MultiBuf.

This operation does not move any data and is O(front.Chunks().size()).

void PushSuffix(MultiBuf &&tail)#

Pushes tail onto the end of this MultiBuf.

This operation does not move any data and is O(Chunks().size()).

void PushFrontChunk(OwnedChunk &&chunk)#

Pushes Chunk onto the front of the MultiBuf.

This operation does not move any data and is O(1).

void PushBackChunk(OwnedChunk &&chunk)#

Pushes Chunk onto the end of the MultiBuf.

This operation does not move any data and is O(Chunks().size()).

OwnedChunk TakeFrontChunk()#

Removes the first Chunk.

This operation does not move any data and is O(1).

ChunkIterator InsertChunk(ChunkIterator position, OwnedChunk &&chunk)#

Inserts chunk into the specified position in the MultiBuf.

This operation does not move any data and is O(Chunks().size()).

Returns an iterator pointing to the newly-inserted Chunk.

std::tuple<ChunkIterator, OwnedChunk> TakeChunk(ChunkIterator position)#

Removes a Chunk from the specified position.

This operation does not move any data and is O(Chunks().size()).

Returns an iterator pointing to the Chunk after the removed Chunk, or Chunks().end() if this was the last Chunk in the MultiBuf.

inline constexpr ChunkIterable Chunks()#

Returns an iterable container which yields the Chunks in this MultiBuf.

inline constexpr const ChunkIterable Chunks() const#

Returns an iterable container which yields the const Chunks in this MultiBuf.

inline constexpr ChunkIterator ChunkBegin()#

Returns an iterator pointing to the first Chunk in this MultiBuf.

inline constexpr ChunkIterator ChunkEnd()#

Returns an iterator pointing to the end of the Chunks in this MultiBuf.

inline constexpr ConstChunkIterator ConstChunkBegin()#

Returns a const iterator pointing to the first Chunk in this MultiBuf.

inline constexpr ConstChunkIterator ConstChunkEnd()#

Returns a const iterator pointing to the end of the Chunks in this MultiBuf.

class ChunkIterable#

An iterable containing the Chunk s of a MultiBuf.

Public Functions

inline Chunk &front()#

Returns a reference to the first chunk.

The behavior of this method is undefined when size() == 0.

Chunk &back()#

Returns a reference to the final chunk.

The behavior of this method is undefined when size() == 0.

NOTE: this method is O(size()).

size_t size() const#

Returns the number of Chunks in this iterable.

class ChunkIterator#

A std::forward_iterator over the Chunks of a MultiBuf.

class const_iterator#

A const std::forward_iterator over the bytes of a MultiBuf.

Public Functions

inline constexpr const Chunk *chunk() const#

Returns the current Chunk pointed to by this iterator.

inline constexpr size_t byte_index() const#

Returns the index of the byte pointed to by this iterator within the current Chunk.

class ConstChunkIterator#

A const std::forward_iterator over the Chunks of a MultiBuf.

class iterator#

An std::forward_iterator over the bytes of a MultiBuf.

Public Functions

inline constexpr Chunk *chunk() const#

Returns the current Chunk pointed to by this iterator.

inline constexpr size_t byte_index() const#

Returns the index of the byte pointed to by this iterator within the current Chunk.

class MultiBufAllocator#

Interface for allocating MultiBuf objects.

A MultiBufAllocator differs from a regular pw::allocator::Allocator in that they may provide support for:

  • Asynchronous allocation.

  • Non-contiguous buffer allocation.

  • Internal header/footer reservation.

  • Size-range allocation.

In order to accomplish this, they return MultiBuf objects rather than arbitrary pieces of memory.

Additionally, MultiBufAllocator implementations may choose to store their allocation metadata separately from the data itself. This allows for things like allocation headers to be kept out of restricted DMA-capable or shared-memory regions.

NOTE: MultiBufAllocators must outlive any futures created from them.

Subclassed by pw::multibuf::SimpleAllocator

Public Functions

MultiBufAllocator(MultiBufAllocator&) = delete#

`MultiBufAllocator is not copyable or movable.

std::optional<MultiBuf> Allocate(size_t size)#

Attempts to allocate a MultiBuf of exactly size bytes.

Memory allocated by an arbitrary MultiBufAllocator does not provide any alignment requirments, preferring instead to allow the allocator maximum flexibility for placing regions (especially discontiguous regions).

Return values:

  • ``MultiBuf`` – if the allocation was successful.

  • ``nullopt_t`` – if the memory is not currently available.

std::optional<MultiBuf> Allocate(size_t min_size, size_t desired_size)#

Attempts to allocate a MultiBuf of at least min_size bytes and at most desired_size bytes.

Memory allocated by an arbitrary MultiBufAllocator does not provide any alignment requirments, preferring instead to allow the allocator maximum flexibility for placing regions (especially discontiguous regions).

Return values:

  • ``MultiBuf`` – if the allocation was successful.

  • ``nullopt_t`` – if the memory is not currently available.

std::optional<MultiBuf> AllocateContiguous(size_t size)#

Attempts to allocate a contiguous MultiBuf of exactly size bytes.

Memory allocated by an arbitrary MultiBufAllocator does not provide any alignment requirments, preferring instead to allow the allocator maximum flexibility for placing regions (especially discontiguous regions).

Return values:

  • ``MultiBuf`` – with a single Chunk if the allocation was successful.

  • ``nullopt_t`` – if the memory is not currently available.

std::optional<MultiBuf> AllocateContiguous(size_t min_size, size_t desired_size)#

Attempts to allocate a contiguous MultiBuf of at least min_size bytes and at most desired_size bytes.

Memory allocated by an arbitrary MultiBufAllocator does not provide any alignment requirments, preferring instead to allow the allocator maximum flexibility for placing regions (especially discontiguous regions).

Return values:

  • ``MultiBuf`` – with a single Chunk if the allocation was successful.

  • ``nullopt_t`` – if the memory is not currently available.

MultiBufAllocationFuture AllocateAsync(size_t size)#

Asynchronously allocates a MultiBuf of exactly size bytes.

Memory allocated by an arbitrary MultiBufAllocator does not provide any alignment requirments, preferring instead to allow the allocator maximum flexibility for placing regions (especially discontiguous regions).

Return values:

A – MultiBufAllocationFuture which will yield a MultiBuf when one is available.

MultiBufAllocationFuture AllocateAsync(size_t min_size, size_t desired_size)#

Asynchronously allocates a MultiBuf of at least min_size bytes and at most `desired_size bytes.

Memory allocated by an arbitrary MultiBufAllocator does not provide any alignment requirments, preferring instead to allow the allocator maximum flexibility for placing regions (especially discontiguous regions).

Return values:

A – MultiBufAllocationFuture which will yield a MultiBuf when one is available.

MultiBufAllocationFuture AllocateContiguousAsync(size_t size)#

Asynchronously allocates a contiguous MultiBuf of exactly size bytes.

Memory allocated by an arbitrary MultiBufAllocator does not provide any alignment requirments, preferring instead to allow the allocator maximum flexibility for placing regions (especially discontiguous regions).

Return values:

A – MultiBufAllocationFuture which will yield an MultiBuf consisting of a single Chunk when one is available.

MultiBufAllocationFuture AllocateContiguousAsync(size_t min_size, size_t desired_size)#

Asynchronously allocates an OwnedChunk of at least min_size bytes and at most desired_size bytes.

Return values:

A – MultiBufAllocationFuture which will yield an MultiBuf consisting of a single Chunk when one is available.

class SimpleAllocator : public pw::multibuf::MultiBufAllocator#

A simple first-fit MultiBufAllocator.

Public Functions

inline SimpleAllocator(ByteSpan data_area, pw::allocator::Allocator &metadata_alloc)#

Creates a new SimpleAllocator.

Parameters:

  • data_area – [in] The region to use for storing chunk memory.

  • metadata_alloc – [in] The allocator to use for metadata tracking the in-use regions. This allocator must be thread-safe if the resulting buffers may travel to another thread. SynchronizedAllocator can be used to create a thread-safe allocator from a non-thread-safe allocator.

class Stream : public pw::stream::SeekableReaderWriter#

A readable, writable, and seekable Stream implementation backed by a MultiBuf.

Public Functions

inline constexpr const MultiBuf &multibuf() const#

Returns the MultiBuf backing this stream.

Allocator Implementors’ API#

Some users will need to directly implement the MultiBufAllocator interface in order to provide allocation out of a particular region, provide particular allocation policy, fix Chunks to some size (such as MTU size - header for socket implementations), or specify other custom behavior.

These users will also need to understand and implement the following APIs:

class ChunkRegionTracker#

An object that manages a single allocated region which is referenced by one or more Chunk objects.

This class is typically implemented by MultiBufAllocator implementations in order to customize the behavior of region deallocation.

ChunkRegionTracker s have three responsibilities:

  • Tracking the region of memory into which Chunk s can expand. This is reported via the Region method. Chunk s in this region can refer to memory within this region sparsely, but they can grow or shrink so long as they stay within the bounds of the Region.

  • Deallocating the region and the ChunkRegionTracker itself. This is implemented via the Destroy method, which is called once all of the Chunk s in a region have been released.

  • Allocating and deallocating space for Chunk classes. This is merely allocating space for the Chunk object itself, not the memory to which it refers. This can be implemented straightforwardly by delegating to an existing generic allocator such as malloc or a pw::allocator::Allocator implementation.

Subclassed by pw::multibuf::HeaderChunkRegionTracker, pw::multibuf::SingleChunkRegionTracker, pw::multibuf::internal::LinkedRegionTracker

Public Functions

std::optional<OwnedChunk> CreateFirstChunk()#

Creates the first Chunk referencing a whole region of memory.

This must only be called once per ChunkRegionTracker, when the region is first created. Multiple calls will result in undefined behavior.

Returns std::nullopt if AllocateChunkStorage returns nullptr.

A simple implementation of a ChunkRegionTracker is provided, called HeaderChunkRegionTracker. It stores its Chunk and region metadata in a Allocator allocation alongside the data. The allocation process is synchronous, making this class suitable for testing. The allocated region or Chunk must not outlive the provided allocator.

class HeaderChunkRegionTracker : public pw::multibuf::ChunkRegionTracker#

A ChunkRegionTracker which stores its Chunk and region metadata in a allocator::Allocator allocation alongside the data.

This is useful when testing and when there is no need for asynchronous allocation.

Public Functions

inline virtual ByteSpan Region() const final#

Returns the entire span of the region being managed.

Chunk s referencing this tracker will not expand beyond this region, nor into one another’s portions of the region.

This region does not provide any alignment guarantees by default.

This region must not change for the lifetime of this ChunkRegionTracker.

Public Static Functions

static inline std::optional<OwnedChunk> AllocateRegionAsChunk(allocator::Allocator &alloc, size_t size)#

Allocates a new Chunk region of size bytes in alloc.

The underlyiing allocation will also store the HeaderChunkRegionTracker itself. The allocated memory must not outlive the provided allocator alloc.

Returns the newly-created OwnedChunk if successful.

static inline HeaderChunkRegionTracker *AllocateRegion(allocator::Allocator &alloc, size_t size)#

Allocates a new region of size bytes in alloc.

The underlyiing allocation will also store the HeaderChunkRegionTracker itself. The allocated memory must not outlive the provided allocator alloc.

Returns a pointer to the newly-created HeaderChunkRegionTracker or nullptr if the allocation failed.

Another ChunkRegionTracker specialization is the lightweight SingleChunkRegionTracker, which does not rely on Allocator and uses the provided memory view to create a single chunk. This is useful when a single Chunk is sufficient at no extra overhead. However, the user needs to own the provided memory and know when a new Chunk can be requested.

class SingleChunkRegionTracker : public pw::multibuf::ChunkRegionTracker#

A ChunkRegionTracker that uses inline memory to create a single Chunk with the only caveat that the provided Chunk cannot be split. All attempts will result in std::nullopt.

Public Functions

inline explicit SingleChunkRegionTracker(ByteSpan region)#

Constructs a region tracker with a single Chunk that maps to region, which must outlive this tracker and any OwnedChunk it creates.

inline std::optional<OwnedChunk> GetChunk(size_t size)#

Gets a Chunk of a given size, which must be less than or equal to the provided region.

Returns: An OwnedChunk if the Chunk is free, otherwise std::nullopt, in which case GetChunk() can be called again.

inline virtual void Destroy() final#

Destroys the ChunkRegionTracker.

Typical implementations will call std::destroy_at(this) and then free the memory associated with the region and the tracker.

inline virtual ByteSpan Region() const final#

Returns the entire span of the region being managed.

Chunk s referencing this tracker will not expand beyond this region, nor into one another’s portions of the region.

This region does not provide any alignment guarantees by default.

This region must not change for the lifetime of this ChunkRegionTracker.

inline virtual void *AllocateChunkClass() final#

Returns a pointer to sizeof(Chunk) bytes with alignas(Chunk). Returns nullptr on failure.

inline virtual void DeallocateChunkClass(void *chunk) final#

Deallocates a pointer returned by AllocateChunkClass.