pw_blob_store#

pw_blob_store is a storage container library for storing a single blob of data. BlobStore is a flash-backed persistent storage system with integrated data integrity checking that serves as a lightweight alternative to a file system.

Usage#

Most operations on a BlobStore are done using BlobReader and BlobWriter objects that have been constructed using a BlobStore. Though a BlobStore may have multiple open BlobReader objects, no other readers/writers may be active if a BlobWriter is opened on a blob store.

The data state of a blob can be checked using the HasData() method. The method returns true if the blob is currenty valid and has at least one data byte. This allows checking if a blob has stored data without needing to instantiate and open a reader or writer.

Write buffer#

BlobStore uses a write buffer to allow writes smaller than and/or unaligned to the flash write aligment. BlobStore also supports using the write buffer for deferred writes that can be enqueued and written to flash at a later time or by a different thread/context.

BlobStore can be used with a zero-size write buffer to reduce memory requirements. When using zero-size write buffer, the user is required to write maintain write sizes that are a multiple of the flash write size the blob is configured for.

If a non-zero sized write buffer is used, the write buffer size must be a multiple of the flash write size.

Writing to a BlobStore#

BlobWriter objects are pw::stream::Writer compatible, but do not support reading any of the blob’s contents. Opening a BlobWriter on a BlobStore that contains data will discard any existing data if Discard(), Write (), or Erase() are called. There is currently no mechanism to allow appending to existing data.

BlobStore::BlobWriterWithBuffer writer(my_blob_store);
writer.Open();
writer.Write(my_data);

// ...

// A close is implied when a BlobWriter is destroyed. Manually closing a
// BlobWriter enables error handling on Close() failure.
writer.Close();

Erasing a BlobStore#

There are two distinctly different mechanisms to “erase” the contents of a BlobStore:

  1. Discard(): Discards any ongoing writes and ensures BlobReader objects see the BlobStore as empty. This is the fastest way to logically erase a BlobStore.

  2. Erase(): Performs an explicit flash erase of the BlobStore’s underlying partition. This is useful for manually controlling when a flash erase is performed before a BlobWriter starts to write data (as flash erase operations may be time-consuming).

Naming a BlobStore’s contents#

Data in a BlobStore May be named similarly to a file. This enables identification of a BlobStore’s contents in cases where different data may be stored to a shared blob store. This requires an additional RAM buffer that can be used to encode the BlobStore’s KVS metadata entry. Calling MaxFileNameLength() on a BlobWriter will provide the max file name length based on the BlobWriter’s metadata encode buffer size.

SetFileName() performs a copy of the provided file name, meaning it’s safe for the std::string_view to be invalidated after the function returns.

constexpr size_t kMaxFileNameLength = 48;
BlobStore::BlobWriterWithBuffer<kMaxFileNameLength> writer(my_blob_store);
writer.Open();
writer.SetFileName("stonks.jpg");
writer.Write(my_data);
// ...
writer.Close();

Reading from a BlobStore#

A BlobStore may have multiple open BlobReader objects. No other readers/writers may be open/active if a BlobWriter is opened on a blob store.

  1. Create BlobReader instance

  2. BlobReader::Open().

  3. Read data using BlobReader::Read() or BlobReader::GetMemoryMappedBlob(). BlobReader is seekable. Use BlobReader::Seek() to read from a desired offset.

  4. BlobReader::Close().

FileSystem RPC integration#

pw_blob_store provides an optional FileSystemEntry implementation for use with pw_file’s FlatFileSystemService. This simplifies the process of enumerating BlobStore objects as files via pw_file’s FileSystem RPC service.

Size report#

The following size report showcases the memory usage of the blob store.

Label

Segment

Delta

BlobStore

FLASH

+20

[section .code]

+328

main

+4

quorem

-2

pw::kvs::KeyValueStore::UpdateKeyDescriptor()

+88

pw::kvs::KeyValueStore::Get()

+4

pw::kvs::FlashPartition

-2

pw::kvs::Key::Key()

+2

pw::sync::NoOpLock::DoLockOperation()

NEW

+408

pw::blob_store::BlobStore::Write()

NEW

+324

pw::blob_store::BlobStore::BlobWriter::WriteMetadata()

NEW

+220

pw::blob_store::BlobStore::Flush()

NEW

+208

pw::blob_store::BlobStore::Init()

NEW

+184

pw::blob_store::BlobStore::FlushFinalPartialChunk()

NEW

+168

pw::blob_store::BlobStore::LoadMetadata()

NEW

+160

pw::blob_store::BlobStore::BlobWriter::Close()

NEW

+152

pw::blob_store::BlobStore::ValidateChecksum()

NEW

+128

pw::blob_store::BlobStore::Erase()

NEW

+112

pw::blob_store::BlobStore::CalculateChecksumFromFlash()

NEW

+104

pw::blob_store::BlobStore::CommitToFlash()

NEW

+88

pw::blob_store::BlobStore::Invalidate()

NEW

+88

pw::blob_store::BlobStore::Read()

NEW

+84

pw::blob_store::BlobStore::BlobReader::Open()

NEW

+80

pw::blob_store::BlobStore::OpenRead()

NEW

+68

pw::blob_store::BlobStore::BlobWriter::Open()

NEW

+64

pw::blob_store::BlobStore::OpenWrite()

NEW

+62

pw::blob_store::BlobStore::GetMemoryMappedBlob()

NEW

+56

pw::blob_store::BlobStore::CloseRead()

NEW

+52

pw::blob_store::BlobStore::BlobReader::DoSeek()

NEW

+52

pw::blob_store::BlobStore::BlobReader::~BlobReader()

NEW

+46

pw::blob_store::BlobStore::BlobReader::DoRead()

NEW

+44

pw::blob_store::BlobStore::BlobWriter::~BlobWriter()

NEW

+36

pw::blob_store::BlobStore::BlobReader

NEW

+36

pw::blob_store::BlobStore::BlobWriter

NEW

+36

pw::blob_store::BlobStore::WriteBufferBytesUsed()

NEW

+34

pw::ByteBuilder::ResizeForAppend()

NEW

+34

pw::ByteBuilder::append()

NEW

+34

pw::kvs::KeyValueStore::CheckReadOperation()

NEW

+32

pw::blob_store::BlobStore::BlobWriter::DoWrite()

NEW

+28

pw::blob_store::BlobStore::BlobWriter::ConservativeLimit()

NEW

+28

pw::sync::Borrowable<>::acquire()

NEW

+20

pw::blob_store::BlobStore::BlobReader::Close()

NEW

+20

pw::blob_store::BlobStore::BlobReader::ConservativeLimit()

NEW

+20

pw::sync::BorrowedPointer<>::~BorrowedPointer()

NEW

+18

pw::blob_store::BlobStore::HasData()

NEW

+18

pw::blob_store::BlobStore::ValidToWrite()

NEW

+16

pw::blob_store::BlobStore::EraseIfNeeded()

NEW

+16

pw::stream::Reader::DoWrite()

NEW

+16

pw::stream::Writer::DoRead()

NEW

+14

pw::blob_store::BlobStore::BlobReader::DoTell()

NEW

+14

pw::blob_store::BlobStore::WriteBytesRemaining()

NEW

+10

pw::blob_store::BlobStore::MaxDataSizeBytes()

NEW

+10

pw::blob_store::BlobStore::ResetChecksum()

NEW

+6

pw::stream::NonSeekableWriter::DoSeek()

NEW

+6

pw::stream::Stream::DoTell()

+3,896

BlobStore with deferred write

FLASH

+16

[section .code]

+340

main

+4

quorem

-2

pw::kvs::KeyValueStore::UpdateKeyDescriptor()

+88

pw::kvs::KeyValueStore::Get()

+4

pw::kvs::FlashPartition

-2

pw::kvs::Key::Key()

+2

pw::sync::NoOpLock::DoLockOperation()

NEW

+408

pw::blob_store::BlobStore::Write()

NEW

+324

pw::blob_store::BlobStore::BlobWriter::WriteMetadata()

NEW

+220

pw::blob_store::BlobStore::Flush()

NEW

+208

pw::blob_store::BlobStore::Init()

NEW

+184

pw::blob_store::BlobStore::FlushFinalPartialChunk()

NEW

+168

pw::blob_store::BlobStore::LoadMetadata()

NEW

+160

pw::blob_store::BlobStore::BlobWriter::Close()

NEW

+152

pw::blob_store::BlobStore::ValidateChecksum()

NEW

+128

pw::blob_store::BlobStore::Erase()

NEW

+110

pw::blob_store::BlobStore::CalculateChecksumFromFlash()

NEW

+104

pw::blob_store::BlobStore::CommitToFlash()

NEW

+88

pw::blob_store::BlobStore::Invalidate()

NEW

+88

pw::blob_store::BlobStore::Read()

NEW

+86

pw::blob_store::BlobStore::AddToWriteBuffer()

NEW

+84

pw::blob_store::BlobStore::BlobReader::Open()

NEW

+80

pw::blob_store::BlobStore::OpenRead()

NEW

+68

pw::blob_store::BlobStore::BlobWriter::Open()

NEW

+64

pw::blob_store::BlobStore::OpenWrite()

NEW

+62

pw::blob_store::BlobStore::GetMemoryMappedBlob()

NEW

+60

pw::blob_store::BlobStore::WriteBufferBytesFree()

NEW

+56

pw::blob_store::BlobStore::CloseRead()

NEW

+52

pw::blob_store::BlobStore::BlobReader::DoSeek()

NEW

+52

pw::blob_store::BlobStore::BlobReader::~BlobReader()

NEW

+46

pw::blob_store::BlobStore::BlobReader::DoRead()

NEW

+44

pw::blob_store::BlobStore::BlobWriter::~BlobWriter()

NEW

+40

pw::blob_store::BlobStore::DeferredWriter::~DeferredWriter()

NEW

+36

pw::blob_store::BlobStore::BlobReader

NEW

+36

pw::blob_store::BlobStore::BlobWriter

NEW

+36

pw::blob_store::BlobStore::DeferredWriter

NEW

+36

pw::blob_store::BlobStore::WriteBufferBytesUsed()

NEW

+34

pw::ByteBuilder::ResizeForAppend()

NEW

+34

pw::ByteBuilder::append()

NEW

+34

pw::kvs::KeyValueStore::CheckReadOperation()

NEW

+32

pw::blob_store::BlobStore::BlobWriter::DoWrite()

NEW

+32

pw::blob_store::BlobStore::DeferredWriter::DoWrite()

NEW

+28

pw::blob_store::BlobStore::BlobWriter::ConservativeLimit()

NEW

+28

pw::sync::Borrowable<>::acquire()

NEW

+22

pw::blob_store::BlobStore::BlobReader::Close()

NEW

+20

pw::blob_store::BlobStore::BlobReader::ConservativeLimit()

NEW

+20

pw::sync::BorrowedPointer<>::~BorrowedPointer()

NEW

+18

pw::blob_store::BlobStore::DeferredWriter::ConservativeLimit()

NEW

+18

pw::blob_store::BlobStore::HasData()

NEW

+18

pw::blob_store::BlobStore::ValidToWrite()

NEW

+16

pw::blob_store::BlobStore::EraseIfNeeded()

NEW

+16

pw::stream::Reader::DoWrite()

NEW

+16

pw::stream::Writer::DoRead()

NEW

+14

pw::blob_store::BlobStore::BlobReader::DoTell()

NEW

+14

pw::blob_store::BlobStore::WriteBytesRemaining()

NEW

+10

pw::blob_store::BlobStore::MaxDataSizeBytes()

NEW

+10

pw::blob_store::BlobStore::ResetChecksum()

NEW

+6

pw::stream::NonSeekableWriter::DoSeek()

NEW

+6

pw::stream::Stream::DoTell()

+4,176

Note

The documentation for this module is currently incomplete.