pw_blob_store#

pw_blob_store is a storage container library for storing a single blob of data. BlobStore is a flash-backed persistent storage system with integrated data integrity checking that serves as a lightweight alternative to a file system.

Usage#

Most operations on a BlobStore are done using BlobReader and BlobWriter objects that have been constructed using a BlobStore. Though a BlobStore may have multiple open BlobReader objects, no other readers/writers may be active if a BlobWriter is opened on a blob store.

The data state of a blob can be checked using the HasData() method. The method returns true if the blob is currenty valid and has at least one data byte. This allows checking if a blob has stored data without needing to instantiate and open a reader or writer.

Write buffer#

BlobStore uses a write buffer to allow writes smaller than and/or unaligned to the flash write aligment. BlobStore also supports using the write buffer for deferred writes that can be enqueued and written to flash at a later time or by a different thread/context.

BlobStore can be used with a zero-size write buffer to reduce memory requirements. When using zero-size write buffer, the user is required to write maintain write sizes that are a multiple of the flash write size the blob is configured for.

If a non-zero sized write buffer is used, the write buffer size must be a multiple of the flash write size.

Writing to a BlobStore#

BlobWriter objects are pw::stream::Writer compatible, but do not support reading any of the blob’s contents. Opening a BlobWriter on a BlobStore that contains data will discard any existing data if Discard(), Write (), or Erase() are called. Partially written blobs can be resumed using Resume(). There is currently no mechanism to allow appending to completed data write.

BlobStore::BlobWriterWithBuffer writer(my_blob_store);
writer.Open();
writer.Write(my_data);

// ...

// A close is implied when a BlobWriter is destroyed. Manually closing a
// BlobWriter enables error handling on Close() failure.
writer.Close();

Resuming a BlobStore write#

BlobWriter::Resume() supports resuming writes for blobs that have not been completed (writer.Close()). Supported resume situations are after Abandon() has been called on a write or after a crash/reboot with a fresh BlobStore instance.

Resume() opens the BlobWriter at the most recent safe resume point. Resume() finds the furthest written point in the flash partition and then backs up by erasing any partially written sector plus a full sector. This backing up is to try to avoid any corrupted or otherwise wrong data that might have resulted from the previous write failing. Resume() returns the current number of valid bytes in the resumed write.

If the blob is using a ChecksumAlgorithm, the checksum of the resumed blob write instance calculated from the content of the already written data. If it is desired to check the integrity of the already written data, BlobWriter::CurrentChecksum() can be used to check against the incoming data.

Once Resume() has successfully completed, the writer is ready to continue writing as normal.

Erasing a BlobStore#

There are two distinctly different mechanisms to “erase” the contents of a BlobStore:

  1. Discard(): Discards any ongoing writes and ensures BlobReader objects see the BlobStore as empty. This is the fastest way to logically erase a BlobStore.

  2. Erase(): Performs an explicit flash erase of the BlobStore’s underlying partition. This is useful for manually controlling when a flash erase is performed before a BlobWriter starts to write data (as flash erase operations may be time-consuming).

Naming a BlobStore’s contents#

Data in a BlobStore May be named similarly to a file. This enables identification of a BlobStore’s contents in cases where different data may be stored to a shared blob store. This requires an additional RAM buffer that can be used to encode the BlobStore’s KVS metadata entry. Calling MaxFileNameLength() on a BlobWriter will provide the max file name length based on the BlobWriter’s metadata encode buffer size.

SetFileName() performs a copy of the provided file name, meaning it’s safe for the std::string_view to be invalidated after the function returns.

constexpr size_t kMaxFileNameLength = 48;
BlobStore::BlobWriterWithBuffer<kMaxFileNameLength> writer(my_blob_store);
writer.Open();
writer.SetFileName("stonks.jpg");
writer.Write(my_data);
// ...
writer.Close();

Reading from a BlobStore#

A BlobStore may have multiple open BlobReader objects. No other readers/writers may be open/active if a BlobWriter is opened on a blob store.

  1. Create BlobReader instance

  2. BlobReader::Open()

  3. Read data using BlobReader::Read() or BlobReader::GetMemoryMappedBlob(). BlobReader is seekable. Use BlobReader::Seek() to read from a desired offset.

  4. BlobReader::Close()

FileSystem RPC integration#

pw_blob_store provides an optional FileSystemEntry implementation for use with pw_file’s FlatFileSystemService. This simplifies the process of enumerating BlobStore objects as files via pw_file’s FileSystem RPC service.

Size report#

The following size report showcases the memory usage of the blob store.

Label

Segment

Delta

BlobStore

FLASH

+16

[section .code]

+328

main

-2

pw::kvs::KeyValueStore::UpdateKeyDescriptor()

+88

pw::kvs::KeyValueStore::Get()

+4

pw::kvs::FlashPartition

-4

p05.0

+2

pw::sync::NoOpLock::DoLockOperation()

NEW

+416

pw::blob_store::BlobStore::Write()

NEW

+332

pw::blob_store::BlobStore::BlobWriter::WriteMetadata()

NEW

+228

pw::blob_store::BlobStore::Flush()

NEW

+220

pw::blob_store::BlobStore::Init()

NEW

+196

pw::blob_store::BlobStore::FlushFinalPartialChunk()

NEW

+172

pw::blob_store::BlobStore::BlobWriter::Close()

NEW

+164

pw::blob_store::BlobStore::LoadMetadata()

NEW

+160

pw::blob_store::BlobStore::ValidateChecksum()

NEW

+132

pw::blob_store::BlobStore::Erase()

NEW

+122

pw::blob_store::BlobStore::CalculateChecksumFromFlash()

NEW

+100

pw::blob_store::BlobStore::CommitToFlash()

NEW

+92

pw::blob_store::BlobStore::BlobReader::Open()

NEW

+88

pw::blob_store::BlobStore::Invalidate()

NEW

+88

pw::blob_store::BlobStore::OpenRead()

NEW

+72

pw::blob_store::BlobStore::BlobWriter::Open()

NEW

+72

pw::blob_store::BlobStore::Read()

NEW

+68

pw::blob_store::BlobStore::OpenWrite()

NEW

+64

pw::blob_store::BlobStore::GetMemoryMappedBlob()

NEW

+60

pw::blob_store::BlobStore::CloseRead()

NEW

+52

pw::blob_store::BlobStore::BlobReader::DoSeek()

NEW

+52

pw::blob_store::BlobStore::BlobReader::~BlobReader()

NEW

+46

pw::blob_store::BlobStore::BlobReader::DoRead()

NEW

+44

pw::blob_store::BlobStore::BlobWriter::~BlobWriter()

NEW

+40

pw::blob_store::BlobStore::WriteBufferBytesUsed()

NEW

+36

pw::ByteBuilder::ResizeForAppend()

NEW

+36

pw::blob_store::BlobStore::BlobReader

NEW

+36

pw::blob_store::BlobStore::BlobWriter

NEW

+34

pw::kvs::KeyValueStore::CheckReadOperation()

NEW

+32

pw::ByteBuilder::append()

NEW

+32

pw::blob_store::BlobStore::BlobWriter::DoWrite()

NEW

+30

pw::blob_store::BlobStore::BlobWriter::ConservativeLimit()

NEW

+28

pw::sync::Borrowable<>::acquire()

NEW

+22

pw::blob_store::BlobStore::BlobReader::ConservativeLimit()

NEW

+20

pw::sync::BorrowedPointer<>::~BorrowedPointer()

NEW

+18

pw::blob_store::BlobStore::ValidToWrite()

NEW

+16

pw::blob_store::BlobStore::BlobReader::Close()

NEW

+16

pw::blob_store::BlobStore::EraseIfNeeded()

NEW

+16

pw::stream::Reader::DoWrite()

NEW

+16

pw::stream::Writer::DoRead()

NEW

+14

pw::blob_store::BlobStore::BlobReader::DoTell()

NEW

+14

pw::blob_store::BlobStore::WriteBytesRemaining()

NEW

+10

pw::blob_store::BlobStore::MaxDataSizeBytes()

NEW

+10

pw::blob_store::BlobStore::ResetChecksum()

NEW

+6

pw::stream::NonSeekableWriter::DoSeek()

NEW

+6

pw::stream::Stream::DoTell()

+3,960

BlobStore with deferred write

FLASH

+16

[section .code]

+340

main

-4

quorem

-2

pw::kvs::KeyValueStore::UpdateKeyDescriptor()

+88

pw::kvs::KeyValueStore::Get()

+4

pw::kvs::FlashPartition

+2

pw::kvs::Key::Key()

+2

pw::sync::NoOpLock::DoLockOperation()

NEW

+416

pw::blob_store::BlobStore::Write()

NEW

+332

pw::blob_store::BlobStore::BlobWriter::WriteMetadata()

NEW

+228

pw::blob_store::BlobStore::Flush()

NEW

+220

pw::blob_store::BlobStore::Init()

NEW

+196

pw::blob_store::BlobStore::FlushFinalPartialChunk()

NEW

+172

pw::blob_store::BlobStore::BlobWriter::Close()

NEW

+164

pw::blob_store::BlobStore::LoadMetadata()

NEW

+160

pw::blob_store::BlobStore::ValidateChecksum()

NEW

+132

pw::blob_store::BlobStore::Erase()

NEW

+120

pw::blob_store::BlobStore::CalculateChecksumFromFlash()

NEW

+100

pw::blob_store::BlobStore::CommitToFlash()

NEW

+92

pw::blob_store::BlobStore::BlobReader::Open()

NEW

+88

pw::blob_store::BlobStore::Invalidate()

NEW

+88

pw::blob_store::BlobStore::OpenRead()

NEW

+78

pw::blob_store::BlobStore::AddToWriteBuffer()

NEW

+72

pw::blob_store::BlobStore::BlobWriter::Open()

NEW

+72

pw::blob_store::BlobStore::Read()

NEW

+68

pw::blob_store::BlobStore::OpenWrite()

NEW

+64

pw::blob_store::BlobStore::GetMemoryMappedBlob()

NEW

+64

pw::blob_store::BlobStore::WriteBufferBytesFree()

NEW

+60

pw::blob_store::BlobStore::CloseRead()

NEW

+52

pw::blob_store::BlobStore::BlobReader::DoSeek()

NEW

+52

pw::blob_store::BlobStore::BlobReader::~BlobReader()

NEW

+46

pw::blob_store::BlobStore::BlobReader::DoRead()

NEW

+44

pw::blob_store::BlobStore::BlobWriter::~BlobWriter()

NEW

+40

pw::blob_store::BlobStore::DeferredWriter::~DeferredWriter()

NEW

+40

pw::blob_store::BlobStore::WriteBufferBytesUsed()

NEW

+36

pw::ByteBuilder::ResizeForAppend()

NEW

+36

pw::blob_store::BlobStore::BlobReader

NEW

+36

pw::blob_store::BlobStore::BlobWriter

NEW

+36

pw::blob_store::BlobStore::DeferredWriter

NEW

+34

pw::kvs::KeyValueStore::CheckReadOperation()

NEW

+32

pw::ByteBuilder::append()

NEW

+32

pw::blob_store::BlobStore::BlobWriter::DoWrite()

NEW

+32

pw::blob_store::BlobStore::DeferredWriter::DoWrite()

NEW

+30

pw::blob_store::BlobStore::BlobWriter::ConservativeLimit()

NEW

+28

pw::sync::Borrowable<>::acquire()

NEW

+22

pw::blob_store::BlobStore::BlobReader::ConservativeLimit()

NEW

+20

pw::sync::BorrowedPointer<>::~BorrowedPointer()

NEW

+18

pw::blob_store::BlobStore::DeferredWriter::ConservativeLimit()

NEW

+18

pw::blob_store::BlobStore::ValidToWrite()

NEW

+16

pw::blob_store::BlobStore::BlobReader::Close()

NEW

+16

pw::blob_store::BlobStore::EraseIfNeeded()

NEW

+16

pw::stream::Reader::DoWrite()

NEW

+16

pw::stream::Writer::DoRead()

NEW

+14

pw::blob_store::BlobStore::BlobReader::DoTell()

NEW

+14

pw::blob_store::BlobStore::WriteBytesRemaining()

NEW

+10

pw::blob_store::BlobStore::MaxDataSizeBytes()

NEW

+10

pw::blob_store::BlobStore::ResetChecksum()

NEW

+6

pw::stream::NonSeekableWriter::DoSeek()

NEW

+6

pw::stream::Stream::DoTell()

+4,240

Note

The documentation for this module is currently incomplete.