Repository navigation
feat(persistent_data): versioned file-backed storage of a data structure with forward migration - #857
Merged
Merged
Conversation
…ure with forward migration New header-only component espp::PersistentData<Versions...>: stores one serializable aggregate in a file (LittleFS / SPIFFS / SD through the VFS) with a 16-byte header (magic, schema version, payload length, CRC32) in front of the alpaca payload, so a load distinguishes "no file", "corrupt or truncated", "written by newer firmware" and "older version, migrated" instead of guessing by trial parsing. Versions form a type list, oldest first; each declares VERSION and every type after the first provides static T migrate(const Previous &), so an old file is walked up the chain one step at a time. Saves are atomic (temporary file + fsync + rename). Every operation reports a std::error_code (PersistentDataErrc); get() copies, update(fn) does a locked read-modify-write; encode()/decode() expose the same bytes for other backends. The codec (format, version list, migration chain, file helpers) lives in detail/persistent_data_codec.hpp (standard library + alpaca only) and is covered by a host test (100 checks) with its own CI workflow. The example stores a settings struct on LittleFS, migrates a v1 file and validates every operation with a pass / fail summary; added to the build matrix and the storage docs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DFjjnq3XCTRAXENSxsCxJU
`settings = s;` is set() and `Settings s = settings;` is get(), for the common read-modify-write flow; save() stays explicit because it can fail and reports an error code. Example validates both (37 checks), README shows the operators. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DFjjnq3XCTRAXENSxsCxJU
Add the component's include dir to espp.cmake and the header to espp.hpp so PersistentData (and its file format) is usable on the host through the installed espp::espp target: desktop tools that read / write a device's settings file, or host-side migration tests. Not bindable to Python (class template over the application's own version types), noted alongside the other template-only headers in autogenerate_bindings.py. Verified with a find_package consumer on macOS (load_or_create, update, operators, auto-load, v1 -> v2 migration and rewrite all pass). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DFjjnq3XCTRAXENSxsCxJU
PersistentData<Current, Older..., Oldest>: the first type is the current one, VERSION strictly decreases along the list, and every type except the last migrates from the type after it. Adding a version is a prepend and dropping the oldest is removing the last entry, which reads better at the use site (the current type is what you see first). The structs themselves still have to be declared oldest first (C++ completeness), noted in the README and example. Host test, example and docs updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DFjjnq3XCTRAXENSxsCxJU
|
✅Static analysis result - no issues found! ✅ |
cppcheck: VersionList::supports() uses std::any_of; the example's pass/fail counter is an if/else cppcheck can follow; the host test checks fopen() results before use. GCC: the build line in the host test's header comment no longer ends lines with a backslash (-Werror=comment), which is what failed the new host-test workflow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DFjjnq3XCTRAXENSxsCxJU
std::filesystem::path::c_str() is wchar_t on Windows, so the fopen / remove / rename calls in the codec did not compile in the lib build matrix. fopen now takes path.string(); remove and rename go through std::filesystem (which replaces an existing file atomically on POSIX and Windows), with the remove-then-retry fallback kept for filesystems that refuse to rename over an existing file. Host test (clang + gcc), cppcheck, macOS lib + consumer smoke and the esp32s3 example all re-verified. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DFjjnq3XCTRAXENSxsCxJU
… with_version alpaca's with_version hash folds sizeof(struct) into the type hash, and that differs between a 32-bit ESP32 and a 64-bit host (a struct of a string, a vector and an integer is 40 bytes on esp32 / esp32p4, 56 on macOS clang, 64 on Linux gcc), so a file written on the host would not load on the device. The payload now uses fixed_length_encoding | strict only, and the header carries a layout fingerprint computed by the codec: a CRC32 over alpaca's field_type codes walked recursively through the type (struct = struct_ + u16 field count, array = array + u32 length, enum = enum_class + underlying type, ...), with nothing platform dependent. The grammar is documented in the README so a reader without the C++ type can compute it from a field spec. Header grows to 20 bytes; a reused version number with a different layout now loads with the new layout_mismatch error. Host test covers the grammar bytes (114 checks). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DFjjnq3XCTRAXENSxsCxJU
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DFjjnq3XCTRAXENSxsCxJU
… list Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DFjjnq3XCTRAXENSxsCxJU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New header-only component
espp::PersistentData<Current, Older..., Oldest>: file-backed storage of one serializable aggregate (a settings struct, calibration data, a paired-device list) on any VFS filesystem (LittleFS, SPIFFS, SD card) with explicit schema versioning and forward migration.ESPD, schema version, layout fingerprint, payload length, CRC32) in front of the alpaca payload (fixed_length_encoding | strict). A load reportsfile_not_found,bad_header(truncated / wrong magic / trailing bytes),bad_checksum,unknown_version(newer firmware, or support dropped),layout_mismatch(a version number reused for a different layout),deserialize_failedor success withwas_migrated(). Trying to parse with each type in turn cannot do this: alpaca ignores trailing bytes, so a newer file "succeeds" as an older type and a layout change parses as garbage, even withstrict | with_checksum(verified on host).with_version. alpaca's type hash foldssizeof(struct)in, which differs between a 32-bit ESP32 and a 64-bit host (a struct of a string, a vector and an integer is 40 bytes on esp32 / esp32p4, 56 on macOS, 64 on Linux; verified with the three toolchains), so a file written on the host would not load on the device. The header's fingerprint is a CRC32 over alpaca'sfield_typecodes walked through the type with nothing platform-dependent; the grammar is in the README so a reader without the C++ type (the planned Python tool) can compute it from a field spec.static constexpr uint32_t VERSION(strictly decreasing) and every type except the last providesstatic T migrate(const Older &)from the type after it. An old file is decoded as its version and walked one step at a time, so each version only knows the one before it. Adding a version is a prepend, dropping the oldest is removing the last entry; a wrong order or missingmigrateis a static_assert with a plain message.<file>.tmp+ fflush + fsync + rename, with a remove-and-retry fallback for filesystems that refuse to rename over an existing file (SPIFFS), so a reset mid-save keeps the previous contents.std::error_codeeverywhere, a mutex,get()returning a copy,update(fn)for a locked read-modify-write,load_or_create(defaults)that creates onfile_not_foundonly (a corrupt or newer file is never silently replaced), assignment / conversion operators as sugar forset()/get(), and staticencode()/decode()so the same bytes can live in an NVS blob or go over a transport.The codec (format, version list, migration chain, file helpers) is
include/detail/persistent_data_codec.hpp, standard library + alpaca only, so it has a host test. The class adds the logger, mutex and Config on top. The component is also part of the cross-platform C++ library (lib/, included byespp.hpp); it is not in the Python bindings (class template over the app's own types), noted next to the other template-only headers inautogenerate_bindings.py.Example
components/persistent_data/examplestores a settings struct on LittleFS: a short day-to-day section (load or create,update, operators), then 37 pass / fail checks covering create, auto-load,update(), a v1 file written "by old firmware" migrated v1 → v2 → v3 and rewritten, a newer-firmware file refused with memory untouched, corruption / truncation detection andencode()/decode(). Added to the build matrix (esp32s3) and the storage docs.Test plan
components/persistent_data/test/persistent_data_host_test.cpp: 114 checks, 0 failures (clang and gcc,-std=c++23 -Wall -Wextra -Werror), cppcheck clean with the CI arguments; new workflowpersistent_data_tests.ymlruns it in CIlib/,ESPP_BUILD_PYTHON=OFF) builds and installs; afind_package(espp)consumer on macOS runs the class end to end (missing file,load_or_create,update, operators, auto-load, v1 → v2 migration and rewrite)Persistent data validation: 37/37 checks passed - PASS, and the boot counter increments across resets (expected output in the example README is derived from the checks, not captured yet)Follow-ups (not in this PR)
pyalpacastyle, derived from dataclasses with width hints) rather than litgen. The upstream prerequisite is open: pyalpaca: options argument, byte-identical container sizes, errors, tests and CI p-ranav/alpaca#55 (pyalpacaoptionsargument, byte-identical container sizes, tests, CI). The layout fingerprint above is what the Python side will compute from the spec.🤖 Generated with Claude Code
https://claude.ai/code/session_01DFjjnq3XCTRAXENSxsCxJU