Repository navigation
Derive: recalculate a night from where the last sync stopped, and survive an app restart - #657
Draft
mikenrafter wants to merge 11 commits into
Draft
mikenrafter wants to merge 11 commits into
mikenrafter wants to merge 11 commits into
Conversation
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
mikenrafter
force-pushed
the
pr/edge-analytics-adopt
branch
from
November 1, 2026 14:48
0459ac9 to
7e32d6c
Compare
- Schema 59 adds input_rev(bucket, rev) with insert, update and delete triggers on decoded_onehz and decoded_rr. A bucket is rec_ts / 900; every write to a bucket advances its revision, including an identical INSERT OR REPLACE. No backfill: existing input stays at revision 0. - inputDayFingerprint(dayId) returns SUM and MAX of the day's bucket revisions plus a store generation that advances on open, after a committed wipe, and after an import that committed a page. - Substrate.fromTransfer adopts the worker's typed columns instead of copying them; the derivation engine receives its prepared substrate through it. - input_rev is copied whole in the diagnostic slice. kAlgoVersion is unchanged (113): no derived number moves.
…lk only the days the target can use - A per-engine cache (14 days, LRU) keeps the staging worker's candidate and observation JSON. The key covers the search bounds, each local date's input revisions and store generation, resolved ownership and priority, the substrate shape, zone offsets over the window and at stored midsleeps, overrides, the warmed profile, stored sleep windows that can reach the history threshold, and vendor nights. A result computed without folding never serves a folding pass. - A pass that cannot fold the profile skips earlier days while the history could not reach 14 distinct nights (the habitual-midsleep prior is null either way) and stops after the target. Folding passes keep the full walk. Output is unchanged; kAlgoVersion stays 113.
…ts inputs are unchanged The engine retains a bounded number of loaded day substrates, keyed like the staged-sleep cache by input revisions, store generation and ownership, and reuses an exact prefix or tail instead of reloading the day from SQLite. Raw and monotonized beat axes, starts and bucket provenance are kept as loaded. Retention is dropped when the app goes to the background, on memory pressure, on dispose, at finalization and eviction, and after a failed or unprepared pass. Output is unchanged; kAlgoVersion stays 113.
…hanged; index wearable days once per pass - A finished session's HR-ceiling envelope and unrounded winning bpm are reused while the day's input revisions, store generation, device family, priority and HR/acceleration ownership are unchanged. Session identity and window writes advance a session_ceiling_rev counter (rung 59, beside input_rev); derived score writes do not. Live sessions always recompute. - Wearable days are indexed once per pass instead of rescanned per day; the index is cleared in finally on every exit. Readiness deferral unchanged. - The E18 HR-ownership test gives the primary coverage as well, so the edit moves part of the window instead of removing all HR input. Output is unchanged; kAlgoVersion stays 113.
- Wearable construction and overlay loops for Garmin, Oura, Colmi and Ultrahuman run through the existing cancellable isolate with typed results instead of on the UI isolate. - getWorkouts reads session windows once and prepares the HR stats and samples together in one worker, instead of two reads. Windows stay inclusive; two-device behavior is unchanged. Output is unchanged; kAlgoVersion stays 113.
… 120 The pin moves 3a7ca34 -> f8e7d3c, the head of analytics OpenStrap#99 (re-point at its merge commit once it lands). Outputs move, so kAlgoVersion goes 119 -> 120 with a changelog entry: the RR artifact-run cap of 32 beats, toleranced batch aggregators (last digits), the Lomb-Scargle origin, near-singular cosinor abstention, non-finite guards and irregular-screen gaps. Goldens regenerated: on the WHOOP freeze fixtures TRIMP, SDNN and dyn_p90 move by at most 2.7e-15 relative, on the two-device fixture one SDNN value by 3.6e-14; everything else that changes is the version stamp and the payload hashes that follow from it. Test-only seams for the following commits are inert here.
… from the target night - Coverage and ownership are queried and resolved through `to + 1`: the callers pass an inclusive `to` into half-open resolution, so a non-owner's rows in the last second were kept. - A folding pass records and folds only the target night's observation (none when the target has none) instead of the longest night it walked, and restores the recording flags. Forward-only: existing profiles and folded_days are not reset. The restricted walk may now skip unused days on folding passes too. Neither change moves a value in the committed fixtures.
…n memory; hash shared lists once per pass - One CardioRowCache serves every staging path: the engine keeps the encoded bytes, workers decode, merge and prune the rows. - The night's corrected RR stays in a RAM owner between passes, keyed by the day's input revisions, store generation and algorithm. - Shared lists are fingerprinted once per pass; the identity memo is cleared between passes. Outputs are unchanged.
…A across passes Built on the night RR owner and analytics OpenStrap#99's streams: a pass feeds only the beats it has not seen. CVHR drops the tail-only fallback because OpenStrap#99 carries the fresh-stream fix; PRSA keeps sequential slices. Results equal one batch call on the same input, within OpenStrap#99's documented tolerances.
One union load per pass, cut into the windows in workers, borrowing overlap from the retained day data. Exact given the to+1 ownership endpoint.
… restarts - day_checkpoint keeps a versioned, checksummed night/cardio state for at most three days (8 MiB cap per state), written after the day result commits. Resuming from it equals a cold derive byte for byte; a bad, old or foreign checkpoint is refused whole and the night is recomputed. - store_identity holds a random id. Wipe, a committed import or restore, salvage/rebuild and a replaced file get a new id and clear checkpoints, so a checkpoint survives an app restart but never a different store. RAM owners still key on the per-open generation; an algorithm change never adopts a disk checkpoint. - Rung 60 is additive and idempotent, repeated by open repair, with no scan or backfill. day_checkpoint is left out of diagnostic exports.
mikenrafter
force-pushed
the
pr/edge-analytics-adopt
branch
from
November 1, 2026 14:59
7e32d6c to
79bea7b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every sync recalculates the whole night, so each 15-minute sync costs more than the one before. On a desktop AOT build of my own night, analytics#99 measured about 109 s of RR correction, HRV frequency, respiration, CVHR and staging added up over a night of syncs, nearly all of it redoing beats Edge had already seen. Edge also loads the same decoded rows several times per day (sleep search, nap, calendar day, night), and it keeps what it learned only in memory, so a restart starts the night over.
This PR makes Edge use the streams and row caches that analytics#99 adds. A sync folds only the beats that arrived since the last pass, the overlapping windows are loaded once, and the night's running state is saved to the database so it carries over a restart. A pass that adopts a saved night reprocesses none of its closed beats. Results match a cold batch pass (PRSA to 1e-9; everything else exactly).
kAlgoVersionmoves once, 113 to 114, because analytics#99 changes some outputs and because two correctness fixes ride on the same bump. On my 199 MiB export, a warm forced 3-day recalculation takes about 3 s, against 10 s on ED-2a and 21 s on main, and a pass after a new 15-minute chunk 4.3–5.4 s against 13 s and 21 s. The first forced pass after an app restart resumes from the saved nights in 11–13.5 s instead of 20–21 s. The light pass after a sync stays at ED-2a's numbers.What a user sees. Every device recalculates once after updating, because of the 113 to 114 bump. After that, later syncs on a night should finish sooner, and the first sync after restarting the app can resume the night instead of redoing it. Values stay the same to the last digits on the committed fixtures (on the six WHOOP freeze fixtures TRIMP, SDNN and
dyn_p90move by at most 2.7e-15 relative; on the two-device fixture one SDNN value moves by 3.6e-14; everything else that changes is the 113 to 114 stamp and the payload hashes that follow from it). Analytics#99's own output changes (RR artifact-run cap 32, Lomb-Scargle origin, singular-cosinor abstention, non-finite guards, irregular-screen gaps) can move more on inputs that exercise them; see its Output changes. Two fixes can change a number: a non-owner row in the last second of a window no longer leaks in, and the sleep profile learns from the target night only. Phone timings are unmeasured.What changes:
c32007f83094c1fb39553301e5967052f8ba9dd5(re-pointed at the merge commit once docs: add AI Coach BYOK setup guide #99 merges).kAlgoVersion113 to 114, with a changelog entry. The committed golden fixtures are regenerated for the new stamps and last digits.to + 1; the sleep profile folds only the target night's observation, or none if the target has none.day_checkpointand a one-rowstore_identity. Night and cardio state are saved after the day result commits, keyed on that durable identity, so they survive a restart but never a wipe, import or restore, rebuild, or replaced file. At most 3 retained days, at most 8 MiB per state.Design
Correctness fix: ownership through
to + 1. The loader reads rows through the secondtoinclusive, but ownership and coverage were resolved over the half-open range ending atto. A non-owner row in the last second was therefore not filtered, and endpoint-only coverage could not take part in priority._resolveOwnershipnow computesend = to + 1once and uses it for bothcoverageIntervalsandresolveOwnershipon all five exclusive signals. Priority and flag filtering are unchanged. The serializedseries.coveragespans of a multi-owner day extend toto + 1, so a payload hash can change even when no metric does. This is why the fix needs the bump.Correctness fix: the sleep profile learns from the target night. With ED-2a's restricted walk, a folding pass still ran segmentation on earlier days and folded the longest observation of the whole walk, so a short target night could train on a longer neighbour, and a target with no accepted sleep could still fold. Now only the target's segmentation pass records an observation (
recordTargetOnly), and nothing folds unless the target candidate is present. The restricted walk now also runs on passes that may fold. D36 is forward-only: nothing resets or replays. Existing profiles andfolded_daysare untouched, and days already tracked stay ineligible. Profiles trained before this PR keep whatever they learned.Shared row cache and night owner. The staging worker receives at most four encoded cardio chunks (8 MiB total, at most 8,000 row variants each) with their revision metadata, decodes and merges them once, shares the cache across every staging call in the request, and returns one pruned chunk. The caller never parses rows.
NightRrStateowns #99's cappedRrCorrector, the folded raw prefix and the settled NN, compares the whole held prefix bit for bit, and folds only appended beats. A changed onset, day, shortened or edited prefix, or a changed revision, generation or algorithm resets it. Inputs that are empty, non-finite, mismatched or over 200,000 beats use the original batch path. The five batch consumers share two corrected-night lists that are hashed once per pass into 128-bit proofs; the memo is released infinally, including when the end-of-pass callback throws.Night streams. Frequency, RSA and CVHR each keep an independent settled-prefix feed count and snapshot with the current provisional tail. RSA bins keep the batch bin width (1,800,000 ms), the first-corrected-beat origin, the 60-beat minimum and sorted order; only resolved closed-bin rates are stored. PRSA appends settled slices with
prsaAggregator.addSlicein order and never callscombine; snapshots copy the state, add the tail and finish. Frequency, RSA, CVHR and RSA bins must equal the v114 batch result exactly. PRSA uses #99's allowance (1e-9 absolute plus relative on capacity and each profile entry, 1e-12 on confidence; anchors, kind and notes exact). All owners belong to the per-dayNightRrState, at most two days, and are cleared by retention disable, background trim, memory pressure, dispose, and any discarded, partial or finalized day.Union loading. For an ordinary live day, one load covers the sleep search window and the buffered calendar day; staging reads an exact search cut, and one later worker cuts nap, calendar day and selected night. Cuts select beats by the original
rr_startand restart the running maximum fromrr_ts_raw, so backward placements match separate loads. A retained union is borrowed only for unchanged revision buckets with the same algorithm, store generation, timezone and ownership; otherwise the gap is read, or the original separate loaders run. A cut or join that would change the first temperature unit is refused. Revisions are read before rows, so a racing write cannot certify the next pass. Unions live in ED-2a's retention map (two days, one union per day, at most 500,000 sample slots and 2,000,000 beats).Checkpoints. After the day result is persisted, the engine writes one
day_checkpointrow (state= SHA-256 followed by UTF-8 JSON, format 1) holding the capped corrector, the raw and settled prefix, the frequency, RSA and CVHR streams, RSA-bin state, PRSA aggregates, and the pinned cardio row cache (at most 8,000 variants). The saved prefix ends at a closed 15-minute boundary, andrev_vecrecords every closed bucket revision, with zeros for missing buckets. The open bucket is excluded, so an append there resumes an unchanged prefix. Validity needs the same algorithm, day, context signature, boundary, revision vector, raw prefix and selected onset. Any failure in any section returns neither section and the pass runs cold.Validity is keyed on a durable
store_identity(one row, 16 random bytes as 32 hex characters), not on the per-open generation. The generation is an in-memory counter that advances on every open, so a check against it would reject every restart. Identity rotates onwipeAll, on the first committed page of any import or restore, and on salvage or rebuild, and it is never imported or restored, so a replaced database file carries its own. Rotation happens inside the same transaction as the first page, so a process killed before the import'sfinallycannot keep valid old scratch. Rotation also deletesday_checkpoint; the closed-bucket revision vector covers ordinary writes. Both tables areSliceKeep.noneand are outside the restore and salvage allow-lists. The checkpoint is written after the day result commits and never fails a successful derive. Finalized or partial days delete their checkpoint; a write keeps the newest three days and the current algorithm. A different algorithm never adopts a disk checkpoint, even when a stale RAM owner is present. RAM owners still key on the per-open generation.What does not change. Stored shapes, staging labels, the derive scheduler, commit-before-ACK,
INSERT OR REPLACEondecoded_onehz, ED-2a's input revisions and reuse layers, and export and restore of user data. The rung is additive: oneCREATE TABLE IF NOT EXISTSper table plus anINSERT OR IGNOREof the identity row, repeated by open repair, with no backfill or scan.Not in this PR: faster import and sync (#617); screen reads during writes (#620); the ED-2a reuse layers, which this builds on.
Benchmarks
Same method as ED-2a's tables: desktop Linux 7.0.13 aarch64, 8 cores, 7 GB RAM, Flutter 3.41.6 / Dart 3.11.4,
flutter testwithsqflite_common_ffiunder JIT, one process at a time; my 199 MiB export restored into a fresh database before each scenario (untimed). Mainc8e63556, ED-2a05f63eee, this PR0459ac91, run in the order main, ED-2a, this PR, this PR, ED-2a, main. Seconds per pass, two runs each (run 1 / run 2), medians of 3 passes per run. These are not phone measurements. This PR's outputs differ from main and ED-2a by design (the 114 bump), so outputs are compared only within this PR: the stored day results were identical after the final warm pass and both passes after a restart, in every run.Forced 3-day recalculation (
runDays(force: true), the override re-analysis path):The pass Edge runs after a sync (
run(heavy: false), one day):The restart gain is on the forced path only: the light pass alone did not write a checkpoint in this run (
day_checkpointhad no rows after the light scenario), so a restart followed by light passes recomputes the night as on main. The first forced pass after a restart is still 3–4 times a warm pass; the second is warm again. The cold forced pass came out faster than main and ED-2a, but its outputs differ by design and I did not isolate why, so I do not claim it. One ED-2a restart pass measured 15.1 s in run 2 (10.2 s in run 1); I left it as an outlier.Peak RSS (
VmHWMof the test process, both scenarios): main 809 / 767 MB, ED-2a 720 / 769 MB, this PR 839 / 793 MB, about 3–10% above ED-2a and within its spread.For the analytics side (night-of-syncs and day-of-syncs speedups, 15.3x on a night in a desktop AOT build), see #99's Benchmarks. Those are analytics numbers, not Edge pass times.
Database size
Rung 60 adds no triggers and no index. At most three
day_checkpointrows (one per retained day) are kept. Measured standalone worker payloads, not full on-device DB sizes:That last row adds 373,195 bytes for the day (about 364 KiB) before SQL, index and WAL overhead; a real night is that order. The hard bounds are 8 MiB per state and 24 MiB of state BLOBs for three days, plus the revision vector (63 to 889 bytes in these probes). Measured on the same restored export after a WAL checkpoint: 196.7 MB after the warm forced passes, against 194.35 MB on ED-2a and 194.33 MB on main; 197.42 MB after the restart passes.
day_checkpointwas 1,888,256 bytes and then 3,055,616 bytes, andstore_identity4,096 bytes. So up to about 3 MB on a 194 MB database. I have not yet looked into why the checkpoint table grew between those two points.Tests
New files, each covering one change:
test/ed2b_algo_version_114_test.dart: the version/changelog guard.ed2a_substrate_transfer_test.dartanded2a_input_revision_test.dartnow assert 114 andLocalDb.schemaVersioninstead of fixed numbers.test/ed2b_d35_ownership_endpoint_test.dart: 6. Endpoint-only HR and RR coverage; a non-owner last-second row and beat excluded; a wide window equals a single-second window; interior and absence guard.test/ed2b_d36_target_profile_test.dart: 9. Target-only full walk; production fold input; absent target folds nothing; forward-only preservation of profile bytes andfolded_days; flag restored on success and on error; fold order replay; sorted bookkeeping.test/ed2b_cardio_cache_test.dart: shared cache across all four staging paths, refusal of a malformed chunk, revision, generation and algorithm invalidation, byte and window bounds. 17 tests.test/ed2b_streams_test.dart: night RR owner, memoized lists and cleanup on every exit; frequency, RSA, CVHR, RSA bins and PRSA against a cold batch at append, mutation, shifted onset and absence, including real isolate dispatch. 44 tests.test/ed2b_e14_union_loading_test.dart: search, nap, day and night equality with separate loads; overlap borrowing; invalidation; temperature-unit refusal. 7 tests.test/ed2b_checkpoints_test.dart: 32. Includes restart resume, and rejection after a wipe plus re-import with repeated revision sums, a committed import, a replaced file and a rebuild. Rung 60 upgrade, repair and no-scan; codec refusals (old or future format, wrong algorithm, day, context or boundary, damaged vector, torn or checksum-damaged bytes, malformed nested sections); append and restart equality with zero reprocessed closed beats; a close and reopen keeps the checkpoint (identity unchanged, generation advanced); rotation on a committed import, a later-page failure, wipe plus full re-import with identical revisions, a replaced file with identical contents, and the ladder-brick rebuild and salvage; a stale blob put back after each is rejected and the output equals cold; empty and first-page-rollback imports keep identity and scratch; export classification.Existing tests changed:
ed2a_e15_restricted_target_walk_test.dart(the folding case D36 deliberately supersedes: one target observation, one segmentation),multidevice_coverage_derive_test.dart(the expected spans now end atto + 1),data_ownership_test.dart(wipe leaves exactly onestore_identityrow, with a new id),ed2a_input_revision_test.dartanded2a_substrate_transfer_test.dart(version numbers), anded2b_cardio_cache_test.dart's generation variant disables checkpoints so it measures the RAM owner. Fixtures regenerated:two_device_day_expected.jsonand the sixwhoop_freezefiles. In those, only the algorithm stamp, payload hashes and last digits moved; no other test or fixture was edited.Each commit passes its own test files on its own. Full suite,
flutter test --concurrency=1on this branch: 6,390 passed, 23 skipped, 0 failed (Linux aarch64, 47 min). The skips are environment-gated, as on main; this PR adds none.I have run this on my own device: Pixel 9a on Android/GrapheneOS.
Suggested review order
Commits are by concern, smallest first. Each carries its own tests and passes on its own.
f45215ab(repin): analytics pin,kAlgoVersion114 and its changelog, the version guard, and the regenerated golden fixtures. Check that only stamps, hashes and last digits moved.08681f8b(ownership endpoint and target-night profile):_resolveOwnershipthroughto + 1, with its 6 tests and the multi-device span expectations; thenrecordTargetOnlyincalendarDays, thecandidate.presentgate in_sleepCandidateForDay, and the updated E15 case.e25c60ad(cardio cache, night owner, shared lists):cardio_rows_cache.dart,night_rr_state.dart, the digest memo inonehz_pipeline.dart, and the retention clearing points.021c7027(night streams): the five stream consumers innight_rr_state.dart,night_resp_bins.dart,night_prsa.dart.404e5259(union loading):substrate_union.dart,_loadUnion, the provenance runs inderive_prepare.dart.0459ac91(checkpoints): rung 60 andstore_identityindb.dart, the rotation points (wipeAll,_mergeFromDbFile, rebuild),night_checkpoint.dart, the engine lookup and write. Read the rotation points first; the codec second.Change distribution breakdown
Against the ED-2a head
05f63eee: 37 files, +3,781 / -257 lines;lib/is 16 files, +1,818 / -50, and the rest is tests and the regenerated fixtures.lib/test/(including six regenerated fixtures)pubspec.*Most of
lib/isderivation_engine.dart(+581 net) and the new filesnight_rr_state.dart,substrate_union.dart,night_checkpoint.dartandnight_resp_bins.dart.Known limits
Questions for reviewers
store_identityrow that rotates on wipe, committed import or restore, rebuild and salvage, and is never imported. Keying on the per-open generation instead would reject every restart. Is a random row the right shape, or would you prefer something derived from the file (for example a header field) or a different rotation point?folded_dayskeep whatever they learned, with no reset or replay on adoption. Do you want a one-time reset or rebuild instead, accepting that every user's profile would warm up again?day_checkpoint,store_identity, restart resume) be split into its own PR after the in-memory streams and union loading? The streams and union loading leave stored output unchanged and need no schema, while the rung adds a table and a trust boundary.