Derive: stop recalculating unchanged days from scratch on every pass by mikenrafter · Pull Request #656 · OpenStrap/edge

mikenrafter · 2026-11-01T00:15:58Z

Draft status. Rebased onto main f4e3b3ec (2026-11-01); it merges cleanly. Rung 59 is still free and kAlgoVersion stays at main's 119. New in the rebase: main's raw-archive restore path also advances the store generation after each committed page (four tests), and the existing derive_unchanged_skip_test fixture derives its first night before both parity runs. Targeted runs on the rebased stack: 4,419 passed, 9 skipped, 0 failed (391 files, before the last two main commits) plus the head checks after them; flutter analyze clean. The benchmark and full-suite numbers below are still from the old base c8e63556; I will rerun them before marking this ready. New commit SHAs, in review order: 748ff8bf, a6fbecb8, f693d588, 11b944ab, 2ae2847d (the SHAs in the review order below are the pre-rebase ones, same order).

Every sync starts a recalculation pass, and each pass redoes work for days whose inputs have not changed. It reloads the day's data from SQLite, restages sleep for the day and the days before it, rescans every wearable day once per day it derives, and recomputes the HR ceiling of sessions that finished long ago. Some of the work also runs on the UI isolate: building Garmin, Colmi, Oura and Ultrahuman inputs, and preparing workout HR stats from two separate reads. On a desktop restore of my 199 MiB export, the pass Edge runs after a sync takes about 5.5 s on main even when nothing changed.

This PR lets Edge tell when a day's inputs have changed. Each 15-minute bucket of decoded data gets a revision counter, and the store gets a generation number. While those and the rest of a result's inputs stay the same, the pass reuses staged sleep, the loaded day data and finished-session ceilings, and moves the remaining UI-isolate preparation into workers. No stored number changes. That repeat pass drops to 0.8 s; a pass after a new 15-minute chunk drops from 5.3 s to 4.7 s, and a forced 3-day recalculation from about 21 s to 10–12 s once warm. The first pass after a start costs the same as main.

What a user sees. After the first pass, later passes over unchanged days should finish faster. Opening workouts no longer prepares HR on the UI isolate. Values, screens and formulas stay the same: kAlgoVersion stays 113 and the analytics pin is unchanged, so no one recalculates because of this PR. Phone timings are unmeasured.

What changes:

Schema rung 59 adds two small revision tables: input_rev (one row per 15-minute bucket, advanced by triggers on decoded_onehz and decoded_rr) and session_ceiling_rev (one row, advanced when a session is created, deleted, or has its identity or window edited). The rung is additive, with no backfill or scan at open.
Four in-memory reuse layers, each keyed on those revisions and the other inputs it reads: staged sleep per day, the last loaded day data, finished-session HR ceilings, and a wearable-day index that lives for one pass.
Passes that cannot fold the sleep profile walk only the days the target day can use. Passes that can fold keep the full walk.
Wearable input construction and workout session-window preparation run in the existing cancellable isolate. Worker arrays are adopted as typed columns instead of being copied again.

Design

Input revisions and store generation. Main has no cheap way to tell that a day's input changed: a MAX(rec_ts)/count fingerprint misses an identical INSERT OR REPLACE, a moved row or a late RR beat. Rung 59 adds input_rev(bucket INTEGER PRIMARY KEY, rev INTEGER NOT NULL), where a bucket is rec_ts / 900. Insert, update and delete triggers on both decoded tables advance the bucket, including an identical replace and both buckets of a moved row. Revisions commit and roll back with the rows. inputDayFingerprint(dayId) returns SUM and MAX over the LOCAL day's buckets (DST-aware, about 96 buckets on an ordinary day) plus a store generation. The generation is in memory and advances after a successful open, a committed wipe, and any import that committed a page. Existing data reads as revision 0 until it is written again, so the first pass after upgrade runs cold, as it does today.

Reuse layers. Each layer checks its full key on every lookup, compares the key as a JSON string (no hash), and never hashes sensor arrays on the UI isolate. A miss runs the original code path unchanged. A worker error or timeout never stores an entry.

Layer	Key, besides algorithm version and store generation	Bound	Dropped on
Staged sleep	Search bounds; each local date's revision SUM/MAX; per-signal ownership and priority; substrate shape; zone offsets over the window and at stored midsleeps; overrides; the warmed profile; stored sleep windows that can reach the history threshold; vendor nights	14 days per engine, LRU. Stores the worker's JSON only	Key mismatch. An entry computed without folding never serves a folding pass
Loaded day data	Day and role (search, day, sleep); exact bounds; every bucket revision in range; ownership spans; zone	2 days × 3 loads per engine, LRU	Background, memory pressure, dispose, finalization, eviction, a failed or unprepared pass. Background engines never retain
Finished-session ceiling	Day and session ID; the bucket revisions in the session window; `session_ceiling_rev` and the session's type, status and window; device family; priority; HR and acceleration ownership	128 sessions per engine	Key mismatch. Live sessions and heavy, forced, selected, rescan and import passes always recompute
Wearable-day index	Device ID and family	One pass	`finally` on every pass exit

When only the newest bucket of a loaded day changed (new data arriving), the loader keeps the unchanged prefix and rereads from that bucket's start. A change to any earlier bucket reloads the whole range. The raw beat axis is kept as loaded, and the existing running-maximum monotonization runs once over the assembled axis. Per-bucket family, device and first-temperature-unit metadata restores the same decisions the cold load makes.

Derived writes (HRR, scores, VO2max) do not advance session_ceiling_rev, so the score write at the end of a pass does not invalidate every session. Any session identity or window edit invalidates all sessions; this is conservative but simple.

Restricted walk. The pinned analytics habitualMidsleepSecFromHistory returns null below 14 distinct nights. Stored nights plus the number of earlier days the walk could add is an upper bound on what the target can see. Below 14, the target's prior is null with or without the earlier days, so the walk skips them and stops after the target. This runs only on passes that cannot fold the profile. Every pass that may fold keeps the full upstream walk, so profile training is unchanged.

Workers. Wearable builders keep their bounded SQL reads on the caller and send only the query results to runCancellableIsolate, which now runs in production and has a 30-second timeout per stage. The construction bodies are unchanged. getWorkouts reads session windows once and prepares HR stats and samples together in one worker, instead of two SQL reads. Windows stay inclusive, and multi-device ownership is unchanged. With workout sensors off, the original SQL paths run as before.

What does not change. kAlgoVersion (113), the analytics pin, every stored value, the golden two_device_day_expected.json, commit-before-ACK, INSERT OR REPLACE on decoded_onehz, sync durability, the derive scheduler, ownership resolution, profile folding, and readiness backfill and deferral. There are no checkpoints, no persistent caches and no heavy-calculation guard framework.

Not in this PR: faster import and sync (#617); screen reads during writes (#620). A later PR adopts OpenStrap/analytics#99 once it merges, with one kAlgoVersion bump. That PR also carries the two held-back correctness fixes (session-ownership end boundary, profile-training night), shared row streams, and persisted day checkpoints at rung 60.

Benchmarks

Main is c8e63556; this PR is 05f63eee. Desktop Linux 7.0.13 aarch64, 8 cores, 7 GB RAM, Flutter 3.41.6 / Dart 3.11.4, flutter test with sqflite_common_ffi under JIT, one process at a time. These are not phone measurements. Input: my 199 MiB phone export restored through LocalDb.importFromDbFile into a fresh database (untimed). Times are seconds per pass; derived day_result rows were byte-identical between main and this PR in every run.

The pass Edge runs after a sync (DerivationEngine.run(profile, heavy: false), one day, as derive_coordinator calls it):

	Main	This PR	Change
First pass after start	7.2	7.1	same
Repeat, no new data	5.5	0.8	about 7x faster
After one new 15-minute chunk	5.3	4.7	12% faster

A forced recalculation of the latest 3 days (runDays(force: true), the override re-analysis path), medians over 3–10 passes per run, 3–4 runs per side:

	Main	This PR
Cold, first pass	21.8–22.6	22.8–23.6
Warm, no new data	20.4–21.1	10.3–12.0
After one new chunk	20.3–21.4	10.2–12.6

The first warm pass after a restore re-stages all three days (about 18 s): the cold pass writes new sleep windows, which correctly changes the staging key once. With every new cache switched off, this PR matches main within about 1 s, which is the always-on cost of reading the day fingerprints and building the keys. Switching one feature off at a time: staged-sleep reuse is worth about 10 s of a warm forced pass and day-data retention about 5–6 s; the rest move it by 2 s or less. Session-ceiling reuse only runs on light passes, so it does not show in the forced table. About 10 s of a warm forced pass is work this PR does not touch (cross-day rollup, baselines, scoring and writes).

An earlier pair of runs of this PR measured a 35.5 s cold pass; seven later runs, including the unmodified original probe, did not reproduce it. I report it rather than drop it; my best guess is load on the machine at the time.

Peak RSS (`VmHWM` of the test process): main 811 and 827 MB, this PR 794 and 739 MB, so no increase within noise. Inserting one 15-minute chunk of decoded rows costs 7.2 ms instead of 4.7 ms because of the revision triggers.

Database size

After restoring the same export and a WAL checkpoint, the database is 193,454,080 bytes against 193,433,600 on main, 20 KiB more: input_rev 12 KiB (544 rows) and session_ceiling_rev 4 KiB (one row).

By construction, input_rev holds at most one row per occupied 15-minute bucket (about 96 per day of wear), rows are kept after deletes, and session_ceiling_rev holds one row. Both tables are rebuildable bookkeeping. They are copied whole in the diagnostic slice, and they are not restored from backups: restored rows fire the local triggers instead.

Tests

New files, each covering one change (the expanded test counts are below), with a shared fixture in test/support/ed2a_recalc_fixture.dart:

test/ed2a_input_revision_test.dart and test/ed2a_substrate_transfer_test.dart: 60. Rung 59 upgrade and repair with no scan or backfill; insert, replace, update, delete and move on both decoded tables; late RR beats; SUM/MAX on ordinary and DST days; generation on reopen, wipe, and successful, partial and failed imports; typed column adoption with defaults and the golden unchanged.
test/ed2a_e12_staging_cache_test.dart: 47, including one that builds an engine whose callback holds an unsendable object, so a staging closure that captures the engine fails. For every key part: a changed key misses once and gives exactly the cold output, and the new key then hits. Also folding versus non-folding, and absent sleep reused as a result.
test/ed2a_e13_substrate_retention_test.dart: 24. Full hits read no rows; an append rereads only the open bucket; beat order and first-unit decisions match a cold load; every clearing boundary.
test/ed2a_e15_restricted_target_walk_test.dart: 12. Walk counts, the 13+1 and 14-night boundaries, a different-day override, and full walks on folding passes.
test/ed2a_e18_session_reuse_test.dart: 24. The invalidation matrix; live sessions and non-light passes always compute; derived writes do not invalidate.
test/ed2a_e19_wearable_day_index_test.dart: 12. One scan per device and family per pass, including empty results; the index is released on errors; readiness backfill is unchanged.
test/ed2a_e20_worker_preparation_test.dart: 14. Construction runs in a worker for each family; device separation; strap overlay timing; Oura temperature-only input; one combined workout read; inclusive two-device samples.

Each file passes under TZ=UTC in targeted serial runs, together with the existing tests most at risk for that change. One E18 fixture was corrected during development. Its HR-ownership edit moved all HR to a second device that had no rows, so the existing guard that keeps a day with no input stopped the pass before the ceiling ran. The edit now also gives the primary device coverage, so it moves part of the window instead of removing all input. No existing test was edited.

Full suite, flutter test --concurrency=1 on this branch: 6,273 passed, 23 skipped, 0 failed (Linux aarch64, 56 min). The skips are environment-gated, as on main; this PR adds none.

I have run this on my own device: Pixel 9a on Android/GrapheneOS.

Suggested review order

Each commit carries its own tests and passes on its own.

39146127: input revisions and typed transfer, with their 60 tests. Check the trigger SQL and wipe ordering in db.dart, the rung 59 and open-repair calls, the generation bumps in import's finally, and Substrate.fromTransfer. Everything after builds on this.
8eae5997: staged_sleep_cache.dart, _sleepCandidateForDay, and the walk restriction in calendarDays. Check that hits still pass through the banked-night check and the locked profile fold, and that the staging closure reads only locals (an engine captured into it cannot be sent to the isolate; a test pins this). The shared test fixture and the inert test hooks for later commits land here.
5e85dbb1: substrate_retention.dart, prefix seeding in derive_prepare.dart, and the lifecycle wiring in app.dart and app_state.dart.
99918749: session_ceiling_rev, _ceilingSignatures, the light-pass-only reuse flag, and the pass-scoped wearable index in canonical.dart.
05f63eee: the wearable builders and sessionHrReads. Construction bodies should be unchanged apart from where they run.

Change distribution breakdown

Computed from git diff --numstat c8e63556: 24 files, +3,523 / -210 lines.

	Files	Added	Deleted
`lib/`	15	1,356	210
`test/`	9	2,167	0

Most of lib/ is compute/derivation_engine.dart (+597 / -130), data/db.dart (+240 / -42) and the two new cache files (staged_sleep_cache.dart, substrate_retention.dart). Part of the engine diff is indentation from wrapping existing pass finalizers in finally.

Known limits

The bounds count days and entries, not bytes. A fully populated 24-hour load is about 10 MiB of numeric buffers, so the retention bound is about 60 MiB in the worst case. That is an estimate from column sizes, not a measured RSS.
The store generation lives in memory per isolate. It does not tell another isolate that the store was replaced. Background, iOS BGTask and shortcut engines never retain loads.
The first pass after upgrade, and any pass after a restart, runs cold: caches are in memory only.
Each revision trigger adds a small insert and update to every decoded row write. Ingest overhead is not yet measured (see Benchmarks).
No phone measurements yet. The beatless-import gap in AGENTS.md and other restore behavior are unchanged.

Questions for reviewers

Is one PR acceptable for these reuse layers and worker moves, or would you prefer them split? Each is a separate commit on the same revision foundation, and all of them leave output unchanged.
Should both revision tables share rung 59, or should session_ceiling_rev get its own rung? Both are additive and empty at upgrade.
Are day-count bounds (14 staged days, 2 retained days, 128 sessions) acceptable for now, or do you want a byte budget before this merges?

coderabbitai · 2026-11-01T00:16:27Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

- Schema 59 adds input_rev(bucket, rev) with insert, update and delete triggers on decoded_onehz and decoded_rr. A bucket is rec_ts / 900; every write to a bucket advances its revision, including an identical INSERT OR REPLACE. No backfill: existing input stays at revision 0. - inputDayFingerprint(dayId) returns SUM and MAX of the day's bucket revisions plus a store generation that advances on open, after a committed wipe, and after an import that committed a page. - Substrate.fromTransfer adopts the worker's typed columns instead of copying them; the derivation engine receives its prepared substrate through it. - input_rev is copied whole in the diagnostic slice. kAlgoVersion is unchanged (113): no derived number moves.

…lk only the days the target can use - A per-engine cache (14 days, LRU) keeps the staging worker's candidate and observation JSON. The key covers the search bounds, each local date's input revisions and store generation, resolved ownership and priority, the substrate shape, zone offsets over the window and at stored midsleeps, overrides, the warmed profile, stored sleep windows that can reach the history threshold, and vendor nights. A result computed without folding never serves a folding pass. - A pass that cannot fold the profile skips earlier days while the history could not reach 14 distinct nights (the habitual-midsleep prior is null either way) and stops after the target. Folding passes keep the full walk. Output is unchanged; kAlgoVersion stays 113.

…ts inputs are unchanged The engine retains a bounded number of loaded day substrates, keyed like the staged-sleep cache by input revisions, store generation and ownership, and reuses an exact prefix or tail instead of reloading the day from SQLite. Raw and monotonized beat axes, starts and bucket provenance are kept as loaded. Retention is dropped when the app goes to the background, on memory pressure, on dispose, at finalization and eviction, and after a failed or unprepared pass. Output is unchanged; kAlgoVersion stays 113.

…hanged; index wearable days once per pass - A finished session's HR-ceiling envelope and unrounded winning bpm are reused while the day's input revisions, store generation, device family, priority and HR/acceleration ownership are unchanged. Session identity and window writes advance a session_ceiling_rev counter (rung 59, beside input_rev); derived score writes do not. Live sessions always recompute. - Wearable days are indexed once per pass instead of rescanned per day; the index is cleared in finally on every exit. Readiness deferral unchanged. - The E18 HR-ownership test gives the primary coverage as well, so the edit moves part of the window instead of removing all HR input. Output is unchanged; kAlgoVersion stays 113.

- Wearable construction and overlay loops for Garmin, Oura, Colmi and Ultrahuman run through the existing cancellable isolate with typed results instead of on the UI isolate. - getWorkouts reads session windows once and prepares the HR stats and samples together in one worker, instead of two reads. Windows stay inclusive; two-device behavior is unchanged. Output is unchanged; kAlgoVersion stays 113.

mikenrafter mentioned this pull request Nov 1, 2026

Derive: recalculate a night from where the last sync stopped, and survive an app restart #657

Draft

mikenrafter force-pushed the pr/edge-recalc-exact branch 2 times, most recently from c093c95 to 9481dec Compare November 1, 2026 14:48

mikenrafter added 5 commits November 1, 2026 07:48

mikenrafter force-pushed the pr/edge-recalc-exact branch from 9481dec to 2ae2847 Compare November 1, 2026 14:59

Navigation Menu

Repository navigation

Derive: stop recalculating unchanged days from scratch on every pass - #656

Derive: stop recalculating unchanged days from scratch on every pass#656
mikenrafter wants to merge 5 commits into
OpenStrap:mainfrom
mikenrafter:pr/edge-recalc-exact

mikenrafter commented Nov 1, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Nov 1, 2026

Draft PR not reviewed

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mikenrafter commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Design

Benchmarks

Peak RSS (VmHWM of the test process): main 811 and 827 MB, this PR 794 and 739 MB, so no increase within noise. Inserting one 15-minute chunk of decoded rows costs 7.2 ms instead of 4.7 ms because of the revision triggers.

Database size

Tests

Suggested review order

Change distribution breakdown

Known limits

Questions for reviewers

Uh oh!

coderabbitai Bot commented Nov 1, 2026

Draft PR not reviewed

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mikenrafter commented Nov 1, 2026 •

edited

Loading

Peak RSS (`VmHWM` of the test process): main 811 and 827 MB, this PR 794 and 739 MB, so no increase within noise. Inserting one 15-minute chunk of decoded rows costs 7.2 ms instead of 4.7 ms because of the revision triggers.