D-STO-5: history as a fold over resident log records (0.72 ms, was 336 ms)

.claude/board/ISSUES.md

@@ -5,7 +5,7 @@
 
 ## ISS-HIST-SCAN-COST — property history scans the whole log
 
-- **Status:** OPEN (D-STO-5 budget; correctness shipped)
+- **Status:** RESOLVED 2026-10-10 without either lever: history folds over the resident log records instead of scanning Lance (0.72 ms vs 336 ms); Lance is read once, by replay. No index, no projection, no Lance filter expression.
 - **Where:** `hubspo-store::log::Log::history`.
 - **What:** history reads every log record and runs the Quack program over them. Measured 2026-10-10 (`examples/probe_history.rs`, release, 100k records, 1,000 versions × 1,000 updates, 1.2M log records, one fragment per version): 336 ms, 320 ms of it the raw scan. P-HIST-1's budget is < 20 ms. The program is not the cost.
 - **Not a lever:** compaction. DECISION 2026-10-10 (operator): the log is never compacted, because the append order gives version order for free. (A one-off measurement compacted to one fragment read in 57 ms, still over budget.)

.claude/board/LATEST_STATE.md

@@ -145,9 +145,13 @@
 - `Log::history(specs, row, col) -> Vec<Entry>`: every write of one
   property on one record, oldest first — the inserted value (absent
   included), each update, the deletion — each with its version and owner.
-  No history store: one Quack program per log batch,
+  No history store and no Lance read: one Quack program,
   `row = r ∧ ((lane = c ∧ op = SET) ∨ op = DELETE)`, on mask-risc over the
-  log's Arrow buffers (borrowed); only matches are decoded. The log's `op`
+  log's resident records; only matches are decoded. `Log` keeps every record
+  it commits or replays resident (29 B each). Lance is the persistence
+  clock: `replay` reads it once, `commit` writes one version per batch.
+  Rule applied: one write costs about a million folds (lance-graph
+  2026-09-24); a Lance scan is recovery cost. The log's `op`
   and `lane` columns widened to `u32` for this (no data existed yet).
 - Tests: for every record and both lanes over 30 random batches, the
   history equals one written down from the batches (anti-vacuity: updates,
@@ -156,8 +160,13 @@
   DELETE arm, drop `op = SET`, lose the I32 sign. A sort by version was
   inert under a disable run and is removed: append order is version order.
   Replay dropped its sort too; a log out of order is still refused there,
-  as a batch whose version is not the next one.
-- Timing: fragmented log (1,001 fragments, 1.2M records): **336 ms**, of which the raw scan is 320 ms. Fails P-HIST-1's < 20 ms; the Quack program is 3–15 ms of it (release, `examples/probe_history.rs`, 7 reads, 2026-10-10). The scan, not the program, is the cost. Open: ISS-HIST-SCAN-COST.
+  as a batch whose version is not the next one. Resident: a second log
+  replayed from disk gives identical histories; a log whose records are not
+  at the table's version refuses to commit; a refused replay leaves the
+  resident records unchanged, so a corrupt record can never reach history.
+  Disable runs 2026-10-10, all red: resident-version check off, records
+  installed before replay validates, commit not appending to them.
+- Timing (release, `examples/probe_history.rs`, 100k records, 1,000 versions × 1,000 updates, 1.2M log records, 2026-10-10): history **0.72 ms** median over 101 reads (0.64–1.99), inside P-HIST-1's < 20 ms; replay from Lance 532 ms, once at startup. Superseded: reading history from Lance took 336 ms (#24), 320 ms of it the scan.
 
 ## Board view (D-QRY-3, board half)
 

.claude/board/STATUS_BOARD.md

@@ -39,7 +39,7 @@
 | D-STO-2 | KV value store for variable-length properties; zero-copy lens reads | M | Queued |  |
 | D-STO-3 | Edge lanes + association labels (H-ADR-4); cardinality invariants as Quack `GroupReduce` | L | Queued |  |
 | D-STO-4 | Tenant layout + P-TEN-1 probe | M | Queued | P-TEN-1 passed 2026-10-10 (7.8 ms p99 at 10k tenants, local disk) |
-| D-STO-5 | Property history via version log + P-HIST-1 probe | M | In progress | read shipped in PR: `Log::history` (one Quack program per log batch, no index); fragmented log (1,001 fragments, 1.2M records): **336 ms**, of which the raw scan is 320 ms. Fails P-HIST-1's < 20 ms; the Quack program is 3–15 ms of it (release, `examples/probe_history.rs`, 7 reads, 2026-10-10). Budget open: ISS-HIST-SCAN-COST. P-HIST-1 failed on in-place updates (46 ms); P-SPARSE-1 passes with the split-tunnel sparse delta: 16.9 ms indexed, 3.3–5.9 ms compacted |
+| D-STO-5 | Property history via version log + P-HIST-1 probe | M | In PR | `Log::history` folds one Quack program over the resident log records; Lance is read once, by replay. **0.72 ms** for 1,001 entries over 1.2M records (release, `examples/probe_history.rs`, 101 reads, 2026-10-10) — passes P-HIST-1's < 20 ms; replay 532 ms once at startup. #24 shipped the read (Lance scan, 336 ms); this PR makes it resident |
 | D-STO-6 | Importer (value oracle, part 1): sandbox portal → crm-sim plan → commit; counts compared per object type | L | Queued |  |
 
 ## Wave 3 — Query, lists, views, reports

.claude/plans/hubspo-integration-v1.md

@@ -861,8 +861,15 @@ rewritten, so nothing depends on keeping old Lance versions.
 
 *Decision 2026-10-10:* the log is never compacted. The append order gives
 version order for free, and readers (history, replay) rely on it instead of
-sorting. History speed comes from reading fewer columns or an index on
-`row` (ISS-HIST-SCAN-COST), not from compaction.
+sorting.
+
+*Decision 2026-10-10:* history is a fold over the log records held
+resident, not a read of Lance. One write costs about a million folds
+(lance-graph 2026-09-24), and a Lance scan is recovery cost; so `replay`
+reads Lance once and every read after it folds over memory. Measured:
+0.72 ms for 1,001 entries over 1.2M records, against 336 ms scanning Lance.
+New persistent state comes only from learning, recorded after as one
+merged, auditable outcome; derived reads like history write nothing.
 
 ### 5.3 Timeline
 
@@ -1108,7 +1115,7 @@ with identical responses modulo ids/timestamps.
 |---|---|---|
 | P-CUSTOM-1 | custom property layout | fold over 500 custom properties × 1M rows within 2× of a standard property fold — **lanes pass (1.61× at 500), packed fails (18×)**, 2026-10-10 |
 | P-TEN-1 | dataset per tenant×type overhead | open + first read at 10k tenants < 50 ms p99 — **passes: 7.8 ms fresh / 5.0 ms shared session, local disk**, 2026-10-10 |
-| P-HIST-1 | property history from version log | one property, one record, 1,000 versions < 20 ms — **fails, 2026-10-10: best 46 ms (address index, 101 changes, flat in N); version-log reads 3–147 s**. **Passes with the split-tunnel sparse delta (P-SPARSE-1): 16.9 ms indexed, 3.3–5.9 ms compacted** |
+| P-HIST-1 | property history from version log | one property, one record, 1,000 versions < 20 ms — **fails, 2026-10-10: best 46 ms (address index, 101 changes, flat in N); version-log reads 3–147 s**. **Passes with the split-tunnel sparse delta (P-SPARSE-1): 16.9 ms indexed, 3.3–5.9 ms compacted**. **Passes as shipped (D-STO-5): 0.72 ms as a fold over resident log records** |
 | P-LIST-1 | active list incremental re-eval | delta of 1k changed rows on 10M population < 10 ms; allocation independent of population — **time passes (2.6 ms at 10M); allocation grows with fragments touched (~200 KiB each), not rows**, 2026-10-10. **Both pass with the sparse delta (P-SPARSE-1): 1.93 ms, 257 KiB flat at 10M** |
 | P-QDB-1 | Quack as the database for tickets / mail / delivery / documents | every read through a bound Quack program over Lance-resident columns, answers equal a plain-loop oracle, < 50 ms at 3M tickets — **passes with fk-ordered storage (all ≤ 25 ms at 3M except Q9 58.6 ms, whose fk is random); random fk layouts fail (253 ms / 1.9 s)**, 2026-10-10 |
 | P-DEDUP-1 | blocking recall | ≥ 99% of labelled duplicates share a block |

crates/hubspo-store/examples/probe_history.rs

@@ -1,6 +1,7 @@
 //! D-STO-5 timing: history of one property of one record, read from a log of
 //! `VERSIONS` batches of `UPDATES` updates each over `ROWS` records, every
-//! batch also writing the probed property. No index on `row`.
+//! batch also writing the probed property. History folds over the resident
+//! records; replay is the one Lance read, timed separately.
 //!
 //! `cargo run --release -p hubspo-store --features lance --example probe_history`
 
@@ -24,7 +25,7 @@ const SPECS: [LaneSpec; 1] = [LaneSpec {
 async fn main() {
     let dir = tempfile::tempdir().expect("tempdir");
     let uri = dir.path().join("probe.lance");
-    let log = Log::new(uri.to_str().expect("utf8"));
+    let mut log = Log::new(uri.to_str().expect("utf8"));
     let mut t = Table::new(OWNER, &SPECS).expect("table");
     let started = Instant::now();
     let seed = Batch {
@@ -52,50 +53,41 @@ async fn main() {
         started.elapsed().as_secs_f64()
     );
     // One fragment per version: the log is never compacted.
-    measure(&log, uri.to_str().expect("utf8"), "fragmented").await;
-}
-
-fn median(mut ms: Vec<f64>) -> (f64, f64, f64) {
-    ms.sort_by(f64::total_cmp);
-    (ms[ms.len() / 2], ms[0], ms[ms.len() - 1])
-}
+    let uri = uri.to_str().expect("utf8");
 
-/// Time `history`, and a raw scan of every column with no program, 7 times.
-async fn measure(log: &Log, uri: &str, label: &str) {
-    use futures::TryStreamExt;
+    // Live clock: history folds over the resident records.
     let mut hist = Vec::new();
-    let mut scan = Vec::new();
     let mut n = 0;
-    let mut records = 0;
-    for _ in 0..7 {
+    for _ in 0..101 {
         let s = Instant::now();
         n = log
             .history(&SPECS, PROBE_ROW, Col(0))
-            .await
             .expect("history")
             .len();
         hist.push(s.elapsed().as_secs_f64() * 1e3);
-        let s = Instant::now();
-        let ds = lance::Dataset::open(uri).await.expect("open");
-        let rows: usize = ds
-            .scan()
-            .try_into_stream()
-            .await
-            .expect("stream")
-            .try_collect::<Vec<_>>()
-            .await
-            .expect("collect")
-            .iter()
-            .map(|b| b.num_rows())
-            .sum();
-        records = rows;
-        scan.push(s.elapsed().as_secs_f64() * 1e3);
     }
     assert_eq!(n, VERSIONS as usize + 1, "one entry per version");
     let (h, hmin, hmax) = median(hist);
-    let (r, rmin, rmax) = median(scan);
     println!(
-        "{label}: {records} log records; history {n} entries, median {h:.1} ms ({hmin:.1}..{hmax:.1}); \
-         raw scan alone median {r:.1} ms ({rmin:.1}..{rmax:.1}); 7 reads each"
+        "history (resident, {} records): {n} entries, median {h:.3} ms ({hmin:.3}..{hmax:.3}), 101 reads",
+        log.len()
     );
+
+    // Recovery clock: replay reads the whole log once from Lance.
+    let mut rep = Vec::new();
+    for _ in 0..5 {
+        let mut cold = Log::new(uri);
+        let s = Instant::now();
+        let back = cold.replay(OWNER, &SPECS).await.expect("replay");
+        rep.push(s.elapsed().as_secs_f64() * 1e3);
+        assert_eq!(back.version(), t.version());
+        assert_eq!(cold.len(), log.len());
+    }
+    let (r, rmin, rmax) = median(rep);
+    println!("replay (Lance, cold): median {r:.1} ms ({rmin:.1}..{rmax:.1}), 5 runs");
+}
+
+fn median(mut ms: Vec<f64>) -> (f64, f64, f64) {
+    ms.sort_by(f64::total_cmp);
+    (ms[ms.len() / 2], ms[0], ms[ms.len() - 1])
 }