D-STO-5 (read): property history from the Lance log, with its measured cost

.claude/board/ISSUES.md

@@ -3,6 +3,16 @@
 > Open blockers and cross-repo follow-ups. Append new entries at the top;
 > change only the `Status` line of an existing entry.
 
+## ISS-HIST-SCAN-COST — property history scans the whole log
+
+- **Status:** OPEN (D-STO-5 budget; correctness shipped)
+- **Where:** `hubspo-store::log::Log::history`.
+- **What:** history reads every log record and runs the Quack program over them. Measured 2026-10-10 (`examples/probe_history.rs`, release, 100k records, 1,000 versions × 1,000 updates, 1.2M log records, one fragment per version): 336 ms, 320 ms of it the raw scan. P-HIST-1's budget is < 20 ms. The program is not the cost.
+- **Not a lever:** compaction. DECISION 2026-10-10 (operator): the log is never compacted, because the append order gives version order for free. (A one-off measurement compacted to one fragment read in 57 ms, still over budget.)
+- **Levers:**
+  1. Read only `row`, `lane`, `op` for the program and fetch the other columns for the matches by position.
+  2. An index on `row`. P-SPARSE-1 measured 16.9 ms with Lance's BTree on an uncompacted log of 1,000 fragments, inside the budget. Using it takes a Lance filter expression, which runs Lance's internal DataFusion; whether that sits within iron rule 5 (no `datafusion*` dependency, queries are Quack) is an operator decision. The plan's alternative (§5.2) is our own change index — positions per record, an index, not a second truth.
+
 ## ISS-LG-MASKRISC-PATH-NDARRAY — Quack could not be consumed by git
 
 - **Status:** RESOLVED 2026-10-10 by lance-graph #1457 (`e724d83`): mask-risc names ndarray by git, and lance-graph's workspaces patch it back to the sibling checkout.

.claude/board/LATEST_STATE.md

@@ -137,7 +137,27 @@
   append. Green, and kept on purpose: the post-append version check — it
   catches only a writer that appends between the check and the write, a
   race no sequential test can schedule.
-- Not yet: folding the log into a base dataset (D-STO-5), object stores.
+- Not yet: an index on `row`, object stores. The log is never compacted
+  (decision 2026-10-10): the append order is the version order.
+
+## Property history (D-STO-5, feature `lance`)
+
+- `Log::history(specs, row, col) -> Vec<Entry>`: every write of one
+  property on one record, oldest first — the inserted value (absent
+  included), each update, the deletion — each with its version and owner.
+  No history store: one Quack program per log batch,
+  `row = r ∧ ((lane = c ∧ op = SET) ∨ op = DELETE)`, on mask-risc over the
+  log's Arrow buffers (borrowed); only matches are decoded. The log's `op`
+  and `lane` columns widened to `u32` for this (no data existed yet).
+- Tests: for every record and both lanes over 30 random batches, the
+  history equals one written down from the batches (anti-vacuity: updates,
+  deletions, negative and absent values); unknown lane refused; empty log
+  reads empty. Disable runs 2026-10-10, red: drop the row term, drop the
+  DELETE arm, drop `op = SET`, lose the I32 sign. A sort by version was
+  inert under a disable run and is removed: append order is version order.
+  Replay dropped its sort too; a log out of order is still refused there,
+  as a batch whose version is not the next one.
+- Timing: fragmented log (1,001 fragments, 1.2M records): **336 ms**, of which the raw scan is 320 ms. Fails P-HIST-1's < 20 ms; the Quack program is 3–15 ms of it (release, `examples/probe_history.rs`, 7 reads, 2026-10-10). The scan, not the program, is the cost. Open: ISS-HIST-SCAN-COST.
 
 ## Board view (D-QRY-3, board half)
 

.claude/board/STATUS_BOARD.md

@@ -35,11 +35,11 @@
 
 | D-id | Deliverable | Size | Status | Evidence |
 |---|---|---|---|---|
-| D-STO-1 | `hubspo-store` write lane: batch image → one Lance version per publish; owner-stamped (`mailbox_owner`) | L | In PR | resident half shipped (#22); persistence half in PR: `hubspo-store::log` (one Lance version per batch, replay, feature `lance`) |
+| D-STO-1 | `hubspo-store` write lane: batch image → one Lance version per publish; owner-stamped (`mailbox_owner`) | L | Shipped | resident half #22; persistence half #23: `hubspo-store::log` (one Lance version per batch, replay, feature `lance`) |
 | D-STO-2 | KV value store for variable-length properties; zero-copy lens reads | M | Queued |  |
 | D-STO-3 | Edge lanes + association labels (H-ADR-4); cardinality invariants as Quack `GroupReduce` | L | Queued |  |
 | D-STO-4 | Tenant layout + P-TEN-1 probe | M | Queued | P-TEN-1 passed 2026-10-10 (7.8 ms p99 at 10k tenants, local disk) |
-| D-STO-5 | Property history via version log + P-HIST-1 probe | M | Queued | P-HIST-1 failed on in-place updates (46 ms); P-SPARSE-1 passes with the split-tunnel sparse delta: 16.9 ms indexed, 3.3–5.9 ms compacted |
+| D-STO-5 | Property history via version log + P-HIST-1 probe | M | In progress | read shipped in PR: `Log::history` (one Quack program per log batch, no index); fragmented log (1,001 fragments, 1.2M records): **336 ms**, of which the raw scan is 320 ms. Fails P-HIST-1's < 20 ms; the Quack program is 3–15 ms of it (release, `examples/probe_history.rs`, 7 reads, 2026-10-10). Budget open: ISS-HIST-SCAN-COST. P-HIST-1 failed on in-place updates (46 ms); P-SPARSE-1 passes with the split-tunnel sparse delta: 16.9 ms indexed, 3.3–5.9 ms compacted |
 | D-STO-6 | Importer (value oracle, part 1): sandbox portal → crm-sim plan → commit; counts compared per object type | L | Queued |  |
 
 ## Wave 3 — Query, lists, views, reports

.claude/plans/hubspo-integration-v1.md

@@ -859,6 +859,11 @@ Writing the 1,000 versions takes 6.6–12.4 s instead of ~300 s. The delta
 is append-only, so compacting it loses no history; the base is never
 rewritten, so nothing depends on keeping old Lance versions.
 
+*Decision 2026-10-10:* the log is never compacted. The append order gives
+version order for free, and readers (history, replay) rely on it instead of
+sorting. History speed comes from reading fewer columns or an index on
+`row` (ISS-HIST-SCAN-COST), not from compaction.
+
 ### 5.3 Timeline
 
 The record timeline is spear's `activity` table (D-HUB-4) joined on

crates/hubspo-store/Cargo.toml

@@ -29,3 +29,7 @@ lance = ["dep:lance", "dep:arrow", "dep:futures"]
 [dev-dependencies]
 tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
 tempfile = "3"
+
+[[example]]
+name = "probe_history"
+required-features = ["lance"]

crates/hubspo-store/examples/probe_history.rs

@@ -0,0 +1,101 @@
+//! D-STO-5 timing: history of one property of one record, read from a log of
+//! `VERSIONS` batches of `UPDATES` updates each over `ROWS` records, every
+//! batch also writing the probed property. No index on `row`.
+//!
+//! `cargo run --release -p hubspo-store --features lance --example probe_history`
+
+use std::time::Instant;
+
+use hubspo_store::log::Log;
+use hubspo_store::table::{Batch, LaneKind, LaneSpec, Table, Value};
+use lance_graph_quack::Col;
+
+const OWNER: u32 = 7;
+const ROWS: u32 = 100_000;
+const VERSIONS: u32 = 1_000;
+const UPDATES: u32 = 1_000;
+const PROBE_ROW: u32 = 4_242;
+const SPECS: [LaneSpec; 1] = [LaneSpec {
+    kind: LaneKind::U32,
+    nullable: false,
+}];
+
+#[tokio::main(flavor = "multi_thread")]
+async fn main() {
+    let dir = tempfile::tempdir().expect("tempdir");
+    let uri = dir.path().join("probe.lance");
+    let log = Log::new(uri.to_str().expect("utf8"));
+    let mut t = Table::new(OWNER, &SPECS).expect("table");
+    let started = Instant::now();
+    let seed = Batch {
+        owner: OWNER,
+        inserts: (0..ROWS).map(|i| vec![Some(Value::U32(i))]).collect(),
+        ..Batch::default()
+    };
+    log.commit(&mut t, &seed).await.expect("seed");
+    for v in 0..VERSIONS {
+        let mut updates: Vec<_> = (0..UPDATES - 1)
+            .map(|k| ((v * 7_919 + k * 97) % ROWS, Col(0), Some(Value::U32(v))))
+            .filter(|(r, _, _)| *r != PROBE_ROW)
+            .collect();
+        updates.push((PROBE_ROW, Col(0), Some(Value::U32(v))));
+        let b = Batch {
+            owner: OWNER,
+            updates,
+            ..Batch::default()
+        };
+        log.commit(&mut t, &b).await.expect("commit");
+    }
+    println!(
+        "wrote {} versions in {:.1} s",
+        VERSIONS + 1,
+        started.elapsed().as_secs_f64()
+    );
+    // One fragment per version: the log is never compacted.
+    measure(&log, uri.to_str().expect("utf8"), "fragmented").await;
+}
+
+fn median(mut ms: Vec<f64>) -> (f64, f64, f64) {
+    ms.sort_by(f64::total_cmp);
+    (ms[ms.len() / 2], ms[0], ms[ms.len() - 1])
+}
+
+/// Time `history`, and a raw scan of every column with no program, 7 times.
+async fn measure(log: &Log, uri: &str, label: &str) {
+    use futures::TryStreamExt;
+    let mut hist = Vec::new();
+    let mut scan = Vec::new();
+    let mut n = 0;
+    let mut records = 0;
+    for _ in 0..7 {
+        let s = Instant::now();
+        n = log
+            .history(&SPECS, PROBE_ROW, Col(0))
+            .await
+            .expect("history")
+            .len();
+        hist.push(s.elapsed().as_secs_f64() * 1e3);
+        let s = Instant::now();
+        let ds = lance::Dataset::open(uri).await.expect("open");
+        let rows: usize = ds
+            .scan()
+            .try_into_stream()
+            .await
+            .expect("stream")
+            .try_collect::<Vec<_>>()
+            .await
+            .expect("collect")
+            .iter()
+            .map(|b| b.num_rows())
+            .sum();
+        records = rows;
+        scan.push(s.elapsed().as_secs_f64() * 1e3);
+    }
+    assert_eq!(n, VERSIONS as usize + 1, "one entry per version");
+    let (h, hmin, hmax) = median(hist);
+    let (r, rmin, rmax) = median(scan);
+    println!(
+        "{label}: {records} log records; history {n} entries, median {h:.1} ms ({hmin:.1}..{hmax:.1}); \
+         raw scan alone median {r:.1} ms ({rmin:.1}..{rmax:.1}); 7 reads each"
+    );
+}