Skip to content

D-STO-5: history as a fold over resident log records (0.72 ms, was 336 ms) - #25

Merged
AdaWorldAPI merged 2 commits into
mainfrom
ccr-1d749cc5-q8s5qs
Oct 10, 2026
Merged

AdaWorldAPI merged 2 commits into
mainfrom
ccr-1d749cc5-q8s5qs

Conversation

@AdaWorldAPI

Copy link
Copy Markdown
Owner

What

#24 read property history by scanning the Lance log on every call. Measured, that took 336 ms, 320 ms of it the scan, against P-HIST-1's < 20 ms budget.

The rule this applies (lance-graph, 2026-09-24): one write costs about a million folds, and a Lance scan is recovery cost. So history should be a fold over records already in memory, not a read of Lance.

  • Log keeps its records resident. Every record it commits or replays stays in memory, at 29 B each.
  • replay is the one Lance read. On any error nothing becomes resident; the log keeps what it held before.
  • commit appends to the resident records after the Lance append. It refuses with Diverged when the records are not at the table's version, because otherwise history would silently miss earlier batches.
  • history is synchronous now. It runs the same Quack program over memory and never opens the dataset.
  • No separate corruption check. A corrupt record cannot become resident, because replay refuses it. So the required-lane check from D-STO-5 (read): property history from the Lance log, with its measured cost #24's review holds structurally, and the per-match check is gone.

No index, no column projection, no Lance filter expression, and no compaction. ISS-HIST-SCAN-COST is resolved. Plan §5.2 records the decision: derived reads write nothing, and only learning adds persistent state, recorded after as one merged, auditable outcome.

Evidence

Timing. Release, examples/probe_history.rs, 100k records, 1,000 versions × 1,000 updates, 1.2M log records:

read median
history, resident (101 reads) 0.72 ms (0.64–1.99)
replay from Lance (5 cold runs) 532 ms, once at startup
history read from Lance (#24, for reference) 336 ms

Tests (cargo test -p hubspo-store --features lance; the log suite dropped from 16 s to 0.44 s):

  • Histories equal an independently written oracle for every record and both lanes.
  • A second log replayed from disk gives identical histories.
  • A log without resident records refuses to commit.
  • A corrupt record is refused by replay and never reaches history: the resident records are unchanged.

Disable runs, all red:

disabled red
resident-version check yes
records installed before replay validates yes
commit not appending to the resident records yes
row = r term yes
op = DELETE arm yes
op = SET term yes
I32 sign bit yes

Local: fmt, clippy (-D warnings, default and lance) and the workspace tests pass. No other HubSPO-rs PR is open; the branch is current with main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z


Generated by Claude Code

…replay

One write costs about a million folds (lance-graph 2026-09-24), and a
Lance scan is recovery cost (~320 ms over 1,000 fragments). So the log
keeps every record it commits or replays resident, and history runs its
Quack program over memory instead of scanning Lance.

- Log owns the resident records; commit appends to them after the Lance
  append and refuses when they are not at the table's version.
- replay is the one Lance read; on any error nothing becomes resident.
- history is synchronous. A corrupt record cannot become resident (replay
  refuses it), so the per-match required-lane check is structural now.
- Tests: a second log replayed from disk gives identical histories; a log
  without resident records refuses to commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
ISS-HIST-SCAN-COST resolved without an index or projection. Plan 5.2
records the decision: history folds over resident records; Lance is read
once, by replay; only learning adds persistent state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-10T19:24:44.727307Z ffc56bf PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@AdaWorldAPI
AdaWorldAPI merged commit 4c5b495 into main Oct 10, 2026
2 checks passed
AdaWorldAPI pushed a commit that referenced this pull request Oct 10, 2026
AdaWorldAPI pushed a commit that referenced this pull request Oct 14, 2026
…#22-#25

D-QRY-1 points at the D-CMR set algebra (relalg prototype, #1546),
D-QRY-5 at the D-CMR-5 preimage hop, D-FLOW-1 at graph-flow-lance.
New ask ISS-LG-GRAPHFLOW-LANCE-PATHS; the preflight pin issue is
still open at c21bd165.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
AdaWorldAPI added a commit that referenced this pull request Oct 14, 2026
board: synergies from lance-graph #1530–#1546, rs-graph-llm #27, spear #22–#25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants