Skip to content

D-STO-5 (read): property history from the Lance log, with its measured cost - #24

Merged
AdaWorldAPI merged 6 commits into
mainfrom
ccr-1d749cc5-q8s5qs
Oct 10, 2026
Merged

AdaWorldAPI merged 6 commits into
mainfrom
ccr-1d749cc5-q8s5qs

Conversation

@AdaWorldAPI

@AdaWorldAPI AdaWorldAPI commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

What

hubspo-store::log::Log::history(specs, row, col) (feature lance) returns every write of one property on one record, oldest first:

  • the value given at insert (absent included);
  • each update;
  • the record's deletion.

Each entry carries its version and the owner the batch wrote on behalf of (Entry { version, owner, change: Change::{Set(Option<Value>), Deleted} }).

There is no history store; history is the log. Each log batch runs one Quack program, row = r ∧ ((lane = c ∧ op = SET) ∨ op = DELETE), on mask-risc over the log's Arrow buffers. The buffers are borrowed, not copied, and only matching records are decoded.

To make that one program over three U32 lanes, the log's op and lane columns widen to u32. The format shipped in #23 and no data has been written with it, so nothing needs migrating.

The log is never compacted (decision 2026-10-10). The append order is the version order, so history and replay no longer sort. History's sort was inert under a disable run. Replay still refuses an out-of-order log, as a batch whose version is not the next one. The decision is recorded in plan §5.2.

The board marks D-STO-1 Shipped and D-STO-5 In progress.

Evidence

Tests (cargo test -p hubspo-store --features lance):

  • For every record and both lanes over 30 random batches, the history equals one written down independently from the batches. Anti-vacuity: the comparison covers histories with updates, deletions, negative values and absent values.
  • An unknown lane is refused.
  • An empty log reads empty.

Disable runs:

disabled guard result
row = r term red
op = DELETE arm red
op = SET term (BORN/DELETE leak in) red
I32 sign bit red
sort by version green, and now removed

The budget fails, and it is stated, not hidden

Measured with examples/probe_history.rs: release, 100k records, 1,000 versions × 1,000 updates, 1.2M log records, one fragment per version.

  • History reads in 336 ms, of which the raw scan is 320 ms.
  • P-HIST-1's budget is < 20 ms, so this fails it.
  • The scan is the cost; the program is 3–15 ms of it.

ISS-HIST-SCAN-COST lists the levers:

  1. Projecting only row, lane and op.
  2. An index on row. P-SPARSE-1 measured 16.9 ms with Lance's BTree on an uncompacted log of 1,000 fragments, which is inside the budget. That index goes through Lance's internal DataFusion. Whether it fits iron rule 5 is an operator decision, and the plan's alternative is our own change index.

D-STO-5 stays open until one of these lands.

Local: fmt, clippy (-D warnings, with and without lance) and the workspace tests pass. No other HubSPO-rs PR is open.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z

Log::history(specs, row, col) returns every write of one property on one
record, oldest first, with version and owner: the inserted value, each
update, and the record's deletion. It runs one Quack program per log batch
over the log's own Arrow buffers and decodes only the matches.

The log's op and lane columns widen to u32 so the whole predicate is one
mask-risc program over three U32 lanes. The log format shipped in #23 with
no data written yet, so no migration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
examples/probe_history.rs times one property's history over 1,000
versions of 1,000 updates each on 100k records, with no index on row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
History is correct but fails P-HIST-1's 20 ms budget: 336 ms fragmented,
57 ms compacted, almost all of it the raw scan. ISS-HIST-SCAN-COST names
the levers (compaction after decoupling the version check, column
projection, an index on row).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-10T18:10:19.074510Z de52dd3 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: de52dd31db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/hubspo-store/src/log.rs Resolved
The append order is the version order, so history and replay drop their
sorts (history's was inert under a disable run). Replay still refuses an
out-of-order log as a batch whose version is not the next one. The probe
drops its compaction arm; ISS-HIST-SCAN-COST and plan §5.2 record the
decision, leaving projection and an index on row as the levers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
Replay refuses that record through Table::apply, so history must not
report it as Set(None). Found by review on #24.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
@AdaWorldAPI
AdaWorldAPI merged commit 571975c into main Oct 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants