Repository navigation
D-STO-5 (read): property history from the Lance log, with its measured cost - #24
Merged
Merged
Conversation
Log::history(specs, row, col) returns every write of one property on one record, oldest first, with version and owner: the inserted value, each update, and the record's deletion. It runs one Quack program per log batch over the log's own Arrow buffers and decodes only the matches. The log's op and lane columns widen to u32 so the whole predicate is one mask-risc program over three U32 lanes. The log format shipped in #23 with no data written yet, so no migration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
examples/probe_history.rs times one property's history over 1,000 versions of 1,000 updates each on 100k records, with no index on row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
History is correct but fails P-HIST-1's 20 ms budget: 336 ms fragmented, 57 ms compacted, almost all of it the raw scan. ISS-HIST-SCAN-COST names the levers (compaction after decoupling the version check, column projection, an index on row). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: de52dd31db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The append order is the version order, so history and replay drop their sorts (history's was inert under a disable run). Replay still refuses an out-of-order log as a batch whose version is not the next one. The probe drops its compaction arm; ISS-HIST-SCAN-COST and plan §5.2 record the decision, leaving projection and an index on row as the levers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
Replay refuses that record through Table::apply, so history must not report it as Set(None). Found by review on #24. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
hubspo-store::log::Log::history(specs, row, col)(featurelance) returns every write of one property on one record, oldest first:Each entry carries its version and the owner the batch wrote on behalf of (
Entry { version, owner, change: Change::{Set(Option<Value>), Deleted} }).There is no history store; history is the log. Each log batch runs one Quack program,
row = r ∧ ((lane = c ∧ op = SET) ∨ op = DELETE), on mask-risc over the log's Arrow buffers. The buffers are borrowed, not copied, and only matching records are decoded.To make that one program over three
U32lanes, the log'sopandlanecolumns widen tou32. The format shipped in #23 and no data has been written with it, so nothing needs migrating.The log is never compacted (decision 2026-10-10). The append order is the version order, so history and replay no longer sort. History's sort was inert under a disable run. Replay still refuses an out-of-order log, as a batch whose version is not the next one. The decision is recorded in plan §5.2.
The board marks D-STO-1 Shipped and D-STO-5 In progress.
Evidence
Tests (
cargo test -p hubspo-store --features lance):Disable runs:
row = rtermop = DELETEarmop = SETterm (BORN/DELETE leak in)The budget fails, and it is stated, not hidden
Measured with
examples/probe_history.rs: release, 100k records, 1,000 versions × 1,000 updates, 1.2M log records, one fragment per version.ISS-HIST-SCAN-COSTlists the levers:row,laneandop.row. P-SPARSE-1 measured 16.9 ms with Lance's BTree on an uncompacted log of 1,000 fragments, which is inside the budget. That index goes through Lance's internal DataFusion. Whether it fits iron rule 5 is an operator decision, and the plan's alternative is our own change index.D-STO-5 stays open until one of these lands.
Local: fmt, clippy (
-D warnings, with and withoutlance) and the workspace tests pass. No other HubSPO-rs PR is open.🤖 Generated with Claude Code
https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z