Skip to content

D-STO-1 (persistence half): one Lance version per batch, rebuilt by replay - #23

Merged
AdaWorldAPI merged 2 commits into
mainfrom
ccr-1d749cc5-q8s5qs
Oct 10, 2026
Merged

AdaWorldAPI merged 2 commits into
mainfrom
ccr-1d749cc5-q8s5qs

Conversation

@AdaWorldAPI

Copy link
Copy Markdown
Owner

What

hubspo-store::log::Log (behind the lance feature) persists the resident Table from #22 as an append-only Lance log. It uses P-SPARSE-1's delta shape: nothing is updated in place, so there are no deletion vectors and the log is safe to compact.

  • Record format: three record kinds, BORN (row), SET (row, lane, present, the lane's 32 bits) and DELETE (row). Every record is stamped with its version and owner.
  • commit(table, batch): check the batch, append it as exactly one Lance version, then apply it to the table. The Lance version equals the table version. The log's version is compared before the append, which refuses a stale second writer, and again after it.
    • Two Lance appends never conflict, so this comparison is what enforces one writer per log.
    • Durable first: a failed append leaves the table unchanged.
  • replay(owner, specs): rebuilds the table batch by batch through the same Table::apply. It refuses a foreign owner, an undecodable record, or a log whose Lance version is not its batch count.
  • Table changes: it gains check() and specs(), and it refuses an empty batch (StoreError::EmptyBatch), since an empty batch would be a version that changes nothing.
  • CI: a new job, store-lance, installs protoc (lance-encoding needs it at build time) and runs clippy and the tests with the feature. The default job stays fast.

The PR also records option (d) for OD-12 in the plan. lance-graph #1453 merged a read-only LDAP v3 handler over dir-sim, so stalwart's own LDAP directory backend could read dir-sim through it with no stalwart code change. The approved option (c) stands until two questions are settled: password checks (they go through the host's Authority) and dir-sim persistence.

Evidence

Tests (cargo test -p hubspo-store --features lance):

  • 25 committed batches replay to an identical image, including deletes and negative values, and the log ends at exactly one Lance version per batch.
  • A second, stale writer is refused before it applies anything, and the log afterwards holds only the first writer's batches.
  • Refused and empty batches leave the log at its version.
  • Replay under another owner is refused.

Disable runs:

disabled guard result
pre-append version check red
owner check on replay red
I32 sign bit red
deletes logged red
apply only after the append red
post-append version check green

The post-append check stays green when disabled, and I kept it on purpose. It only fires when another writer appends between the pre-check and the write, a race no sequential test can schedule. The source says so at the check.

Locally: workspace fmt, clippy (--all-targets --no-deps -D warnings, default features and --features lance) and tests all pass. Rebased on main after #22; no other HubSPO-rs PR is open.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z


Generated by Claude Code

…'s LDAP handler

lance-graph #1453 merged a read-only LDAP v3 handler over a dir-sim
projection. stalwart's own LDAP directory backend could read dir-sim
through it with no stalwart code change. Recorded as an option to
evaluate; the approved (c) stands until passwords (the host's Authority)
and dir-sim persistence are settled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
…eplay

hubspo-store::log (feature lance): an append-only Lance log in
P-SPARSE-1's delta shape. Log::commit checks the batch, appends it as
exactly one Lance version (version checked before and after), then
applies it; Log::replay rebuilds the table through the same Table::apply
and refuses a foreign owner, an undecodable record, or a version that is
not the batch count. Table gains check(), specs() and refuses an empty
batch. New CI job store-lance installs protoc and runs the feature.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@AdaWorldAPI
AdaWorldAPI merged commit df12fc2 into main Oct 10, 2026
2 checks passed
AdaWorldAPI pushed a commit that referenced this pull request Oct 10, 2026
Log::history(specs, row, col) returns every write of one property on one
record, oldest first, with version and owner: the inserted value, each
update, and the record's deletion. It runs one Quack program per log batch
over the log's own Arrow buffers and decodes only the matches.

The log's op and lane columns widen to u32 so the whole predicate is one
mask-risc program over three U32 lanes. The log format shipped in #23 with
no data written yet, so no migration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z
AdaWorldAPI added a commit that referenced this pull request Oct 10, 2026
…d cost (#24)

* D-STO-5: property history read from the Lance log

Log::history(specs, row, col) returns every write of one property on one
record, oldest first, with version and owner: the inserted value, each
update, and the record's deletion. It runs one Quack program per log batch
over the log's own Arrow buffers and decodes only the matches.

The log's op and lane columns widen to u32 so the whole predicate is one
mask-risc program over three U32 lanes. The log format shipped in #23 with
no data written yet, so no migration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z

* D-STO-5: history timing probe; note why the sort is kept

examples/probe_history.rs times one property's history over 1,000
versions of 1,000 updates each on 100k records, with no index on row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z

* D-STO-5 probe: split raw scan from history; measure after compaction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z

* Board: D-STO-1 shipped; D-STO-5 history read with its measured cost

History is correct but fails P-HIST-1's 20 ms budget: 336 ms fragmented,
57 ms compacted, almost all of it the raw scan. ISS-HIST-SCAN-COST names
the levers (compaction after decoupling the version check, column
projection, an index on row).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z

* Never compact the log: readers rely on append order

The append order is the version order, so history and replay drop their
sorts (history's was inert under a disable run). Replay still refuses an
out-of-order log as a batch whose version is not the next one. The probe
drops its compaction arm; ISS-HIST-SCAN-COST and plan §5.2 record the
decision, leaving projection and an index on row as the levers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z

* History refuses an absent value in a required lane

Replay refuses that record through Table::apply, so history must not
report it as Set(None). Found by review on #24.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants