Skip to content

feat(runtime-host): page semantic transcript turns - #4295

Draft
Sun-GLiang wants to merge 11 commits into
apache:mainfrom
Sun-GLiang:codex/4123-pr2-host-semantic-pager
Draft

feat(runtime-host): page semantic transcript turns#4295
Sun-GLiang wants to merge 11 commits into
apache:mainfrom
Sun-GLiang:codex/4123-pr2-host-semantic-pager

Conversation

@Sun-GLiang

@Sun-GLiang Sun-GLiang commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the first production consumer of the semantic transcript position snapshots introduced by PR #4258. Runtime Host now exposes subscription-scoped semantic position paging and complete semantic Turn windows through strict, additive protocol operations, while keeping exact Storage identities, leases, watermarks, and shared-only data private to the Host.

Refs #4123

Issue: #4123

Depends on apache/maka#4258. Do not merge or mark this PR Ready before PR1. This is an upstream cumulative Draft targeting main because apache/maka does not contain the fork-only PR1 base ref. It temporarily exposes the cumulative PR1 + PR2 diff. After PR1 merges, the controller must revalidate the exact resulting history and, if the project merge method requires it, explicitly authorize the rebase/update that narrows this PR to PR2-only. This delivery follows review 5060816416.

  • Plan packet: issue-4123-pr2-semantic-transcript-v1.4-controller-approved-2026-08-31
  • Exact embedded PR1 predecessor: b00436209373c0d8453c51f1c0cbad9f53dfeb8d
  • Exact head: 6044ea430df4ef45045d1c8667519e12f6e5d830
  • apache/maka main ref observed immediately before creation: ac59b47b72cdfd63b737dac6946715789ce277c5

Protocol and client surface

  • session.transcript.positions.query accepts the strict acquire | page | continue | replace | release union and returns page | building | capacity | snapshot_stale | anchor_not_found | released.
  • session.transcript.turn_window.page accepts open | continue and returns page | building | capacity | snapshot_stale | anchor_not_found | position_too_large.
  • Wire positions expose only dense { ordinal, key } values, where the key is turn | note | empty. Storage snapshot triples, leases, projection, sequences, recovery progress, raw counts, and retained generations remain Host-private.
  • Snapshot tokens, position cursors, and window cursors are opaque, subscription-bound, incarnation-bound, domain-separated HMAC-SHA256 values. Snapshot handles map to the exact Storage triple and lease in Host state; cursors bind the snapshot-token digest and their ordinal/window continuation state.
  • ClientSessionSubscription now provides queryTranscriptPositions(...), loadTranscriptTurnWindow(...), and decodeTranscriptTurnWindowPage(...). The existing fragment assembler was extracted and reused; callers receive only a complete digest- and decode-validated logical window.

Retryable building and capacity results carry only a fixed retry hint. A valid opaque handle whose internal exact snapshot is gone returns snapshot_stale; invisible/missing anchors return anchor_not_found; an individually oversized position returns position_too_large with no fragments. Tampered/replayed inputs map to invalid_request, access failures to non-disclosing not_found, recovery corruption to persistence_failed, and failed replacement CAS to operation_conflict.

Lifecycle, privacy, and bounds

  • Semantic acquisition uses a separately delivered semanticEligibleThroughSequence; v34 subscription open performs no semantic Storage work.
  • One Host-owned opaque lease is tied to each active semantic snapshot. Acquire is idempotent, replace prepares before CAS, release is idempotent, and subscription/connection/access/Host shutdown paths await cleanup.
  • Shared subscriptions use projectSharedSessionTranscriptMessage as the only sanitizer. Dense shared positions reveal neither hidden identities nor hidden counts; hidden records still count against Storage body limits.
  • A logical window contains at most 10 complete positions, 256 records, and 16 MiB raw/encoded data. Each retained immutable window is encoded and hashed once, fragments are at most 512 KiB, and continuation performs slicing only.
  • Existing and semantic transcript users share the 64 MiB retained ledger, 48 MiB preparation reservation, 64-waiter cap, one active semantic window per subscription, 64-window maximum, and 5-minute idle TTL.

PR1 consumer-invariant mapping

PR1 mechanism Production PR2 invariant
normalized membership/order metadata dense position pages and exact per-key body probes
bounded legacy/imported/chunked recovery one bounded Storage step per Host request; v34 chunk recovery resumes after Host reopen
persisted SHA/scanner state >4 MiB recovery continuation survives owner replacement without retaining bodies
exact snapshot triple and immutable Ready rows private handle state and signed continuation stability
two retained generations replacement can be prepared while the current snapshot remains usable
opaque leases/refcounts subscription lifecycle, shared-generation ownership, and awaited release
capacity/backpressure privacy-neutral typed Host result, without eviction/fallback
monotonic generation stale-handle/cursor ABA rejection after release, reopen, and reallocation

Out of scope

No Desktop/Preload/Renderer production activation, continuous layout, navigation, export, compatibility-epoch switch, legacy-path removal, Storage schema redesign, or PR3/PR4 behavior is included. The two Desktop/CLI changes are test-fixture compile stubs only. Current Desktop does not call either operation. Any post-PR1-merge history update or narrowing remains controller-owned.

Verification

  • npm --workspace @maka/storage test — 1072 total, 1065 passed, 7 skipped, 0 failed.
  • npm --workspace @maka/runtime-host test — 1449 total, 1440 passed, 9 skipped, 0 failed.
  • Focused Storage semantic index tests — 55/55 passed.
  • Focused compiled Runtime Host semantic/protocol/reader/client/coordinator/connection/composition tests — 184/184 passed.
  • npm test — all workspace tests passed on the final exact-command rerun.
  • npm run build, npm run format:check, npm run lint, npm run typecheck — passed.
  • npx knip --workspace apps/desktop and npx knip --workspace packages/ui — passed.
  • ASF headers, repository notices, and CLI notices checks — passed.
  • Epoch guard against exact stacked base — declared compatible extension at epoch 73; guard tests 12/12 and semantic protocol codec tests 6/6 passed.
  • Both exact-base and cumulative git diff --check — passed.

Benchmark evidence (npm --workspace @maka/runtime-host run benchmark:transcript, plus MAKA_TRANSCRIPT_BENCHMARK_FULL_SEMANTIC=1):

  • 10,000 alternating-visibility positions: owner 10,000 / shared 5,000; two generations contain 20,000 position rows; 100 warm exact tail/ordinal/Turn-anchor reads kept the exact key stable with WAL delta 0 and data-version delta 0.
  • 15 MiB single position: one body read/project/encode, 31 fragments, wire amplification 1.335, no oversized result.
  • 17 MiB single position: position_too_large, zero fragments.
  • 17 MiB / 64 MiB multi-position fixtures: 428 / 1609 logical windows, wire amplification 1.383, peak retained accounting about 0.04 MiB, and cleanup returned retained accounting to zero.
  • Production integration covers real SQLite -> Runtime Host -> production client, shared privacy, bodyless admissions, visible/hidden notes, two-slot capacity, lease refcount, monotonic ABA, and persisted v34 >4 MiB chunk recovery across Host reopen.

Independent Standards and Spec/Composition reviews both passed the final head with no blocking or high-severity findings.

Cumulative Draft

This upstream Draft targets main only because apache/maka lacks the fork-only PR1 base ref. Its current review surface is intentionally cumulative PR1 + PR2 and depends on upstream PR #4258. It must not merge or be marked Ready before PR1. After PR1 merges, the controller must revalidate the resulting history and explicitly authorize any rebase/update required to narrow the diff to PR2-only.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: OpenAI Codex implemented the Runtime Host protocol/client/server changes, tests, benchmark coverage, and review fixes. The commit includes the required Generated-by: OpenAI Codex trailer.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

@github-actions github-actions Bot added the effort/XL Over 1000 readable lines label Aug 31, 2026
@Sun-GLiang
Sun-GLiang force-pushed the codex/4123-pr2-host-semantic-pager branch from 9d9c67d to 6044ea4 Compare August 31, 2026 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/XL Over 1000 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant