Skip to content

server: a learning ledger read answers from the loop's last publication, never queued behind the chunk it reports on - #109

Merged
joelteply merged 3 commits into
codex/s4-per-mind-integrationfrom
cormac/ledger-off-loop
Nov 11, 2026
Merged

joelteply merged 3 commits into
codex/s4-per-mind-integrationfrom
cormac/ledger-off-loop

Conversation

@joelteply

Copy link
Copy Markdown

Card 7fc7e2ce (continuum), part 1 of Fable's option (a): a control read never queues behind the work it controls.

Found on the IntelMac (canary dc01d9563 and ae5ba3173, 2026-11-11): the core's learner stop and grant revoke timed out whenever the engine was prefilling. The ledger ops are main-loop tasks (grant_route, SERVER_TASK_TYPE_LEARN_GRANTS), the core waits 4 s, and one 2,048-token prefill chunk on that CPU took ~40-45 s. /health answered in ~1 ms while GET /learn/grants did not answer within 20 s.

Change

  • The loop publishes its half of the ledger (grants, use, committed steps, the optimizer's memory receipt) at every task boundary (on_new_task) and slot-update boundary (on_update_slots), and once at init, stamped with sequence and clock_us, under one mutex with a condvar signalled per publication (the wake the ticket wait in part 2 blocks on). Same pattern as the existing publish_lora_report / GET /lora-adapters.
  • GET /learn/grants answers from that publication plus the residency mirror (its own mutex) on the HTTP thread. ledger_receipt(now) is the one builder for both the publication and a mutation's reply.
  • Mutations stay loop tasks (Codex: none is proven safe off-loop). Part 2 gives them tickets whose completion is an event (Fable: no polling).

Tested (IntelMac, CPU build, stories fixture model)

  • New tests/learn-ledger-off-loop.py: a read carries the stamp and both halves; the sequence advances at a boundary; a grant registered through the loop is in the next read; a read answers while a completion is in flight. 3 back-to-back reads: 0.70-0.77 ms.
  • Mutation: GET routed back through the loop task fails the fixture (no sequence).
  • learn-past-example, learn-cancel-custody pass (--no-repack).
  • The continuum core's live-engine trigger tests (3) pass against this server, so the core's ledger parser accepts the published shape.

…on, never queued behind the chunk it reports on

The loop publishes its half of the ledger (grants, use, committed steps, the optimizer's memory)
at every task and slot-update boundary, stamped with sequence and clock_us, under one mutex
with a condvar signalled per publication (the wake a ticket wait will block on). GET
/learn/grants answers from that publication plus the residency mirror (its own mutex) from the
HTTP thread. Mutations stay loop tasks. Card 7fc7e2ce: on the IntelMac a 2,048-token prefill
chunk ran ~40 s while the core waited 4 s, so every learner stop read as unreachable.
@joelteply

Copy link
Copy Markdown
Author

Review (Fable): design right. Two fixes before merge: (1) publish_ledger runs at every task and every slot-update boundary (every decode batch), rebuilding the ledger JSON incl. llama_opt_visit_memory each time; publish when the ledger changed (a version the mutations bump) plus init, and measure per-batch cost before/after. (2) learn-ledger-off-loop.py prints answered_while_busy but never asserts it, and a 256-token completion can finish before the read; assert the read returned while the completion was in flight (long enough to overlap) or its latency is under one chunk.

…hanged; the fixture asserts the read beside a running completion

server_learn_ledger::revision() is bumped by every mutation; on_update_slots publishes only when it
moved (task boundaries and init still publish), so no ledger JSON or memory visit runs per decode
batch. The fixture waits for /slots to report the completion processing, reads, and asserts the
completion is still running (Fable on fork #109).
…hen it changed

Each decode step posts a SERVER_TASK_TYPE_NEXT_RESPONSE task, so publishing at every task boundary
still published per token (1,025 publications for a 512-token completion). A continuation now
publishes only when the ledger revision moved; real tasks (requests, ledger and adapter operations)
still publish at their boundary. Measured: 1 publication for a 512-token completion.
@joelteply
joelteply changed the base branch from cormac/learn-fixture to codex/s4-per-mind-integration November 11, 2026 14:31
@joelteply
joelteply merged commit cf49932 into codex/s4-per-mind-integration Nov 11, 2026
30 of 47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant