Skip to content

tests: one learn fixture owns custody, grants and release for every learn script - #107

Merged
joelteply merged 2 commits into
codex/s4-per-mind-integrationfrom
cormac/learn-fixture
Nov 11, 2026
Merged

joelteply merged 2 commits into
codex/s4-per-mind-integrationfrom
cormac/learn-fixture

Conversation

@joelteply

Copy link
Copy Markdown

tests: one learn fixture owns custody, grants and release for every learn script

Every learn now runs inside custody the core registers (weights, optimizer and workspace
residencies from the engine's own preflight) under a ledger grant, and every lifecycle verb names
the engine incarnation. Seven scripts predated that; each copied its own transport and none could
run. tools/server/tests/learn_fixture.py is the one owner:

  • Engine(url, name): the incarnation, an idle-only grant, request/post/ok/get with the engine
    added on bound routes;
  • fresh()/load(): dry-run plan, registered weights residency, seat;
  • learn()/experience(): preflight THIS example, optimizer residency once per adapter, a
    workspace per example, split exactly as admit_learning_residency splits rows;
  • shared_grant(measured, ...): a contended share whose latency allowance covers the measured
    chunks and boundary, max(entry, restoration);
  • filling(): the longest reply the slot window holds, counted by the server's tokenizer;
  • close(), also at exit: retire adapters, release every reference, revoke grants. Without it the
    engine-wide workspace refused the next script's owner (owner_mismatch, reproduced here).

Migrated: dream-adapter-routes.py (CI), learn-past-example.py, learn-unload-during-learn.py,
learn-every-format.py (all six modes), learn-cancel-custody.py, learn-chunk-yields.py,
learn-refuses-repacked.py. No script needs a hand-placed shadow-05.gguf any more: shadows are
minted fresh (two from one seed are the "copy" two-minds needs); the F16 copy is still made by
make-f16-shadow.py, from a fresh shadow.

Measured on a CPU llama-server built from this branch (IntelMac, stories260K, --no-repack):
dream-adapter-routes twice then learn-past-example twice on ONE server: pass (this failed with
owner_mismatch before close existed); learn-every-format default, --shadow-retirement,
--two-minds, --non-f32-refused: pass; learn-cancel-custody: pass; learn-refuses-repacked
(server without --no-repack): pass.

Not passing, reported, not hidden:

  • learn-unload-during-learn.py aborts the engine 2 of 3 runs: ggml-opt.cpp:969
    GGML_ASSERT(!allocation_pending && !graph_pending) in ggml_opt_free <- opt_release, freeing idle
    B while A learns (confirmed from source by Codex; the native fix is assigned). This script is
    the regression for it.
  • --fairness and learn-chunk-yields learn under contention: a contended graph chunk (1.5 s on
    chunk-yields) exceeds an allowance only idle steps can measure, so the step is refused mid-way
    (chunk_exceeds_latency). How a contended learn obtains its allowance is the ledger owner's open
    question.
  • --template-replies needs a model whose template renders tool_calls; stories260K's does not.

Stacked on #106 (cormac/inherited-ci-repair); retarget to codex/s4-per-mind-integration after it lands. Card f1d9d2cf.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

…earn script

Every learn now runs inside custody the core registers (weights, optimizer and workspace
residencies from the engine's own preflight) under a ledger grant, and every lifecycle verb names
the engine incarnation. Seven scripts predated that; each copied its own transport and none could
run. tools/server/tests/learn_fixture.py is the one owner:

- Engine(url, name): the incarnation, an idle-only grant, request/post/ok/get with the engine
  added on bound routes;
- fresh()/load(): dry-run plan, registered weights residency, seat;
- learn()/experience(): preflight THIS example, optimizer residency once per adapter, a
  workspace per example, split exactly as admit_learning_residency splits rows;
- shared_grant(measured, ...): a contended share whose latency allowance covers the measured
  chunks and boundary, max(entry, restoration);
- filling(): the longest reply the slot window holds, counted by the server's tokenizer;
- close(), also at exit: retire adapters, release every reference, revoke grants. Without it the
  engine-wide workspace refused the next script's owner (owner_mismatch, reproduced here).

Migrated: dream-adapter-routes.py (CI), learn-past-example.py, learn-unload-during-learn.py,
learn-every-format.py (all six modes), learn-cancel-custody.py, learn-chunk-yields.py,
learn-refuses-repacked.py. No script needs a hand-placed shadow-05.gguf any more: shadows are
minted fresh (two from one seed are the "copy" two-minds needs); the F16 copy is still made by
make-f16-shadow.py, from a fresh shadow.

Measured on a CPU llama-server built from this branch (IntelMac, stories260K, --no-repack):
dream-adapter-routes twice then learn-past-example twice on ONE server: pass (this failed with
owner_mismatch before close existed); learn-every-format default, --shadow-retirement,
--two-minds, --non-f32-refused: pass; learn-cancel-custody: pass; learn-refuses-repacked
(server without --no-repack): pass.

Not passing, reported, not hidden:
- learn-unload-during-learn.py aborts the engine 2 of 3 runs: ggml-opt.cpp:969
  GGML_ASSERT(!allocation_pending && !graph_pending) in ggml_opt_free <- opt_release, freeing idle
  B while A learns (confirmed from source by Codex; the native fix is assigned). This script is
  the regression for it.
- --fairness and learn-chunk-yields learn under contention: a contended graph chunk (1.5 s on
  chunk-yields) exceeds an allowance only idle steps can measure, so the step is refused mid-way
  (chunk_exceeds_latency). How a contended learn obtains its allowance is the ledger owner's open
  question.
- --template-replies needs a model whose template renders tool_calls; stories260K's does not.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
…what is held, and its own need

The step workspace is engine-wide and its custody passes from one reference to the next; claim_plan
refuses ('uncovered') a replacement that does not cover every row the previous reference was
registered with. Measured on CPU (stories260K, -np 2): A's long learn registered 7,026,688 bytes,
the scheduler then held 4,907,008, and the reloaded B's short reply needed 2,573,344; sized to its
own need (or even to need plus held) B's workspace was refused every time. The fixture now
remembers the rows it last registered and covers the union of those, shared_compute_held and the
example's own requirement, each buffer at the largest. learn-unload-during-learn.py then passes
10/10 on the fixed engine (fork #108).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@joelteply

Copy link
Copy Markdown
Author

Review (Fable): approve. One owner of the learn lifecycle for every learn script (grants, custody registered from the engine's own dry-run rows, seating, release), closed on atexit so a failed assertion never leaves custody bound for the next script; seven scripts drop their hand-rolled copies (+419/-276).

@joelteply
joelteply changed the base branch from cormac/inherited-ci-repair to codex/s4-per-mind-integration November 11, 2026 14:31
@joelteply
joelteply merged commit c6f6277 into codex/s4-per-mind-integration Nov 11, 2026
7 of 10 checks passed
joelteply added a commit that referenced this pull request Nov 11, 2026
…shadow unloads while another mind learns

c5551ed made ggml_opt_free assert that the SHARED scheduler had no pending graph. A serving
context keeps one optimizer per shadow on one scheduler, and llama_context::opt_release frees an
idle shadow's optimizer while another shadow's step is in flight (#85's contract: "ggml_opt_free
touches no shared scheduler"). That assert then aborted llama-server whenever the unload landed
mid-step (fork #106/#107 reproducer, learn-unload-during-learn.py, 2 of 3 runs on CPU).

ggml_opt_free releases only this context's own buffers and metadata. The check is now
context-local: ggml_backend_sched_graph_pending_for(sched, graph), true only when the scheduler is
part-way through THAT graph (graph_step already keys its continuation on the caller's graph
pointer). Freeing a context whose own graph is in flight still aborts. ggml_opt_reset and
ggml_opt_prepare_alloc keep the scheduler-wide guard: reset zeroes the last graph's gradients,
which in a dynamic graph can sit in the shared compute buffer another mind now uses, and
prepare_alloc allocates onto the scheduler.

Tests (test-opt-dynamic-accum, built with -fsanitize=address on CPU, IntelMac):
- run_frees_idle_mind_while_another_steps: B learns a step, A stops one node into its step
  (scheduler pending), B is freed, A completes with exactly the update it makes alone. PASS.
- run_free_own_pending_graph_aborts: freeing a context whose own graph is in flight aborts
  (observed in a forked child; skipped with a message where fork() is unavailable). PASS.
- the existing suite: PASS.
- Mutation: restoring the scheduler-wide check aborts the first test at the same assert the
  engine hit (exit 134).
End to end: the reproducer passes 10/10 on the fixed CPU server, but it is timing-dependent (it
aborts only if the unload lands mid-graph; 0/6 on the unfixed engine in a later run), so the unit
test is the proof.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant