Repository navigation
tests: one learn fixture owns custody, grants and release for every learn script - #107
Merged
Merged
Conversation
…earn script Every learn now runs inside custody the core registers (weights, optimizer and workspace residencies from the engine's own preflight) under a ledger grant, and every lifecycle verb names the engine incarnation. Seven scripts predated that; each copied its own transport and none could run. tools/server/tests/learn_fixture.py is the one owner: - Engine(url, name): the incarnation, an idle-only grant, request/post/ok/get with the engine added on bound routes; - fresh()/load(): dry-run plan, registered weights residency, seat; - learn()/experience(): preflight THIS example, optimizer residency once per adapter, a workspace per example, split exactly as admit_learning_residency splits rows; - shared_grant(measured, ...): a contended share whose latency allowance covers the measured chunks and boundary, max(entry, restoration); - filling(): the longest reply the slot window holds, counted by the server's tokenizer; - close(), also at exit: retire adapters, release every reference, revoke grants. Without it the engine-wide workspace refused the next script's owner (owner_mismatch, reproduced here). Migrated: dream-adapter-routes.py (CI), learn-past-example.py, learn-unload-during-learn.py, learn-every-format.py (all six modes), learn-cancel-custody.py, learn-chunk-yields.py, learn-refuses-repacked.py. No script needs a hand-placed shadow-05.gguf any more: shadows are minted fresh (two from one seed are the "copy" two-minds needs); the F16 copy is still made by make-f16-shadow.py, from a fresh shadow. Measured on a CPU llama-server built from this branch (IntelMac, stories260K, --no-repack): dream-adapter-routes twice then learn-past-example twice on ONE server: pass (this failed with owner_mismatch before close existed); learn-every-format default, --shadow-retirement, --two-minds, --non-f32-refused: pass; learn-cancel-custody: pass; learn-refuses-repacked (server without --no-repack): pass. Not passing, reported, not hidden: - learn-unload-during-learn.py aborts the engine 2 of 3 runs: ggml-opt.cpp:969 GGML_ASSERT(!allocation_pending && !graph_pending) in ggml_opt_free <- opt_release, freeing idle B while A learns (confirmed from source by Codex; the native fix is assigned). This script is the regression for it. - --fairness and learn-chunk-yields learn under contention: a contended graph chunk (1.5 s on chunk-yields) exceeds an allowance only idle steps can measure, so the step is refused mid-way (chunk_exceeds_latency). How a contended learn obtains its allowance is the ledger owner's open question. - --template-replies needs a model whose template renders tool_calls; stories260K's does not. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
…what is held, and its own need
The step workspace is engine-wide and its custody passes from one reference to the next; claim_plan
refuses ('uncovered') a replacement that does not cover every row the previous reference was
registered with. Measured on CPU (stories260K, -np 2): A's long learn registered 7,026,688 bytes,
the scheduler then held 4,907,008, and the reloaded B's short reply needed 2,573,344; sized to its
own need (or even to need plus held) B's workspace was refused every time. The fixture now
remembers the rows it last registered and covers the union of those, shared_compute_held and the
example's own requirement, each buffer at the largest. learn-unload-during-learn.py then passes
10/10 on the fixed engine (fork #108).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
Author
|
Review (Fable): approve. One owner of the learn lifecycle for every learn script (grants, custody registered from the engine's own dry-run rows, seating, release), closed on atexit so a failed assertion never leaves custody bound for the next script; seven scripts drop their hand-rolled copies (+419/-276). |
joelteply
changed the base branch from
cormac/inherited-ci-repair
to
codex/s4-per-mind-integration
November 11, 2026 14:31
joelteply
merged commit Nov 11, 2026
c6f6277
into
codex/s4-per-mind-integration
7 of 10 checks passed
joelteply
added a commit
that referenced
this pull request
Nov 11, 2026
…shadow unloads while another mind learns c5551ed made ggml_opt_free assert that the SHARED scheduler had no pending graph. A serving context keeps one optimizer per shadow on one scheduler, and llama_context::opt_release frees an idle shadow's optimizer while another shadow's step is in flight (#85's contract: "ggml_opt_free touches no shared scheduler"). That assert then aborted llama-server whenever the unload landed mid-step (fork #106/#107 reproducer, learn-unload-during-learn.py, 2 of 3 runs on CPU). ggml_opt_free releases only this context's own buffers and metadata. The check is now context-local: ggml_backend_sched_graph_pending_for(sched, graph), true only when the scheduler is part-way through THAT graph (graph_step already keys its continuation on the caller's graph pointer). Freeing a context whose own graph is in flight still aborts. ggml_opt_reset and ggml_opt_prepare_alloc keep the scheduler-wide guard: reset zeroes the last graph's gradients, which in a dynamic graph can sit in the shared compute buffer another mind now uses, and prepare_alloc allocates onto the scheduler. Tests (test-opt-dynamic-accum, built with -fsanitize=address on CPU, IntelMac): - run_frees_idle_mind_while_another_steps: B learns a step, A stops one node into its step (scheduler pending), B is freed, A completes with exactly the update it makes alone. PASS. - run_free_own_pending_graph_aborts: freeing a context whose own graph is in flight aborts (observed in a forked child; skipped with a message where fork() is unavailable). PASS. - the existing suite: PASS. - Mutation: restoring the scheduler-wide check aborts the first test at the same assert the engine hit (exit 134). End to end: the reproducer passes 10/10 on the fixed CPU server, but it is timing-dependent (it aborts only if the unload lands mid-graph; 0/6 on the unfixed engine in a later run), so the unit test is the proof. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tests: one learn fixture owns custody, grants and release for every learn script
Every learn now runs inside custody the core registers (weights, optimizer and workspace
residencies from the engine's own preflight) under a ledger grant, and every lifecycle verb names
the engine incarnation. Seven scripts predated that; each copied its own transport and none could
run. tools/server/tests/learn_fixture.py is the one owner:
added on bound routes;
workspace per example, split exactly as admit_learning_residency splits rows;
chunks and boundary, max(entry, restoration);
engine-wide workspace refused the next script's owner (owner_mismatch, reproduced here).
Migrated: dream-adapter-routes.py (CI), learn-past-example.py, learn-unload-during-learn.py,
learn-every-format.py (all six modes), learn-cancel-custody.py, learn-chunk-yields.py,
learn-refuses-repacked.py. No script needs a hand-placed shadow-05.gguf any more: shadows are
minted fresh (two from one seed are the "copy" two-minds needs); the F16 copy is still made by
make-f16-shadow.py, from a fresh shadow.
Measured on a CPU llama-server built from this branch (IntelMac, stories260K, --no-repack):
dream-adapter-routes twice then learn-past-example twice on ONE server: pass (this failed with
owner_mismatch before close existed); learn-every-format default, --shadow-retirement,
--two-minds, --non-f32-refused: pass; learn-cancel-custody: pass; learn-refuses-repacked
(server without --no-repack): pass.
Not passing, reported, not hidden:
GGML_ASSERT(!allocation_pending && !graph_pending) in ggml_opt_free <- opt_release, freeing idle
B while A learns (confirmed from source by Codex; the native fix is assigned). This script is
the regression for it.
chunk-yields) exceeds an allowance only idle steps can measure, so the step is refused mid-way
(chunk_exceeds_latency). How a contended learn obtains its allowance is the ledger owner's open
question.
Stacked on #106 (cormac/inherited-ci-repair); retarget to codex/s4-per-mind-integration after it lands. Card f1d9d2cf.
🤖 Generated with Claude Code
https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE