Repository navigation
Monitoring: build the restricted stall-prediction pilot and frozen baseline #11
Description
Current status (2026-09-30)
A restricted simulator, journal, fixtures, and a scripted pilot were committed in the v1.2.2 line. The scripted policy received fixture target files and reported synthetic token accounting, so that run did not produce a qualifying frozen learned baseline. A separate baseline-only DeepSeek preflight used the real session hook on 12 training fixtures; it found 7/12 verified successes and three trajectories with a qualifying stall window, including one invalid run. It did not train, calibrate, or freeze the predictor specified here, and no held-out data should be used to tune it. Keep this issue open for the original restricted-executor, artifact, budget, and frozen-learner acceptance criteria. The monitor-first sequencing decision in #13 has been superseded by the active RSI proof #15–#20; this monitor experiment is independent of that proof.
Original pilot specification (historical context)
The design below records the original requested scope and acceptance checks. The current status above identifies what remains to be completed.
What problem are you trying to solve?
Before adding a learned monitor to real coding sessions, HeadlessCode needs a reproducible environment that can measure early stall prediction without exposing the host or confusing final failure with the correctness of every earlier step.
This issue depends on #10. It builds the restricted pilot, development fixtures, authoritative records, and one frozen logistic predictor. It does not run the held-out four-arm evaluation.
Proposed behavior
Create a monitor-pilot package under scripts/monitor-pilot/ with:
-
Restricted simulator
- In-memory virtual JSON/text files only.
- Typed
read_file,list_files,apply_diff, andattempt_completionoperations. - A simulated
execute_commandthat accepts only the exactvalidatetoken and never spawns a process. - No host paths, shell, Git, network, dynamic imports, child sessions, runtime tool registration, checkpoints, browser, vision, or auxiliary model calls.
- Unknown or forbidden operations fail closed with zero host effects.
-
Authoritative experiment records
- Versioned manifest, boundary, prediction, intervention, and outcome records.
- Experiment/run/attempt IDs, ordered sequence numbers, model and fixture hashes, budgets, feature and predictor hashes, and final oracle result.
- Append-only records with an exclusive runner lock and corruption detection.
- Interrupted attempts become aborted; uncertain model calls are not replayed. A retry receives a new attempt ID and consumes the remaining global budget.
-
Fixtures
- 24 hand-authored configuration-repair fixtures, six in each family:
- wrong key or path assumption;
- revision-mismatched patch requiring a re-read;
- failed validation with independent constraints;
- legitimate multi-file investigation requiring several reads before an edit.
- Split each family into three training, one calibration, and two held-out test fixtures. Freeze sibling variants in the same split.
- Each fixture has a visible validator and a separate deterministic final oracle that also checks preservation of unrelated state.
- 24 hand-authored configuration-repair fixtures, six in each family:
-
Frozen learner
- One L2 logistic predictor using only: identical-call streak, same-tool-error streak, read-only streak, unique-call fraction over four turns, error fraction over four turns, turns since successful virtual write, repeated-result fraction, and iteration-budget fraction.
- Standardization fitted on training data only.
- Boundary label: the following three uncorrected turns contain a redundant or invalid retry and no oracle milestone progress. Mask shorter terminal windows.
- Calibration-only probability scaling and threshold selection over 0.50–0.95 in 0.05 steps, requiring precision at least 0.80 and false-positive rate at most 0.10.
- No online updates, feature search, alternate model class, temporal network, LLM monitor, or SNN.
- Use the existing TypeScript runtime or a standalone build-time script without adding a runtime dependency. Export a hashed, versioned artifact and test inference determinism.
Development collection budget
Run only the training and calibration fixtures after all fake-client and boundary tests pass:
- 16 development fixtures × two repetitions = 32 scheduled runs.
- At most 16 model calls, 20,000 total input-plus-output tokens, and five minutes per run.
- Seeds 101 and 202 control arm ordering and provider sampling where supported; record when the provider cannot honor a sampling seed.
- Pin the resolved model digest and settings; do not use an unresolved mutable
latestidentity.
Stop as data-insufficient if training has fewer than 20 positive windows across eight trajectories or calibration has fewer than 10 positive windows across four trajectories. Do not automatically add fixtures, tune features, or try another learner.
Alternative approaches
- Train directly from current RSI trajectory exports. Rejected because histories can be duplicated and their trusted label is derived from the current gates rather than independent verification.
- Use real repositories immediately. Rejected because host effects and uncontrolled task variation would make a failed first result difficult to interpret.
- Start with an SNN or recurrent model. Rejected until a linear baseline demonstrates transferable signal.
Security / least-privilege impact
The simulator, not prompts or command filters, is the pilot authority boundary. Model output is data. The final oracle, fixture split, journal, budgets, and predictor artifact remain supervisor-owned. The predictor may request one fixed correction but cannot authorize tools or modify the experiment.
Do not route pilot requests through the normal ToolExecutor or candidate test execution path.
Acceptance criteria
- The restricted executor has tests showing attempted shell, absolute path, traversal, network, delegation, evaluator access, and dynamic tool registration have zero host effects.
- Train/calibration/test separation and fixture-family grouping are validated before collection.
- Feature extraction has no future-label or oracle-state leakage.
- Fake-client tests cover successful long investigations so a read-only streak is not automatically treated as failure.
- Journal corruption, duplicate sequence, stale attempt, concurrent runner, and interrupted-call cases fail closed or become explicitly aborted.
- The development collection stays within its admitted budget.
- Predictor coefficients, scaling, feature order, threshold, model digest, splits, templates, and hashes are frozen before the held-out test.
- Insufficient label counts or no qualifying threshold produces a recorded STOP result rather than an architecture search.
npx tsc --noEmitand the fullnpm testsuite pass.
Additional context
The held-out evaluation is a separate issue so test outcomes cannot silently change the learner or experiment design.
Activity
w4ffl35 commentedon Sep 22, 2026
Tracked by #13. This depends on #10 and must freeze the predictor and experiment artifacts before #12 begins.
w4ffl35 commentedon Oct 11, 2026
The restricted live collector has been implemented and reviewed, with additional fixes for native token accounting, billing on aborted admission, fixed intention-to-treat denominators, calibration timing, and source freezing. Integration PR and full final qualification are still pending.
Two development attempts used only OpenRouter
meta/muse-spark-1.3-contributorthrough Meta, with fallback disabled. Both stopped on the first assigned training slot; no training/calibration trajectory completed and no held-out fixture was accessed.max_tokens=512. The existing completion guard stopped before dispatching that response's tools. Its cache receipt reported 1,137 cached tokens out of 1,267 prompt tokens. No cap was relaxed and no predictor or threshold was fitted.Both full per-run reservations remain charged: 40,000 tokens and ten minutes in the shared experiment budget. Recorded actual cost across both attempts is $0.000460142; aborted attempts are retained and were not classified as free calls. The second source freeze carries the first budget journal forward rather than resetting the allowance. A focused regression reproduces 519 > 512 and checks that billing persists while the reservation remains unsettled. Abort journaling now emits bounded failure codes without raw provider errors.
Status: STOP on a provider output-budget admission result, not a qualifying development predictor. Keep this issue open; #12 remains gated. No local GPU tests were run.
w4ffl35 commentedon Oct 11, 2026
The restricted collector and receipt/accounting fixes shipped in v1.7.0 and passed the full local suite with real PostgreSQL/S3 plus CI. The live experiment status remains STOP: the authoritative Muse receipt reported 519 completion tokens against the frozen 512-token limit. Its billed rejected response and previous reservations remain in the journal. No predictor or threshold has been fitted, and no held-out fixtures were used. This issue remains open for a qualifying collection under a separately frozen protocol; shipping the collector does not qualify the experiment.
w4ffl35 commentedon Oct 12, 2026
The frozen Muse Spark Contributor/Meta development schedule reached 32/32 slots, with 230/230 billed responses and no unresolved requests. Reported provider cost was $0.025320744; 226,172 of 310,790 prompt tokens were cached. Per-run and shared budgets stayed within their frozen ceilings, including three prior unsettled reservations.
The specified gate recorded STOP: calibration had 9 positive windows, below its minimum of 10. No predictor was frozen and no held-out trial ran.
Independent audit also found all 230 saved response events fail hash verification; reopening the journal rejects record 3. The original records remain unchanged. Journal SHA-256:
86c0db521f8bf1709ff7f1dcc4b827053194d3945e17077bbecbbf8660cdbc84. Manifest SHA-256:fa4c2d7d980cae679bfd76174e124874dc3a71e417bed20c2521326c86be7f6f. Frozen source hashes matched. The session exited 143; finalized artifacts do not prove clean process exit.This issue stays open. #107 tracks the serialization fix and offline regression tests. These observations do not qualify the experiment, authorize more collection, or satisfy #12's admission condition.
Merge pull request #125 from Capsize-Games/issues/w1-2026-10-12
w4ffl35 commentedon Oct 13, 2026
Blocker status (2026-10-13): simulator/journal/fixtures/scripted pilot exist, but no qualifying frozen learned baseline has been trained, calibrated, or frozen (prior runs used scripted policies or baseline-only preflights). Needs a real training run with frozen artifacts under the original budget/registrar criteria, without touching held-out data. Blocked on model spend + operator scheduling.
w4ffl35 commentedon Oct 14, 2026
Live collection launched (2026-10-14 ~02:20Z). Command:
npx tsx scripts/monitor-pilot/live.ts collect-development --output-dir /home/joe/.local/share/headlesscode/monitor-pilot(detached, survives session end). Pinned modelmeta/muse-spark-1.3-contributorvia OpenRouter, provider Meta, no fallbacks; budgets per preregistration (16 calls / 20k tokens / 5min per run, 2M global). 32 scheduled runs (16 development fixtures x seeds 101/202). Journal: output-dirdevelopment/journal.jsonl(hash-chained from GENESIS); first rundevelopment:wrong-key-path-1:101budget-reserved. Do not touch held-out data; collection writes only to the private output dir. Will train/freeze/threshold the predictor from the journal when collection completes, then run the scripted qualification.w4ffl35 commentedon Oct 14, 2026
Dev-collection update: the 512-cap pilot self-halted fail-closed after 1/32 runs (journal pilot_stopped seq 61). Cause: attempt 2 iter 3 returned 537 native completion tokens (363 reasoning) against request maxTokens=512 — provider reasoning-token overrun tripped the completion-exceeded-request-cap guard (attempt_aborted seq 60, failureCode completion-exceeded-request-cap). Attempt 1 passed with a max of exactly 512, so the default cap sits exactly on the observed distribution — resume-at-512 would keep aborting.
Decision: supersede with a fresh development collection at --max-output-tokens 1024 (2x headroom over observed max 537; per-run 20k budget still fits 7+ iters worst-case). The manifest pins the cap, so this is a new output lineage; the 512 journal (~$0.001 spend, 1 completed run) is preserved as the aborted-pilot record and its data will NOT be mixed into the 1024 analysis. No harness code changed mid-pilot. Observation for later: the guard compares native (reasoning-inclusive) tokens against requested max; worth a design note but not a mid-pilot change.
w4ffl35 commentedon Oct 14, 2026
Update supersedes the 1024-cap restart: that lineage also self-halted on the same seed-202 slot (native 1033 vs cap 1024 — but visible only 122, with 884 reasoning tokens). Root cause is systematic, not a parameter choice: max_tokens constrains visible output while the guard compared native (reasoning-inclusive) counts. Fix in PR #167 (guard on visible, budget still native; regression test included). Both aborted lineages (512, 1024) preserved as calibration evidence; fresh collection launches on the fixed harness. No keeper data existed in either lineage (both halted at run 2).
w4ffl35 commentedon Oct 14, 2026
Third halt root-caused and fixed in PR #168: all three provider-or-unclassified-error aborts were Meta 400s ('arguments must be valid JSON') on history echo — truncated (2x) or empty (1x) tool-call arguments echoed verbatim. Reproduced live with a control ('{}' succeeds). Fix sanitizes the wire echo only; journal keeps raw; runs continue via the designed error-nudge path. Fresh lineage launches once #167+#168 merge.
Sanitize malformed tool-call arguments before echoing history (#11)
w4ffl35 commentedon Oct 14, 2026
Both harness fixes merged (#167 visible-cap guard, #168 echo sanitization). Fresh 32-run development collection launched from master c5e7b1b (default 512 cap) in a new lineage; three aborted calibration lineages preserved. Slot 1 completed; seed-202 in progress.
w4ffl35 commentedon Oct 14, 2026
32-run lineage complete: 32/32 runs, 0 aborted, 198/198 billed ($0.018). Predictor STOP: 19/20 positive train windows (15 trajectories, gate needs 8) and 8/10 calibration positives — narrow miss on both window counts.
Root cause of the miss is instrument clipping, not model behavior: 26 of 198 responses (13%) are EXACTLY 512 native / 509 reasoning tokens with null content and no tool calls — the provider hard-stops mid-reasoning at max_tokens=512, the harness reads empty-model-response, and the run ends. Most of the 32 trajectories were artificially shortened this way (10 produced zero label-eligible windows).
Calibration amendment (documented before seeing new labels): the 512 default cap is incompatible with this reasoning model (observed reasoning needs reach 884). Re-running the full 32-slot schedule at --max-output-tokens 2048 (2x observed max, within the 4096 configured bound) as a SEPARATE lineage — no mixing with 512 data. The 512 lineage stands as reported above; whatever the 2048 lineage yields (QUALIFIED or STOP) will be reported as-is.
w4ffl35 commentedon Oct 14, 2026
Development collection: final report (two lineages)
512 lineage (master pre-amendment): 32/32 runs, 0 aborted, 198/198 billed ($0.018). STOP: train 19/20 positives (15 trajectories), calibration 8/10 (4 trajectories). Post-hoc analysis found 26/198 responses (13%) clipped at exactly 512 native / 509 reasoning tokens with null content — the provider hard-stops mid-reasoning, ending runs early (10 trajectories produced zero label-eligible windows).
2048 lineage (calibration amendment, documented before any 2048 labels existed): 32/32 runs, 0 aborted, 236/236 billed ($0.025). Clipping fully eliminated (0 responses at cap, 0 empty). STOP: train gate PASSED (27 positives, 18 trajectories), calibration gate failed at 8/10 positives (6 trajectories).
Structural finding: the positive rate is consistent across splits (train 27/24 slots = 1.1/slot; calibration 8/8 = 1.0/slot), but the preregistered gates demand 0.83/slot for train vs 1.25/slot for calibration. The calibration gate is ~50% stricter per-slot against the same underlying rate.
legitimate-investigation-4yields 0 positives in all 4 runs (both seeds × both lineages — long wandering trajectories ending in token-budget exhaustion, never stalling per the label rule); every other calibration slot yields 1–2.No legitimate in-flight fix: gates, schedule, splits, and the label rule are all preregistered; the 512 lineage stands as reported and no data is mixed. Options for the owner: (a) new preregistration with an expanded calibration split; (b) a justified gate revision through the normal protocol process; (c) accept the negative result. All four journals (2 aborted calibration + 512 + 2048) are preserved with hash-chained manifests. Held-out fixtures remain untouched.