Add structured Puzzletron results and progress

examples/puzzletron/README.md

@@ -11,6 +11,12 @@ inputs:
 Puzzletron resolves the full stage configuration, validates it, and stores an
 immutable copy with the run.
 
+- [Start here: lifecycle smoke](#quickstart)
+- [Choose the next task](#choose-a-recipe)
+- [Results and live progress](docs/campaign_reports.md)
+- [Central results catalog](reports/catalog.yaml)
+- [Documentation map](#documentation)
+
 ## Quickstart
 
 Use Python 3.10 through 3.14 for the lightweight controller environment:
@@ -62,9 +68,19 @@ the run, not the current recipe or site file.
 ```bash
 python examples/puzzletron/puzzletron.py resume /shared/puzzle_runs/my-run
 python examples/puzzletron/puzzletron.py inspect /shared/puzzle_runs/my-run
+python examples/puzzletron/puzzletron.py results inspect /shared/puzzle_runs/my-run
 ```
 
-Use a new run directory whenever an authored input changes.
+Use a new run directory whenever an authored input changes. Completed
+compatible stages are not rerun, and `resume` always uses the sealed inputs
+under the existing run directory. See [run and
+recovery](docs/orchestration_operations.md) for allocation replacement and
+retry behavior.
+
+Puzzletron stores scheduler state and resolved bundles under
+`<run-root>/orchestration/`. The authoritative run, stage, progress, subject,
+metric, artifact, provenance, and timing evidence is atomically refreshed in
+`<run-root>/results/result.json`.
 
 ## Choose a recipe
 
@@ -74,13 +90,28 @@ model, workflow, and mode combinations. Five checked-in recipes cover Qwen 3.5
 campaign routes. See [maintained recipes](docs/maintained_recipes.md) for their
 requirements and interpretation limits.
 
+## Read results and progress
+
+After the selected plan completes cleanly, Puzzletron finalizes `result.json`,
+then generates the optional HTML view at
+`<run-root>/artifacts/campaign_report/campaign_report.html`. The HTML contains
+no unique evidence and can be regenerated from the result. See the
+[central results catalog](reports/catalog.yaml) for retained runs and
+[campaign reports](docs/campaign_reports.md) for detached inspection, export,
+refresh, evidence drill-down, and interpretation. See
+[run and recovery options](docs/orchestration_operations.md) for individual
+stages, `--once`, logging controls, security options, and recovery details. For
+a failed or interrupted run, follow the actionable checks in
+[run and recovery options](docs/orchestration_operations.md#progress-and-interruption).
+
 Recipes are the public run interface. Files under `configs/families/` are
 internal composition templates and should not be edited or launched directly.
 
 ## Custom models
 
 If no maintained recipe matches the model, use the existing setup wizard
-through the same main command:
+through the same main command. Review its custom-model inputs first with
+`python examples/puzzletron/puzzletron.py setup --help`, then run:
 
 ```bash
 python examples/puzzletron/puzzletron.py setup
@@ -105,6 +136,8 @@ compatibility, but they are not additional maintained-recipe workflows.
   evaluation, serving, and distillation stages.
 - [Campaign reports](docs/campaign_reports.md): generate and interpret the
   cumulative report.
+- [Central results catalog](reports/catalog.yaml): discover retained structured
+  results and qualified historical reports.
 
 Additional evaluator and implementation references live under
 `examples/puzzletron/docs/`.

examples/puzzletron/configs/families/qwen3_5/qwen3p5_0p8b/runs/vlm_campaign.yaml

@@ -170,7 +170,7 @@ post_mip:
           input: post_kd_eval
           mode: aggregate_rank
           metrics:
-            - metric: post_kd_eval.modelopt_vlm_benchmark_realworldqa.exact_match_flexible-extract
+            - metric: post_kd_eval.modelopt_vlm_benchmark_realworldqa.exact_match_none
               direction: maximize
             - metric: post_kd_eval.modelopt_vlm_benchmark_mmmu_val.mmmu_acc_none
               direction: maximize

examples/puzzletron/configs/site.example.yaml

@@ -6,6 +6,7 @@ site:
     venv: REPLACE_WITH_WORKER_VISIBLE_MODELOPT_VENV
     container:
     container_mounts:
+    # Puzzletron defaults runtime caches below TMPDIR; most sites need no cache hooks.
     prerun_commands: []
     postrun_commands: []
   paths:

examples/puzzletron/docs/campaign_reports.md

@@ -1,47 +1,117 @@
-# Puzzletron Campaign Reports
+# Puzzletron results and campaign progress
 
-After a campaign completes cleanly, Puzzletron attempts to generate a
-cumulative HTML report through the configured runner. A report submission,
-polling, or artifact failure does not invalidate completed stages, but it is
-recorded in the run result and the command exits nonzero. Inspect the campaign
-logs, then regenerate the report without rerunning model work:
+Puzzletron stores each run's evidence in one structured JSON result. That
+document drives live and detached status, portable result export, the central
+results catalog, and the optional HTML summary. The HTML is a replaceable view:
+it does not contain evidence that is absent from the structured result.
+
+Use the generated [results catalog](../reports/catalog.yaml) to find every
+retained run. It is the single central listing and points to each structured
+result and any available human-readable summary. Regenerate it after
+adding or updating checked-in result leaves:
+
+```bash
+python examples/puzzletron/generate_results_catalog.py
+```
+
+The catalog is YAML so it is easy to scan and review. Run results remain JSON
+because their canonical bytes are validated, hashed, and atomically replaced.
+
+## Inspect a running or detached campaign
+
+The controller atomically refreshes `<run-root>/results/result.json`. It
+contains the run, DAG stages, attempts, completed and total work, native
+evaluator dimensions, timing, freshness, and qualified ETA. Detached
+inspection reads that same file and does not require the original controller
+process:
 
 ```bash
-python examples/puzzletron/generate_campaign_progress_report.py \
-  --puzzle-dir /shared/puzzle_runs/my_campaign \
-  --model-name 'My model'
+python examples/puzzletron/puzzletron.py results inspect /shared/puzzle_runs/my_campaign
+python examples/puzzletron/puzzletron.py results inspect /shared/puzzle_runs/my_campaign --json
+```
+
+Treat `attachment: detached` separately from scheduler state: work may still be
+running after its launching terminal exits. Check `freshness` before acting on
+a status. An ETA is qualified only after the producer has a stable total and
+observed progress; otherwise its reason explains why no estimate is shown.
+Completed progress remains visible so engineers can reconstruct what finished,
+not only what is active now.
+
+Evaluator records distinguish repetitions, evaluator iterations, tasks,
+samples, and optimizer steps. The controller polling count is never presented
+as evaluator work. A failed evaluator with zero processed samples stays a
+failure with its diagnostic artifacts; it cannot become a numeric score.
+
+## Export and refresh
+
+After clean completion, Puzzletron finalizes `result.json` before it generates
+any presentation. The result is already portable. Validate and locate it
+without rerunning model work:
+
+```bash
+python examples/puzzletron/puzzletron.py results export /shared/puzzle_runs/my_campaign
+```
+
+Regenerate the optional HTML from that structured evidence:
+
+```bash
+python examples/puzzletron/puzzletron.py results refresh /shared/puzzle_runs/my_campaign
 ```
 
 The output is
-`<puzzle-dir>/artifacts/campaign_report/campaign_report.html`. Section inputs
-and configuration fingerprints are cached under
-`<puzzle-dir>/artifacts/campaign_report/section_cache`. Use
-`--rebuild-section aiperf` to rebuild one section, or `--no-cache` to rebuild
-the whole report.
-
-This page also catalogs retained Puzzletron campaign reports and the status of
-their evidence. The compact [campaign report index](../reports/campaign_report_index.yaml)
-records each report's producer state, reproduction and support status, metadata
-origin, current-configuration relationship, and known limitations. Detailed
-run facts remain in the reports.
-
-Retained reports are self-contained HTML files and may be hundreds of MB.
-Download them and open them locally. Interpret their results together with the
-reproduction status and unresolved findings below.
-
-## Report status
-
-| Model | Report | Producer state | Reproduction | Support | Current configuration relationship |
-|---|---|---|---|---|---|
-| Nemotron-3 Nano 30B-A3B | [Campaign report](../reports/nemotron3_nano_30b_a3b.html) | `development_snapshot`; revision `unknown` | `not_reproduced` | `not_established` | `migration`: [default.yaml](../configs/families/nemotron3/nano_30b_a3b_bf16/runs/default.yaml) is not the executed configuration |
-| Qwen3.5-9B | [Campaign report](../reports/qwen3p5_9b.html) | `development_snapshot`; revision `unknown` | `not_reproduced` | `not_established` | `reconstruction`: [default.yaml](../configs/families/qwen3_5/qwen3p5_9b/runs/default.yaml) is not the executed configuration; the report records additional overrides and width values |
-
-## Evidence boundary
-
-| Record | Status |
-|---|---|
-| Retained reports | Preserve the detailed configuration, stage, result, and warning data from their producing development state. |
-| Campaign report index | Records only curated status, metadata origin, current-configuration relationship, and known limitations. |
-| Current configuration references | Provide migration or reconstruction starting points, not frozen executed configurations. |
-| Reproduction status | No reproduction is recorded for the listed reports. |
-| Support status | Not established while reproduction and unresolved correctness findings remain open. |
+`<run-root>/artifacts/campaign_report/campaign_report.html`; its neighboring
+`report_manifest.json` records `source_result_digest`, the renderer revision,
+output digest, and validation status. The summary visibly includes
+run and subject identity, teacher and candidate roles, heterogeneous
+architecture axes, execution and attachment state, freshness, timing, DAG
+parents and phases, active and completed progress, metric values and qualified
+comparisons, artifacts, provenance, and limitations. If HTML generation fails,
+the sealed structured result remains valid and usable.
+
+## Metric and comparison boundaries
+
+Teacher, candidate, and control checkpoints use the same subject, architecture,
+metric, artifact, and limitation fields. Each teacher/candidate metric pair
+with the same name and producer execution produces a comparison entry that
+names both source metric IDs. A numeric delta is emitted only when the unit,
+direction, aggregation, workload contract, task, row manifest, prompt template,
+decoding contract, and dimensions match and both values are numeric. Dimensions
+carry denominators, evaluator repetitions, and sample counts when producers
+record them. Otherwise, the entry records explicit exclusion reasons.
+
+Language-model loss is `quality.lm_loss` in `nats_per_target_token` and uses
+`lower_is_better`. Producers must distinguish a target-token-weighted mean over
+all unmasked target tokens from the current scoring route's unweighted mean of
+per-sample token means. Those aggregation and denominator contracts are not
+interchangeable. Record teacher loss and candidate loss under the same frozen
+workload when both were explicitly measured; do not infer teacher loss from a
+candidate loss or teacher-relative metric. Token accuracy follows the same
+rule: preserve whether ratios are token-weighted or averaged per sample.
+`training.effective_tokens` is cumulative loss-bearing exposure after masking
+and packing. It records tokenizer and data identity and whether it was measured
+or derived; it is not inferred from optimizer steps times maximum sequence
+length. Requested input/output tokens, observed sequence lengths, aggregate
+output-token throughput, per-user throughput, token accuracy, examples,
+samples, steps, latency, and GPU-hours remain distinct measures.
+
+## Historical evidence
+
+Older retained runs use the provisional
+`modelopt.puzzletron-result-record/v1` schema and retain their historical
+`result_record.json` filenames. Their summaries and structured files remain
+available through the central catalog, but the catalog marks them as qualified
+historical evidence and does not translate nested historical values into new
+tidy metrics. This preserves the original claim boundaries, including bespoke
+or superseded selection policies, unmatched teacher/student conditions,
+missing row manifests, missing repetitions, and incomplete runtime provenance.
+
+The two standalone historical HTML reports have structured legacy wrappers.
+Those wrappers carry their producer, reproduction, support, current-config
+relationship, and known limitations. They do not infer missing values from the
+HTML. A current configuration linked from a legacy record is a migration or
+reconstruction starting point, not proof of the executed configuration.
+
+Run summaries remain useful derived explanations of recorded results and
+limitations. There are no reports-level or campaign-level README indexes;
+navigation belongs to `reports/catalog.yaml`, and this guide owns the shared
+operational and interpretation instructions.

examples/puzzletron/docs/slurm_configuration.md

@@ -29,12 +29,16 @@ profiles have been qualified only in the documented eight-GPU node
 environment.
 
 This mode assumes a Slurm site where one containerized task can see the full
-node allocation, shared campaign paths are mounted identically, and runtime
-caches are writable for the lifetime of the outer job. The outer job uses the
+node allocation and shared campaign paths are mounted identically. Puzzletron
+places XDG, Triton, FlashInfer, Torch extension, and vLLM caches in an isolated
+task-local directory below `TMPDIR` (or the platform temporary directory when
+unset). Explicit cache
+environment variables from the launch environment or site hooks take
+precedence. The outer job uses the
 site's CPU and memory defaults for its GPU request, which must be sufficient for
 the concurrent workers. Other sites may need to adapt the site's account,
 partition, container integration, mounts, time limit, GPU capacity, and cache
-hooks. Use per-attempt mode when those assumptions do not hold.
+root. Use per-attempt mode when those assumptions do not hold.
 
 ### Stage instances
 
@@ -178,7 +182,12 @@ existing variable such as `${API_KEY:?set API_KEY}`, retrieve it from a secret
 command, or source a permission-protected `setup_env` file. This check catches
 common mistakes but is not a shell parser or a complete credential scanner.
 
-For containerized workers, use `prerun_commands` when the site needs to place
-`TMPDIR` or runtime caches in a short, worker-local writable directory. vLLM
-uses Unix-domain sockets with a platform path limit, and a read-only container
-home prevents runtime caches from being initialized.
+Puzzletron gives each task writable defaults for `XDG_CACHE_HOME`,
+`TRITON_CACHE_DIR`, `FLASHINFER_WORKSPACE_BASE`, `TORCH_EXTENSIONS_DIR`, and
+`VLLM_CACHE_ROOT` below `TMPDIR`, falling back to the platform temporary
+directory. Existing values are preserved, so most containerized workers need
+no cache hooks, including when the
+container home is read-only. Set `TMPDIR` with `prerun_commands` only when the
+site's `/tmp` is unsuitable or when vLLM needs a shorter root for its
+Unix-domain sockets. Override an individual cache variable only for a site that
+requires a different location.

examples/puzzletron/docs/v2_architecture.md

@@ -6,8 +6,9 @@ seals the full configuration, then the orchestrator compiles and executes a
 resumable stage graph.
 
 Architecture support does not by itself establish that a model, pruning axis,
-or topology has been validated end to end. See the [campaign report
-catalog](campaign_reports.md) for recorded runs and their evidence status.
+or topology has been validated end to end. See the [results
+catalog](../reports/catalog.yaml) for recorded runs and their evidence status,
+and the [campaign reports guide](campaign_reports.md) for interpretation.
 
 ## Components