fix: [no-ticket] harbor - honest measurement signals (summary qualifiers, versioned re-evals, dead-attempt causes)

[shehabyasser-scale]

Stacked on #34 (harbor-7-access-tiers). Part of the pre-paper hardening series; this is the "recorded numbers say what they are" group.

Changes

1. summary.json qualifies its mean (server.py). The agent-visible aggregate now carries n_scored, n_errored, and score_se beside mean_score. A mean over 3-of-18 scored samples, or one dominated by errored zero-fills, is a different measurement than a clean full-split mean; the agent (deciding whether a delta is real) and any auditor (reading the run afterward) should both see that without per-sample access. All three are label-safe aggregates, so they ride the same non_viewable-safe summary as before.

2. Enum value, not repr (server.py). "status": str(experiment.result.status) serialized as "ExperimentResultStatus.SUCCESS" (str-mixin enums lost their value-__str__ in py3.11). Now writes .value -> "success".

3. Versioned result dirs (server.py). Result dirs were keyed on (split, commit[:12]) and wiped on re-eval, so a repeat measurement of the same commit erased the agent's earlier evidence; repeat measurements (multifidelity confirms, champion re-evals under a noisy scorer) are exactly the ones worth comparing. Each metered eval now gets {split}__{commit12}__eN and result_path in the response names the dir for that eval. The stale-file hazard the wipe used to prevent is gone structurally: no two evals share a dir.

4. Dead-attempt causes (runner.py). Mean-mode collation zero-fills attempts that died before scoring and counts them in n_dead; it now also records dead_exception_types (a per-cause count) in the sample output. Causes are not interchangeable: rate-limit deaths are infra noise that retunes with capacity, crashes point at the candidate, and deaths cluster hard by cause in practice (measured on 46 historical runs: 110 of 129 UnicodeDecodeError deaths sat on two never-solved tasks). Kept out of metrics (float-valued by contract) and rides the same viewable-only exposure path as attempt detail.

Tests

• summary qualifiers: n_scored/n_errored/score_se/mean asserted on a known score vector; status == "success".
• versioned re-evals: two evals of one commit -> __e1 and __e2, both dirs alive.
• dead causes: recorded exception type + no-exception death bucketed as no_rewards_recorded; all-clean samples carry no key.
• 81 passed on the affected files; existing dest assertions updated to the versioned naming.

🤖 Generated with Claude Code

Greptile Summary

This PR hardens the measurement-signal layer across four independent axes: summary.json now carries n_scored, n_errored, and mean_score_se so a thin-sample or error-dominated mean is legible without per-sample access; metered result dirs gain a per-eval __eN ordinal so repeat measurements (multifidelity confirms, champion re-evals) coexist on disk rather than the second erasing the first; dead attempt causes are recorded per exception type in dead_exception_types so n_dead is no longer opaque to infra vs candidate failures; and a bounded, off-by-default infra-retry path lets trusted-candidate evaluations recover from transient outages without exposing an adversarial re-roll surface.

• summary.json qualifiers: n_scored/n_errored/mean_score_se are zero-fill–consistent (SE is of the same population as mean_score), and status now serialises as .value fixing the ExperimentResultStatus.SUCCESS string regression on Python 3.11+.
• Versioned result dirs + restart resume: _eval_seq is scanned from surviving __eN dirs on the first route call after a restart, so old-session evidence is never silently wiped.
• Dead-attempt causes + infra retry: the conservative allowlist, adversarial opt-out (infra_retry_rounds = 0 by default), sibling-dir isolation, and per-recovery audit markers (infra_retry) are all in place; key-budget exhaustion is correctly classified as non-retryable infra.
• Additional clean-ups: free-baseline claim-before-await race fix, ledger fail-closed on corrupt file, k_anonymity_floor default aligned to sidecar enforcement, transfer-target model override wired end-to-end, instruct_exhaust_budget ablation lever.

Confidence Score: 5/5

Safe to merge; all changes add measurement-signal qualifiers, fix a serialisation regression, and introduce an opt-in recovery path — none touch any critical hot path without proper guards.

The four axes of change (summary qualifiers, versioned dirs, dead-cause classification, infra retry) are each independently scoped and extensively tested (81 assertions on affected files). Security-sensitive properties — adversarial opt-out for infra retry, zero-score-fill regardless of classification, conservative exception allowlist, sibling-dir isolation — are correct and explicitly verified. Two minor audit-record issues exist but neither affects scores, recovery correctness, or the agent-visible data.

runner.py _retry_transient_infra — the discard_history accumulation timing is worth a follow-up in a multi-round, no-output-round scenario.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/server.py	Versioned result dirs (__eN), summary.json qualifiers (n_scored/n_errored/mean_score_se), enum .value fix, free-baseline claim-before-await race fix — logic all correct and well-tested.
vero/src/vero/harbor/runner.py	Dead-attempt cause classification, n_dead_infra metric, key-budget alarm, and bounded infra retry — infra vs candidate distinction is sound; security guards (adversarial-opt-out, no score excusal, no key-budget retry) are all in place.
vero/src/vero/harbor/serve.py	Ledger fail-closed on corrupt file (was: silent refund), instruct_exhaust_budget field added — both changes are correct and the fail-closed behaviour is now tested.
vero/src/vero/harbor/verifier.py	_admin_eval_score now returns (score, cause) tuple, _dominant_sample_errors produces a task-normalised frequency summary, transfer-target model override wired through baseline and champion evals.
vero/src/vero/evaluation/engine.py	evaluate_admin gains model= parameter — correctly uses model_copy to produce a one-shot params object without mutating shared run_constraints.
vero/src/vero/harbor/config.py	infra_retry_rounds / infra_retry_delay_s added with correct validation (negative rounds rejected; zero delay forbidden when retries enabled).
vero/src/vero/harbor/protocol.py	k_anonymity_floor default corrected from 1 to 5 to match EvaluationSidecar's enforcement default — prevents agents seeing a laxer floor than the sidecar enforces.
vero/src/vero/harbor/build/config.py	TargetSpec.model and instruct_exhaust_budget fields added; both are nullable/defaulted correctly and flow through to serve.json.
vero/tests/test_harbor_runner.py	Comprehensive TestInfraResilience suite covering off-by-default, sibling-dir invariant, no-trial-round persistence, partial recovery, multi-round backoff, mixed-cause guard, and candidate-crash/key-budget non-retry.
vero/tests/test_harbor_server.py	TestHonestSummary covers qualifiers + SE value, versioned re-eval dirs, and volume-reuse ordinal resume; TestFreeBaselineEval gains the claim-before-await concurrency test.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[produce_sample_results] --> B[_run_harbor\ninitial jobs_dir]
    B --> C[_collate\njobs_dir]
    C --> D{infra_retry_rounds > 0?}
    D -- No --> Z[Done]
    D -- Yes --> E[_retry_transient_infra]
    E --> F{for each round}
    F --> G[_transient_infra_dead\nper sample]
    G --> H{any retryable?}
    H -- No --> Z
    H -- Yes --> I[asyncio.sleep\nN x delay_s]
    I --> J[_run_harbor\nsibling round_dir]
    J --> K[_load_trials\nround_dir]
    K --> L{any produced?}
    L -- No --> F
    L -- Yes --> M[_collate\nround_dir only]
    M --> N[_mark_recovered\nadd infra_retry audit marker]
    N --> F

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[produce_sample_results] --> B[_run_harbor\ninitial jobs_dir]
    B --> C[_collate\njobs_dir]
    C --> D{infra_retry_rounds > 0?}
    D -- No --> Z[Done]
    D -- Yes --> E[_retry_transient_infra]
    E --> F{for each round}
    F --> G[_transient_infra_dead\nper sample]
    G --> H{any retryable?}
    H -- No --> Z
    H -- Yes --> I[asyncio.sleep\nN x delay_s]
    I --> J[_run_harbor\nsibling round_dir]
    J --> K[_load_trials\nround_dir]
    K --> L{any produced?}
    L -- No --> F
    L -- Yes --> M[_collate\nround_dir only]
    M --> N[_mark_recovered\nadd infra_retry audit marker]
    N --> F

Loading

_{Reviews (2): Last reviewed commit: "Merge pull request #36 from scaleapi/har..." | Re-trigger Greptile}

[varunursekar]

this seems tied to the particular instruction template? should this be in a general config?