fix: [no-ticket] harbor - scoring integrity (dead attempts, best-rank monotonicity, strict reward key)

[shehabyasser-scale]

What

Three scoring-integrity fixes in HarborRunner, each closing a channel a live optimizer exploited or could exploit:

1. Dead attempts count 0.0 in mean aggregation (new n_dead metric). Attempts dying before the verifier scored were silently dropped from the mean, so the score estimated P(pass | attempt survived). Measured live in the Pawn replication: a no-retry candidate won selection at an inflated 0.233 while its retry-hardened successors measured an honest ~0.19 and lost; the optimizer documented the artifact in its own notes ("retry HELPS true quality but can LOWER measured train score"). All-dead samples error loudly instead of scoring 0.0.
2. 'best' trial ranking is monotone in the reward. The rank was (clean, has_rewards, recency), so with concurrent attempts 'best' meant "last clean attempt to finish" and a later clean 0.0 could clobber an earlier clean 1.0, violating the invariant documented in _load_trials. Reward now precedes recency in the key.
3. Strict reward_key extraction. A configured reward_key missing from a rewards dict no longer silently falls back to 'pass'/'reward' (attempts within one mean could be scored on different metrics), and several unrecognized keys are refused instead of averaged (a candidate could inflate the unweighted mean by emitting easy auxiliary metrics beside the real one). Sole-key dicts stay accepted.

Tests

tests/test_harbor_runner.py: 48 pass, including new pins for zero-fill (0.5 not 1.0), all-dead-errors, rank monotonicity, strict key, and refuse-to-average.

Stack

Based on #31 (harbor-4-mode-config-split). Part of the pre-paper hardening series; found by an 18-agent adversarial code sweep + live optimizer behavior.

🤖 Generated with Claude Code

Greptile Summary

This PR hardens scoring integrity in HarborRunner with three targeted fixes — dead-attempt zero-fill, reward-monotone best ranking, and strict reward_key extraction — all backed by documented live exploitation evidence and 48 tests. It also lands a bounded infra-retry mechanism (off by default), fail-closed ledger recovery on corrupt persistence, k-anonymity floor enforcement, versioned result directories, and a free-vs-admin flag separation in EvaluationEngine.

• Scoring fixes (runner.py): dead attempts now count 0.0 in mean aggregation (with n_dead/n_dead_infra metrics and cause-labeled exception types); _trial_rank inserts reward before recency so a later 0.0 cannot clobber an earlier 1.0; _extract_reward refuses silent fallback when reward_key is configured.
• Infra resilience (runner.py, config.py): bounded within-eval retry for samples whose every attempt died of a transient infra cause, gated by infra_retry_rounds=0 default with a clear adversarial-optimizer caveat; recovered samples carry an infra_retry audit marker naming the discarded rounds.
• Server/engine hardening (serve.py, server.py, engine.py, verifier.py): ledger unreadable → spend treated as exhausted (not refunded); free eval flag separated from admin to prevent free-baseline path from evaluating no_access splits; k-anonymity floor rejects small subset evals on non-viewable splits; finalize is now idempotent with a lock+cache.

Confidence Score: 5/5

Safe to merge. The three scoring-integrity fixes are logically correct, tightly scoped, and each directly backed by a live exploitation artifact. The infra-retry feature is off by default and surrounded by explicit adversarial-optimizer warnings.

The zero-fill, rank-monotonicity, and strict-key changes are all straightforward and well-tested. The verifier finalize idempotency and ledger fail-closed change are conservative and correct. The two observations flagged are audit-trail edge cases in an off-by-default retry path and an unlikely None-value disambiguation — neither affects scores or correctness under any realistic input.

No files require special attention. runner.py carries the most new logic but is thoroughly covered by the new test class.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/runner.py	Core scoring fixes: dead-attempt zero-fill with n_dead/n_dead_infra metrics, reward-monotone _trial_rank, strict _extract_reward; infra-retry feature (off by default). Logic is sound and well-commented.
vero/src/vero/harbor/verifier.py	Finalize idempotency (lock+cache), _admin_eval_score with bounded retry, tree-pooled shortlisting, baseline floor fail-safe, and target model override.
vero/src/vero/harbor/serve.py	Ledger fail-closed on unreadable file (corrupt copy preserved), free-vs-admin flag split, split_accesses wired to engine before sidecar, k_anonymity_floor threaded through.
vero/src/vero/harbor/server.py	k-anonymity floor enforcement on non_viewable subset evals; versioned result directories; free-baseline refund on exception; n_scored/n_errored/mean_score_se added to summary.json.
vero/src/vero/evaluation/engine.py	Separates free (budget-waived agent eval) from admin (tier-bypass); passes model override via task_params for transfer targets; split_accesses now armed on engine side.
vero/src/vero/harbor/config.py	Adds harbor_requirement pin, infra_retry_rounds/delay_s with validation, and model override support.
vero/tests/test_harbor_runner.py	48 tests covering zero-fill, all-dead-errors, rank monotonicity, strict key, infra classification, retry flow edge cases, and versioned directories.
vero/src/vero/harbor/protocol.py	Adds min_subset_samples to the per-split status block so agents see the k-anonymity floor before burning budget on rejected requests.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[attempt list] --> B{aggregate_attempts == mean?}
    B -- yes --> C[loop over all attempts]
    C --> D{rewards present AND _extract_reward != None?}
    D -- yes --> E[append reward / n_scored++]
    D -- no --> F[append 0.0 / n_dead++ / record exception type]
    E & F --> G{n_scored > 0?}
    G -- yes --> H[mean = sum divided by len measured / return SampleResult]
    G -- no --> I[fall through to single-trial path]
    B -- no --> J[_load_trials: pick best trial via _trial_rank]
    J --> K[rank = clean + has_rewards + reward + finished_at + mtime]
    I & K --> L{rewards present?}
    L -- no --> M[error: No verifier rewards with dead_exception_types]
    L -- yes --> N{_extract_reward != None?}
    N -- no --> O[error: no usable metric / reward_key strict]
    N -- yes --> P[return SampleResult score=reward]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[attempt list] --> B{aggregate_attempts == mean?}
    B -- yes --> C[loop over all attempts]
    C --> D{rewards present AND _extract_reward != None?}
    D -- yes --> E[append reward / n_scored++]
    D -- no --> F[append 0.0 / n_dead++ / record exception type]
    E & F --> G{n_scored > 0?}
    G -- yes --> H[mean = sum divided by len measured / return SampleResult]
    G -- no --> I[fall through to single-trial path]
    B -- no --> J[_load_trials: pick best trial via _trial_rank]
    J --> K[rank = clean + has_rewards + reward + finished_at + mtime]
    I & K --> L{rewards present?}
    L -- no --> M[error: No verifier rewards with dead_exception_types]
    L -- yes --> N{_extract_reward != None?}
    N -- no --> O[error: no usable metric / reward_key strict]
    N -- yes --> P[return SampleResult score=reward]

Loading

_{Reviews (3): Last reviewed commit: "Merge pull request #33 from scaleapi/har..." | Re-trigger Greptile}