fix: [no-ticket] harbor - ops integrity (exhaust-budget instruction lever, fail-closed ledger restore)

[shehabyasser-scale]

Stacked on #35 (harbor-8-honest-signals). Closes out the pre-paper hardening series (#34 access tiers, #35 honest signals, #36 ops).

Changes

1. instruct_exhaust_budget build-config lever (build/config.py, compiler.py, instruction.md.j2, mirrored into serve.json like instruct_multifidelity). The compiled instruction's "unspent budget is wasted: re-measure your champion or try one more variant" bullet was unconditional; it is now a flag, default on (current behavior byte-for-byte in spirit; the bullet splits from the "scores are noisy" fact, which stays unconditional). Off makes stopping-early the agent's own choice. This is the ablation arm for the persistence experiments: with the exhortation always baked in, there is no way to measure what it contributes to optimizer persistence vs what the optimizer does on its own.

2. Ledger restore fails closed (serve.py). _load_or_build_ledger restored the configured (full) budgets whenever the persisted ledger.json existed but could not be parsed. The persisted ledger exists precisely so a sidecar restart cannot refill spent budget; falling back to full on a parse failure refunds the agent everything already spent, so any crash that corrupts the flush mints budget. Now: a missing file is still a fresh boot (configured budgets), but an unreadable file restores every metered split with zero remaining, preserves the unreadable original as ledger.corrupt for the operator, and logs at error level with the recovery instruction (delete ledger.json deliberately to boot fresh). Admin and finalize paths are unaffected; only agent metering closes.

Tests

• Instruction lever: default renders the bullet; instruct_exhaust_budget=False omits it while keeping "Scores are noisy".
• Ledger: missing file boots configured; unparseable file restores zero-remaining and preserves ledger.corrupt byte-for-byte; structurally-malformed JSON entries (KeyError path) also fail closed.
• 48 passed, 1 skipped (pre-existing) on the affected files.

🤖 Generated with Claude Code

Greptile Summary

This PR closes out the pre-paper hardening series with two focused integrity fixes and several supporting signal improvements.

• Ledger fail-closed: _load_or_build_ledger now treats an unreadable-but-present ledger.json as fully spent (zeroes all metered budgets) rather than refunding configured budgets, closing the crash-to-budget-refund path. Missing files still boot fresh. The corrupt file is copied to ledger.corrupt as forensic evidence.
• instruct_exhaust_budget flag: Splits the "Unspent budget is wasted" instruction bullet from the unconditional "Scores are noisy" fact; defaults on (current behavior) and enables an ablation arm that removes the persistence exhortation to measure the optimizer's own stopping behaviour.
• Supporting changes: per-target executor-model override for transfer probes, infra dead-attempt classification and bounded within-eval retry (off by default, adversarial-candidate warning documented), freebie-baseline race-condition fix (claim before await, refund on BaseException), volume-reuse ordinal resumption, score_se → mean_score_se rename, and k_anonymity_floor default corrected from 1 → 5.

Confidence Score: 5/5

Safe to merge; all changes harden existing invariants rather than relaxing them.

The ledger fail-closed logic is correct and well-tested (missing, unparseable, and malformed-entry paths all covered). The infra-retry feature is disabled by default with documented adversarial-candidate risks. The freebie race-condition fix correctly uses the claim-before-await / refund-on-exception pattern. The only notable nit is that the error log in _load_or_build_ledger claims the corrupt file is preserved before the copy is attempted, so a disk-full failure would leave an incorrect log trace — but this does not affect the core zeroing logic.

No files require special attention; serve.py has the minor log-ordering nit but the fail-closed behaviour itself is correct.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/serve.py	Ledger fail-closed logic is correct; log message claims corrupt-file preservation before copy is attempted, which can mislead operators if the copy fails.
vero/src/vero/harbor/server.py	Race-condition fix for free baseline (claim before await, refund on BaseException) and volume-reuse ordinal scan are both correct.
vero/src/vero/harbor/runner.py	Dead-attempt classification, infra retry (OFF by default, sibling dirs, audit marker), and feedback for no-rewards error samples all look well-guarded.
vero/src/vero/harbor/build/config.py	New instruct_exhaust_budget flag and TargetSpec.model field; validation for infra_retry config is thorough.
vero/src/vero/harbor/verifier.py	Transfer-target model override, cause string in floored rewards, and dominant-error clustering are all correct; _dominant_sample_errors defensively catches exceptions.
vero/src/vero/evaluation/engine.py	model_copy avoids mutating shared run_constraints; per-eval override correctly isolated to the call scope.
vero/src/vero/harbor/config.py	infra_retry_rounds and infra_retry_delay_s added with appropriate guards; security implications documented clearly in comments.
vero/tests/test_harbor_serve.py	TestLedgerFailClosed covers missing file, unparseable JSON, and malformed entries; all three fail-closed paths are tested.
vero/tests/test_harbor_runner.py	TestInfraResilience is comprehensive: infra labeling, forged-suffix neutralization, key-budget alarm, retry default-off, sibling dir isolation, audit markers, multi-round backoff, partial recovery.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[sidecar startup: _load_or_build_ledger] --> B{ledger.json exists?}
    B -- No --> C[Fresh boot: configured budgets\nremaining = total]
    B -- Yes --> D[Parse JSON]
    D -- Success --> E[Restore persisted remaining_* values\nReturn BudgetLedger]
    D -- Fail\njson.JSONDecodeError / KeyError / OSError --> F[logger.error: failing CLOSED]
    F --> G[shutil.copyfile to ledger.corrupt]
    G --> H[Zero out all metered remaining budgets]
    H --> I[Return BudgetLedger - all splits exhausted]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[sidecar startup: _load_or_build_ledger] --> B{ledger.json exists?}
    B -- No --> C[Fresh boot: configured budgets\nremaining = total]
    B -- Yes --> D[Parse JSON]
    D -- Success --> E[Restore persisted remaining_* values\nReturn BudgetLedger]
    D -- Fail\njson.JSONDecodeError / KeyError / OSError --> F[logger.error: failing CLOSED]
    F --> G[shutil.copyfile to ledger.corrupt]
    G --> H[Zero out all metered remaining budgets]
    H --> I[Return BudgetLedger - all splits exhausted]

Loading

_{Reviews (3): Last reviewed commit: "Merge pull request #37 from scaleapi/har..." | Re-trigger Greptile}