[2.0] Add new Frontier-CS 2.0 problem vllm_llm_serving_optimization#145
Merged
Conversation
joyemang33
changed the title
wenhaochai
added a commit
that referenced
this pull request
Jun 13, 2026
The CI validator (scripts/validate_problems.py -> get_language_config) only
supports {python, cpp, rust}; `language: patch` raised
`ValueError: Unsupported language: patch` and crashed the whole validate
step with a traceback before any per-problem logic ran.
Mirror vllm_llm_serving_optimization (#145), the canonical Modal-GPU 2.0
task: declare `language: python`, keep the real solution in reference.patch,
and make reference.py a docstring-only placeholder. Also align the tag to
`systems` (matches #145 and the sibling systems-optimization tasks
duckdb_e2e_query_optimization / vector_db_ann).
Submission contract is unchanged: agents still submit /app/solution.patch;
the static patch policy + Modal-GPU scoring in {speedup,stability}_eval are
untouched. The judge stays CPU-only (no runtime.docker.gpu), exactly like
#145. Verified locally: language resolves to python, reference.py is found,
both evaluators return 1.0 on the no-GPU smoke path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
momoway
added a commit
that referenced
this pull request
Jun 28, 2026
…ld Models) (#150) * nanowm_rollout_speedup: core — scoring/settings/runner (apply patch -> CSGO rollout -> speedup + LPIPS guardrail) + bf16 reference.patch * nanowm_rollout_speedup: full #145-style scaffold — config, readme, DESIGN, evaluator (patch policy validated: accepts ref, rejects metric-edit+env-leak), orchestrate+modal_app (Modal GPU, judge CPU), harbor/app, docker agent+judge, infra patch, evaluate.sh. Patch-policy + smoke validated on CPU; GPU runner validating on H100 * nanowm_rollout_speedup: 2.0/README entry * nanowm_rollout_speedup: record validated reference result (1.17x bf16, score 22.4, qmult 1.0) — end-to-end H100 local backend * nanowm_rollout_stability: drift task (dual of speedup) — minimize 80-frame tail-drift at iso-wall-clock. Reference (stab=0.20) reliably beats baseline (t~2.5/22 clips). Reuses framework; patch policy + smoke validated * nanowm_rollout_stability: validated end-to-end (stab=0.20 ref 5.5% tail-drift reduction @iso-wall-clock, score 5.54 > baseline) * nanowm_rollout_speedup: infra patch = frame-chunked decode (24) for long rollouts * fix(2.0/nanowm): use language: python so validate-benchmark20 resolves The CI validator (scripts/validate_problems.py -> get_language_config) only supports {python, cpp, rust}; `language: patch` raised `ValueError: Unsupported language: patch` and crashed the whole validate step with a traceback before any per-problem logic ran. Mirror vllm_llm_serving_optimization (#145), the canonical Modal-GPU 2.0 task: declare `language: python`, keep the real solution in reference.patch, and make reference.py a docstring-only placeholder. Also align the tag to `systems` (matches #145 and the sibling systems-optimization tasks duckdb_e2e_query_optimization / vector_db_ann). Submission contract is unchanged: agents still submit /app/solution.patch; the static patch policy + Modal-GPU scoring in {speedup,stability}_eval are untouched. The judge stays CPU-only (no runtime.docker.gpu), exactly like #145. Verified locally: language resolves to python, reference.py is found, both evaluators return 1.0 on the no-GPU smoke path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(2.0/nanowm): deterministic seeding + region timer (audit P1/P3/P4) An adversarial audit found the rollout metric was unseeded and the baseline cached-not-paired: ~2-3% run-to-run noise (≈ the effect sizes) let a no-op patch score nonzero (gameable), and the bf16 reference showed an impossible -4.97% LPIPS "improvement" that was pure noise. Fix (judge infrastructure, outside the agent's editable sampling scope; the shared 2.0 adapter/template is untouched): - Deterministic clip-keyed seeding in the rollout harness so the baseline (unpatched) and patched arms draw identical per-clip initial noise (common random numbers); batch boundaries are clip-aligned so QUICK is a noise-identical prefix of FINAL. Regenerated into infra_patches/0001-rollout-judge-infra.patch (was chunked-decode only). - Real per-region sampling timer (writes NWM_TIME_FILE) so the speedup metric isolates the patchable region (was dead code -> full-process wall-clock). - *_eval: settings add SEED; runner passes --seed; orchestrate baseline cache keyed by (== clips, seed) so a cached baseline is a valid CRN partner. Re-validated on Della H100 (3 seeds): a no-op patch now scores 0.15 / 0.000 (ungameable), residual wall noise 0.24%, bf16 LPIPS delta +1.7% (expected), stability drift reduction 6.8% +/- 1.2% pooled paired t=5.15 p<1e-4. Canonical numbers reconciled across DESIGN/PR_SUMMARY (speedup 1.17x/22.3, stability 6.8%/6.8); LPIPS tables labeled. Standing audit conclusions C1-C6 unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(2.0/nanowm): include the adversarial-audit report in each task Adds AUDIT_REPORT.md (the multi-agent adversarial audit + the "fixes applied + validated" table) to both task dirs so reviewers can see the provenance: the refuted strong attacks (fair fp32 baseline, no leakage, real frontier), the one HIGH finding (unseeded/cached metric) and its deterministic-seeding fix, and the post-fix validation numbers. Absolute working-repo paths rewritten to relative. PR body reference updated to point here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(2.0/nanowm): reconcile residual pre-audit numbers + harden build helper Post-audit cleanup a maintainer review would flag. No change to validated logic (rollout / scoring / infra-patch / evaluator untouched; canonical results stand: speedup 1.17x/22.3, stability 6.8%/6.8). Doc reconciliation (audit P2 — retire leftover pre-audit single-seed numbers to the canonical CRN values: t=5.15, p<1e-4, 74% win, 6.8% +/- 1.2%): - stability config.yaml + stability_eval/settings.py inline comments: t~2.5/73% win - stability readme "Reference & difficulty": 73% win / t~2.5 -> 74% / t=5.15 / p<1e-4 build_images.sh (both tasks; maintainer-only image-build helper): - drop hardcoded personal scratch path as the NWM_ASSETS default; require it via :? (clear error if unset instead of silently pointing at a nonexistent path) - correct the asset-layout doc to match what the script actually reads (ckpts/nanowm-l2-csgo-100k, csgo/, csgo_subset/ -- not the old data/ paths) - fail-fast if the required checkpoint / held-out data didn't land (was silently swallowed by `|| true`, producing an empty judge image that fails at runtime) - fix stability script header (had copy-pasted "speedup") Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(2.0/nanowm): close two tail-targeting reward-hacks + the agent-side judge-config leaks Hardens the paired nanowm_rollout_{speedup,stability} 2.0 tasks against the reward-hacks surfaced by the codex Harbor trials, validated end-to-end on H100. stability #7 (tail-targeting via a hardcoded module counter): - The scored (role=final) rollout horizon is now drawn at random per run from [ROLLOUT_LENGTH_MIN, ROLLOUT_LENGTH_MAX]=[64,72] (< the agent-measurable nominal 80), threaded once to both CRN arms and every fan-out chunk; the scored tail = horizon - TAIL_FRAMES. CRN pairing + chunk bit-identity preserved; agent-role QUICK + cache stay at nominal. A submission that hardcodes the horizon (codex's %76 / frame-60 ramp) now misfires off the tail. Proof: stability_eval/ test_antihack_horizon.py (no GPU) — expected targeting collapses +0.064 -> ~0. speedup temp_embed gray area + model-mutation: - frozen-model guard, actively injected into the copied rollout.py at apply-time ({speedup,stability}_eval.runner.inject_frozen_guard): a restored transient reshape (causal-prefix) passes, a persistently mutated model hard-errors -> 0. Fail-closed if the rollout.py anchors move. Unit-tested without torch. - speedup faithfulness fail-closed: role==final + faithfulness_lpips is None -> 0, not a silent faithfulness_mult=1.0 free pass (the primary model-mutation defense). agent-side judge-config LEAKS (how codex read 80/60 in the first place): - adapter agent template Dockerfile no longer copies config.yaml/task_config.json (full evaluation block: band, seed, drift_tail_start, val paths) into /app. - {speedup,stability}/docker/agent/Dockerfile no longer bakes the eval package (task_pkg/*_eval with settings.py) into /app/task. The judge keeps its own /judge/task_config.json + /opt/nanowm/task. De-leaked the agent readme/config environment string (no exact horizon/tail). Also: graduated wallclock-multiplier comment corrected; #6 (task collapses to the history_stabilization scalar) documented as a depth limitation (H100-sweep-gated), not a reward-hack. AUDIT_REPORT/DESIGN updated in both tasks. Validated on Modal H100 (stability, regenerated package + rebuilt images): scored at rollout_length=65 / tail>=45 (random, not 80/60), agent could not read the band, frozen guard clean, reward 0.0338 (22 clips, iso-wall-clock). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: momoway <3499622023@qq.com>
…ng/gate fixes Updates the vLLM serving-latency task with this iteration's work: - BFCL switched from AST function-calling to the multi-turn `memory` category (vendored kv backend + 5 pre-baked per-scenario snapshots); real, non-zero, non-ceilinged accuracy used as a guardrail. - Continuum job-FCFS reference.patch added (wins ~1.1-1.6x on SWE, single H100). - Single H100 (was H100:2): creates the KV contention scheduling needs; H100:2 was measured to over-provision (no queueing -> codex ~1.0x). - BFCL at jps=1.0 + down-weighted to 0.2 (SWE 0.8): measured to have no reproducible latency signal at any arrival rate (batch-numerics non-determinism swamps it), so it mainly serves as correctness gate + accuracy guardrail. - BFCL correctness gate uses a 5% abs-OR-rel tolerance to absorb that non-determinism instead of falsely failing good patches. - modal_app serves Qwen3-Coder-30B-A3B; dynamic serving_harness label; misc fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Type of Change
Testing
Checklist
CI Validation (for new problems)