From dbce3b36e23d0621e14a9c27a116e4144c82eb65 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Mon, 21 Sep 2026 22:05:50 +0800 Subject: [PATCH 01/80] docs(issue26): record correctness branch base and Step 0 status --- .../progress.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index d28f42fd..e2edd5e4 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -10,18 +10,18 @@ | Field | Value | | --- | --- | -| Correctness branch | `fix/issue26-correctness-pr` (to be created from `refactor/oversized-module-split` after the refactor branch's Step 0 push) | -| Base at creation | pending | +| Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | +| Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | Refactor PR fidelity matrix PASS (see `task_memory/task_2026-09-21_oversized_module_split/progress.md`) | -| Current step | Step 0 (records landed; branch creation pending) | -| Publication | LOCAL_ONLY | -| Next action | Create the correctness worktree from the refactor branch, push, and stop for user review (Q6=a). | +| Current step | Step 0 complete pending user review | +| Publication | PUSHED_VERIFIED (records) | +| Next action | After user review: refactor task Step 1 (fidelity harness + baseline capture). Correctness Step 1 (candidate/vLLM audit, draft PR) may start in parallel because it is read-only. | ## Step status | Step | Work package | Status | Test | Publication | User review | | --- | --- | --- | --- | --- | --- | -| 0 | Worktree, references, baseline | IN_PROGRESS | NOT_RUN | LOCAL_ONLY | NOT_REVIEWED | +| 0 | Worktree, references, baseline | PASS | PASS (baseline recorded) | PUSHED_VERIFIED | NOT_REVIEWED | | 1 | Candidate/vLLM audit | NOT_STARTED | — | — | — | | 2 | RR DP rotation | NOT_STARTED | — | — | — | | 3 | Shared monolithic forward | NOT_STARTED | — | — | — | From a1f8e454a8f54afebaba2bf6a7198072d9b5c578 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Mon, 21 Sep 2026 22:05:50 +0800 Subject: [PATCH 02/80] docs(issue26): land the candidate and vLLM 0.10.2 source audit Step 1 of the correctness PR is a source audit at pinned revisions; no code runs and no accuracy is claimed. Findings that change the plan: - The round-robin DP lane defect is still present on main, and main already implements the intended formula for the DECODE role only, so the fix is an internal-consistency repair. - The shared monolithic forward is still split by local request phase at three layers, and the candidate's decode metrics hunk calls a method main has deleted, so that hunk is blocked pending a rewrite. - The opt-in DP placement strategy reuses a step identity that is not valid for dense models or for MoE with DP>1 before the shared forward fix, so the order becomes RR, then shared forward, then DP placement. - Two routing implementations can already select each other's cost model on main through a shared training signature that carries no routing term, reachable from the public CLI in a PDD run. - The legacy fused-MoE path omits gated activation and the local top-k reduction; adding the reduction changes what the measurement contains and no existing column separates old rows, so it needs a decision. Also records a defect in the candidate itself: it deletes a method the SGLang scheduler still calls. --- .../audit_predictor_profiling.md | 204 +++++++ .../audit_scheduler.md | 287 ++++++++++ .../progress.md | 7 +- .../reference_vllm_0_10_2.md | 522 ++++++++++++++++++ .../review.md | 117 +++- .../validation.md | 22 +- 6 files changed, 1147 insertions(+), 12 deletions(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/audit_predictor_profiling.md create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/audit_scheduler.md create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/reference_vllm_0_10_2.md diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/audit_predictor_profiling.md b/task_memory/task_2026-09-21_issue26_correctness_pr/audit_predictor_profiling.md new file mode 100644 index 00000000..248e721e --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/audit_predictor_profiling.md @@ -0,0 +1,204 @@ +# Audit: candidate PREDICTOR and PROFILING changes (W5 routing identity, W6 fused-MoE arithmetic) + +Read-only source audit. No file under `/data/ycfeng/Frontier` was modified. + +## Revisions + +| Label | SHA | Subject | +| --- | --- | --- | +| BASE (main) | `1f694f7c549aa3aeeb7c5bbae04e119c09167a77` | `updapte doc` | +| CAND (`bug/ttft-check`) | `a7b3320fe9b8b083ee86b91dae3d6838f4443d91` | `Record Issue 26 H200 TTFT task docs and numerical results` | +| MERGE_BASE | `d71ad80b0800880808a0857fd30477e6d96592c6` | `chore: keep repair receipts and task memory local-only` | + +`git merge-base BASE CAND` = `d71ad80b...`, i.e. the supplied MERGE_BASE is the true merge base (verified). + +### Divergence sizing (line counts of the audited files at each revision) + +| Path | MB | BASE | CAND | MB->CAND (true candidate delta) | MB->BASE (main-only work) | +| --- | --- | --- | --- | --- | --- | +| `frontier/moe_routing_runtime.py` | 67 | 67 | 74 | +16/-9 | none | +| `frontier/execution_time_predictor/sklearn_moe_execution_time_predictor.py` | 3523 | 3539 | 3518 | +2/-7 | +217/-201 | +| `frontier/execution_time_predictor/shared_prediction_model_manager.py` | 4570 | 4614 | 4606 | +89/-53 | +169/-125 | +| `frontier/config/config.py` | 5663 | 5720 | 5648 | +25/-40 | +57/-0 | +| `frontier/profiling/moe/moe_vllm_kernel.py` | 674 | 1051 | 684 | +22/-12 | **+419/-42** | +| `tests/unit/test_moe_routing_runtime.py` | 31 | 31 | 88 | +57 | none | +| `tests/unit/test_moe_routing_runtime_model_sharing.py` | 0 | 0 | 181 | +181 | none | +| `tests/unit/test_moe_fused_expert_numerical_parity.py` | 0 | 0 | 52 | +52 | none | + +**Critical framing fact.** The raw `BASE..CAND` diff for `moe_vllm_kernel.py` reads `-431` lines, and for `config.py` reads `-97` lines. Those deletions are *not* candidate reverts: the candidate branched at MERGE_BASE and never received main's later work. Every disposition below is therefore judged against the `MERGE_BASE->CAND` delta (the real candidate intent), not against `BASE->CAND`. + +Main-only work that a naive file-level take would revert, verified present on BASE and absent on CAND: + +- `moe_vllm_kernel.py`: functional `fused_experts` API adapter (`BASE:116-140`, `BASE:434-454`, `BASE:798-869`), MXFP4/AITER online path (`BASE:43-58`, `BASE:61-114`, `BASE:457-601`, `BASE:808-833`), device-event timing family (`BASE:603-626`), profile-method platform validation (`BASE:722-728`), `accelerator_platform`/`TimerStatsStore` imports (`BASE:28-33`). +- `config.py`: GDN runtime guards (`BASE:2085-2096`, `BASE:5322`, `BASE:5361-5398`), `gdn_input_file` (`BASE:2154-2157`), cluster-scheduler configs still inline on BASE (`BASE:2107-2140`) — the candidate instead imports them from `frontier/config/cluster_scheduler_config.py`, which does **not** exist on BASE. +- `shared_prediction_model_manager.py`: the `device_event` measurement family as a third registry (`BASE:4159-4163`, `BASE:4171-4175`, `BASE:4181-4185`, `BASE:4193-4197`, `BASE:4402`, `BASE:544-551`). The candidate's rewrite of those helpers only lists `eager` and `kernel_only`. + +--- + +## (a) Disposition table + +| # | Candidate path | Change summary (MB->CAND) | Old defect it targets | Current main behavior (file:line on BASE) | Disposition | Rationale | Proposed owner on main | Planned test | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | `frontier/moe_routing_runtime.py` | Adds `DEFAULT_MOE_GATING_ROUTING_RUNTIME_PATHS` map (`CAND:12-18`); `resolve_moe_gating_routing_runtime_path` gains optional `requested_runtime_path=""` second arg, validates the distribution first, then returns the explicit runtime if given else the distribution default (`CAND:30-43`). | Routing implementation identity is *derived* from the load distribution, so the two cannot be chosen independently. | `BASE:24-36`: pure `distribution -> runtime` map, no override. Existing defaults `balanced/skewed/zipf -> standard_fused_topk`, `random -> uniform_topk`. | **PORT** | Small, additive, default-preserving (`resolve(d)` unchanged for all four distributions). No main-only code in this file (MB==BASE). Same two-value vocabulary already validated by `validate_moe_gating_routing_runtime_path` (`BASE:14-21`). | `frontier/moe_routing_runtime.py` | `tests/unit/test_moe_routing_runtime.py` — keep the 3 existing cases verbatim, add the candidate's `test_explicit_runtime_preserves_distribution_validation` (`CAND test:34-41`). | +| 2 | `frontier/config/config.py` (routing-runtime fields only) | Adds `ReplicaConfig.moe_gating_routing_runtime_path: str = ""` (`CAND:1955-1962`); validates non-empty values in `__post_init__` (`CAND:2039-2041`); threads it through the per-role replica builder (`CAND:4269`) and `_create_replica_config_copy` (`CAND:5018`); imports the validator (`CAND:48`). | No config surface exists to select a routing implementation; only `moe_routing_distribution_type` exists and it also decides expert-load skew. | `BASE:1937-1946` (`moe_routing_distribution_type` field), `BASE:2005-2018` (validation), `BASE:4257-4273`/`BASE:4302` (per-role resolution), `BASE:5050` (copy). No routing-implementation field anywhere. | **PORT (config hunks only) / DROP (rest of file)** | The four routing hunks are additive and empty-default. Everything else in the `BASE..CAND` diff for this file is main-only work the candidate lacks and must not be taken. Because `get_field_value` (`BASE:4242-4247`) synthesizes `{cluster_prefix}_replica_config_{field}` lookups, the per-role CLI flags come for free with no registry change. **Naming**: main's standalone trainer already calls this concept `routing_runtime_path` (`frontier/training/cli.py:153`, `frontier/training/moe_trainer.py:176`); the candidate's `moe_gating_routing_runtime_path` is a third spelling. Reconcile before creating the public flag. | `frontier/config/config.py` | `tests/unit/test_moe_routing_runtime.py::test_runtime_config_copy_and_predictor_keep_balanced_loads` (`CAND test:44-69`) — **needs adaptation**, see note N1. Plus the subprocess CLI check (`CAND test:72-88`). | +| 3 | `sklearn_moe_execution_time_predictor.py` | `_get_requested_moe_gating_routing_runtime_path` now passes `replica_config.moe_gating_routing_runtime_path` as the override (`CAND:270-274`); deletes the dead `self._moe_gating_routing_runtime_path = resolve(...)` assignment (`CAND` removes what is `BASE:731-735`). | (i) predictor cannot honour an explicit routing runtime; (ii) `_moe_gating_routing_runtime_path` is written and never read. | `BASE:279-282` (helper, distribution-only); `BASE:731-735` (assignment). **Dead-write confirmed**: a repo-wide `git grep -n "_moe_gating_routing_runtime_path" 1f694f7c` returns the write at `BASE:731` and no read anywhere in `frontier/`, `tests/`, `examples/`, `docs/`. | **PORT** | 2-line change plus a 5-line dead-code removal. Ordering is safe on main: `self._replica_config` is set by the parent `__init__` (`BASE:737-746`) and the helper is only called during training at `BASE:1236` and `BASE:1371`, both after init. The `getattr(getattr(self,"_replica_config",None), ..., "")` form is defensive but tolerable; main's style would allow a direct attribute read. | `frontier/execution_time_predictor/sklearn_moe_execution_time_predictor.py` | Extend `tests/unit/test_moe_share_expert_operator_families.py` / `tests/unit/test_profiling_governance_minimal_red.py`, which already stub this helper (`BASE tests:341`, `BASE tests:1209`), with an explicit-override case. | +| 4 | `shared_prediction_model_manager.py` — dataset admission | `_validate_moe_dataset_contract` passes the explicit runtime into the resolver (`CAND:1236-1239`). | Dataset row admission is keyed to the distribution-derived runtime, so an explicit uniform profile cannot be admitted. | `BASE:1294-1296` resolves from the distribution only; `BASE:1317-1326` filters `moe_gating_routing_topk` rows and appends `routing_runtime_path=` to the requirement text. | **PORT** | One-line argument addition; filtering/error machinery unchanged. | same file | `tests/unit/test_moe_routing_runtime_model_sharing.py::test_dataset_validation_requires_explicit_runtime_rows` (`CAND test:164-181`). | +| 5 | `shared_prediction_model_manager.py` — training signature | `ffn_signature += f"_routing{path}"` when `is_moe_model` (`CAND:1327-1333`); the local resolve inside the MoE branch is deleted and the hoisted value reused (`CAND:1436`). | Two clusters whose only difference is routing runtime collide on one `ffn_signature`, so the second cluster's MoE FFN training is skipped and it silently inherits the first cluster's routing-cost model. | `BASE:1377-1382` builds `ffn_signature` from device/model/tp/is_moe/arch_profile/typed-contract-hash/measurement-family — **no routing term**. `BASE:1384-1386` returns early when the signature was already trained. `trained_model_signatures` is a single set shared across all clusters (`BASE:702`, loop `BASE:707`, passed at `BASE:778`). | **PORT with a scoping caveat** | This is the load-bearing fix; see (b) for the reachability proof. **Caveat**: appending to `ffn_signature` separates *every* MoE FFN model (`moe_grouped_gemm`, `moe_shuffling`, `moe_gating_linear`, MoE layernorm) by routing runtime, not just `moe_gating_routing_topk`. That is correctness-safe but retrains and re-registers models whose measured work does not depend on the routing implementation. A tighter fix keys only the routing-topk model. Decide before porting; the wide form is what the candidate verified. | same file | `tests/unit/test_moe_routing_runtime_model_sharing.py::test_shared_training_separates_runtime_and_reuses_same_runtime` (`CAND test:56-94`) — needs the three-family adaptation, note N2. | +| 6 | `shared_prediction_model_manager.py` — per-model training identity | In `_train_single_model`, derive `routing_runtime_path` from the training frame for routing-topk models and require exactly one non-NaN value (`CAND:2768-2773`); attach it to both the cache-hit model (`CAND:2811`) and the freshly fitted model (`CAND:2915`) via the new `_model_routing_runtime_identity` (`CAND:4113-4127`). | A trained/cached estimator carries no record of which routing implementation it measured, so nothing downstream can tell two apart. | `BASE:2765` `_train_single_model`; `BASE:2820-2832` computes `profiling_precision`, `measurement_type` and `model_hash`; `BASE:4130-4154` `_model_contract_identity` attaches only `_frontier_layer_cache_identity`. No routing attribute exists. | **PORT** | Mirrors the established `_model_contract_identity` pattern exactly (attach-or-conflict). Deriving the identity from the frame rather than from config is the right choice: it makes disk-cache hits carry the identity too (`CAND:2811`). | same file | `tests/unit/test_moe_routing_runtime_model_sharing.py::test_runtime_survives_fresh_training_and_cache_hit` (`CAND test:99-123`) and `::test_routing_training_rejects_mixed_runtime_rows` (`CAND test:155-161`). | +| 7 | `shared_prediction_model_manager.py` — registry key widening | Registry key goes `(model_name, identity)` -> `(model_name, identity, runtime_path)` (`CAND:4145`, `CAND:4157`, `CAND:4237`, `CAND:4243-4248`); `_get_family_model` gains `requested_runtime_path` and is rewritten around a single filtered `candidates` dict (`CAND:4234-4275`); `_models_view_for_family` derives the cluster's runtime and passes it (`CAND:4348-4372`); `get_model` gains an optional `routing_runtime_path` kwarg (`CAND:4381-4399`). | Two trained routing models with the same layer identity overwrite each other in the registry; the cluster-scoped view can only ask by layer identity. | `BASE:4238` `model_key = (model_name, identity)`; `BASE:4244-4302` `_get_family_model`; `BASE:4358-4388` `_models_view_for_family`; `BASE:4390-4410` `get_model`; `BASE:4510-4550` `get_models_for_cluster`. | **ADAPT** | Correct in direction, but the candidate's helper rewrite enumerates only `{"eager","kernel_only"}` (`CAND:4146-4149`, `CAND:4158-4161`); main has three families including `device_event` (`BASE:4160-4162`, `BASE:4172-4174`, `BASE:544-551`). Taking the hunk verbatim breaks every ROCm/device-event run. Also the candidate collapses main's explicit `requested_identity`/legacy branch structure (`BASE:4276-4302`) into a set-filter; behaviour is close but the refactor is larger than the fix requires. Port the key widening and the `requested_runtime_path` plumbing; keep main's branch structure and all three families. Error text keeps the `multiple layer contracts` prefix, so the two existing assertions at `BASE tests/unit/test_profiling_governance_minimal_red.py:1809,1843` still match. | same file | `tests/unit/test_moe_routing_runtime_model_sharing.py::test_runtime_variants_preserve_precision_and_measurement_isolation` (`CAND test:128-152`), extended to iterate `CUDA_EVENT`, `DEVICE_EVENT`, `KERNEL_ONLY`. | +| 8 | `shared_prediction_model_manager.py` — persistent cache key | **No change.** `_get_model_hash` is untouched by the candidate. | (claimed) routing-blind disk cache | `BASE:4008-4056` `_get_model_hash` — hash inputs are hash-relevant config, model name, `df.to_json()` md5 (`BASE:4033`), precision, measurement type, optional layer-contract identity. **No routing term.** | **ALREADY_PRESENT (implicitly) — no port needed, but record the reasoning** | The disk cache does *not* collide across routing runtimes, but only incidentally: `_load_moe_df` filters rows and never drops columns (`BASE:3628`, `BASE:3712-3720`), so the `routing_runtime_path` column survives into the frame, and `filter_moe_gating_routing_topk_rows` (`BASE frontier/moe_routing_runtime.py:39-67`) is applied at `BASE:1518-1523` before training. Different runtimes therefore yield different `df_hash_str` -> different `model_hash` -> different cache file `{model_name}_{model_hash}.pkl` (`BASE:4448`, `BASE:4457`). The candidate relies on exactly this and re-attaches identity on cache hit (`CAND:2811`). An explicit routing term in `_get_model_hash` would be belt-and-braces only. | — | Covered by `test_runtime_survives_fresh_training_and_cache_hit` asserting `cached._frontier_model_hash == trained._frontier_model_hash` (`CAND test:118`). | +| 9 | `frontier/profiling/moe/moe_vllm_kernel.py` | Legacy `_run_fused_moe_iteration`: replaces the bare first-half slice with `torch.ops._C.silu_and_mul(activated, ic1_flat)` (`CAND:310-313`) and appends `ops.moe_sum(intermediate_cache2, output)` (`CAND:340`); signature gains `activated`/`output` and drops `expert_hidden_dim_per_partition` (`CAND:276-284`); caller allocates a shared W1/W2 workspace plus separate activation and output buffers outside the timed step (`CAND:533-550`); adds `from vllm import _custom_ops as ops` (`CAND:31`). | The legacy path performs no gated activation at all (it discards the up-projection half) and no local top-k reduction, so `moe_grouped_gemm` measures work that is not the model's expert computation. | `BASE:365-431` `_run_fused_moe_iteration`; defect at `BASE:404-405`; FP8 activation quant already present at `BASE:407-413`; routing weights already applied in GEMM2 at `BASE:423`; no `moe_sum` anywhere in the file. | **ADAPT (activation) / BLOCKED (moe_sum)** | See (c). The activation hunk lands cleanly on main's legacy branch (main's functional/MXFP4 branch returns at `BASE:869`, before the legacy allocations at `BASE:904`), so there is no code conflict. Three adaptations required: (i) take only the hunks, never the file; (ii) move `from vllm import _custom_ops as ops` into the version-detection try-block so it cannot break main's newer-vLLM import path (`BASE:116-140`); (iii) main already owns a `SiluAndMul` wrapper over the same kernel at `frontier/profiling/common/layers/activation.py:8-34` — the candidate bypasses it, contradicting the donor's own instruction to extend rather than duplicate. The `moe_sum` line changes the measured scope and is **BLOCKED** on a user decision (see (c)). | `frontier/profiling/moe/moe_vllm_kernel.py` (+ possibly `frontier/profiling/common/layers/activation.py` for an out-parameter variant) | New GPU test derived from `tests/unit/test_moe_fused_expert_numerical_parity.py` (`CAND test:1-52`) plus a CPU-runnable call-order test (mock `_invoke_kernel`, assert W1 -> activation -> optional FP8 quant -> W2 and an `ehdpp`-wide W2 input). | +| 10 | `tests/unit/test_moe_routing_runtime.py` | +57 lines: explicit-runtime resolver cases, a config copy + predictor case, and a subprocess CLI case. | — | `BASE` file is 31 lines, 3 tests. | **ADAPT** | `test_explicit_runtime_preserves_distribution_validation` (`CAND:34-41`) ports verbatim. `test_runtime_config_copy_and_predictor_keep_balanced_loads` (`CAND:44-69`) builds a `SimpleNamespace` predictor without `_cluster_type` and calls `_init_global_routing_allocations` unbound; that works on CAND because CAND/MB use `getattr(self,"_cluster_type",None)` (`CAND:765-768`) but main tightened the same lines to a direct `self._cluster_type` read (`BASE:773`). The test must set `_cluster_type`; main's tightening must not be reverted. `test_runtime_override_reaches_effective_cli_config` (`CAND:72-88`) is a real CLI check and worth keeping — the donor report records that it was moved to a subprocess precisely because in-process `IS_MOE` global initialization leaks between tests. | `tests/unit/test_moe_routing_runtime.py` | itself | +| 11 | `tests/unit/test_moe_routing_runtime_model_sharing.py` (new, 181 lines) | Five tests over real training/dedup/registry/disk-cache/projection behaviour with synthetic frames. | — | no equivalent on main | **ADAPT** | Highest-value test in the candidate: it exercises the exact collision this work package fixes. Two adaptations: (N2) the precision/measurement isolation test iterates only `CUDA_EVENT` and `KERNEL_ONLY` (`CAND:137`) and must add `DEVICE_EVENT`; and `manager_at` (`CAND:18-24`) constructs the manager via `object.__new__` and sets only 5 attributes, so it will need the `*_by_contract`/legacy registry dicts main initializes at `BASE:444-449` (the candidate's version happens to work only because it never touches an uninitialized family). Also note it monkeypatches `_validate_moe_dataset_contract` and `_load_moe_df` (`CAND:65-67`), so shape admission is *not* covered by this file. | `tests/unit/test_moe_routing_runtime_model_sharing.py` | itself | +| 12 | `tests/unit/test_moe_fused_expert_numerical_parity.py` (new, 52 lines) | GPU test comparing `_run_fused_moe_iteration` output against vLLM `fused_experts` at `rtol=0, atol=0`. | — | no equivalent on main | **ADAPT** | Correct shape of test, but: (i) it asserts **bitwise** equality against `fused_experts`, which is brittle if vLLM chunks large `M` differently from the single-shot workspace the test builds — recommend a declared tolerance, exactly as the donor's own plan demands ("record raw per-iteration values and a declared tolerance before deciding numerical parity"); (ii) it depends on GPU + vLLM 0.10.x `fused_moe.fused_experts`, which is precisely the import path main treats as optional (`BASE:120-138`) — it must skip when `VLLM_API_VERSION != "0.10.x"`, not just when vLLM is absent; (iii) `data/config/models/qwen3-a3b-30b-moe.json` exists on BASE (verified), `routed_mlp_hidden_dim` exists (`BASE frontier/profiling/common/model_config.py:91-101`) and `uniform_topk` exists (`BASE frontier/profiling/moe/moe_impl.py:75`), so the fixtures resolve. Note that this test's assertion only holds **with** `moe_sum` in the profiled path, because `fused_experts` returns the reduced tensor — i.e. the test presupposes the blocked scope decision. | `tests/unit/test_moe_fused_expert_numerical_parity.py` | itself | + +### Notes referenced above + +- **N1** — `ReplicaConfig(model_name="Phi-tiny-MoE-instruct", moe_gating_routing_runtime_path="uniform_topk")` at `CAND test:46-49` will additionally run main's GDN runtime guard (`BASE frontier/config/config.py:2085-2096`), which does not exist on CAND. Verify the fixture still constructs on main. +- **N2** — three measurement families on main (`BASE:544-551`): `CUDA_EVENT->eager`, `DEVICE_EVENT->device_event`, `KERNEL_ONLY->kernel_only`. + +--- + +## (b) W5 chain: does routing identity reach each stage on main today? + +Legend for "reaches on main": **NO** = the stage sees only the *distribution*, from which the runtime is derived, so an explicit routing implementation cannot be expressed or distinguished. + +| # | Stage | Routing identity reaches it on main? | Evidence (BASE file:line) | What the candidate does | What would be required | +| --- | --- | --- | --- | --- | --- | +| 1 | Config creation (CLI -> `ReplicaConfig`) | **NO** — only `moe_routing_distribution_type` exists | field `config.py:1937-1946`; validation `config.py:2005-2018`; per-role resolution `config.py:4257-4273` and `:4302`; the `get_field_value` helper that synthesizes `{prefix}_replica_config_{field}` lookups `config.py:4242-4247` | Adds `moe_gating_routing_runtime_path: str = ""` (`CAND config.py:1955-1962`) validated in `__post_init__` (`CAND:2039-2041`) and wired at `CAND:4269` | One new `ReplicaConfig` field with an empty default plus one `get_field_value` line. No registry change: the flat-dataclass CLI and the per-role flags derive automatically. Reconcile the name with `--routing_runtime_path` already used by `frontier/training/cli.py:153`. | +| 2 | Config copying (`_create_replica_config_copy`) | **NO** (nothing to copy) | `config.py:5050` copies `moe_routing_distribution_type` | Adds the parallel copy line (`CAND config.py:5018`) | One line. Omitting it is a silent-drop bug class: `ClusterConfig` builds role replicas by copy, so an uncopied field reverts to default for PDD/PD-AF roles. | +| 3 | Resolved routing runtime | **NO** — resolution is a total function of the distribution | `frontier/moe_routing_runtime.py:24-36` | Adds an optional explicit override, distribution still validated first (`CAND:30-43`), defaults table extracted (`CAND:12-18`) | Optional second parameter, empty string = current behaviour. Every existing call site keeps its result unchanged. | +| 4 | Dataset row selection and validation | **partially — the *derived* runtime does reach it** | shared manager `spmm.py:1294-1296` -> `spmm.py:1317-1326`; per-predictor `sklearn_moe_...py:1235-1237` -> `:1257-1265`; row filter `frontier/moe_routing_runtime.py:39-67` (fails fast when the column is missing or no row matches) | Passes the explicit runtime into both resolvers (`CAND spmm.py:1236-1239`, `CAND sklearn_moe_...py:270-274`) | Two argument additions. This stage is already correct in structure — it is the only stage where routing identity is honoured today. | +| 5 | Training identity / signature | **NO** | `spmm.py:1377-1382` — `ffn_signature` = device, model_name, `tp_size`, `moe{bool}`, `arch_profile{id}`, `layer_contracts{hash}`, `family{name}`. Dedup gate `spmm.py:1384-1386`. Shared set across clusters: created `spmm.py:702`, cluster loop `spmm.py:707`, passed `spmm.py:778` | Appends `_routing{path}` for MoE models (`CAND spmm.py:1327-1333`) | A routing term in the signature (or a narrower per-model signature for `moe_gating_routing_topk` only). Without it the second cluster never trains and inherits the first cluster's model. | +| 6 | Per-call dataset cache (inside one cluster's FFN training) | **YES** | `spmm.py:1440-1442` resolves; `spmm.py:1486-1488` sets `runtime_path_key`; `spmm.py:1499-1505` puts it in the 5-tuple `moe_df_cache` key; log at `spmm.py:1535`. Scope: one invocation of `_train_ffn_models_for_cluster`; the dict is created at `spmm.py:1444-1453` and discarded when the call returns | Reuses the hoisted value (`CAND spmm.py:1436`) instead of re-resolving | Nothing — this stage already isolates correctly, but only *within* one cluster call, which is why it never prevents the cross-cluster collision. | +| 7 | Trained-model registry and precision / measurement-family selection | **NO** | `_store_model_precision` `spmm.py:4219-4242`, key built at `spmm.py:4238` as `(model_name, identity)`; `_get_family_model` `spmm.py:4244-4302`, typed candidates keyed by identity alone at `spmm.py:4260-4264`; families `spmm.py:4156-4200` (`eager`, `device_event`, `kernel_only`); precision bucket `spmm.py:4202-4217` | Widens the key to `(model_name, identity, runtime_path)` and adds `requested_runtime_path` to the lookup (`CAND spmm.py:4237`, `:4243-4248`, `:4234-4275`) | Key widening + a lookup parameter. **Must preserve all three families**; the candidate's helper rewrite drops `device_event`. | +| 8 | Persistent (disk) model cache key | **NO as an explicit term; YES incidentally via the frame hash** | `_get_model_hash` `spmm.py:4008-4056` — inputs are hash-relevant config, model name, `df.to_json()` md5 (`spmm.py:4033`), precision, measurement type, optional contract identity. `hash_args` assembled at `spmm.py:2822-2828`. Cache file `{model_name}_{model_hash}.pkl` (`spmm.py:4448`, `spmm.py:4457`). The frame reaching the hash still carries the `routing_runtime_path` column because `_load_moe_df` filters rows only (`spmm.py:3628`, `spmm.py:3712-3720`) and the routing filter runs before training (`spmm.py:1518-1523`) | No change to `_get_model_hash`. Instead attaches `_frontier_routing_runtime_path` to the cache-loaded estimator from the frame (`CAND spmm.py:2768-2773`, `:2811`) | Nothing strictly required for cache separation. Attaching the identity on cache load *is* required, otherwise a cache hit returns a model with no routing identity and the widened registry key degrades to `None`. | +| 9 | Runtime lookup / query cache | **NO** | Population: `sklearn_execution_time_predictor.py:571-580` calls `model_manager.get_models_for_cluster(cluster_type)` (`spmm.py:4510-4550`), which calls `_models_view_for_family` (`spmm.py:4358-4388`); that asks `_get_family_model(..., requested_identity=identity)` at `spmm.py:4376-4380` with **no routing term**. Active dict swap `sklearn_execution_time_predictor.py:919-931`. Prediction-time access `sklearn_moe_...py:1756-1779` -> `self._models[model_name]` (`sklearn_execution_time_predictor.py:3177`). Query cache `_get_prediction_cache_hash` `sklearn_execution_time_predictor.py:3143-3164` keys on `_frontier_model_hash` (set at `spmm.py:2949`), so it inherits whatever separation the training hash had | `_models_view_for_family` resolves the cluster's routing runtime from `self._cluster_configs[cluster_type].replica_config` and passes it (`CAND spmm.py:4348-4372`); `get_model` gains the kwarg (`CAND spmm.py:4381-4399`) | Derive the runtime at the cluster-view boundary and pass it down. Note `get_model` has **no production caller** (`git grep -n "\.get_model(" 1f694f7c -- 'frontier/**'` returns nothing) — it is an external/test API, so the cluster-view path is the one that matters. | + +### Explicit answer: can two different routing implementations with the same layer shape select each other's routing-cost model on main today? + +**Yes — through two independent mechanisms, and the first one is reachable from the public CLI.** + +**Mechanism A — training dedup (dominant).** + +1. Routing distribution is settable *per role*: `config.py:4242-4247` resolves `{cluster_prefix}_replica_config_moe_routing_distribution_type` before falling back to the base config, and `config.py:4257-4259` / `:4271-4273` feed it into the role's `ReplicaConfig` (`:4302`). So a PDD run can set `PREFILL=balanced` and `DECODE=random`. +2. Those resolve to different runtimes: `frontier/moe_routing_runtime.py:29-32` maps `balanced -> standard_fused_topk` and `random -> uniform_topk`. +3. `trained_model_signatures` is one set shared across every cluster: created at `spmm.py:702`, the cluster loop at `spmm.py:707` reuses it, and it is handed to `_train_ffn_models_for_cluster` at `spmm.py:778`. +4. `ffn_signature` (`spmm.py:1377-1382`) contains device, model name, TP size, `is_moe`, architecture-profile id, typed-contract hash and measurement family — and **no routing term**. For two roles of the same model on the same device with the same MoE TP, the two signatures are byte-identical. +5. `spmm.py:1384-1386` therefore returns early for the second cluster: `if ffn_signature in trained_model_signatures: ... return models`. This early return is **before** `_validate_moe_dataset_contract` at `spmm.py:1433-1439`, so there is no fail-fast either — the second cluster's routing requirement is never checked against the dataset. +6. The second cluster then reads the registry through `_models_view_for_family` (`spmm.py:4358-4388`), which asks only for the layer identity (`spmm.py:4376-4380`) and receives the first cluster's model. Both clusters predict `moe_gating_routing_topk` from the `standard_fused_topk` estimator. + +**Mechanism B — registry overwrite.** Even if the two clusters' `ffn_signature` values differed (for example different devices, so both actually train), `_store_model_precision` keys the registry as `(model_name, identity)` at `spmm.py:4238`. `identity` is `_serialize_selected_layer_cache_identity(layer_contract)` — a function of the *layer shape/domain*, not of routing. Two routing models of the same layer shape produce the same key, so the second silently overwrites the first in `self._trained_models_*_by_contract` and in the precision registry (`spmm.py:4239-4242`). Subsequent lookups for either cluster return the survivor. + +**What is *not* broken:** the disk cache. Because the frame is routing-filtered before training (`spmm.py:1518-1523`) and `_get_model_hash` hashes the frame content (`spmm.py:4033`), the two runtimes land in different `.pkl` files. This separation is incidental to the content hash, not an identity guarantee, and it does not help either mechanism above because both operate on the in-memory registry. + +--- + +## (c) W6: arithmetic steps, main's legacy path, candidate additions, conflicts, and measured scope + +### C.1 Step-by-step table + +| # | Arithmetic step the corrected legacy path must perform | What main's legacy path does today (BASE file:line) | What the candidate adds (CAND file:line) | Conflicts with main-only code? | +| --- | --- | --- | --- | --- | +| 1 | First expert GEMM: `A @ w1^T` into a `2 * ehdpp`-wide buffer, no routing weight | Done. `_invoke_kernel(..., B=w1, C=intermediate_cache1, mul_routed_weight=False, top_k=top_k)` at `moe_vllm_kernel.py:386-402`. `w1` is `(num_experts, 2*ehdpp, hidden_dim)` (`:779-785`), `intermediate_cache1` is `(num_tokens, top_k, w1.shape[1])` (`:905-911`) | unchanged (`CAND:292-308`) | No | +| 2 | Gated SiLU: `silu(x[..., :ehdpp]) * x[..., ehdpp:]` -> `ehdpp`-wide activation | **MISSING.** `moe_vllm_kernel.py:404-405` takes `intermediate_cache1_flat[:, :expert_hidden_dim_per_partition].contiguous()` — the first half only, with no SiLU and no elementwise multiply by the second half. The up-projection half is computed and then discarded | `torch.ops._C.silu_and_mul(activated, intermediate_cache1.view(-1, ic1.shape[-1]))`, then `intermediate_cache2_input = activated` (`CAND:310-313`); `activated` is `(num_tokens*top_k, ehdpp)` allocated outside the step at `CAND:545-549` | No code conflict. **Reuse concern**: main already wraps the identical kernel in `frontier/profiling/common/layers/activation.py:8-34` (`self._op(out, x)` at `:33`); the candidate calls `torch.ops._C.silu_and_mul` raw, contradicting the donor proposal's own instruction to extend that module with a supplied-output variant rather than duplicate it | +| 3 | Optional activation quantization (FP8 W8A8), applied to the activated tensor and never to a raw slice | Present, but consumes the wrong input. `moe_vllm_kernel.py:407-413` quantizes `intermediate_cache2_input` with `quantize_activations_to_fp8(..., group_size=block_dims[1] or 128)`. Correct *order*, wrong *operand* | Unchanged code; the operand becomes `activated` because of step 2 (`CAND:313-321`). Order W1 -> activation -> quantize -> W2 is preserved | No. This is why the activation fix is genuinely a 4-line change: main's FP8 ordering already satisfies the donor's requirement once the operand is fixed | +| 4 | Second expert GEMM `act @ w2^T` with routing weights applied | Done. `_invoke_kernel(..., B=w2, C=intermediate_cache2, mul_routed_weight=True, top_k=1)` at `moe_vllm_kernel.py:415-431`. `w2` is `(num_experts, hidden_dim, ehdpp)` (`:786-792`) | unchanged (`CAND:323-339`) | No. Note the pre-existing latent hazard at `BASE:418` / `CAND:326`: passing `C=intermediate_cache2.contiguous()` would silently write into a throwaway copy if the tensor were ever non-contiguous. Harmless today (both are contiguous) but the candidate's workspace views make it worth re-checking | +| 5 | Local top-k output reduction `out[t] = sum_k ic2[t,k,:]` into a `(num_tokens, hidden)` buffer | **MISSING.** `intermediate_cache2` is `(num_tokens, top_k, hidden_dim)` (`moe_vllm_kernel.py:912-918`) and is never reduced. No `moe_sum` appears anywhere in the file (`git grep -n "moe_sum" 1f694f7c -- 'frontier/**'` returns nothing) | `ops.moe_sum(intermediate_cache2, output)` at `CAND:340`, inside the timed iteration; `output` is `(num_tokens, hidden_dim)` allocated outside the step at `CAND:550`; `from vllm import _custom_ops as ops` at `CAND:31` | **Yes, two ways.** (i) Import placement: `CAND:31` puts the import in the same top-level `try` as the 0.10.x kernel imports, which on main is a two-branch detection block (`BASE:116-140`); it must move inside the 0.10.x branch or be guarded so it cannot alter which API main selects. (ii) **Scope conflict** — see C.3; this line is the blocked one | +| 6 | Workspace layout matching vLLM (shared W1/W2 output buffer, separate activation buffer) | Two disjoint buffers, `intermediate_cache1` and `intermediate_cache2`, allocated separately at `moe_vllm_kernel.py:904-918`, both outside the timed step | One `workspace` flat tensor of `num_tokens*top_k*max(w1.shape[1], hidden_dim)` elements with `intermediate_cache1` and `intermediate_cache2` as overlapping views from offset 0 (`CAND:533-544`), plus separate `activated` and `output` | No code conflict. The aliasing is **safe only because `activated` is a separate buffer**: GEMM2 reads `activated` and writes `intermediate_cache2`, which overlaps `intermediate_cache1` — correct given the ordering at `CAND:310-339`. The claim in the comment at `CAND:533` that this "matches vLLM's shared W1/W2 output workspace" is **UNKNOWN from this repo**: no vLLM source is checked in here. Missing evidence: the `fused_experts` workspace allocation in the pinned vLLM revision (the donor cites `fused_moe.py:1721-1798` in an external checkout at `/data/ycfeng/tmp/issue26-vllm-diagnostics-20260908`, which is not part of this repository) | +| 7 | Respect non-gated MoE configurations | Not handled, and not handled by the candidate either. `profile_fused_moe_kernel` hard-codes `w1` width as `2 * expert_hidden_dim_per_partition` (`moe_vllm_kernel.py:781`) and passes `use_gated=True` unconditionally to the MXFP4 layout planner (`:770`), while `MoEWrapper._profile_with_vllm_kernel` (`moe_wrapper.py:657-690`) never forwards `self.use_gated` (set at `moe_wrapper.py:127` from `model_config.use_gated_mlp`) | Candidate adds no `use_gated` threading; it applies gated SiLU unconditionally | No conflict, but the donor proposal explicitly required "Respect supported non-gated model settings through the existing enum/activation registry; no model-name cases or silent slice fallback." The candidate does not satisfy that clause. Since main's vLLM-kernel path already allocates gated-shaped weights unconditionally, the honest options are to thread `use_gated` through or to fail fast for non-gated MoE on this backend | + +### C.2 What the current measured timing scope includes + +Both collectors wrap the **entire** `_step` closure, so the measured scope is exactly the body of `_run_fused_moe_iteration`: + +- `_collect_cuda_event_stats` (`BASE moe_vllm_kernel.py:603-626`): per-iteration `torch.cuda.Event` record -> `step_fn()` -> record -> `torch.cuda.synchronize()` -> `elapsed_time`, aggregated through `TimerStatsStore.get_stats_from_times` under the key `moe_grouped_gemm` (`:620-622`). +- `_collect_record_function_stats` (`BASE:629-653`): wraps each `step_fn()` in `torch.profiler.record_function("vidur_moe_grouped_gemm")` (`:645`) and reads back the `moe_grouped_gemm` entry. +- Warmup and `torch.cuda.synchronize()` precede measurement for both entry points (`BASE:943-945` legacy, `BASE:851-853` functional). All buffers are allocated outside the step (`BASE:904-918`). + +Therefore, **today on main, `moe_grouped_gemm` in `moe.csv` means different work depending on which entry point ran**: + +| Entry point | Selected when | Measured scope | +| --- | --- | --- | +| Legacy low-level (`_run_fused_moe_iteration`, `BASE:365-431`, driven from `BASE:920-958`) | `VLLM_API_VERSION == "0.10.x"` (`BASE:122-130`) | GEMM1 + first-half slice/`.contiguous()` copy + optional FP8 activation quant + GEMM2. **No activation, no reduction.** | +| Functional (`_run_functional_fused_experts_iteration`, `BASE:434-454`, driven from `BASE:837-869`) | `VLLM_API_VERSION == "functional_fused_experts"` (`BASE:131-138`); FP8 rejected here (`BASE:737-742`) | Whatever vLLM's `fused_experts` does end-to-end, which per the donor's inspection of the pinned vLLM includes `silu_and_mul` **and** `moe_sum`. | +| Functional MXFP4 (`BASE:820-833`) | `use_mxfp4` with the functional API | `mxfp4_method.moe_kernel.apply(...)` end-to-end | + +So the scope divergence the candidate is accused of introducing **already exists on main, between main's own two entry points.** Adding activation + `moe_sum` to the legacy path *converges* the two, it does not create a new split. + +### C.3 Does the local top-k reduction make existing CSV rows or cached models semantically incompatible? + +**Yes for semantics, and today nothing in the data can tell the two apart. This needs a user decision.** + +Metadata that exists today to distinguish rows, from the checked-in `moe.csv` header (`BASE data/profiling/compute/h800/Qwen3-30B-A3B-tiny/moe.csv`, 56 columns): + +| Column (position) | Producer | Can it distinguish measured arithmetic scope? | +| --- | --- | --- | +| `moe_grouped_gemm_backend` (51) | `moe_wrapper.py:649` via `resolve_grouped_gemm_backend` (`moe_wrapper.py:50-59`) | **No.** Its whole value set is `{"frontier_loop", "vllm_fused", "vllm_aiter_mxfp4"}`, a function of `use_vllm_kernel` and `use_mxfp4` only. Both of main's vLLM entry points emit `vllm_fused`. | +| `measurement_type` (52) | profiling metadata | No — `CUDA_EVENT` / `DEVICE_EVENT` / `KERNEL_ONLY` is the *timer*, not the scope. Consumed as the measurement family at `spmm.py:544-551`. | +| `profiling_precision` (53) | `spmm.py:4058+` consumer | No. | +| `quant_signature` (55) | `frontier/profiling/moe/main.py:394-401`, `:426` | No — quantization identity only. | +| `model_architecture_profile` (56) | validated at `spmm.py:3635-3638` | No — model-architecture identity. | +| `typed_operator_contracts` (`frontier/operators/typed_contracts.py:14`) | not present in the checked-in MoE CSVs; admission logic at `spmm.py:1256-1259`, `:3659` | No — it records operator family / profile / TP-EP semantics, not the arithmetic contained in one timed region. | +| `use_gated` (37), `routing_runtime_path` (28), `gating_runtime_context` (32) | profiling metadata | No — model/routing properties, not scope. | +| `VLLM_API_VERSION` | `moe_vllm_kernel.py:37`, `:130`, `:138` | **Never written to the CSV.** `git grep -n "VLLM_API_VERSION" 1f694f7c -- 'frontier/**'` shows it only inside `moe_vllm_kernel.py` and the README. | + +Consequences, precisely: + +1. **Existing rows become non-comparable with new rows, and nothing flags it.** A row measured before the change (GEMM1+GEMM2) and a row measured after (GEMM1+SiLU+GEMM2+`moe_sum`) are byte-indistinguishable in every metadata column. They will be mixed into one training frame by `_load_moe_df` (`spmm.py:3628`, `:3712-3720`) and fitted as one model. +2. **Cached models silently survive.** `_get_model_hash` (`spmm.py:4008-4056`) hashes the *frame content*, so a regenerated CSV does invalidate the cache — but a cache directory retained alongside an un-regenerated CSV will keep serving the old-scope model with no warning. There is no scope term in the hash and no scope attribute on the estimator. +3. **The predictor has nowhere to put the extra work.** `MOE_FAMILY` (`frontier/operators/families.py:77-133`) declares exactly four routed operators — `moe_gating_linear`, `moe_gating_routing_topk`, `moe_shuffling`, `moe_grouped_gemm` — with no routed activation and no reduction term. `MoETime` (`frontier/entities/time_components.py:503-547`) sums no such term; `share_expert_act_time` (`:515`, `:526`, `:539`) belongs to the distinct shared-expert family. So `moe_sum` can only enter the model by being folded into `moe_grouped_gemm`, i.e. by redefining that operator's meaning. +4. **The two sub-steps are not symmetric, and the donor says so.** `activation_profile_repair_proposal.md` (read from CAND) recommends only the activation fix and explicitly states: *"A separate proven gap is local `moe_sum`, outside vLLM's grouped scope. That requires an explicit operator ownership decision and is not silently included in this proposal."* It lists "Fold activation and local reduction into the existing grouped timing" under **Alternatives considered — rejected** for this sub-step: *"rejected for this sub-step because vLLM's current grouped scope excludes reduction. A consistently broadened scope would require a separate cross-side operator-contract decision."* `moe_profile_contract_review.md` (also read from CAND) corroborates with the vLLM call sites: activation is inside the named `moe_grouped_gemm` scope, `ops.moe_sum` is called after that scope closes. **The candidate's `CAND:340` does the thing its own donor proposal explicitly declined to do.** + +**Therefore:** + +- **Step 2 (gated SiLU) — ADAPT and port.** It restores work that the same-named vLLM scope contains. Old CSV rows for gated-SiLU MoE were already measuring the wrong thing; regenerating them is a repair, not a contract change. Still requires regeneration of affected `moe.csv` rows and cache invalidation, and still deserves a visible note in the profiling docs, but it needs no new operator and no cross-side renegotiation. +- **Step 5 (`ops.moe_sum`) — BLOCKED on a user decision.** It broadens `moe_grouped_gemm` past the boundary of the vLLM scope it is compared against, there is no metadata column that can mark the change, there is no `moe_sum` operator in `MOE_FAMILY` or `MoETime`, and there is no way to detect stale rows or stale caches afterwards. The decision to put to the user is: (i) add an explicit `moe_sum` operator across profiler / registry / predictor / `MoETime` / vLLM instrumentation so it is counted exactly once, or (ii) consistently broaden and rename/document `moe_grouped_gemm` on both sides, or (iii) port activation only and defer reduction. Note that the candidate's own parity test (`CAND tests/unit/test_moe_fused_expert_numerical_parity.py:48-52`) compares against `fused_experts`, whose return value *is* reduced — so that test only passes under option (i)/(ii) or with a reduction applied outside the timed region. +- **Recommended de-risking regardless of the choice:** record the entry point (`VLLM_API_VERSION`, or a scope label such as `moe_grouped_gemm_scope`) as a profiling metadata column. Main already has two entry points with different scopes and no way to tell their rows apart; that gap exists today, independent of this change. + +**UNKNOWN items for W6** (source in this repo cannot settle them): + +- Whether `vllm._custom_ops.moe_sum` exists and is correct on ROCm, which main now supports (`moe_vllm_kernel.py:28` imports `accelerator_platform`, `:728` validates the profile method per platform). Missing evidence: a ROCm vLLM build. Neither `torch` nor `vllm` is importable in this audit environment. +- The exact vLLM `fused_experts` workspace layout that `CAND:533` claims to match. Missing evidence: the pinned vLLM source (the donor's checkout `/data/ycfeng/tmp/issue26-vllm-diagnostics-20260908` at `361d941c97fcec52e544f74b7ab91c54192de9c9` is outside this repository). +- The magnitude and sign of the timing change. The donor's own reports state this is unmeasured: *"The timing impact, sign, and contribution to official TTFT remain Unknown and are not accepted as the primary TTFT RCA"* (`moe_profile_contract_review.md`, Verdict). No controlled A/B exists. + +--- + +## (d) Candidate changes that are experiment scaffolding and should be DROPped + +Scoped to the audited paths and their immediate blast radius. Everything here is a consequence of the candidate branching at MERGE_BASE, not deliberate removal — which is exactly why a file-level take must not happen. + +1. **The whole-file content of `frontier/profiling/moe/moe_vllm_kernel.py` from CAND.** Taking the file reverts 419 lines of main-only work: the functional `fused_experts` adapter (`BASE:434-454`, `:798-869`), MXFP4/AITER (`BASE:43-114`, `:457-601`), `device_event` timing (`BASE:603-626`), profile-method platform validation (`BASE:722-728`). Port hunks only (items 2, 5, 6 of C.1). +2. **`from vllm import _custom_ops as ops` at CAND `moe_vllm_kernel.py:31` as written.** Placed in the top-level `try`, it sits above main's two-branch API detection (`BASE:116-140`) and can change which branch main selects. Drop the placement; re-add inside the 0.10.x branch if `moe_sum` is approved. +3. **The `frontier/config/cluster_scheduler_config` import block at CAND `config.py:11-19`.** That module does not exist on BASE; the classes are still defined inline at `BASE config.py:2107-2140`. Pure branch-divergence artifact. +4. **All non-routing hunks in the CAND `config.py` diff**: removal of the GDN runtime guard call (`BASE:2085-2096`), removal of `gdn_input_file` (`BASE:2154-2157`), removal of `_validate_gdn_runtime_guards` and its call site (`BASE:5322`, `:5361-5398`). Main-only work; not candidate intent. +5. **The `{"eager", "kernel_only"}`-only family tables in the CAND rewrite of `_contract_model_registry` / `_contract_precision_registry`** (`CAND spmm.py:4146-4149`, `:4158-4161`). Dropping `device_event` would break ROCm/device-event prediction on main. Keep main's three-family tables and apply only the key widening. +6. **The wholesale restructuring of `_get_family_model`** (`CAND spmm.py:4250-4275`). The behaviour-equivalent parts are incidental refactor; the fix only needs a third key component and a `requested_runtime_path` filter. Main's explicit `requested_identity` / legacy-identity branch structure (`BASE:4276-4302`) carries readable error messages that two existing tests assert against. +7. **`rtol=0, atol=0` in `tests/unit/test_moe_fused_expert_numerical_parity.py:52`.** Bitwise equality against `fused_experts` is an experiment-grade assertion; the donor's own corrected plan requires "a declared tolerance before deciding numerical parity." +8. **The `SimpleNamespace`-based unbound-method invocation in `tests/unit/test_moe_routing_runtime.py:61-69`.** It depends on the pre-tightening `getattr(self, "_cluster_type", None)` form that only exists on MB/CAND (`CAND sklearn_moe_...py:765-768`); main reads `self._cluster_type` directly (`BASE:773`). Either construct a real predictor or set `_cluster_type` explicitly — do not soften main's code to fit the fixture. +9. **The `object.__new__` manager fixture in `tests/unit/test_moe_routing_runtime_model_sharing.py:18-24`.** It sets 5 attributes and skips the registry dicts main initializes at `BASE spmm.py:444-449`; it works on CAND only because the tests never touch an uninitialized family. Needs a real or fully initialized manager before it can cover three families. +10. **The entire `task_memory/task_2026-09-07_issue26_ttft_h200/` tree carried by CAND.** ~1.7M inserted lines of run artifacts, JSON receipts and worker scripts. The two evidence documents cited here should be preserved as references in a local task directory, not merged onto main. (Main's own history already shows `26b490a chore: remove local task memory from published repository`.) + +--- + +## Donor evidence documents read (from CAND) + +| Path (all under `task_memory/task_2026-09-07_issue26_ttft_h200/`) | What it contributed | +| --- | --- | +| `test_report_2026-09-08_d012_routing_runtime.md` | D012 acceptance record for W5: 54 config/resolver/predictor/CLI checks + 20 shared-registry/cache checks. States distribution defaults and allocations are unchanged, explicit uniform runtime survives CLI construction and replica copying, distinct runtimes stay distinct through dedup/registration/projection/disk-cache. Explicitly limits the claim to correctness, not numerical TTFT parity. | +| `analysis/activation_profile_repair_proposal.md` | The W6 proposal. Recommends activation only; names `torch.ops._C.silu_and_mul`, the out-of-timing activation buffer, and preservation of the activation-before-FP8-quant order. Explicitly excludes `moe_sum` as requiring a separate operator-ownership decision, and lists fold-both-together under rejected alternatives. Also requires non-gated support via the existing activation registry and prefers extending `SiluAndMul` over calling the raw op — two clauses the candidate code does not satisfy. | +| `analysis/moe_profile_contract_review.md` | Independent review with an evidence ledger. Establishes the contract mismatch as Direct/Corroborated and the timing magnitude and TTFT attribution as Unknown. Confirms `MOE_FAMILY`, `_build_moe_operator_times` and `MoETime.total_time()` contain no routed-activation and no `moe_sum` term (re-verified on BASE: `frontier/operators/families.py:77-133`, `frontier/entities/time_components.py:503-547`). Gives the vLLM-side boundary: activation inside the named grouped scope, `ops.moe_sum` after it closes. | +| `analysis/routing_import_capability.md` | Context for W5: confirms `moe_routing_trace_path` has no consumer, and that `_init_global_routing_allocations` produces static per-layer ratios. Re-verified on BASE at `sklearn_moe_...py:766-801`. | +| `test_report_2026-09-08_uniform_routing.md` | The reachability receipt for the W5 defect: running the real `resolve_moe_gating_routing_runtime_path('balanced')` plus `filter_moe_gating_routing_topk_rows` against a `uniform_topk` CSV produced the expected `ValueError: No moe_gating_routing_topk profiling rows match the requested routing_runtime_path='standard_fused_topk'`. Explicitly refuses relabeling or fallback as a workaround, and states "Standard-fused-topk profiles cannot substitute for uniform_topk timing rows." | + +Sibling analysis files matching moe/routing/activation that were **listed but not read** (out of scope for these two work packages): `d019-moe-repair.md`, `d019-moe-review.md`, `d019-moe-integration/report.md`, `d021-post-moe-input-shape-audit.md`, `moe-subphase-code-audit.md`, `profile_coverage_plan_d019.md`, `dp_routing_endpoint_proposal.md`, `dp_routing_repair_proposal.md`, `routing_global_counts_proposal.md`, `d012_module_scope.md`, `cpu-master-runtime-01/d012-implementation.md`, and the `test_report_2026-09-08_*` / `2026-09-09_*` / `2026-09-1x_*` series on post-MoE attribution and fresh profiles. + +## Anchor-fact verification summary (requested in the task) + +| Anchor fact as given | Verdict | Corrected/confirmed citation on BASE | +| --- | --- | --- | +| `sklearn_moe_execution_time_predictor.py` sets `_moe_routing_distribution_type` around L715-724 | **Confirmed** | `1f694f7c:frontier/execution_time_predictor/sklearn_moe_execution_time_predictor.py:715-724` | +| ...derives `_moe_gating_routing_runtime_path` around L731-734 which is then never read | **Confirmed, with a one-line correction** | The statement spans `:731-735` (the closing paren is on 735). Dead-write verified by repo-wide `git grep` at BASE: the only occurrence of the attribute is the write at `:731`. | +| `shared_prediction_model_manager.py` resolves the runtime path at ~L1294-1296 and ~L1440-1442 | **Confirmed exactly** | `:1294-1296` (in `_validate_moe_dataset_contract`) and `:1440-1442` (in `_train_ffn_models_for_cluster`) | +| ...uses `runtime_path_key` only inside a per-call `moe_df_cache` tuple at ~L1488-1502 | **Confirmed, with a refined range** | `runtime_path_key` assigned at `:1486-1488`; used in the cache key tuple at `:1499-1505`; used in the filter at `:1518-1523`; used in the log at `:1535`. The `moe_df_cache` dict is declared at `:1444-1453` and is local to one call. | +| `_get_model_hash` (~L4008-4057) has no routing-runtime term | **Confirmed** | `:4008-4056`. Hash inputs enumerated at `:4029-4045`. It does include the frame-content md5 at `:4033`, which incidentally separates routing variants — see (b) stage 8. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/audit_scheduler.md b/task_memory/task_2026-09-21_issue26_correctness_pr/audit_scheduler.md new file mode 100644 index 00000000..4c065f3c --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/audit_scheduler.md @@ -0,0 +1,287 @@ +# Scheduler-side audit of candidate `bug/ttft-check` (W2 / W3 / W4) + +Read-only source audit. No file under `/data/ycfeng/Frontier` was modified. + +## Revisions + +| Label | SHA | Subject | +| --- | --- | --- | +| BASE (current main) | `1f694f7c549aa3aeeb7c5bbae04e119c09167a77` | `updapte doc` | +| CAND (`bug/ttft-check`) | `a7b3320fe9b8b083ee86b91dae3d6838f4443d91` | `Record Issue 26 H200 TTFT task docs and numerical results` | +| MERGE_BASE | `d71ad80b0800880808a0857fd30477e6d96592c6` | `chore: keep repair receipts and task memory local-only` | + +`git merge-base BASE CAND` = `d71ad80b0800880808a0857fd30477e6d96592c6` (verified). + +Diff volume on the audited paths (`+`/`-` lines): + +| Path | MB→CAND | MB→BASE | Notes | +| --- | --- | --- | --- | +| `frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py` | 5/4 | 0/0 | main untouched since merge base | +| `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py` | 12/2 | 12/23 | both sides changed | +| `frontier/scheduler/cluster_scheduler/cluster_scheduler_registry.py` | 6/0 | 0/0 | | +| `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py` | 57/0 (new) | — | | +| `frontier/scheduler/request_load.py` | 10/0 (new) | — | | +| `frontier/scheduler/utils/vllm_dp_load_balancer.py` | 84/0 (new) | — | | +| `frontier/scheduler/utils/forward_collective.py` | 39/0 (new) | — | | +| `frontier/scheduler/utils/forward_sync_state.py` | 2/1 | 0/0 | | +| `frontier/scheduler/utils/sync_state.py` | 1/1 | 0/0 | | +| `frontier/scheduler/utils/ep_wave_inputs.py` | 6/0 | 0/0 | | +| `frontier/scheduler/utils/ep_wave_schedule.py` | 22/11 | 7/0 | both sides changed | +| `frontier/scheduler/utils/prefill_collective.py` | 19/1 | 15/8 | both sides changed | +| `frontier/scheduler/utils/decode_collective.py` | 25/2 | 3/4 | both sides changed | +| `frontier/scheduler/replica_scheduler/base_replica_scheduler.py` | 7/0 | 28/0 | both sides changed | +| `frontier/scheduler/replica_scheduler/vllm_v1_engine_replica_scheduler.py` | 10/5 | 80/15 | both sides changed | +| `frontier/events/replica_stage_schedule_event.py` | 4/0 | 0/0 | | +| `frontier/events/cluster_schedule_event.py` | 1/1 | 0/0 | | +| `frontier/events/global_batch_end_event.py` | 3/0 | 0/0 | | +| `frontier/config/config.py` | 25/40 | 57/0 | both sides changed | +| `frontier/config/cluster_scheduler_config.py` | 53/0 (new) | — | | +| `frontier/types/cluster_scheduler_type.py` | 1/0 | 0/0 | | + +--- + +## (a) Disposition table + +Column `main:line` cites BASE `1f694f7`. Column `cand:line` inside "change summary" cites CAND `a7b3320`. + +### W2 — Round-robin DP placement rotation across `schedule()` calls + +| candidate path | change summary | old defect | current main behavior (file:line on main) | disposition | rationale | proposed owner on main | planned test | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py` | `_schedule_batch_mode` computes `ordinal = self._request_counter + request_idx`, then `replica_idx = ordinal % self._num_replicas` and `dp_id = (ordinal // self._num_replicas) % self._replica_dp_size`, carries `(dp_id, request)` in the per-replica bucket, and emits that `dp_id` instead of a per-call `enumerate` index (cand:369-389). Net +5/-4. | DP-lane assignment restarts at lane 0 on every `schedule()` call. In online mode where each `ClusterScheduleEvent` carries 1 request, every request lands on `dp_id = 0`; lane 1..N-1 of each Replica never receive work from `_schedule_batch_mode`. | Defect is **still present**. `frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py:385-386` uses `for local_idx, request in enumerate(requests): dp_id = local_idx % self._replica_dp_size`, where `requests` is the per-call bucket built at :366-374. Only the *replica* index uses the persistent counter (:372, :376). See (b) for the full quote. | **PORT** | Minimal, self-contained, no dependency on W3/W4. It makes `_schedule_batch_mode` agree with the formula main already uses in `_schedule_decode_lane_round_robin` (`round_robin_cluster_scheduler.py:438-439`), i.e. it removes a real inconsistency inside the same class rather than introducing a new policy. Numeric results change for co-location/PREFILL with `attn_dp > 1`, so it is a fidelity fix, not a pure refactor. | `RoundRobinClusterScheduler._schedule_batch_mode` in `frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py` (single method; `_request_counter` semantics at :27 unchanged). | Extend `tests/unit/test_cluster_scheduler_dp_lanes.py` with the candidate's parametrized case (see next row). Add an equivalence assertion that `_schedule_batch_mode` and `_schedule_decode_lane_round_robin` produce the same `(replica_id, dp_id)` sequence for the same counter/queue, since they now share one formula. Add one online co-location `attn_dp=2` E2E smoke asserting both lanes are non-empty in `request_metrics.csv` / batch logs. | +| `tests/unit/test_cluster_scheduler_dp_lanes.py` | Adds `test_round_robin_preserves_lane_rotation_across_schedule_calls`, parametrized over chunk shapes `(1,)*8`, `(0,2,0,3,3)`, `(8,)`, counter offsets `0`/`5`, and 1- or 2-replica clusters; asserts the global `(replica, lane)` cycle and `_request_counter == offset + len(requests)` (cand:76-108). | No regression pinned the cross-call rotation. | Main's copy of this file is byte-identical to MERGE_BASE (`git diff --numstat d71ad80 1f694f7 -- tests/unit/test_cluster_scheduler_dp_lanes.py` → empty), and contains no `RoundRobinClusterScheduler` case. | **PORT** | Directly encodes the acceptance criterion for the W2 fix and uses only the existing `_lane_scheduler` helper already in that file. | Same file on main. | It *is* the test. | + +### W3 — Shared monolithic forward execution (mixed prefill/decode source lanes in one EP wave) + +| candidate path | change summary | old defect | current main behavior (file:line on main) | disposition | rationale | proposed owner on main | planned test | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `frontier/scheduler/utils/sync_state.py` | For `MONOLITHIC`, `_decode_sync_waiting_room` is aliased to the same object as `_prefill_sync_waiting_room` instead of a second `_new_sync_waiting_room()` (cand:30-34). Net +1/-1. | The same admitted forward group is held in two disjoint waiting rooms when its lanes have different local phases, so neither room ever reaches `expected_lanes` and the EP wave is never dispatched (documented reproduction: `design_shared_forward_sync.md` §"Source findings"; `review_shared_forward_sync.md` §4, group 5 / layer 0 / decode 240 lane 0 vs prefill 241 lane 1). | **Still present.** `frontier/scheduler/utils/sync_state.py:29-34`: `scheduler._prefill_sync_waiting_room = _new_sync_waiting_room()` then `scheduler._decode_sync_waiting_room = (_new_sync_waiting_room() if cluster_type is ClusterType.MONOLITHIC else None)` — two distinct `defaultdict` trees. | **ADAPT** | Required for the fix, but a bare alias is not sufficient on its own: `ep_wave_schedule.py:184-188` still selects the room by `mode`, and `forward_sync_state` still partitions the *step-id namespace* by kind. Adapt = land it together with the `forward_sync_state` + `base_cluster_scheduler` rows, and re-read `ep_wave_schedule.py:189-196` (the "post_moe room already contains a batch" guard) under a single shared room. | `initialize_sync_waiting_rooms` in `frontier/scheduler/utils/sync_state.py`. | `tests/unit/test_monolithic_mixed_forward_sync.py::test_all_local_phases_join_one_wave` (9 phase pairs × 2 lane orders). Plus an assertion that `PREFILL`/`DECODE` clusters still get independent (or `None`) rooms. | +| `frontier/scheduler/utils/forward_sync_state.py` | Adds a third sync kind `"forward"` to `_open_steps_by_kind` and to `_validate_kind` (cand:38-41, :56-59). Net +2/-1. | Step-id bindings for the two lanes of one monolithic forward live in different namespaces, so `resolve_step` allocates two different cohort IDs from `_next_step_id_by_replica`. | **Still present.** `frontier/scheduler/utils/forward_sync_state.py:38-41` declares only `{"prefill": {}, "decode": {}}`; `:57-59` rejects anything else. `_next_step_id_by_replica` (:42) is keyed by `replica_id` only, so two lanes of the same wall-clock forward consume two consecutive IDs (`:133-139`). | **PORT** | Two-line additive change with no behavior change for PDD/PD-AF, which keep passing `"prefill"`/`"decode"`. | `ForwardSyncState.__init__` / `ForwardSyncState._validate_kind` in `frontier/scheduler/utils/forward_sync_state.py`. | Extend `tests/unit/test_forward_sync_state.py` (exists on main) with a `"forward"`-kind case: two lanes, different local phases, same `(replica, stage, layer, sync_stage, provisional_id)` → one resolved step id; and that an unknown kind still raises. | +| `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py` | `_resolve_forward_step` and `_close_forward_step` substitute `"forward"` for the caller's `sync_kind` when `self._cluster_type == ClusterType.MONOLITHIC` (cand:903, :926). Net +12/-2 on this file (the other +10 is the W4 hook pair). | Callers in `sync_entry.py` hard-code the kind from the entry event's phase. | **Still present.** `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py:889-898` forwards `sync_kind=sync_kind` verbatim; `:912-919` likewise. Callers: `frontier/scheduler/utils/sync_entry.py:52` (`sync_kind="prefill"`) and `:190` (`sync_kind="decode"`), with matching `_close_forward_step` at `:133` and `:270`. | **ADAPT** | The mechanism is right but the placement is a ternary inside two delegating wrappers of a 1,860-line module. On main the same normalization is better expressed once — e.g. a single `_sync_kind_for(cluster_type, entry_phase)` helper owned by `forward_sync_state.py` and called from `sync_entry.py`, so `base_cluster_scheduler.py` stays a thin seam (which is exactly what `design_shared_forward_sync.md` §5 states it wants). Also note main's `base_cluster_scheduler.py` changed independently in this region (+12/-23, MB→BASE), so the hunk will not apply cleanly. | `frontier/scheduler/utils/sync_entry.py` (`enter_prefill_sync`, `enter_decode_sync`) with the mapping helper in `frontier/scheduler/utils/forward_sync_state.py`. | Same `test_monolithic_mixed_forward_sync.py` case, plus existing `tests/unit/test_shared_forward_group_admission.py` to confirm PDD/PD-AF kinds are untouched. | +| `frontier/scheduler/utils/forward_collective.py` (new, 39 lines) | `complete_forward_collective(scheduler, time, replica_id, stage_id, step_id, layer_id, metrics_store)`: pops the whole `step_id` sub-tree from `_prefill_sync_waiting_room[replica_id][stage_id]`, reads `[layer_id]["post_moe"]["batches"]`, calls `_restore_forward_step_full_stage_owners` **once** for the whole cohort with `operation_kind = "attention" if layer_id+1 < layer_end else "final"`, then re-enters `handle_prefill_sync_collective` / `handle_decode_sync_collective` per live source batch with `direct_batch=batch, owners_restored=True`, skipping idle lanes. | Completion helpers previously predicted continuation/final timing from a single `sample_batch` and restored ownership per phase-room, so a second lane with a different shape borrowed the first lane's timing, and ownership promotion/restoration was not 1:1 with the wave. | No equivalent exists on main. Main restores owners inside each phase helper: `frontier/scheduler/utils/prefill_collective.py:96` region and `frontier/scheduler/utils/decode_collective.py:103-110`, each over its own `dp_batches`. Main's `decode_collective.py:147-150` still predicts `full_execution` from `sample_batch` for the whole cohort; `prefill_collective.py:235-238` predicts metrics timing from `batch` (the per-source loop variable), so main already fixed the *metrics* side of the sample-batch borrow for prefill but not for decode final timing. | **ADAPT** | The invariant set is correct and needed (see (c)), but this file as written is coupled to candidate-era internals: it calls `scheduler._restore_forward_step_full_stage_owners(...)` with the pre-main signature, it assumes `_prefill_sync_waiting_room` is the single room (depends on the `sync_state.py` alias), and it uses `scheduler._predictor._num_layers_per_pipeline_stage` directly instead of the `get_pipeline_stage_layer_bounds` bounds helper main uses defensively (`decode_collective.py:89-101`). It also does `stage_rooms.pop(step_id)` with no guard, which will `KeyError` rather than raise the explicit "no matching waiting room" error main raises at `decode_collective.py:50-55`. | New `frontier/scheduler/utils/forward_collective.py` on main, invoked from `prefill_collective.handle_prefill_sync_collective` / `decode_collective.handle_decode_sync_collective`. | `tests/unit/test_monolithic_mixed_forward_sync.py::test_completion_preserves_each_source_and_its_attention_time` (predictor spy asserts each original batch and its own `num_tokens` reach the predictor) and `::test_mixed_batch_finishes_requests_once_without_rewriting_old_ttft`. Add a ticket-lifecycle assertion: exactly one promote + one restore per wave, and empty rooms/queues at end of run. | +| `frontier/scheduler/utils/prefill_collective.py` | (1) Early-dispatch: when `_cluster_type == MONOLITHIC and direct_batch is None`, delegate to `complete_forward_collective` (cand:46-52). (2) New `owners_restored: bool = False` kwarg short-circuiting `_restore_forward_step_full_stage_owners` (cand:30, :113). (3) For `MONOLITHIC`, advance `mb_on_step_layer_count_increment(1)` for the *decode subset* (`r.is_prefill_complete`) of a prefill-entry source batch, guarded by `validate_decode_layer_advance` (cand:99-106). | Decode requests riding inside a mixed (prefill-entry) source batch never had their per-layer decode counters advanced, because layer advance lived only in the decode helper. | `frontier/scheduler/utils/prefill_collective.py` on main has **no** `ClusterType` import and no decode-layer advance (`git grep -n "MONOLITHIC\|ClusterType" 1f694f7 -- frontier/scheduler/utils/prefill_collective.py` → empty). Main's counterpart lives only in `frontier/scheduler/utils/decode_collective.py:81-86`. | **ADAPT** | Items (1) and (2) are correct and needed. Item (3) duplicates the exact `validate_decode_layer_advance` + `mb_on_step_layer_count_increment` pair already at `decode_collective.py:81-86`; on main it should be one shared helper (candidate leaves two copies). Conflict risk: main rewrote `prefill_collective.py:230-243` to gate metrics on `metrics_store.stage_execution_reporting_enabled` and to re-predict from `stage_layer_start`; the candidate's hunk at the same `_restore_forward_step_full_stage_owners` call site is adjacent. | `handle_prefill_sync_collective` in `frontier/scheduler/utils/prefill_collective.py`; the shared layer-advance helper belongs in `frontier/scheduler/utils/collective_timing.py` next to `validate_decode_layer_advance` (`collective_timing.py:42`). | `test_monolithic_mixed_forward_sync.py::test_mixed_batch_finishes_requests_once_without_rewriting_old_ttft`. Additionally assert a 4096/1024 request ends with exactly 1024 output tokens (the acceptance row in `design_shared_forward_sync.md` §4). | +| `frontier/scheduler/utils/decode_collective.py` | (1) Same MONOLITHIC early-dispatch to `complete_forward_collective` (cand:46-52). (2) Same `owners_restored` kwarg (cand:31, :113). (3) For MONOLITHIC, append `next_execution.get_single_layer_attention_scope_time()` into `batch._decode_model_execution_components_ms_by_stage[stage_id]` per layer (cand:137-140). (4) For MONOLITHIC, replace `override_model_execution_time(full_execution.model_time)` with `math.fsum(components) * 1e-3 + final_timing.pipeline_time + final_timing.draft_proposer_time` (cand:195-203). | Decode had no explicit per-source model-component ledger (prefill had one), so the reported decode model time for a shared forward came from one `sample_batch` aggregate prediction rather than the actual per-source layer components. | Main still uses the aggregate: `frontier/scheduler/utils/decode_collective.py:147-150` predicts `full_execution` from `sample_batch` over `num_layers`, and `:181` calls `batch_stage.override_model_execution_time(full_execution.model_time)` inside the per-batch loop. No `_decode_model_execution_components_ms_by_stage` symbol exists anywhere on main (`git grep` → 0 hits at `1f694f7`). | **ADAPT** (items 1–3) / **BLOCKED** (item 4 as written) | Item 4 sits in a block main rewrote: candidate calls `scheduler._create_corrected_execution_time_for_metrics(full_execution, actual_execution, original_start)` (cand:205-207), but main **deleted that method** (MB→BASE removes `_create_corrected_execution_time_for_metrics` from `base_cluster_scheduler.py`; no definition remains at `1f694f7`) and `decode_collective.py:182-184` now calls `_create_prefill_corrected_execution_time_for_metrics(batch, stage_id, full_execution)` whose signature also changed (`base_cluster_scheduler.py:1138`). Porting item 4 verbatim raises `AttributeError`. The decode-ledger idea still applies, but it must be re-expressed against main's current metrics contract, which now separates stage-attention reporting from EP-lane wave records. | `handle_decode_sync_collective` in `frontier/scheduler/utils/decode_collective.py`; the ledger accessor should mirror `_prefill_model_execution_components_ms_by_stage` and be owned by `frontier/entities/batch.py` rather than set ad hoc on the event. | Component-reconciliation test: per-source wall time equals the event interval, and explicit modeled components plus join waits reconcile without double counting (gate "Timing accounting" in `design_shared_forward_sync.md` §6). Plus existing `tests/comm_backend_tests/` and monolithic MoE E2E smokes to confirm numerics for pure-decode are unchanged. | +| `frontier/scheduler/utils/ep_wave_schedule.py` | Adds `_source_mode(scheduler, batch, mode)` returning `"prefill" if batch.num_prefill_tokens else "decode"` under MONOLITHIC (cand:15-18); uses it to pick the dense continuation event class payload (cand:95, :148) and to select the per-source ledger; hoists `wave_time_ms` out of the `if mode == "prefill"` branch so **both** prefill and decode sources record wave time (cand:157-186). | The wave-level barrier time was recorded only when the *group* mode was `"prefill"`; a decode source in a shared forward recorded no wave component, and every source was labelled with the group's mode. | Main still keys everything off the group `mode`: `frontier/scheduler/utils/ep_wave_schedule.py:126` (`if mode == "prefill"` for the dense ledger), `:145` (passes `mode` into the dense event), `:155-182` (`wave_time_ms` computed and recorded only in the prefill branch; the decode branch at `:180-182` records only `_decode_ep_wave_lane_times_ms`). | **ADAPT** | Correct direction, but main added `metrics_store` plumbing to this exact function (`:22`, `:69-74` `capture_lane_timings` / `metrics_store.on_ep_wave_schedule`) that the candidate does not have, so the hunks overlap. Also, `_source_mode` is a free function reading `scheduler._cluster_type`; on main this is better derived from the source batch alone and passed in, keeping the "phase is a diagnostic label, not group identity" rule from `review_shared_forward_sync.md` §5. | `schedule_layer_wave` in `frontier/scheduler/utils/ep_wave_schedule.py`. | Parametrized wave test over P/P, D/D, P/D, mixed/P, mixed/D, mixed/mixed asserting exactly one wave per key and one ledger append per source per layer. | +| `frontier/scheduler/utils/ep_wave_inputs.py` | `prepare_ep_wave_inputs` now rejects the same `request_id` appearing in two non-idle source lanes: builds `owned_requests` and raises `"a request cannot belong to two EP source lanes"` (cand:51-57). | Cross-lane duplicate request ownership would be silently deduplicated downstream instead of failing. | Not present on main. `frontier/scheduler/utils/ep_wave_inputs.py:44-50` validates only that all sources share one step id and that at least one is non-idle. | **PORT** | 6 additive lines, no behavior change for any valid input, and it is the guard that turns a silent double-count into a fail-fast. Applies cleanly (main did not touch this file since the merge base). | `prepare_ep_wave_inputs` in `frontier/scheduler/utils/ep_wave_inputs.py`. | `test_monolithic_mixed_forward_sync.py::test_ep_inputs_reject_cross_lane_request_duplication`. | +| `frontier/events/replica_stage_schedule_event.py` | In the MONOLITHIC decode branch, initialize `batch._decode_model_execution_components_ms_by_stage = {self._stage_id: [attention_time_ms]}` (cand:540-543), mirroring the prefill ledger init. | Decode had no component ledger to append into. | Main initializes only the prefill ledger: `frontier/events/replica_stage_schedule_event.py:247-249`. The decode branch at `:516-537` computes `attention_time_ms` but stores no ledger. | **ADAPT** | Depends on the decode-ledger decision in the `decode_collective.py` row; ship or drop the two together. As written it uses a plain `dict` with one pre-seeded stage key, which only works for PP1 — the prefill twin at `:247` has the same limitation, so it is at least consistent, but the PP>1 behavior is **UNKNOWN** (no test exercises a monolithic MoE PP>1 shared forward in either tree). | `ReplicaStageScheduleEvent.handle_event` in `frontier/events/replica_stage_schedule_event.py`; ledger ownership preferably moved to `frontier/entities/batch.py`. | Same component-reconciliation test as the `decode_collective.py` row. Add an explicit PP2 monolithic MoE case or an explicit fail-fast if PP>1 is unsupported. | +| `tests/unit/test_monolithic_mixed_forward_sync.py` (new, 4 tests) | `test_all_local_phases_join_one_wave` (3×3 phases × 2 orders), `test_completion_preserves_each_source_and_its_attention_time` (4 phase pairs, predictor spy), `test_mixed_batch_finishes_requests_once_without_rewriting_old_ttft`, `test_ep_inputs_reject_cross_lane_request_duplication`. | No regression covered a mixed-phase monolithic forward. | Not present on main. | **PORT** (after the production rows land) | Each test maps 1:1 to a gate in `design_shared_forward_sync.md` §6. The first two construct the scheduler via `object.__new__` + `SimpleNamespace` stubs, which is brittle against main's newer `metrics_store` parameters on `_on_prefill_ep_wave_ready` / `_on_decode_ep_wave_ready` (`base_cluster_scheduler.py:1005`, `:1057`) — the stub lambdas take `**kwargs`, so they should survive, but this needs a run to confirm. | `tests/unit/test_monolithic_mixed_forward_sync.py`. | It *is* the test. | + +### W4 — vLLM-style opt-in DP request placement + +| candidate path | change summary | old defect | current main behavior (file:line on main) | disposition | rationale | proposed owner on main | planned test | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `frontier/types/cluster_scheduler_type.py` | Adds `VLLM_LOAD_BALANCING = 5` (cand:10). | — (new capability, not a defect fix) | `frontier/types/cluster_scheduler_type.py:1-9` defines `ROUND_ROBIN..STICKY_LOR` (0-4). | **PORT** (gated on the whole W4 decision) | One enum value; `BaseIntEnum.__str__` (`frontier/types/base_int_enum.py:5-6`) makes the CLI token `vllm_load_balancing`. | `frontier/types/cluster_scheduler_type.py`. | `tests/unit/test_vllm_dp_load_balancer.py::test_all_cluster_policy_configs_and_registry_entries_remain_discoverable`. | +| `frontier/config/cluster_scheduler_config.py` (new, 53 lines) + `frontier/config/config.py` | Moves the six `*ClusterSchedulerConfig` dataclasses out of `config.py` into a new module and re-imports them into `config.py` (cand `config.py`:11-18, and deletion of `config.py`:2106-2145 region); adds `VllmLoadBalancingClusterSchedulerConfig`. | `config.py` is oversized (5,720 lines on main, above the 2,000-line gate in `AGENTS.md` §Development Gates). | Main still defines all five in `frontier/config/config.py:2099-2136`; the field is `ClusterConfig.cluster_scheduler_config` at `config.py:2474-2475` with `default_factory=RoundRobinClusterSchedulerConfig`. | **ADAPT** | The extraction itself is desirable and matches the gate, but it is an orthogonal refactor bundled into a behavior change. On main it should land as its own commit (pure move + re-export, zero diff in generated CLI surface), verified by dumping `--help` before/after. Discovery is safe because `frontier/config/flat_dataclass.py:132-139` resolves poly types via `get_all_subclasses(field_type)`, which only needs the module imported — and `config.py:11-18` imports it. | New `frontier/config/cluster_scheduler_config.py`; `frontier/config/config.py` keeps the re-export so existing `from frontier.config.config import ...` imports do not break. | Snapshot test: the set of accepted `--cluster_scheduler_config_type` values before/after the move is identical plus exactly one new value. Existing `tests/unit/test_open_source_release_arch_guard.py` for the PD-AF guard at `config.py:4081-4086`. | +| `frontier/scheduler/request_load.py` (new, 10 lines) | `RequestLoad(NamedTuple)` with `waiting: int`, `running: int`. | Waiting and running counts were only co-located inside one log payload dict. | Not present on main. | **PORT** | 10 lines, no dependencies, removes the duplicated count-selection at `vllm_v1_engine_replica_scheduler.py:2180-2182`. | `frontier/scheduler/request_load.py`. | Covered indirectly by `test_vllm_dp_load_balancer.py::test_load_accessor_keeps_running_separate_from_schedulable_pending`. | +| `frontier/scheduler/utils/vllm_dp_load_balancer.py` (new, 84 lines) | Pure state machine reproducing vLLM v0.10.2 `core_client.py`/`coordinator.py` count publication: `engine_counts`, `frontend_counts`, `last_step_counts`, `last_report_step`, `stats_changed`, `last_publish_ms=-5000`, `next_publish_ms=50`; `_poll_deadline` uses 100 ms (changed) / 5000 ms (unchanged) intervals and a 50 ms initial collection wait; `select()` picks `argmin(4*waiting + running)` with first-index tie-break and applies a local `waiting+1` reservation; `report()` suppresses unchanged reports and latches the previous snapshot when a newer step arrives. | — (new capability) | Not present on main. Main's only DP placement policies are the RR/LOR/random/sticky family in `frontier/scheduler/cluster_scheduler/`. | **PORT** as an opt-in module, **BLOCKED** on the `step` identity question | The module itself is pure, deterministic, unit-tested (11 tests), and has no import into any default path, so it cannot perturb existing results. What is blocked is its *input*: `report(..., step=ForwardSyncState.get_step_id(batch), ...)` — see (d). Until the step key is a genuine per-engine forward counter, the `step > self.last_report_step` latch at cand:64-67 is not well-defined on main. | `frontier/scheduler/utils/vllm_dp_load_balancer.py`. | `tests/unit/test_vllm_dp_load_balancer.py` (11 tests). Add a determinism test that two runs with identical arrivals produce identical placements, and a trace-comparison harness against a recorded vLLM coordinator log (the design note explicitly says source equivalence alone does not establish identical placement — `design_dp_load_balancing.md` §"Snapshot timing"). | +| `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py` (new, 57 lines) | `VllmLoadBalancingClusterScheduler(BaseClusterScheduler)`. Constructor hard-rejects anything other than `MONOLITHIC` + `num_replicas == 1` + `num_pipeline_stages == 1` + `ReplicaSchedulerType.VLLM_V1` (cand:17-26). `schedule_at(time)` stores `self._routing_time` and calls `schedule()`; `schedule()` assigns every queued request a lane from `VllmDPLoadBalancer.select(self._routing_time)` (cand:31-46). `on_replica_batch_end` reports `lane.get_request_load()` keyed by `ForwardSyncState.get_step_id(batch)` (cand:48-57). | — (new capability) | Not present on main. | **PORT** (gated with the rest of W4) | Fail-fast topology validation is present and correct; `self._replica_scheduler_type` is set by `frontier/scheduler/utils/replica_state.py:21`, which is byte-identical on main, so the guard works. One gap: `on_replica_batch_end` passes `replica_local_id` straight into `VllmDPLoadBalancer.report`, which requires `0 <= engine < num_engines` (cand `vllm_dp_load_balancer.py`:56-57). A `None` lane identity (`get_replica_scheduler` treats `None` as the full-stage identity — `base_cluster_scheduler.py:437-440`) would raise `TypeError`, not the intended `ValueError`. | `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py`. | Topology-rejection tests for each of the four guard conditions; a DP1 case asserting `select()` always returns 0 and `report()` is still well-formed; an `attn_dp=1` monolithic run asserting `replica_local_id` is an `int`, not `None`. | +| `frontier/scheduler/cluster_scheduler/cluster_scheduler_registry.py` | Registers `ClusterSchedulerType.VLLM_LOAD_BALANCING -> VllmLoadBalancingClusterScheduler` (cand:35-37) and adds the import (cand:16-18). | — | `frontier/scheduler/cluster_scheduler/cluster_scheduler_registry.py:1-31` registers the five existing types; file unchanged since the merge base. | **PORT** | Registry-entry addition through the unchanged registry mechanism. | Same file. | `test_vllm_dp_load_balancer.py::test_all_cluster_policy_configs_and_registry_entries_remain_discoverable`. | +| `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py` | Adds two default-no-op seams: `schedule_at(self, time) -> ...: return self.schedule()` and `on_replica_batch_end(self, time, replica_id, replica_local_id, batch) -> None: return None` (cand:418-427). | The cluster-scheduler interface exposed no DES time to routing and no post-step feedback. | Not present on main. `BaseClusterScheduler.schedule` is abstract at `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py:1858-1859`; `sort_requests`/`_schedule_batch_mode` sit at `:412-420`. | **PORT** | Two default implementations that are provably inert for all five existing policies (`schedule_at` delegates; `on_replica_batch_end` returns `None`). | `BaseClusterScheduler` in `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py`. | Assert all five existing policies produce byte-identical mappings through `schedule_at` and `schedule`; assert `on_replica_batch_end` is a no-op for RR/LOR (existing monolithic + PDD smokes unchanged). | +| `frontier/events/cluster_schedule_event.py` | `cluster_scheduler.schedule()` → `cluster_scheduler.schedule_at(self.time)` (cand:42). | The routing call had no access to DES time. | `frontier/events/cluster_schedule_event.py:42` calls `schedule()`. It is the only caller of the *cluster* scheduler's `schedule()` in `frontier/` (`global_schedule_event.py:22` calls the *global* scheduler's `schedule()`, a different object). | **PORT** | Single call site; behavior-preserving given the `schedule_at` default. | `ClusterScheduleEvent.handle_event` in `frontier/events/cluster_schedule_event.py`. | Existing co-location/PDD/PD-AF example smokes must produce identical `request_metrics.csv` before/after. | +| `frontier/events/global_batch_end_event.py` | After `replica_scheduler.on_batch_end(self._batch)`, call `cluster_scheduler.on_replica_batch_end(self.time, self._replica_id, self._replica_local_id, self._batch)` (cand:181-183). | No post-step load feedback reached the cluster scheduler. | `frontier/events/global_batch_end_event.py:180` calls `replica_scheduler.on_batch_end(self._batch)` and nothing else. `cluster_scheduler` is already bound at `:96` and `self._replica_local_id` at `:63`. | **PORT** | Three lines, correct ordering (after the request-state transition, as `design_dp_load_balancing.md` §"Scope and interfaces" item 3 specifies), and inert without a policy that overrides the hook. | `GlobalBatchEndEvent.handle_event` in `frontier/events/global_batch_end_event.py`. | Assert the hook fires exactly once per completed batch and zero times for idle batches; assert RR/LOR runs are numerically unchanged. | +| `frontier/scheduler/replica_scheduler/base_replica_scheduler.py` | Adds `get_request_load(self) -> "RequestLoad"` raising `NotImplementedError` with the concrete class name, plus a `TYPE_CHECKING` import (cand:479-485, :24). | No shared accessor for waiting/running populations. | Not present on main. Nearest neighbours: `num_pending_requests` at `frontier/scheduler/replica_scheduler/base_replica_scheduler.py:506` and `peek_waiting_requests` at `:509`. | **PORT** | Additive; every existing subclass keeps working because nothing calls it by default. Note main added 28 unrelated lines to this file (`_get_memory_planner_max_num_seqs`, `_admitted_request_capacity`) in a different region — no textual conflict. | `BaseReplicaScheduler` in `frontier/scheduler/replica_scheduler/base_replica_scheduler.py`. | `test_vllm_dp_load_balancer.py::test_load_accessor_keeps_running_separate_from_schedulable_pending`; add a test that an unsupported scheduler raises `NotImplementedError` with its class name. | +| `frontier/scheduler/replica_scheduler/vllm_v1_engine_replica_scheduler.py` | Renames `_get_num_waiting_reqs_for_decision_log()` to `get_request_load() -> RequestLoad`, returning `(waiting, len(self._running_requests))` with the same `DECODE`/`DECODE_ATTN` branch; the decision logger now consumes `request_load.running` / `.waiting` (cand:2105-2111, :2142, :2154-2155). | Waiting and running counts were selected at two unrelated points in one payload dict. | Main keeps `_get_num_waiting_reqs_for_decision_log` at `frontier/scheduler/replica_scheduler/vllm_v1_engine_replica_scheduler.py:2136-2139` and uses it at `:2182`, with `len(self._running_requests)` read separately at `:2181`. | **ADAPT** — **the candidate's version is broken as-is** | `frontier/scheduler/replica_scheduler/sglang_style_replica_scheduler.py:65` calls `self._get_num_waiting_reqs_for_decision_log()`, and `SGLangStyleReplicaScheduler` subclasses `VLLMv1EngineReplicaScheduler` (`sglang_style_replica_scheduler.py:15`). The candidate deletes the method but leaves that caller intact — verified at CAND: `git grep _get_num_waiting_reqs_for_decision_log a7b3320` returns exactly one hit, the *caller*, with no definition anywhere. That is an `AttributeError` on the SGLang decision-log path. (The file is present in all three revisions; it is not a main-only addition.) Port the accessor, but update `sglang_style_replica_scheduler.py:64-65` in the same change, or keep `_get_num_waiting_reqs_for_decision_log` as a thin wrapper over `get_request_load().waiting`. | `VLLMv1EngineReplicaScheduler.get_request_load` and `_emit_schedule_decision_event` in `frontier/scheduler/replica_scheduler/vllm_v1_engine_replica_scheduler.py`; plus `SGLangStyleReplicaScheduler._emit_schedule_decision_event` in `frontier/scheduler/replica_scheduler/sglang_style_replica_scheduler.py`. | A test that instantiates the SGLang-style scheduler and emits one decision-log event (this exact break would have been caught by importing and calling it). Plus `test_vllm_dp_load_balancer.py::test_load_accessor_keeps_running_separate_from_schedulable_pending` and the same accessor across admission / preemption / completion. | +| `tests/unit/test_vllm_dp_load_balancer.py` (new, 131 lines, 11 tests) | Covers weighted score, fixed tie-break, local reservations, changed-count publication, previous-step snapshot, unchanged-count heartbeat, duplicate/suppressed reports, report-at-deadline tie, the load accessor, registry/config discoverability, and invalid-time rejection. | — | Not present on main. | **PORT** with W4 | Covers every branch of the state machine; pure and fast. | `tests/unit/test_vllm_dp_load_balancer.py`. | It *is* the test. | + +### Out of scope for W2/W4 but present in the audited config diff + +| candidate path | change summary | disposition | rationale | +| --- | --- | --- | --- | +| `frontier/config/config.py` (MoE routing runtime) | Adds `ReplicaConfig.moe_gating_routing_runtime_path` (cand:1955-1962), validates it in `__post_init__` (cand:2039-2042), and threads it through `ClusterConfig` construction (cand:4269, :5018); imports `validate_moe_gating_routing_runtime_path` from `frontier.moe_routing_runtime` (cand:48). | **DROP from this audit** (route to the MoE-routing work package) | Unrelated to W2/W3/W4. Main already has `resolve_moe_gating_routing_runtime_path` consumed by `frontier/execution_time_predictor/sklearn_moe_execution_time_predictor.py:279-280`, `:731-733` and `frontier/execution_time_predictor/shared_prediction_model_manager.py:1294`, `:1440`, but exposes no `ReplicaConfig` field. Whether main *should* gain the field is a separate decision with its own CLI-surface implications. | + +--- + +## (b) W2 detail — does the rotation-restart defect still exist on main? + +**Yes.** Verbatim from BASE `1f694f7`, `frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py:359-389`: + +```python +359 def _schedule_batch_mode(self) -> List[Tuple[int, int, Request]]: +360 """ +361 Original batch processing logic for prefill cluster and other cluster types. +362 Processes all requests in the queue at once using traditional round-robin. +363 """ +364 +365 # First, distribute requests to replicas using round-robin +366 replica_requests = [[] for _ in range(self._num_replicas)] +367 replica_ids = list(self._cluster.replicas.keys()) +368 +369 request_idx = 0 +370 while self._request_queue: +371 request = self._request_queue.pop(0) +372 replica_idx = (self._request_counter + request_idx) % self._num_replicas +373 replica_requests[replica_idx].append(request) +374 request_idx += 1 +375 +376 self._request_counter += request_idx +377 +378 # Distribute requests across logical DP lanes inside each Replica. +379 request_mapping = [] +380 for replica_idx, requests in enumerate(replica_requests): +381 if not requests: +382 continue +383 +384 replica_id = replica_ids[replica_idx] +385 for local_idx, request in enumerate(requests): +386 dp_id = local_idx % self._replica_dp_size +387 request_mapping.append((replica_id, dp_id, request)) +388 +389 return request_mapping +390 +``` + +Exact defect lines: **`:385-386`**. `local_idx` is produced by `enumerate(requests)` over the *per-call* bucket built at `:366-374`, so it restarts at `0` on every `schedule()` invocation. `:372` and `:376` show that only the *replica* index consults the persistent counter. + +Concrete consequence: with `num_replicas == 1` and `attn_dp == 2` (the co-location DP2 topology in the candidate's acceptance case), any scheduling cycle carrying a single request yields `local_idx == 0` → `dp_id == 0`. Lane 1 receives nothing from `_schedule_batch_mode`. This is exactly the one-request-per-cycle regime described in main's own comment at `round_robin_cluster_scheduler.py:416-419` for the DECODE path. + +**Counter semantics on main.** `self._request_counter` is initialized once at `frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py:27` (`self._request_counter = 0`) and is a persistent, monotonically increasing count of requests routed by this cluster scheduler instance. It is advanced in three places with two different meanings: + +- `:376` — `_schedule_batch_mode`: advanced by the number of requests routed this call; used only as a *replica* offset. +- `:443` — `_schedule_decode_lane_round_robin`: same advance, used as a combined replica+lane ordinal. +- `:484` and read at `:474`, `:588` — `_schedule_dynamic` / `_get_least_loaded_replica`: advanced one per request and used purely as a tie-break index among equally-loaded candidates. + +So the counter already carries a mixed meaning on main; the W2 change does not alter its increment rule (`+= request_idx`), only how it is decomposed. + +**Does an existing persistent counter already match the candidate's formula?** **Yes — in a different method.** BASE `frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py:426-444`: + +```python +426 def _schedule_decode_lane_round_robin(self) -> List[Tuple[int, int, Request]]: +427 """Schedule unified PD decode requests across Replica-local DP lanes.""" +428 replica_ids = list(self._cluster.replicas.keys()) +429 if not replica_ids: +430 return [] +431 +432 num_replicas = len(replica_ids) +433 request_mapping: List[Tuple[int, int, Request]] = [] +434 +435 request_idx = 0 +436 while self._request_queue: +437 request = self._request_queue.pop(0) +438 replica_idx = (self._request_counter + request_idx) % num_replicas +439 dp_id = (self._request_counter + request_idx) // num_replicas % self._replica_dp_size +440 request_mapping.append((replica_ids[replica_idx], dp_id, request)) +441 request_idx += 1 +442 +443 self._request_counter += request_idx +444 return request_mapping +``` + +Lines `:438-439` are algebraically identical to the candidate's `ordinal = self._request_counter + request_idx; replica_index = ordinal % num_replicas; dp_lane = (ordinal // num_replicas) % dp_size`. It uses the same persistent `_request_counter`. + +Reach: `_schedule_decode_lane_round_robin` is called only from `_schedule_decode_with_priority` (`:420`), which `schedule()` selects only for `ClusterType.DECODE` (`:75-76`). `_schedule_batch_mode` is the `else` branch at `:85-86`, i.e. it serves **`MONOLITHIC` (co-location) and `PREFILL`**. So main's correct formula exists but is unreachable from the two cluster roles the candidate's fix targets. + +One behavioral difference the port must preserve: the candidate keeps the per-Replica grouped return order (its test asserts `[row[0] for row in mapping] == sorted(...)` at cand `tests/unit/test_cluster_scheduler_dp_lanes.py`:93), whereas `_schedule_decode_lane_round_robin` returns strict arrival order. The candidate's `_schedule_batch_mode` therefore is not a drop-in call to `_schedule_decode_lane_round_robin`; consolidating the two into one helper would change the returned ordering for co-location/PREFILL. Whether downstream consumers depend on that ordering is **UNKNOWN** — settling it requires tracing every consumer of `ClusterScheduleEvent._request_mapping`, which I did not do. + +--- + +## (c) W3 detail — invariants the candidate implements, and their owners on main + +**Does main still split a shared monolithic forward by local request phase? Yes — at three independent layers.** + +1. **Entry routing.** `frontier/events/replica_stage_schedule_event.py:157-184`: `is_monolithic_prefill_moe` requires `batch.num_prefill_tokens > 0`; `is_monolithic_decode_moe` requires `batch.num_prefill_tokens <= 0 and batch.num_decode_tokens > 0`. These select `uses_prefill_sync_path` vs `uses_decode_sync_path`, i.e. `PrefillSyncEvent` vs `DecodeSyncEvent`, purely from the *local lane's* token composition. +2. **Waiting rooms.** `frontier/scheduler/utils/sync_state.py:29-34` allocates two independent `_new_sync_waiting_room()` trees for `MONOLITHIC`. `frontier/scheduler/utils/sync_entry.py:63` writes into `_prefill_sync_waiting_room[...]` and `:201` into `_decode_sync_waiting_room[...]`. +3. **Step-id namespace.** `frontier/scheduler/utils/forward_sync_state.py:38-41` partitions `_open_steps_by_kind` into `"prefill"` and `"decode"`; `sync_entry.py:52` and `:190` pass those kinds. Because `_next_step_id_by_replica` (`forward_sync_state.py:42`, allocated at `:133-139`) is keyed by `replica_id` only, two lanes of the same wall-clock forward with different phases resolve to two *consecutive but different* cohort IDs — the `240`/`241` pair recorded in `review_shared_forward_sync.md` §3. + +Consequence on main: each room waits for `expected_lanes = scheduler._replica_dp_size` (`sync_entry.py:73-76`, `:211-213`) and each holds exactly one of the two required lanes. Neither peer is idle, so the idle-materialization path at `sync_entry.py:76-120` / `:214-258` cannot fill the gap (`_can_supply_idle_lane` at `:8-14` requires the sibling stage to be not-busy). Both rooms stall. This matches the candidate's RCA and is not fixed on main. + +### Invariant → owner mapping + +| # | Invariant the candidate implements | Candidate mechanism (cand `a7b3320`) | Main function that would own it | Already handled on main? | +| --- | --- | --- | --- | --- | +| I1 | The shared layer key is `(cluster context, replica_id, stage_id, admitted_forward_group_id, layer_id)`; local phase is **not** part of the key. | `"forward"` sync kind (`forward_sync_state.py`:38-41, :56-59) + `base_cluster_scheduler.py`:903, :926 substituting it for `MONOLITHIC`. | `ForwardSyncState.__init__` / `_validate_kind` in `frontier/scheduler/utils/forward_sync_state.py`, driven from `enter_prefill_sync` / `enter_decode_sync` in `frontier/scheduler/utils/sync_entry.py:52`, `:190`. | **No.** `forward_sync_state.py:38-41` still has only two kinds. | +| I2 | One waiting room per monolithic forward regardless of entry phase. | `sync_state.py`:30-34 aliases `_decode_sync_waiting_room` to `_prefill_sync_waiting_room`. | `initialize_sync_waiting_rooms` in `frontier/scheduler/utils/sync_state.py`; the room selector at `frontier/scheduler/utils/ep_wave_schedule.py:184-188` must follow. | **No.** `sync_state.py:29-34` allocates two trees. | +| I3 | A request may not appear in two non-idle source lanes of one wave. | `ep_wave_inputs.py`:51-57 raises `"a request cannot belong to two EP source lanes"`. | `prepare_ep_wave_inputs` in `frontier/scheduler/utils/ep_wave_inputs.py`. | **No.** `ep_wave_inputs.py:44-50` validates only shared step id and non-empty non-idle set. | +| I4 | Ownership promotion→restoration happens **once per wave** over the whole source set, never once per phase. | `forward_collective.py`:17-24 restores once; both helpers accept `owners_restored=True` and skip their own restore (`prefill_collective.py`:113, `decode_collective.py`:113). | New `frontier/scheduler/utils/forward_collective.py`, consumed by `handle_prefill_sync_collective` / `handle_decode_sync_collective`. `_promote_forward_step_to_ep_wave` / `_restore_forward_step_full_stage_owners` stay at `base_cluster_scheduler.py:926-960`. | **Partially.** Main already restores once per helper invocation (`prefill_collective.py` and `decode_collective.py:103-110`), but there are two invocations for a mixed forward, so the 1:1 property does not hold. The `restored_full_stage_owners` return value and the `transition_stage_admission_for_layer` fallback (`decode_collective.py:128-132`) already exist and are the right seam. | +| I5 | Each live source predicts its **own** local attention with its own `Batch` (`predict_stage_execution_time(source_batch, ..., num_layers=1, layer_id=layer, include_ffn=False)` → `get_single_layer_attention_scope_time()`), so an intact mixed batch keeps its `attn_decode_in_mixed` feature. | `forward_collective.py`:26-38 loops over `source_batches.values()` and re-enters the per-source helper with `direct_batch=batch`. | `handle_prefill_sync_collective` / `handle_decode_sync_collective`. | **Partially, and only for prefill.** Main's `prefill_collective.py:235-238` predicts metrics timing from `batch` (the loop variable). But main's `decode_collective.py:147-150` still predicts `full_execution` from `sample_batch` and applies it to every batch at `:181`. So the decode side still borrows one source's timing. | +| I6 | Join time is `J = max_d R[d]`; the EP wave is consumed exactly once and each source experiences the common wave duration as latency, not as repeated resource work. | Unchanged planner; `ep_wave_schedule.py`:157-186 hoists `wave_time_ms` so both phases record it once per source. | `schedule_layer_wave` in `frontier/scheduler/utils/ep_wave_schedule.py` (join time already comes from `sync_time = max(sync_room["arrival_times"].values())`, `sync_entry.py:124`, `:261`). | **Partially.** The max-join already exists per room, but with two rooms there are two joins. `wave_time_ms` is still prefill-only on main (`ep_wave_schedule.py:155-160`). | +| I7 | Decode requests inside a mixed source advance their layer counter once per completed layer; prefill requests keep the prefill convention. | `prefill_collective.py`:99-106 (`r.is_prefill_complete` subset, guarded by `validate_decode_layer_advance`). | `handle_prefill_sync_collective`, with the shared guard in `frontier/scheduler/utils/collective_timing.py:42`. | **Only for pure-decode sources.** `decode_collective.py:81-86` does this for the decode entry path; the prefill entry path has no equivalent, so a decode request inside a mixed batch never advances. | +| I8 | Per-source explicit model-component accounting for **decode** as well as prefill; join wait is wall time, not operator compute. | `replica_stage_schedule_event.py`:540-543 seeds `_decode_model_execution_components_ms_by_stage`; `ep_wave_schedule.py`:139-140, :185-186 and `decode_collective.py`:137-140, :195-203 append and consume it. | `ReplicaStageScheduleEvent.handle_event`, `schedule_layer_wave`, `handle_decode_sync_collective`; the ledger itself arguably belongs on `frontier/entities/batch.py` beside the prefill twin. | **No.** `_decode_model_execution_components_ms_by_stage` has zero occurrences at `1f694f7`. Main has only `_prefill_model_execution_components_ms_by_stage` (`replica_stage_schedule_event.py:247`, `ep_wave_schedule.py:123`, `:164`, `prefill_collective.py:129`, `:199`). | +| I9 | One `BatchStageEndEvent` per live source at its own completion time; idle lanes emit none and have no request callbacks. | `forward_collective.py`:27-28 skips `batch.is_idle`; each helper emits its own stage-end. | `handle_prefill_sync_collective` / `handle_decode_sync_collective`. | **Yes, within one phase.** Main's `decode_collective.py:153-160` and the prefill twin both skip idle participants and emit one `BatchStageEndEvent` per live source. The property just does not span a mixed cohort. | +| I10 | Terminal chain `BatchStageEndEvent → ClusterBatchEndEvent → GlobalBatchEndEvent → Batch.on_batch_end → Request.on_batch_end` is unchanged; shared EP events never call `request.on_batch_end`. | Preserved by construction (no new event types). | Unchanged. | **Yes.** No candidate change touches this chain. | +| I11 | Dense layers inside a MoE model keep the existing full-stage FFN path — no EP collective is invented for them. | `ep_wave_schedule.py`:87-149 dense branch retained; only the per-source label changes. | `schedule_layer_wave` dense branch. | **Yes**, structurally — but main labels the dense continuation event with the *group* `mode` (`ep_wave_schedule.py:145`), so a decode source inside a prefill-mode group would be mislabelled. The candidate fixes the label via `_source_mode`. | + +Not implemented by the candidate and still open: nothing in the candidate enforces that a *sealed* group rejects a late real batch beyond the existing `StageExecutionContext.forward_group_sealed` check at `sync_entry.py:14`. The design note (§1 step 3) claims this, but I found no new code for it; it relies entirely on main's existing `_can_supply_idle_lane`. Marking the "sealed groups reject late membership" gate as **UNKNOWN** — evidence missing is a test that exercises a late arrival into a sealed mixed group. + +--- + +## (d) W4 detail — new public surface and integration points + +### New public surface + +| Kind | Symbol / token | Location (CAND) | +| --- | --- | --- | +| Enum value | `ClusterSchedulerType.VLLM_LOAD_BALANCING = 5` | `frontier/types/cluster_scheduler_type.py:10` | +| CLI token | `--cluster_scheduler_config_type vllm_load_balancing` | Derived: `BaseIntEnum.__str__` returns `self.name.lower()` (`frontier/types/base_int_enum.py:5-6`); `frontier/config/flat_dataclass.py:132-139` matches `str(subclass.get_type())` against the `_type` flag. Field is `ClusterConfig.cluster_scheduler_config` (`config.py:2440-2441` CAND / `:2474-2475` BASE). | +| Config dataclass | `VllmLoadBalancingClusterSchedulerConfig` | `frontier/config/cluster_scheduler_config.py:49-53` | +| Config module (moved) | `BaseClusterSchedulerConfig`, `Random…`, `RoundRobin…`, `LOR…`, `StickyRoundRobin…`, `StickyLOR…` relocated from `config.py` | `frontier/config/cluster_scheduler_config.py:9-46`, re-imported at `frontier/config/config.py:11-18` | +| Registry entry | `ClusterSchedulerRegistry.register(ClusterSchedulerType.VLLM_LOAD_BALANCING, VllmLoadBalancingClusterScheduler)` | `frontier/scheduler/cluster_scheduler/cluster_scheduler_registry.py:35-37` | +| Scheduler class | `VllmLoadBalancingClusterScheduler` | `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py:12` | +| Value type | `RequestLoad(NamedTuple)` with `waiting`, `running` | `frontier/scheduler/request_load.py:6-10` | +| Policy engine | `VllmDPLoadBalancer` with `report()`, `select()` | `frontier/scheduler/utils/vllm_dp_load_balancer.py:12-84` | +| Base-class API | `BaseClusterScheduler.schedule_at(time)`, `BaseClusterScheduler.on_replica_batch_end(time, replica_id, replica_local_id, batch)` | `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py:418-427` | +| Base-class API | `BaseReplicaScheduler.get_request_load() -> RequestLoad` | `frontier/scheduler/replica_scheduler/base_replica_scheduler.py:479-485` | +| Renamed (breaking) | `VLLMv1EngineReplicaScheduler._get_num_waiting_reqs_for_decision_log` → `get_request_load` | `frontier/scheduler/replica_scheduler/vllm_v1_engine_replica_scheduler.py:2105` | + +**No new CLI flags beyond the existing `_type` selector.** The design note states this explicitly (`design_dp_load_balancing.md`: "New public flags beyond selecting the policy are unnecessary") and the code agrees — `VllmLoadBalancingClusterSchedulerConfig` declares no fields. + +### Integration points touched + +1. `frontier/events/cluster_schedule_event.py:42` — `schedule()` → `schedule_at(self.time)`. Sole cluster-scheduler `schedule()` call site in `frontier/`. +2. `frontier/events/global_batch_end_event.py:181-183` — new `cluster_scheduler.on_replica_batch_end(...)` immediately after `replica_scheduler.on_batch_end(self._batch)` (main `:180`). `cluster_scheduler` already bound at main `:96`; `self._replica_local_id` at main `:63`. +3. `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py:418-427` — the two default seams. +4. `frontier/scheduler/replica_scheduler/base_replica_scheduler.py:479-485` + `TYPE_CHECKING` import at `:24`. +5. `frontier/scheduler/replica_scheduler/vllm_v1_engine_replica_scheduler.py:2105-2111`, `:2142`, `:2154-2155` — accessor + decision-log consumption. +6. `frontier/scheduler/cluster_scheduler/cluster_scheduler_registry.py:16-18`, `:35-37`. +7. `frontier/config/config.py:11-18` + the deleted `:2106-2145` block. +8. `frontier/scheduler/utils/replica_state.py:21` — **read-only dependency**, unchanged by the candidate. `VllmLoadBalancingClusterScheduler.__init__` relies on `self._replica_scheduler_type` being set there. Byte-identical on main and CAND, so the guard works. +9. **Missed integration point:** `frontier/scheduler/replica_scheduler/sglang_style_replica_scheduler.py:65` still calls the deleted `_get_num_waiting_reqs_for_decision_log()`. See the W4 table row. + +### Is `ForwardSyncState.get_step_id(batch)` used as a report-order key, and is that identity valid on main? + +**Yes, it is used as the ordering key.** `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py:52-56` (CAND) passes `ForwardSyncState.get_step_id(batch)` as the `step` argument of `VllmDPLoadBalancer.report`. That value drives the snapshot latch at `vllm_dp_load_balancer.py:64-67`: + +```python +64 if step > self.last_report_step: +65 if self.stats_changed: +66 self.last_step_counts = list(self.engine_counts) +67 self.last_report_step = step +``` + +`last_report_step` is a **single scalar shared across all engines** (`vllm_dp_load_balancer.py:22`), so the key must be a per-*Replica* (not per-lane) monotonic forward counter for this latch to mean "a newer engine step began". + +**What `get_step_id` actually returns on main** (`frontier/scheduler/utils/forward_sync_state.py:44-54`): `batch._forward_cohort_id` if it is a non-negative `int`, else `batch.global_id`, else `ValueError`. + +Validity per case, all against BASE `1f694f7`: + +- **Dense models — INVALID as an ordering key.** Every batch gets `batch._forward_cohort_id = lane_batch_counter` at `frontier/scheduler/replica_scheduler/base_replica_scheduler.py:463`, where `lane_batch_counter = self._batch_creation_counter` (`:459`, incremented at `:483`) is **per replica-scheduler instance, i.e. per DP lane**. For a dense model `sync_state.py:36-37` sets both waiting rooms to `None`, so `ForwardSyncState.resolve_step` never runs and the value is never promoted to the Replica-scoped `_next_step_id_by_replica` counter. With `attn_dp > 1` both lanes independently emit `0, 1, 2, …`; lane 1's step `0` arriving after lane 0's step `5` fails `step > self.last_report_step`, so the previous-step snapshot is silently never latched. The candidate's constructor does **not** require a MoE model (`vllm_load_balancing_cluster_scheduler.py:17-26` checks only cluster type, replica count, PP, and replica-scheduler type), so dense + DP2 is reachable. +- **MoE with DP > 1 on main (without W3) — INVALID for the "same step" semantics.** `resolve_step` allocates from `_next_step_id_by_replica` (`forward_sync_state.py:133-139`), which is keyed by `replica_id` only, but the *binding key* at `:100-106` includes `sync_stage` and the lane's `provisional_id`, and the open-steps dict is partitioned by `sync_kind` (`:107`). Two lanes of one wall-clock forward with different local phases therefore get two different resolved IDs (`240`/`241`, `review_shared_forward_sync.md` §3). The candidate's design assumes both lanes of one forward report the *same* step (`design_dp_load_balancing.md`: "Use the existing synchronized forward-cohort ID as a monotonic ordering key"). That assumption holds only once W3 lands. **W4's report key is therefore dependent on W3.** +- **MoE with DP == 1 — VALID.** Single lane, and `resolve_step` promotes the ID to the Replica-scoped counter, so the value is monotonic per Replica. +- **Dense with DP == 1 — incidentally valid.** One lane means the lane-local counter is also the Replica counter. +- **Idle participants — not exercised, therefore fine.** Idle batches get `_forward_cohort_id = step_id` at `sync_entry.py:99`/`:237` (a valid int), but they have no requests, are skipped before any `BatchStageEndEvent` is emitted (`decode_collective.py:153-160` and the prefill twin; `forward_collective.py:27-28` in CAND), and therefore never reach `GlobalBatchEndEvent` and never call `on_replica_batch_end`. I found no path that reports an idle batch. Confidence: derived from the emit-site skips, not from a test — I did not find a regression pinning it. +- **Additional semantic caveat (both trees).** `_forward_cohort_id` is *mutated per layer*: `resolve_step` re-resolves a fresh ID for each `(…, layer_id, …)` binding and `close_step` pushes `_next_step_id_by_replica` to `provisional_id + 1` (`forward_sync_state.py:133-139`, `:163-166`); `sync_entry.py:64` documents this as "step_id advances per layer". So the value read at `GlobalBatchEndEvent` time is the *final layer's* cohort ID, and consecutive engine steps are roughly `num_layers` apart. That is still monotonic per Replica, so ordering works, but the value is not a vLLM step counter — which the design note already concedes ("Its numeric value is not a vLLM step counter; only ordering/equality are used"). + +**Net:** the key is usable only for MoE + W3, or DP1. Before porting W4, either restrict the policy's topology guard to those cases, or introduce an explicit per-Replica engine-step counter instead of reusing the layer-advancing cohort ID. I am not proposing which; both are visible from the source. + +--- + +## (e) Candidate changes that are clearly experiment scaffolding → DROP + +None of these are in the audited production paths; they are listed because they dominate the candidate diff and must not be carried along with W2/W3/W4. + +1. **All 52 `tests/e2e/issue26_*` files** (`issue26_h200_*.sh`, `issue26_h800_*.sh`, `issue26_nsys_*.py`, `issue26_first_batch_op_rca_*.py`, `issue26_official_ttft_analysis.py`, `issue26_cpu_frontier_worker.py`, …). Single-case H200/H800 GPU workers and one-off RCA analysis scripts for the frozen Qwen3 4096/1024 calibration case. They are investigation receipts, not repeatable regression checks. +2. **All 5 `tests/integration/issue26_*` files** — `issue26_dp_coordinator_reference.py`, `issue26_dp_snapshot_diagnostic.py`, `issue26_dp_workflow_rca_controls.py`, `issue26_dp_workflow_rca_observe.py`, `issue26_dp_workflow_rca_trace.py`. Diagnostic drivers for the DP-routing RCA. `issue26_dp_snapshot_diagnostic.py:30` reaches into `scheduler.get_replica_scheduler(replica_id, lane).get_request_load()` directly — useful during the investigation, not a durable gate. +3. **All 7 `tests/performance/issue26_*` files** — `issue26_h200_*_worker.sh`, `issue26_attention_exact_profile.py`, `issue26_moe_exact_profile.py`, `issue26_moe_profile_dataset.py`, `issue26_linear_timing_context.py`, `issue26_prefill_clock_drift_probe.py`, `issue26_prefill_endpoint_probe.py`. Profile-collection and clock-drift probes tied to specific hardware runs. +4. **The `task_memory/task_2026-09-07_issue26_ttft_h200/` tree** (design/review/analysis/test-report notes). Main deliberately removed local task memory from the published repository (commit `26b490a`, "chore: remove local task memory from published repository"); re-adding it would reverse that decision. The three notes read for this audit remain valuable as *inputs* but belong in the local task directory, not in the branch. + +Keep from `tests/`: only `tests/unit/test_cluster_scheduler_dp_lanes.py` (W2), `tests/unit/test_monolithic_mixed_forward_sync.py` (W3), and `tests/unit/test_vllm_dp_load_balancer.py` (W4). + +--- + +## Cross-cutting notes and UNKNOWNs + +- **Ordering constraint.** W3 must land before W4 for the report key to be well-defined (see (d)). W2 is independent of both. +- **Merge friction.** Main and the candidate both edited `base_cluster_scheduler.py`, `ep_wave_schedule.py`, `prefill_collective.py`, `decode_collective.py`, `base_replica_scheduler.py`, `vllm_v1_engine_replica_scheduler.py`, and `config.py`. The hard blocker is main's removal of `BaseClusterScheduler._create_corrected_execution_time_for_metrics` and the signature change of `_create_prefill_corrected_execution_time_for_metrics` (now `(sample_batch, stage_id, original_execution_time)` at `base_cluster_scheduler.py:1138`); the candidate's `decode_collective.py` final-metrics hunk calls the removed method. +- **Main's newer metrics work has no candidate counterpart.** `metrics_store.ep_wave_reporting_enabled` / `stage_execution_reporting_enabled` (`frontier/metrics/metrics_store.py:478`, `:493`), `capture_lane_timings`, and `metrics_store.on_ep_wave_schedule` (`ep_wave_schedule.py:69-74`) are all main-only. Any W3 port must thread `metrics_store` through `forward_collective.py`, which the candidate's signature does accept but does not use for wave reporting. +- **UNKNOWN — numerical acceptance.** The candidate's own notes state full-case numerical validation was still pending (`design_shared_forward_sync.md` §"Review result and limits": "full-case numerical validation remains pending"; `review_shared_forward_sync.md` §3 marks "Fix closes numerical TTFT gate" as *Unknown*). I found no committed artifact establishing the ≤10 % mean-TTFT gate. Missing evidence: a completed fresh 100-request 4096/1024 replay with error tables. +- **UNKNOWN — PP > 1 under W3.** Both the prefill (`replica_stage_schedule_event.py:247`) and the candidate's decode ledger are plain dicts seeded with a single stage key. No test in either tree exercises a monolithic MoE shared forward with `num_pipeline_stages > 1`. Missing evidence: a PP2 monolithic MoE run or an explicit guard. +- **UNKNOWN — whether `_schedule_batch_mode`'s per-Replica grouped return order is load-bearing.** Settling it requires tracing every consumer of `ClusterScheduleEvent._request_mapping`; not done in this audit. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index e2edd5e4..d489c1bd 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -13,16 +13,16 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | Refactor PR fidelity matrix PASS (see `task_memory/task_2026-09-21_oversized_module_split/progress.md`) | -| Current step | Step 0 complete pending user review | +| Current step | Step 1 complete (audit); Steps 2-7 wait on the refactor branch | | Publication | PUSHED_VERIFIED (records) | -| Next action | After user review: refactor task Step 1 (fidelity harness + baseline capture). Correctness Step 1 (candidate/vLLM audit, draft PR) may start in parallel because it is read-only. | +| Next action | Open the draft PR, then start Step 2 (RR DP rotation) once the refactor branch has landed its splits. Decisions D1 and D2 in `review.md` are open. | ## Step status | Step | Work package | Status | Test | Publication | User review | | --- | --- | --- | --- | --- | --- | | 0 | Worktree, references, baseline | PASS | PASS (baseline recorded) | PUSHED_VERIFIED | NOT_REVIEWED | -| 1 | Candidate/vLLM audit | NOT_STARTED | — | — | — | +| 1 | Candidate/vLLM audit | PASS | n/a (source audit) | LOCAL_ONLY | NOT_REVIEWED | | 2 | RR DP rotation | NOT_STARTED | — | — | — | | 3 | Shared monolithic forward | NOT_STARTED | — | — | — | | 4 | Opt-in vLLM DP placement | NOT_STARTED | — | — | — | @@ -35,3 +35,4 @@ - 2026-09-21: Baseline on the shared base recorded in the refactor task's Step 0 report; vLLM reference cloned (no tags in the fork; upstream `v0.10.2` comparison deferred to Step 1). - 2026-09-21: Draft specification analyzed; eleven facts verified against main, the candidate, the submodule remote, and the host; planning interview settled twelve decisions (see `requirements.md`). Records landed under `task_memory/`, `.gitignore` narrowed, `plan.md` carries the Amendments table. +- 2026-09-21: Step 1 audit complete. Three pinned-source audits landed as `audit_scheduler.md`, `audit_predictor_profiling.md`, `reference_vllm_0_10_2.md`; dispositions and two decision checkpoints recorded in `review.md`. Execution order corrected to W2 first, then W3, then W4, because the candidate's report-order key depends on the shared forward identity. The vLLM fork was confirmed to be a direct descendant of upstream v0.10.2 with the DP placement files byte-identical to the tag. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/reference_vllm_0_10_2.md b/task_memory/task_2026-09-21_issue26_correctness_pr/reference_vllm_0_10_2.md new file mode 100644 index 00000000..2a0a151e --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/reference_vllm_0_10_2.md @@ -0,0 +1,522 @@ +# Reference Behavior: pinned vLLM checkout + +Authoritative reference table for validating a Frontier simulator change. +All citations are `path:line` relative to the checkout root +`/data/ycfeng/Frontier/.real-engine/vLLM-BS` unless written absolute. +No server was started; this is pure source reading at a fixed revision. + +--- + +## Part 1 — Upstream relationship + +### 1.1 Identity + +| Item | Value | How established | +|---|---|---| +| Checkout | `/data/ycfeng/Frontier/.real-engine/vLLM-BS` | — | +| HEAD (verified) | `ea95f571e20937c7c908c6d59ddd1cd6bf9268f1` | `git rev-parse HEAD` | +| HEAD subject | `Instrument vLLM attention ops for Frontier calibration` | `git log --oneline -1` | +| Working tree | clean (no modified/untracked files) | `git status --short` → empty | +| `origin` | `https://github.com/fwyc0573/vLLM-BS.git` | `git remote -v` | +| `upstream` (added read-only this session) | `https://github.com/vllm-project/vllm.git` | `git remote add upstream …` | +| Tag fetch | **SUCCEEDED** — `refs/tags/v0.10.2` → local `refs/tags/upstream-v0.10.2` | `git fetch --no-tags upstream refs/tags/v0.10.2:refs/tags/upstream-v0.10.2` | +| **Resolved commit of upstream v0.10.2** | **`01efc7ef781391e744ed08c3292817a773d654e6`** (lightweight tag; `git cat-file -t` → `commit`) | `git rev-parse refs/tags/upstream-v0.10.2^{commit}` | +| **merge-base(HEAD, v0.10.2)** | **`01efc7ef781391e744ed08c3292817a773d654e6`** — i.e. the tag commit itself | `git merge-base HEAD refs/tags/upstream-v0.10.2^{commit}` | +| **Is HEAD a descendant of the tag?** | **YES.** `git merge-base --is-ancestor` returns 0 | — | +| Distance | HEAD is **40 commits ahead, 0 behind** | `git rev-list --left-right --count v0.10.2...HEAD` → `0 40` | + +Because the merge-base equals the tag commit and HEAD is behind by zero, the +fork is a **clean linear/merge-free descendant of v0.10.2**: nothing from +upstream after v0.10.2 is present, and no upstream commit was dropped. The tag +comparison below is therefore exact, not approximate. + +`vllm/version.py:5` reads the version from a generated `vllm/_version.py` +(setuptools-scm) and falls back to `"dev"` at `vllm/version.py:13`; it carries +no hard-coded version string, so the git tag comparison — not the source tree — +is the authority here. + +### 1.2 Files differing from v0.10.2, inside the requested paths + +Requested paths: `vllm/v1/engine/`, `vllm/v1/core/sched/`, `vllm/forward_context.py`, +`vllm/v1/worker/gpu_model_runner.py`, `vllm/model_executor/layers/fused_moe/`, +`vllm/model_executor/models/qwen3_moe.py`. + +**Unchanged (byte-identical to v0.10.2) — important for Part 2 A/B/E:** + +| File | Status | +|---|---| +| `vllm/v1/engine/core_client.py` | **UNCHANGED** — every answer in section A is pristine upstream v0.10.2 behavior | +| `vllm/v1/engine/coordinator.py` | **UNCHANGED** — every answer in section B is pristine upstream v0.10.2 behavior | +| `vllm/forward_context.py` | **UNCHANGED** | +| `vllm/v1/core/sched/{async_scheduler,interface,output,request_queue,utils}.py` | **UNCHANGED** | +| `vllm/v1/engine/{__init__,async_llm,llm_engine,utils,detokenizer,logprobs,exceptions,parallel_sampling}.py` | **UNCHANGED** | +| `vllm/distributed/device_communicators/all2all.py` | **UNCHANGED** | + +Verified per-file with `git diff --numstat HEAD -- ` returning no rows. + +**Changed, with characterization** (`+add/-del` from `git diff --numstat`): + +| File | +/- | Characterization | +|---|---|---| +| `vllm/v1/engine/core.py` | +43/-0 | **BEHAVIOR CHANGE, env-gated (default OFF).** Adds a "wait for all initial requests before the first scheduling step" barrier in `EngineCoreProc._process_input_queue`, gated on `VLLM_FRONTIER_WAIT_INITIAL_REQUESTS=="1"` **and** `VLLM_FRONTIER_EXPECTED_NUM_REQUESTS>0` (`core.py:61-67`, guard at `core.py:763-766`). With defaults (`"0"` / `0`) the branch is dead and upstream control flow is bit-identical. | +| `vllm/v1/engine/output_processor.py` | +71/-2 | **Instrumentation.** Populates the previously-unused `RequestOutput.metrics` field with a new `FrontierRequestMetrics` record, built only for finished leaf requests (`output_processor.py:258-262`). It reads existing `RequestStateStats` timestamps; it does not alter token generation, stopping, or detokenization. | +| `vllm/v1/engine/processor.py` | +8/-1 | **BEHAVIOR CHANGE, env-gated (default OFF).** `VLLM_FRONTIER_FORCE_ARRIVAL_TIME_ZERO=="1"` pins `arrival_time = 0.0` instead of `time.time()` (`processor.py:33-35`, `processor.py:342-345`). Default `"0"` preserves upstream. | +| `vllm/v1/core/sched/scheduler.py` | +471/-8 | **Mostly instrumentation + 2 real behavior changes.** See 1.3. | +| `vllm/v1/worker/gpu_model_runner.py` | +514/-65 | **Mixed: env-gated instrumentation + 6 UNGATED behavior changes.** See 1.4. The behavior changes are inert on a non-spec-decode, PP=1 run, but they are inert because the spec-decode predicate is false — **not** because they are flag-gated. | +| `vllm/model_executor/models/qwen3_moe.py` | +206/-21 | See 1.3 / section F. | +| `vllm/model_executor/layers/fused_moe/fused_moe.py` | +230/-146 | See 1.3 / section F. | +| `vllm/model_executor/layers/fused_moe/layer.py` | +125/-89 | See 1.3 / section F. | +| `vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py` | +32/-27 | See 1.3 / section F. | +| `vllm/model_executor/layers/fused_moe/deepep_ll_prepare_finalize.py` | +23/-17 | See 1.3 / section F. | +| `vllm/model_executor/layers/fused_moe/pplx_prepare_finalize.py` | +34/-27 | See 1.3 / section F. | +| `vllm/model_executor/layers/fused_moe/configs/README → configs/specific-README` | 0/0 | **Pure rename**, 100% similarity (`git diff --find-renames --summary`). No content change. | + +### 1.3 `vllm/v1/core/sched/scheduler.py` — the only 3 non-instrumentation edits + +The other ~460 added lines are logging scaffolding: a `VLLM_FLOW_VALIDATION` +logger (`scheduler.py:41-75`), a `VLLM_FRONTIER_SCHED_LOG_PATH` / +`VLLM_FRONTIER_SCHED_DECISION_LOG_PATH` JSONL logger (`scheduler.py:87-96`), and +`_emit_frontier_schedule_decision` (`scheduler.py:283-334`). All are no-ops when +the env vars are unset (`_log_flow` early-returns; `_emit_frontier_schedule_decision` +returns at `scheduler.py:299-300` when the logger is `None`). + +| # | Change | HEAD line | Upstream | Verdict | +|---|---|---|---|---| +| 1 | `skipped_waiting_requests.prepend_request(request)` → `.add_request(request)` at **6 call sites** | `scheduler.py:645, 656, 666, 690, 727, 800` | `prepend_request` | **BEHAVIOR CHANGE, always on.** Commit `dcb9709e0` "Fix skipped-waiting FCFS order in scheduler". The temporary `skipped_waiting_requests` queue is re-prepended wholesale to `self.waiting` at `scheduler.py:882-883`; building it with `prepend_request` reversed the relative order of skipped requests, `add_request` preserves FCFS order. Changes the order requests are retried, not which requests are admitted. | +| 2 | `preempted_req = self.running.pop()` → `preempted_req = self.running[-1]` … `self.running.pop()` | peek at `scheduler.py:493`, pop at `scheduler.py:507` | single `pop()` | **Instrumentation only / behaviorally identical.** Split into peek + pop so the victim's pre-removal state can be logged (`scheduler.py:494-505`); the element is still removed at `:507`. | +| 3 | `num_accepted = len(generated_token_ids) - 1` → `min(_get_num_accepted_spec_tokens(...), num_draft_tokens)` | `scheduler.py:1298-1301`, helper at `scheduler.py:78-84` | bare `len(...) - 1` | **BEHAVIOR CHANGE, always on.** Clamps below at 0 (`return max(len(generated_token_ids) - 1, 0)`, `scheduler.py:84`) and above at `num_draft_tokens`. Feeds `num_rejected` at `scheduler.py:1302`, which decrements `request.num_computed_tokens` at `:1308`. Only reachable with speculative decoding enabled; on the non-spec path `scheduled_spec_token_ids` is falsy and the block is skipped. | + +### 1.4 `vllm/v1/worker/gpu_model_runner.py` — instrumentation vs ungated behavior + +Abbreviated `GMR` below. Instrumentation clusters (all default-off): + +| Cluster | HEAD lines | Gate / default | +|---|---|---| +| Frontier env flags, `frontier_trace` / CUDA-event / MoE-routing logger imports | `GMR:6-7, 13, 17-42, 52, 117-120` | `VLLM_FRONTIER_INSTRUMENTATION`, default `"0"` (`GMR:18-19`) | +| Instrumentation config validation + logger construction in `__init__` | `GMR:537-620` | same; `VLLM_FRONTIER_RUNTIME_META_ENABLED` default `0`, `VLLM_FRONTIER_OP_TIMING_MODE` default `"record_function"` | +| Per-batch trace/metric capture in `execute_model` (CUDA events, JSONL batch log, PP-boundary timestamps, scopes around `set_forward_context`) | `GMR:2240-2245, 2302-2391, 2394-2396, 2407-2459` | `FRONTIER_INSTRUMENTATION_ENABLED and frontier_trace.is_active()` (`GMR:2302-2303`). **When ON it forces `torch.cuda.synchronize()` at `GMR:2428`** — perturbs timing, not values | +| Per-request positions metadata in `_prepare_inputs` | `GMR:1131-1145` | `FRONTIER_RUNTIME_META_ENABLED` (default off) | +| New `get_frontier_batch_metrics()` / `clear_frontier_batch_metrics()` | `GMR:2571-2589` | additive; returns `[]` when off (`GMR:2581-2582`) | +| Attention-backend admission check in `initialize_attn_backend` | `GMR:3638-3671` | `FRONTIER_INSTRUMENTATION_ENABLED` | + +**Ungated behavior changes** — no env var, no config flag: + +| # | Change | HEAD lines | Inert when | +|---|---|---|---| +| 1 | Runner buffers resized by `_get_max_num_input_tokens = max_num_batched_tokens + max_num_reqs * num_speculative_tokens`; applied to `InputBatch`, `input_ids`, `positions`, `inputs_embeds`, `mrope_positions`, `arange_np`, `kv_sharing_fast_prefill_logits_indices`, `make_empty_intermediate_tensors` | def `GMR:191-197`; uses `GMR:319-328, 413, 443, 445, 453, 475, 492, 505, 3196, 3788` | `speculative_config is None` (term becomes `max_num_reqs * 0`) | +| 2 | `_prepare_inputs` folds `num_draft_tokens` into per-request input token counts, changing `req_indices`, `positions`, `input_ids`, slot mapping, `query_start_loc`, `seq_lens`, `num_actual_tokens`, `max_query_len` | `GMR:1102-1114, 1119-1127, 1164-1169, 1178-1190, 1199, 1211-1213, 1255-1281, 1326-1330`; `_calc_spec_decode_metadata` `GMR:1476-1501`; helper `GMR:176-188` | `num_draft_tokens is None` → helper returns input unchanged (`GMR:186-188`) | +| 3 | Cudagraph/DP padding keyed on a draft-inclusive count | `GMR:1977-1998` (`total_num_input_tokens = num_scheduled_tokens + sum(len(token_ids) for token_ids in scheduler_output.scheduled_spec_decode_tokens.values())`), `:2016, :2021` | no scheduled spec tokens → sum is 0 | +| 4 | EAGLE draft request reordering: `drafter.prepare_inputs` returns a 3-tuple incl. `request_order`; `_apply_request_order` / `_restore_request_order` | def `GMR:200-227`; uses `GMR:2648, 2670-2671, 2682-2688, 2699-2700`; paired with forked `vllm/v1/spec_decode/eagle.py:624-629` (upstream returned a 2-tuple) | EAGLE not in use | +| 5 | Mixed dummy-batch layout clamp: upstream `num_decode_tokens = num_tokens // 2` → `min(num_tokens // 2, max(max_num_reqs - 1, 0))` | `GMR:164-173` | `num_tokens // 2 <= max_num_reqs - 1`; affects **warmup shape only** | +| 6 | `hasattr(self, "drafter")` guards around `drafter.dummy_run` and `validate_same_kv_cache_group` | `GMR:3235-3236, 4028-4029` | PP=1; `self.drafter` only exists under `if self.speculative_config and get_pp_group().is_last_rank:` (`GMR:379-390`) — this is a crash fix for PP>1 + spec decode | + +Also outside the requested paths but relevant: `vllm/distributed/parallel_state.py` (+11/-0) wraps +`GroupCoordinator.dispatch`/`combine` in `record_function_or_nullcontext("expert_parallel_alltoall_dispatch" / "..._combine")` when `self.unique_name.startswith("ep:")` (`parallel_state.py:843-848, 856-860`). Instrumentation only — `record_function_or_nullcontext` is a `nullcontext` unless `VLLM_CUSTOM_SCOPES_FOR_PROFILING` or the Frontier contextvar logger is live (`vllm/v1/utils.py:1179-1191`). + +--- + +## Part 2 A — `vllm/v1/engine/core_client.py` (file is byte-identical to v0.10.2) + +Class picked at runtime: `CoreEngineProcManager.make_async_mp_client` returns +`DPLBAsyncMPClient` for `data_parallel_size > 1` **without** external LB +(`core_client.py:96-101`), `DPAsyncMPClient` for external LB (`:99`), and plain +`AsyncMPClient` for DP=1 (`:102`). Only `DPLBAsyncMPClient` scores engines. + +| # | Question | Answer | file:line | Verbatim | +|---|---|---|---|---| +| A1 | How are DP engines **ordered**? | Engines are held in **rank order** in `self.core_engines`, built from `engine_ranks_managed = range(dp_rank, dp_rank + num_ranks)`. The scan is a full linear pass over all of them, but **rotated** to start at `self.eng_start_index` so that different frontends break ties on different engines. Ties keep the **first** engine visited in rotated order (strict `<`). | order: `core_client.py:480-489`; comment "Engines are in rank order." `core_client.py:1134`; rotation `core_client.py:1141-1144`; start index `core_client.py:1129-1130` | `idx = (self.eng_start_index + i) % num_engines` / `self.eng_start_index = (len(self.core_engines) * self.client_index) // client_count` | +| A1 | How are they **scored**? | Each engine carries a `[waiting, running]` pair; the score is a **weighted sum with waiting weighted 4x**, and the minimum score wins. | `core_client.py:1145-1149` | `waiting, running = current_counts[idx]`
`score = waiting * 4 + running`
`if score < min_score:`
` min_score = score`
` eng_index = idx` | +| A2 | What does a single frontend **reserve locally** after picking? | It increments **only the `waiting` element** of its own local `lb_engines` row for the chosen engine, **by `self.client_count`** (not by 1) — a deliberate over-reservation so that N frontends racing between coordinator updates spread out rather than all piling onto the same engine. `running` is not touched. | `core_client.py:1150-1152` | `# Increment local waiting count for better balancing between stats`
`# updates from the coordinator (which happen every 100ms).`
`current_counts[eng_index][0] += self.client_count` | +| A2 | When is that reservation **reset/replaced**? | It is **never decremented**. It is **wholesale replaced** — not merged — the next time the coordinator publishes counts: `self.lb_engines` is rebound to a freshly decoded slice, discarding every local increment accumulated since the last update. Note `current_counts` at `:1136` is a *reference* to `self.lb_engines`, so the `+=` mutates the live list until that rebind. | replace: `core_client.py:1074-1078`; alias: `core_client.py:1136`; init: `core_client.py:979` | `counts, wave, running = msgspec.msgpack.decode(buf)`
`…`
`sliced_counts = counts[count_slice]`
`self.lb_engines = sliced_counts` | +| A3 | Are coordinator updates applied as **replace** or **incremental adjust**? | **Full replace (absolute snapshot).** The coordinator sends absolute `[waiting, running]` counts for *all global* engines; the client slices out the engines it manages and rebinds `self.lb_engines` to that slice. There is no delta arithmetic anywhere. Additionally, the receive loop **drains the socket and keeps only the latest** message before applying it, so intermediate snapshots are dropped. | rebind `core_client.py:1078`; slice `core_client.py:1005-1006`; drain `core_client.py:1065-1073` | `count_slice = slice(self.engine_ranks_managed[0],`
` self.engine_ranks_managed[-1] + 1)`
and `# Drain all stats events (we only care about latest).` | +| A4 | What changes with **multiple frontends**? | Three things, all driven by `client_count`/`client_index`: (a) the scan **start offset** differs per frontend, so empty engines are not all picked in the same order; (b) the local reservation step is **`client_count`** instead of 1, modelling the other frontends' likely concurrent picks; (c) each frontend applies only its own **`count_slice`** of the global count vector. The frontends never talk to each other — the only shared state is the coordinator's periodic broadcast. | (a) `core_client.py:1129-1130`, `:1144`; (b) `core_client.py:1152`; (c) `core_client.py:1005-1006`, `:1077` | `self.eng_start_index = (len(self.core_engines) * self.client_index) // client_count` | +| A4 | What changes with an **explicit DP rank**? | The walrus at `:1135` short-circuits: if `request.data_parallel_rank is not None`, `eng_index` is taken directly from it and **the entire scoring loop AND the local `waiting` reservation are skipped** — a pinned request neither consults nor perturbs the load-balancer state. The rank is range-validated earlier in the frontend. | skip `core_client.py:1135`; chosen `core_client.py:1154`; validation `vllm/v1/engine/processor.py:336-340` | `if (eng_index := request.data_parallel_rank) is None:`
validation: `if data_parallel_rank is not None and not (0 <= data_parallel_rank < data_parallel_size): raise ValueError(...)` | +| A4 | External-LB mode | With `data_parallel_external_lb`, the plain `DPAsyncMPClient` is used and selection is a constant — no scoring at all, and (per `coordinator.py:53-55`) the engines publish no stats. | `core_client.py:97-99`; `core_client.py:1103-1104` | `def get_core_engine_for_request(self, request: EngineCoreRequest):`
` return self.core_engine` | + +Also recorded for abort routing: `self.reqs_in_flight[request.request_id] = chosen_engine` +(`core_client.py:1156`), cleared on finish at `core_client.py:1167-1171`. + +--- + +## Part 2 B — `vllm/v1/engine/coordinator.py` (file is byte-identical to v0.10.2) + +The whole publish loop is `DPCoordinatorProc.process_input_socket`, +`coordinator.py:144-339`. + +| # | Question | Answer | file:line | Verbatim | +|---|---|---|---|---| +| B1 | When are **changed counts** published? | Counts are published **only on a poller timeout** — never directly in response to an engine's stats message. The timeout length itself is what encodes "changed": `wait_for` is the short interval when `stats_changed`, else the long heartbeat. So the publish condition is *"the poller expired, and the expiry deadline was computed from `stats_changed`"*. | condition `coordinator.py:204-219`; deadline `coordinator.py:194-205` | `elapsed = int(time.time() * 1000) - last_publish_time`
`# Send at stats_update_interval_ms interval if the stats have`
`# changed, or otherwise every 5 seconds.`
`wait_for = (self.stats_update_interval_ms`
` if stats_changed else 5000)`
`…`
`events = poller.poll(timeout=max(min_timeout, wait_for - elapsed))`
`if not events:` ← the publish branch | +| B1 | What exactly goes out | A 3-tuple `(counts, wave, engines_running)`. On the timeout path `counts` is a real list. On a **wave/state change** a second, separate message goes out immediately with `counts = None`, which the client at `core_client.py:1076` ignores for LB purposes while still updating `current_wave`/`engines_running`. | timeout publish `coordinator.py:215-218`; state-change publish `coordinator.py:337-339` | `to_publish = (engine_req_counts_list, current_wave, engines_running)`
`publish_front.send(msgspec.msgpack.encode(to_publish))`
and `message = (None, current_wave, engines_running)` | +| B2 | Role of the **previous-step snapshot** (`last_step_counts`) | Prevents publishing a *torn* count vector. Engines report independently; when a stats message arrives that belongs to a **newer step** while `stats_changed` is still set from the previous step, the coordinator **deep-copies the counts as they stood for the older, now-complete step** and publishes that snapshot on the next timeout instead of the live half-updated vector. It is consumed once and cleared. | capture `coordinator.py:296-298`; consume+clear `coordinator.py:208-210`; deep copy `coordinator.py:353-357` | `if stats_changed:`
` last_step_counts = self._get_engine_counts(do_copy=True)`
consume: `if last_step_counts is not None:`
` engine_req_counts_list = last_step_counts`
` last_step_counts = None` | +| B2 | Role of the **minimum collection wait** | Forces the poller to block at least 50 ms **when no snapshot is pending**, giving all DP engines time to report for the current step before the coordinator publishes. Once a snapshot is pending the wait drops to 0 because the data is already known-consistent, so it can go out immediately. | `coordinator.py:200-205` | `# Wait at least 50ms to ensure we've received all stats for`
`# the current step.`
`min_timeout = 50 if last_step_counts is None else 0` | +| B2 | Role of the **unchanged-state heartbeat** | When nothing changed, still republish every 5 s so late-joining / resubscribing frontends (XPUB/XSUB, no history replay) converge, and so a frontend's stale local reservations (A2) are eventually flushed. On the heartbeat path `stats_changed` is reset to `False`. | `coordinator.py:197-198`, `:211-213` | `wait_for = (self.stats_update_interval_ms if stats_changed else 5000)`
and `else:`
` engine_req_counts_list = self._get_engine_counts()`
` stats_changed = False` | +| B2 | Role of **wave/step ordering** | `(wave, step)` is the logical clock used to decide whether an incoming engine report belongs to a *newer* step, which is what triggers the snapshot. It is compared **lexicographically, wave-major**. Out-of-order reports are logged and **do not advance the clock**, but — note — the counts from them are still written (`:308-310` runs unconditionally after the ordering block). | compare `coordinator.py:293-295`; advance `:299-300`; warn `:301-307`; unconditional write `:308-310` | `if (stats_wave > last_stats_wave`
` or stats_wave == last_stats_wave`
` and stats_step > last_stats_step):` | +| B2 | Where wave state itself moves | `wave_complete` from rank 0 → `current_wave = wave + 1`, `engines_running = False` (`:312-322`). `start_wave` from a stale-wave engine → adopt wave, resume, rebroadcast `START_DP_WAVE` (`:323-335`). A frontend `FIRST_REQ` while paused → resume + broadcast (`:264-274`). | `coordinator.py:264-335`; broadcast helper `coordinator.py:341-351` | `engines_running = True`
`wave_state_changed = True`
`self._send_start_wave(publish_back, current_wave, engine_to_exclude)` | + +### B3 — Numeric constants, checked against the Frontier spec expectation + +| Expected by spec | Confirmed? | Actual | Symbol name | file:line | Verbatim | +|---|---|---|---|---|---| +| waiting-count weight **4** | **CONFIRMED** | `4` | *no symbol* — inline literal in the scoring expression. **And it does not live in `coordinator.py` at all**; it is in the frontend client. | `vllm/v1/engine/core_client.py:1146` | `score = waiting * 4 + running` | +| **50 ms** min collection wait | **CONFIRMED** | `50` | *no symbol* — bare literal assigned to the local `min_timeout` | `vllm/v1/engine/coordinator.py:202` | `min_timeout = 50 if last_step_counts is None else 0` | +| **100 ms** changed-stats interval | **CONFIRMED** | `100` | `min_stats_update_interval_ms` (parameter, **default** 100) → stored as `self.stats_update_interval_ms` | default `coordinator.py:116` and `coordinator.py:130`; stored `coordinator.py:122`; used `coordinator.py:197` | `def __init__(self, engine_count: int, min_stats_update_interval_ms: int = 100):`
`self.stats_update_interval_ms = min_stats_update_interval_ms` | +| **5000 ms** unchanged heartbeat | **CONFIRMED** | `5000` | *no symbol* — bare literal in the ternary | `vllm/v1/engine/coordinator.py:198` | `wait_for = (self.stats_update_interval_ms`
` if stats_changed else 5000)` | + +Four caveats a simulator must not gloss over: + +1. **The weight 4 is not a coordinator constant.** It is applied per-frontend at + request-admission time (`core_client.py:1146`). The coordinator transports raw + `[waiting, running]` pairs and never weights them. +2. **Only `100` is a named/overridable symbol.** `50` and `5000` are hard-coded + literals with no name and no env/config override; `DPCoordinator.__init__` + (`coordinator.py:58-93`) does not even forward `min_stats_update_interval_ms` + into the `Process` kwargs (`coordinator.py:81-86`), so in practice the default + `100` at `coordinator.py:130` is always what is used. +3. **100 ms is a floor, not a period.** `poller.poll` returns early on any socket + event, and the publish only happens on the *timeout* branch; a busy coordinator + re-enters the loop with a recomputed `elapsed`, so the effective publish cadence + is ">= 100 ms", not "every 100 ms". +4. The client-side comment at `core_client.py:1151` calls it + "every 100ms", matching constant 3. + +### B4 — Initialization epoch and equal-timestamp ordering + +| Question | Answer | file:line | Verbatim | +|---|---|---|---| +| What is the **initialization epoch**? | `last_publish_time = 0`, i.e. **Unix epoch 0**, set once before the loop. Since `elapsed = int(time.time()*1000) - 0` is ~1.7e12 ms, `wait_for - elapsed` is hugely negative on the very first iteration, so `max(min_timeout, …)` collapses to `min_timeout`. Net effect: **the first poll always waits exactly the 50 ms minimum and then publishes an all-zero count vector**, rather than waiting 5 s. | set `coordinator.py:192`; used `coordinator.py:194`, `:204-205` | `last_publish_time = 0`
`elapsed = int(time.time() * 1000) - last_publish_time` | +| Initial **logical clock** | `current_wave = 0`, `engines_running = False`, `stats_changed = False`, `last_stats_step = -1`, `last_stats_wave = -1`, `last_step_counts = None`. The `-1` sentinels guarantee the very first engine report (wave 0, step >= 0) satisfies the ordering test and is accepted. | `coordinator.py:151-158` | `last_stats_step = -1`
`last_stats_wave = -1`
`last_step_counts: Optional[list[list[int]]] = None` | +| Initial **per-engine counts** | Every engine starts `[waiting=0, running=0]`; the frontend mirrors this with `[[0, 0] for _ in self.core_engines]`, so before any real data every engine scores 0 and the rotated scan order (A1) is the sole tie-break. | `coordinator.py:108-109`, `:120`; client side `core_client.py:979` | `self.request_counts = [0, 0] # [waiting, running]` | +| How are **equal-timestamp events ordered**? | There is no wall-clock timestamp in the ordering at all — ordering is on the **`(wave, step)` logical clock**, compared **wave-major then step**, both **strictly greater**. Consequences: a report with *exactly equal* `(wave, step)` is **not** newer, so it does **not** trigger a snapshot and does **not** log a warning (the `elif` at `:301-302` is false when both are equal) — it silently overwrites that engine's counts and sets `stats_changed`. This is the normal case for the 2nd..Nth engine reporting the same step. `step_counter` resets to 0 on each wave boundary (`core.py:1128-1129`), which is exactly why `wave` must dominate the comparison. | `coordinator.py:293-295`; equal-case fallthrough `:301-302`; unconditional write `:308-310`; reset `vllm/v1/engine/core.py:1127-1129` | `if (stats_wave > last_stats_wave`
` or stats_wave == last_stats_wave`
` and stats_step > last_stats_step):`
reset: `self.current_wave += 1`
`self.step_counter = 0` | +| Ordering of the two publish paths within one loop iteration | A wave-state change publishes at the **end of the same iteration** (`:337-339`) and always carries `counts = None`; count publishes only ever happen at the **top** of an iteration on the timeout branch. They cannot be emitted in the same iteration. | `coordinator.py:206-219` vs `:337-339` | `if wave_state_changed:`
` message = (None, current_wave, engines_running)` | + +--- + +## Part 2 C — `vllm/v1/engine/core.py` and the scheduler stats producers + +There are **two independent stats producers** with different triggers, +different payloads and different suppression rules. Conflating them is the main +modelling hazard. + +| # | Question | Answer | file:line | Verbatim | +|---|---|---|---|---| +| C1 | **Producer 1** — the per-step metrics report | `Scheduler.make_stats()` is called at the **very end of `update_from_output()`**, i.e. **after** all per-request state updates for the step have been applied. It is attached to the `EngineCoreOutputs` of **exactly one** frontend, and an empty `EngineCoreOutputs` is synthesized if no client had outputs this step. | call `vllm/v1/core/sched/scheduler.py:1413-1419` | `if (stats := self.make_stats(spec_decoding_stats)) is not None:`
` # Return stats to only one of the front-ends.`
` if (eco := next(iter(engine_core_outputs.values()), None)) is None:`
` # We must return the stats even if there are no request`
` # outputs this step.`
` engine_core_outputs[0] = eco = EngineCoreOutputs()`
` eco.scheduler_stats = stats` | +| C1 | Producer-1 trigger chain | `run_busy_loop` → `_process_engine_step` → `step_fn()` = `EngineCore.step()` → `schedule()` → `execute_model` → `update_from_output()` (which emits). So **one report per executed engine step**, and `step()` returns early with `{}, False` — producing **no** report — when `self.scheduler.has_requests()` is false. | loop `core.py:1091-1099`; `_process_engine_step` `core.py:803-813`; `step` `core.py:290-309`; early return `core.py:299-300` | `if not self.scheduler.has_requests():`
` return {}, False`
`scheduler_output = self.scheduler.schedule()`
`model_output = self.execute_model_with_error_logging(...)`
`engine_core_outputs = self.scheduler.update_from_output(scheduler_output, model_output)` | +| C1 | Producer-1, extra trigger | A stats object is **also** produced on the model-execution **exception** path, purely to enrich the crash dump — it is passed to `dump_engine_exception` and is not published to any frontend. | `core.py:278-288` | `dump_engine_exception(self.vllm_config, scheduler_output,`
` self.scheduler.make_stats())` | +| C1 | **Producer 2** — the DP load-balance report | `DPEngineCoreProc._maybe_publish_request_counts()` is called from `run_busy_loop` **immediately after `_process_engine_step()` returns**, i.e. after `update_from_output()` has already removed finished requests. It builds a *minimal* `SchedulerStats` carrying only `(num_running_reqs, num_waiting_reqs, step_counter, current_wave)` and pushes it with `client_index = -1` (broadcast to the coordinator, not to a frontend). | call site `core.py:1098-1099`; body `core.py:1075-1087` | `executed = self._process_engine_step()`
`self._maybe_publish_request_counts()`
…
`stats = SchedulerStats(*counts,`
` step_counter=self.step_counter,`
` current_wave=self.current_wave)`
`self.output_queue.put_nowait(`
` (-1, EngineCoreOutputs(scheduler_stats=stats)))` | +| C1 | Note on positional construction | `SchedulerStats(*counts, …)` relies on `get_request_counts()` returning `(running, waiting)` in exactly the field order of the dataclass — `num_running_reqs` then `num_waiting_reqs`. | dataclass `vllm/v1/metrics/stats.py:49-57`; getter `vllm/v1/core/sched/scheduler.py:1502-1504` | `num_running_reqs: int = 0`
`num_waiting_reqs: int = 0`
`# These are used for internal DP load-balancing.`
`step_counter: int = 0`
`current_wave: int = 0`
getter: `return len(self.running), len(self.waiting)` | +| C1 | Which one the coordinator consumes | The coordinator reads `outputs.scheduler_stats.{num_waiting_reqs, num_running_reqs, step_counter, current_wave}` and asserts the message carries **no** request outputs — i.e. it is fed by **Producer 2 only**. | `coordinator.py:282-309` | `assert not outputs.outputs`
`assert outputs.utility_output is None`
…
`stats[0] = scheduler_stats.num_waiting_reqs`
`stats[1] = scheduler_stats.num_running_reqs` | +| C2 | Producer-1 suppression | Suppressed entirely when `log_stats` is false — `make_stats` returns `None` and the `if` at `:1413` never fires. | `vllm/v1/core/sched/scheduler.py:1609-1610` | `if not self.log_stats:`
` return None` | +| C2 | Producer-2 suppression (mode) | Suppressed entirely unless a coordinator exists **and** LB is not external — i.e. only in "internal" and "hybrid" DP LB modes. Also structurally absent for DP=1 (the method lives on `DPEngineCoreProc`). | gate `core.py:1075-1077`; definition `core.py:503-510` | `if not self.publish_dp_lb_stats:`
` return`
set by: `# Only publish request queue stats to coordinator for "internal"`
`# and "hybrid" LB modes .`
`self.publish_dp_lb_stats = (self.has_coordinator and not vllm_config.parallel_config.data_parallel_external_lb)` | +| C2 | Producer-2 suppression (dedup) | **Suppressed whenever the `(running, waiting)` pair is unchanged since the last publish** — a pure equality check against `self.last_counts`, seeded `(0, 0)`. This is why the coordinator needs its 5 s heartbeat: a steady-state engine emits nothing at all. | `core.py:1079-1082`; seed `core.py:1014` | `# Publish our request counts (if they've changed).`
`counts = self.scheduler.get_request_counts()`
`if counts != self.last_counts:`
` self.last_counts = counts`
seed: `self.last_counts = (0, 0)` | +| C2 | Consequence for `step_counter` monotonicity | Because of the dedup, **consecutive published reports can skip step numbers arbitrarily**. `step_counter` is incremented once per call to `_has_global_unfinished_reqs` (`core.py:1133-1135`), not once per publish. A simulator must treat `step_counter` as a sparse, strictly-increasing-within-a-wave tag, not a dense counter. | `core.py:1133-1135`; reset `core.py:1127-1129` | `# Optimization - only perform finish-sync all-reduce every 32 steps.`
`self.step_counter += 1`
`if self.step_counter % 32 != 0:`
` return True` | +| C2 | Batch-queue (pipelined) variant | With `step_with_batch_queue`, a scheduled-but-not-yet-completed batch returns `None, True` early, so **no** `EngineCoreOutputs` and therefore no Producer-1 report is emitted for that iteration. | `core.py:318-353`, early return at `core.py:349-353` | `# Don't block on next worker response unless the queue is full`
`# or there are no more requests to schedule.`
`return None, True` | + +--- + +## Part 2 D — `vllm/v1/core/sched/scheduler.py`: what counts as waiting vs running + +Both numbers are plain `len()` of two live containers — `self.running: list[Request]` +and `self.waiting: RequestQueue` — with **no filtering by status**. Everything +below is therefore a question of *which container an object is in at the moment +of the `len()`*. + +| # | Case | Counted as | Why / file:line | Verbatim | +|---|---|---|---|---| +| D1 | Definition | `num_running_reqs = len(self.running)`, `num_waiting_reqs = len(self.waiting)`. Identical in both producers. | `scheduler.py:1614-1615`; `scheduler.py:1502-1504` | `num_running_reqs=len(self.running),`
`num_waiting_reqs=len(self.waiting),` | +| D1 | Newly arrived request | **waiting** | `add_request` pushes straight onto the waiting queue; nothing else. | `scheduler.py:1506-1508` | `def add_request(self, request: Request) -> None:`
` self.waiting.add_request(request)`
` self.requests[request.request_id] = request` | +| D1 | **Admitted-but-not-yet-scheduled** (admitted this step, model not yet run) | **running** | Admission appends to `self.running` *inside* `schedule()`, before `execute_model` is ever called. Status is set to `RUNNING` a few lines later at `:838`. So from the instant `schedule()` returns, a first-chunk-prefill request already counts as running. | append `scheduler.py:812-813`; status `scheduler.py:838` | `req_index += 1`
`self.running.append(request)`
…
`request.status = RequestStatus.RUNNING` | +| D1 | Running request that `schedule()` chose **not** to schedule this step (budget exhausted) | **running** | It is never removed from `self.running`; the code explicitly asserts the scheduled set may be a strict subset. | `scheduler.py:890-894` | `# Since some requests in the RUNNING queue may not be scheduled in`
`# this step, the total number of scheduled requests can be smaller than`
`# len(self.running).`
`assert (len(scheduled_new_reqs) + len(scheduled_resumed_reqs) + len(scheduled_running_reqs) <= len(self.running))` | +| D1 | **Preempted** request | **waiting** (moves running→waiting **within** `schedule()`) | Victim is popped/removed from `self.running` (`:489` priority, `:507` FCFS), marked `PREEMPTED` with `num_computed_tokens = 0`, then **prepended to the head of the waiting queue**. So a preemption is `running -= 1` and `waiting += 1` atomically, visible in the same step's report. | remove `scheduler.py:489`, `:507`; status `scheduler.py:513-514`; requeue `scheduler.py:537-538` | `preempted_req.status = RequestStatus.PREEMPTED`
`preempted_req.num_computed_tokens = 0`
…
`self.waiting.prepend_request(preempted_req)` | +| D1 | Preempted request **resumed** later | **running** | On re-admission it takes the same `self.running.append` path and is classified `scheduled_resumed_reqs`. | `scheduler.py:813`, `:819-820` | `elif request.status == RequestStatus.PREEMPTED:`
` scheduled_resumed_reqs.append(request)` | +| D1 | Request awaiting **remote KV** (`WAITING_FOR_REMOTE_KVS`) | **waiting** | It is popped from `self.waiting`, parked in the temporary `skipped_waiting_requests` queue, and that whole queue is **prepended back onto `self.waiting`** before `schedule()` returns. Net: still in `self.waiting` when counted. Same for `WAITING_FOR_FSM`, `max_loras` blocking, unknown external-token count, and oversized non-chunked prefill. | park `scheduler.py:644-645`, `:655-656`, `:665-666`, `:689-690`, `:726-727`, `:799-801`; restore `scheduler.py:882-883` | `# Put back any skipped requests at the head of the waiting queue`
`if skipped_waiting_requests:`
` self.waiting.prepend_requests(skipped_waiting_requests)` | +| D1 | **Requests finishing this step** | **neither** — removed before the report | In `update_from_output`, a stopped request is freed, bucketed by its pre-stop status, and then removed from the container it was in — `self.running` for `RUNNING`, `self.waiting` for anything else (a preempted request that hit a stop condition). This happens at `:1381-1385`, i.e. **before** `make_stats()` at `:1413`. | bucket `scheduler.py:1332-1337`; remove `scheduler.py:1381-1385` | `if stopped:`
` kv_transfer_params = self._free_request(request)`
` if status_before_stop == RequestStatus.RUNNING:`
` stopped_running_reqs.add(request)`
` else:`
` stopped_preempted_reqs.add(request)`
…
`if stopped_running_reqs:`
` self.running = remove_all(self.running, stopped_running_reqs)`
`if stopped_preempted_reqs:`
` self.waiting.remove_requests(stopped_preempted_reqs)` | +| D1 | Externally **aborted** requests | removed from whichever queue they were in, same step | `finish_requests` path rebuilds `self.running` via `remove_all`. | `scheduler.py:1556` | `self.running = remove_all(self.running, running_requests_to_remove)` | +| D1 | Finished request still pending an outbound `finished_req_ids` notification | **neither** | `_free_request` only adds the id to `self.finished_req_ids` / `finished_req_ids_dict`; those sets are **not** part of either count. But they *do* keep `has_requests()` true (`interface.py:126-129`), so the engine can still take a step whose report shows `0, 0`. | `scheduler.py:1581-1584`; `interface.py:126-129` | `self.finished_req_ids.add(request_id)`
…
`return self.has_unfinished_requests() or self.has_finished_requests()` | +| D2 | **At what point are the counts observable?** (Producer 1) | At the **end of `update_from_output()`**, after: token append & stop detection, `_free_request`, stopped-request removal from both queues, KV-connector finish handling, and `EngineCoreOutputs` assembly. So the report is a **post-step, post-retirement** snapshot: admissions made by *this* step's `schedule()` are already counted as running, and completions from *this* step are already gone. | `scheduler.py:1413` reached after `:1381-1391` | `if (stats := self.make_stats(spec_decoding_stats)) is not None:` | +| D2 | **At what point are the counts observable?** (Producer 2) | One further step later in the call chain, but on the **same state**: `_maybe_publish_request_counts()` runs after `_process_engine_step()` has fully returned (which includes `update_from_output` **and** `post_step`). Between the two producers' reads, nothing mutates `self.running`/`self.waiting` — `post_step` only forwards draft token ids. So the two reports agree on the counts for a given step. | `core.py:1098-1099`; `post_step` `core.py:311-317` | `executed = self._process_engine_step()`
`self._maybe_publish_request_counts()` | +| D2 | What is **not** observable | There is no hook between `schedule()` and `execute_model`. A simulator cannot observe the mid-step state in which a preemption victim has left `self.running` but the model has not yet run. Preemption is only ever visible as an already-settled `(running, waiting)` delta. | `core.py:301-306` | `scheduler_output = self.scheduler.schedule()`
`model_output = self.execute_model_with_error_logging(…)`
`engine_core_outputs = self.scheduler.update_from_output(…)` | +| D2 | `AsyncScheduler` caveat | `AsyncScheduler` overrides only `_update_after_schedule` and `_update_request_with_output` (`async_scheduler.py:16-47`); it does **not** override `make_stats`, `get_request_counts`, or any queue mutation. All of section D applies unchanged. | `vllm/v1/core/sched/async_scheduler.py:14-47` | — | + + +--- + +## Part 2 E — `vllm/v1/worker/gpu_model_runner.py` (`GMR`) and `vllm/forward_context.py` (`FC`, UNCHANGED from v0.10.2) + +### E1 — Why DP source batches can be in different local phases yet share one expert forward + +| # | Question | Answer | file:line | Verbatim | +|---|---|---|---|---| +| E1a | The one per-step DP agreement collective | A single **all-reduce over a zero-filled `dp_size` vector** — a sum-as-allgather idiom, on the **CPU** process group. Each rank writes only its own slot; the sum reconstructs the full vector. | `FC:78-85` | `num_tokens_across_dp = [0] * dp_size`
`num_tokens_across_dp[dp_rank] = num_tokens`
`num_tokens_tensor = torch.tensor(num_tokens_across_dp,`
` device="cpu",`
` dtype=torch.int32)`
`from vllm.distributed.parallel_state import get_dp_group`
`dist.all_reduce(num_tokens_tensor, group=get_dp_group().cpu_group)`
`return num_tokens_tensor` | +| E1a | What does **not** exist at this revision | **There is no `should_ubatch_across_dp` and no `vllm/v1/worker/dp_utils.py`** in v0.10.2 — repo-wide grep finds nothing. DP micro-batch coordination is a later-vLLM feature. The only other DP-wide collective is the every-32-step liveness all-reduce. | absence verified by grep; liveness `vllm/config/parallel.py:239-250` (`ReduceOp.MAX`), driven from `vllm/v1/engine/core.py:1131-1139` | — | +| E1b | What is **forced to agree** | The **padded row count**. `get_dp_padding` all-reduces, takes the max, and returns a vector in which *every* entry is that max; the caller then inflates its own batch to it. So all ranks execute the same number of rows and the collective buffers line up. | `GMR:1920-1927`; applied `GMR:2001-2002` | `max_tokens_across_dp_cpu = torch.max(num_tokens_across_dp).item()`
`num_tokens_after_padding = torch.tensor([max_tokens_across_dp_cpu] * dp_size,`
` device="cpu", dtype=torch.int32)`
`return max_tokens_across_dp_cpu - num_tokens, num_tokens_after_padding`
caller: `num_pad, num_tokens_across_dp = self.get_dp_padding(num_input_tokens)`
`num_input_tokens += num_pad` | +| E1b | What stays **ragged** | Under `enforce_eager` (or DP=1) `get_dp_padding` early-exits with `None`, and `DPMetadata.make` then recomputes the vector itself and leaves it **unequalized**; the ragged prefix sums in `cu_tokens_across_dp_cpu` are what the naive all2all dispatch slices by. So equal row counts are a *cudagraph* requirement, not an MoE requirement. | early exit `GMR:1916-1918`; recompute `FC:111-113`; prefix sums `FC:115` | `if dp_size == 1 or self.vllm_config.model_config.enforce_eager:`
` # Early exit.`
` return 0, None`
`…`
`cu_tokens_across_dp_cpu = torch.cumsum(num_tokens_across_dp, dim=0)` | +| E1b | The cudagraph **mode** is NOT forced to agree | `uniform_decode` is computed from this rank's own batch, *after* DP padding, and fed to a rank-local dispatcher. A prefilling rank gets `False`, a decoding rank `True`, so `dispatch` may legitimately return FULL on one rank and PIECEWISE/NONE on another. **Only the row count must agree, not the replay mechanism.** | `GMR:2291-2298`; dispatcher `vllm/v1/cudagraph_dispatcher.py:92-121` | `uniform_decode = (max_query_len == self.uniform_decode_query_len) and (`
` num_scheduled_tokens == self.input_batch.num_reqs * max_query_len)`
`batch_descriptor = BatchDescriptor(num_tokens=num_input_tokens,`
` uniform_decode=uniform_decode)`
`cudagraph_runtime_mode, batch_descriptor = \`
` self.cudagraph_dispatcher.dispatch(batch_descriptor)` | +| E1b | The "all ranks must call the collective" invariant | Structural, not a flag: the naive all2all manager issues `dp_group.broadcast` / `all_reduce` **unconditionally** inside dispatch/combine, reached whenever `dp_size > 1`. A rank that skipped the forward would hang every other rank — which is precisely why the idle path (E1d) must exist. | collectives `vllm/distributed/device_communicators/all2all.py:39-44, :64`; reached from `vllm/model_executor/layers/fused_moe/layer.py:1804-1808`; condition `layer.py:1790-1793` | `do_naive_dispatch_combine` is true iff `dp_size > 1` | +| E1c | Where local phase is decided, and why it need not agree | **Each DP rank runs its own `Scheduler` in its own `EngineCore` process** — there is no cross-rank coordination in `schedule()` at all. The prefill-chunk-vs-decode split is a purely rank-local token-budget decision. Downstream, **attention metadata is built per rank from rank-local arrays and never crosses the DP boundary**, so phase disagreement is invisible to attention. Only the MoE/expert path is collective, and it consumes **token counts, not phases** — which is the whole answer to E1. | scheduler per rank `vllm/v1/engine/core.py:133`, class chain `core.py:72, 470, 997`; decode side `vllm/v1/core/sched/scheduler.py:408-424`; prefill chunking `:712-731`; rank-local attn metadata `GMR:1178-1290` | `self.scheduler: SchedulerInterface = Scheduler(...)` | +| E1d | "Even with no work, still participate" | `GPUWorker.execute_dummy_batch` runs a 1-token forward. It is invoked from the DP busy loop whenever the engine is in a running wave but scheduled nothing. | `vllm/v1/worker/gpu_worker.py:556-557`; driver `vllm/v1/engine/core.py:1102-1109`; chain `core.py:403-404` → `vllm/v1/executor/abstract.py:97-98` → `multiproc_executor.py:197-198` | `def execute_dummy_batch(self) -> None:`
` self.model_runner._dummy_run(1)`
driver: `if not executed:`
` if not local_unfinished_reqs and not self.engines_running:`
` # All engines are idle.`
` continue`
` # We are in a running state and so must execute a dummy pass`
` # if the model didn't execute any ready requests.`
` self.execute_dummy_batch()` | +| E1d | Same invariant restated for EPLB | `GMR:3240-3248` comments that the dummy run is needed "to avoid blocking DP… some DP ranks do not have any requests… we still have to trigger EPLB… in synchronization". | `GMR:3240-3248` | — | + +### E2 — Token populations and dummy/idle participants + +| # | Question | Answer | file:line | Verbatim | +|---|---|---|---|---| +| E2a | The DP dataclass carried into the forward pass — **all** its fields | `DPMetadata` has exactly **three** fields. `local_sizes` is transient: set only inside the `chunked_sizes` context manager and cleared in its `finally`. It hangs off `ForwardContext.dp_metadata`, populated only when `data_parallel_size > 1`. | dataclass `FC:65-69`; built `FC:114-116`; `chunked_sizes` `FC:118-155` (clear at `:155`); reader `FC:157-158`; field on context `FC:175`; population `FC:215-220` | `@dataclass`
`class DPMetadata:`
` max_tokens_across_dp_cpu: torch.Tensor` *(FC:67)*
` cu_tokens_across_dp_cpu: torch.Tensor` *(FC:68)*
` local_sizes: Optional[list[int]] = None` *(FC:69)* | +| E2a | Guard that keeps DP metadata alive for an attention-free dummy run | The `num_tokens is not None` disjunct. | `FC:216-217` | `if vllm_config.parallel_config.data_parallel_size > 1 and (`
` attn_metadata is not None or num_tokens is not None):` | +| E2b | `_dummy_run` signature | 8 parameters; the idle DP call uses defaults for all but `num_tokens=1`, so `cudagraph_runtime_mode=NONE` and `skip_eplb=False`. | `GMR:3035-3045`; idle invocation `vllm/v1/worker/gpu_worker.py:557` | `def _dummy_run(self, num_tokens: int, cudagraph_runtime_mode: CUDAGraphMode = CUDAGraphMode.NONE, force_attention: bool = False, uniform_decode: bool = False, skip_eplb: bool = False, is_profile: bool = False, create_mixed_batch: bool = False, remove_lora: bool = True) -> tuple[torch.Tensor, torch.Tensor]:` | +| E2b | What `_dummy_run` sets | DP padding **first** — the same `get_dp_padding` call as the real path, so the idle rank is inflated to `max_tokens_across_dp_cpu`; then a synthetic token/req layout; attention metadata **only** if `force_attention or cudagraph_runtime_mode == FULL`, so for the plain idle case `attn_metadata` stays **`None`**; then slices of the same persistent buffers; then the real model call; then `eplb_step(is_dummy=True, …)`. | padding `GMR:3070-3072`; layout `GMR:3096-3123`; attn gate `GMR:3129-3168`, default `GMR:3125`; buffers `GMR:3175-3188`; model call `GMR:3215-3228`; eplb `GMR:3247-3248` | `# Padding for DP`
`num_pad, num_tokens_across_dp = self.get_dp_padding(num_tokens)`
`num_tokens += num_pad` | +| E2b | **What distinguishes a dummy batch from a real one at the forward-context level** | **Nothing.** `ForwardContext` has **no `is_dummy` / `is_profile` field** (`FC:161-179`: `no_compile_layers`, `attn_metadata`, `virtual_engine`, `dp_metadata`, `cudagraph_runtime_mode`, `batch_descriptor`), and `set_forward_context` is called with the **identical argument set** in both paths. Differences are only indirect: `attn_metadata=None` for a plain dummy run, and optionally randomized `input_ids`. **A simulator cannot distinguish real from dummy participation from inside the MoE layer — and neither can vLLM.** | dataclass `FC:161-179`; dummy call `GMR:3215-3221` vs real call `GMR:2387-2393`; randomization `GMR:2978-3005` | dummy: `with self.maybe_randomize_inputs(input_ids), set_forward_context(`
` attn_metadata, self.vllm_config, num_tokens=num_tokens,`
` num_tokens_across_dp=num_tokens_across_dp,`
` cudagraph_runtime_mode=cudagraph_runtime_mode,`
` batch_descriptor=batch_descriptor):` | +| E2b | Input randomization | Active **only** when `VLLM_RANDOMIZE_DP_DUMMY_INPUTS` is set and `dp_size > 1`; its docstring names the purpose as balancing expert selection during DP dummy runs. | `GMR:2978-3005` (docstring `:2980-2984`) | — | +| E2c | Is cudagraph padding real rows or metadata? | **Real tensor rows.** Two padding stages (local cudagraph bucket, then DP max) both raise `num_input_tokens`, and that padded count then slices the real device buffers. Only `[:total_num_input_tokens]` was ever written, so pad rows carry **stale buffer contents**, are computed on through the GEMMs **and the MoE collective**, and are discarded. They are neutralized only at the metadata level. | bucket `GMR:1982-1998`; DP `GMR:2001-2002`; slices `GMR:2025, 2035, 2041`; writes `GMR:1190, 1199`; neutralization: seq-len fill `GMR:~1184`, slot-mapping fill `GMR:1266-1268` | `num_input_tokens = self.vllm_config.pad_for_cudagraph(total_num_input_tokens)`
`…`
`input_ids = self.input_ids.gpu[:num_input_tokens]`
`positions = self.positions.gpu[:num_input_tokens]`
`# Fill unused with -1. Needed for reshape_and_cache in full cuda graph mode`
`blk_table.slot_mapping[total_num_input_tokens:].fill_(-1)` | +| E2c | Simulator implication | Padding rows are **real MoE work and real collective payload**. A cost model that charges only scheduled tokens will under-count whenever DP ranks are imbalanced or a cudagraph bucket rounds up. | — | — | +| E2d | Who consumes `num_tokens_across_dp` | Passed as a kwarg into `set_forward_context` (both paths), converted to `DPMetadata` with an assertion that this rank's slot matches its batch size. Then: (1) `cu_tokens_across_dp_cpu` → naive all2all buffer sizing and per-rank slice bounds; (2) `max_tokens_across_dp_cpu` → the chunked MoE loop bound; (3) `local_sizes` → `get_local_sizes()` for the flashinfer-cutlass prepare/finalize; (4) CPU runner override. | pass `GMR:2392` (real), `GMR:3219` (dummy); convert+assert `FC:109-110, :218-220`; (1) `vllm/distributed/device_communicators/all2all.py:28-44, :46-55, :57-66`; (2) `vllm/model_executor/layers/fused_moe/layer.py:1743`, loop `:1753-1766`, chunk size `fused_moe/config.py:320` = `envs.VLLM_MOE_DP_CHUNK_SIZE`; (3) `layer.py:1762-1763` → `flashinfer_cutlass_prepare_finalize.py:16-17`; (4) `vllm/v1/worker/cpu_model_runner.py:128` | assert: `assert (num_tokens_across_dp is None`
` or num_tokens_across_dp[dp_rank] == batchsize)`
(2): `max_tokens_across_dispatchers = ctx.dp_metadata.max_tokens_across_dp_cpu`
buffer: `torch.empty((cu_tokens_across_dp_cpu[-1], x.size(1)))` | +| E2d | The lockstep guarantee, verbatim | In the chunked path, a rank that has run out of real tokens is given a **synthetic token** so that it still enters every chunk iteration and every collective. | `FC:50-62` | `if local_size[i] <= 0:`
` local_size[i] = 1 # ensure lockstep even if done` | + +--- + +## Part 2 F — Qwen3-MoE expert path, reductions, and LOCAL vs DISTRIBUTED + +Path selection is fixed by construction: `Qwen3MoeSparseMoeBlock.__init__` builds +`FusedMoE(... reduce_results=True, renormalize=config.norm_topk_prob ...)` +(`vllm/model_executor/models/qwen3_moe.py:146-155`). It passes neither +`apply_router_weight_on_input` nor `activation`, so the `FusedMoE.__init__` +defaults apply: `apply_router_weight_on_input: bool = False`, `activation: str = "silu"` +(`vllm/model_executor/layers/fused_moe/layer.py:787-788`), with +`use_grouped_topk=False` and `custom_routing_function=None`. + +### F1 — Exact call order, gate → returned value + +Abbreviations: `qwen3_moe.py` = `vllm/model_executor/models/qwen3_moe.py`; +`layer.py` / `fused_moe.py` = `vllm/model_executor/layers/fused_moe/{layer,fused_moe}.py`. + +| # | Step | file:line | Verbatim | +|---|---|---|---| +| 1 | Gate linear — `ReplicatedLinear`, **no collective** | `qwen3_moe.py:185`; class `vllm/model_executor/layers/linear.py:352-360` | `router_logits, _ = self.gate(hidden_states)` | +| 2 | FusedMoE call | `qwen3_moe.py:186` | `final_hidden_states = self.experts(hidden_states=hidden_states,` | +| 3 | `CustomOp.forward` → `_forward_method` from `dispatch_forward()` | `vllm/model_executor/custom_op.py:47-48`, dispatch `:85-110` | `return self._forward_method(*args, **kwargs)` | +| 4 | `FusedMoE.forward_cuda` → `forward_native` | `layer.py:1654` | `return self.forward_native(hidden_states, router_logits)` | +| 5 | `forward_native` → registered custom op (`shared_experts is None` for Qwen3) | `layer.py:1634-1635` | `fused_output = torch.ops.vllm.moe_forward(`
` hidden_states, router_logits, self.layer_name)` | +| 6 | `moe_forward` op body | `layer.py:1934` | `return self.forward_impl(hidden_states, router_logits)` | +| 7 | `forward_impl` → quant method | `layer.py:1821` | `final_hidden_states = self.quant_method.apply(` | +| 8 | `UnquantizedFusedMoEMethod.apply` → `self.forward(...)` → CustomOp dispatch → `forward_cuda` | `layer.py:394`, `:423`, `:446` | `return self.forward(` | +| 9 | Routing | `layer.py:470` | `topk_weights, topk_ids = FusedMoE.select_experts(` | +| 9a | → `fused_topk` | `layer.py:1506-1507` (guard `elif custom_routing_function is None:` at `:1506`) | `topk_weights, topk_ids, token_expert_indices = fused_topk(` | +| 9b | → `vllm_topk_softmax` via `dispatch_topk_func()` | `fused_moe.py:1001-1003`; fn `fused_moe.py:884-898` | `topk_weights, topk_ids = topk_func(topk_weights, topk_ids,`
` token_expert_indices,`
` gating_output_float, renormalize)` | +| 10 | `fused_experts` | `layer.py:518` | `return fused_experts(` | +| 11 | → `dispatch_fused_experts_func(inplace)` (`inplace=True` at `layer.py:525`) → `torch_vllm_inplace_fused_experts` → `fused_experts_impl` | `fused_moe.py:1466-1469`, `:1542` | `return dispatch_fused_experts_func(inplace)(` | +| 12 | Input quant — no-op for bf16 | `fused_moe.py:1710-1715` | `qcurr_hidden_states, a1q_scale = moe_kernel_quantize_input(` | +| 13 | **Expert sort + `expert_map` applied here** | `fused_moe.py:1718-1720` | `sorted_token_ids, expert_ids, num_tokens_post_padded = (`
` moe_align_block_size(curr_topk_ids, config['BLOCK_SIZE_M'],`
` global_num_experts, expert_map))` | +| 14 | **w1 grouped GEMM = gate **and** up fused, `N = 2 * intermediate_size`** | `fused_moe.py:1723-1725`; kernel launch `fused_moe.py:621` | `invoke_fused_moe_kernel(qcurr_hidden_states,`
` w1,`
` intermediate_cache1,` | +| 15 | **SiLU*mul — a SEPARATE CUDA op, NOT inside the triton kernel** | `fused_moe.py:1746-1748` | `if activation == "silu" and is_act_and_mul:`
` torch.ops._C.silu_and_mul(intermediate_cache2,`
` intermediate_cache1.view(-1, N))` | +| 16 | **w2 grouped GEMM (down)** | `fused_moe.py:1775-1777` | `invoke_fused_moe_kernel(qintermediate_cache2,`
` w2,`
` intermediate_cache3,` | +| 17 | **Routing weights applied — inside the w2 kernel** | `fused_moe.py:469-473` | `if MUL_ROUTED_WEIGHT:`
` moe_weight = tl.load(topk_weights_ptr + offs_token,`
` mask=token_mask,`
` other=0)`
` accumulator = accumulator * moe_weight[:, None]` | +| 18 | **Local top-k sum reduction** | `fused_moe.py:1797-1798` | `ops.moe_sum(intermediate_cache3.view(*intermediate_cache3.size()),`
` out_hidden_states[begin_chunk_idx:end_chunk_idx])` | +| 19 | Distributed reduce — conditional | `layer.py:1859-1860` | `if self.reduce_results and (self.tp_size > 1 or self.ep_size > 1):`
` states = self.maybe_all_reduce_tensor_model_parallel(states)` | +| 20 | Value returned to the model | `layer.py:1866`, then `qwen3_moe.py:190-191` | `return reduce_output(final_hidden_states)` | + +**Gated/SwiGLU precision.** `w1` is the fused `w13_weight` of shape `[E, N, K]` with +`N = 2 * intermediate_size` (`E, N, _ = w1.size()` at `fused_moe.py:1689`; +`intermediate_cache2` is `(M * top_k_num, N // 2)` at `fused_moe.py:1660`). The triton +kernel writes the **raw `[.., 2*inter]` accumulator and contains no SiLU**. The SwiGLU +nonlinearity is a separate `torch.ops._C.silu_and_mul` launch at `fused_moe.py:1747-1748`, +selected by `if activation == "silu" and is_act_and_mul:` (`fused_moe.py:1746`, with +`is_act_and_mul: bool = True` default at `fused_moe.py:1481`). It is a **bare op call, not +the `SiluAndMul` `CustomOp` module** (`vllm/model_executor/layers/activation.py:59-89`), +so `CustomOp` enable/disable and `forward_native` have **no effect** on this path. + +**Kernel sequence per MoE layer:** `topk_softmax` → `moe_align_block_size` → +grouped-GEMM(w1) → `silu_and_mul` → grouped-GEMM(w2, routing weights folded in) → `moe_sum`. + +### F2 — Routing weights, workspace, expert map, local reduction + +| # | Question | Answer | file:line | Verbatim | +|---|---|---|---|---| +| F2a | Where are routing weights applied? | **Inside `invoke_fused_moe_kernel` via the `MUL_ROUTED_WEIGHT` constexpr, on the w2 GEMM only** — not after, and not on w1. Parameter is positional #11, `mul_routed_weight: bool`. The two call sites pass exactly the expected `False` / `True`. | param `fused_moe.py:503`; **w1 → False** `fused_moe.py:1733`; **w2 → True** `fused_moe.py:1785`; forwarded `fused_moe.py:657` (dense) and `:607` (gptq/awq); applied `fused_moe.py:469-473` (dense) and `:257-261` (gptq/awq); guard `fused_moe.py:514` | w1 site: `apply_router_weight_on_input,` (→ `False`, default `layer.py:787`)
w2 site: `not apply_router_weight_on_input,` (→ `True`)
forward: `MUL_ROUTED_WEIGHT=mul_routed_weight,`
guard: `assert topk_weights is not None or not mul_routed_weight` | +| F2a | Where in the kernel | After the K-loop **and after the bias add** (`fused_moe.py:467-468`), **before** the dtype cast (`fused_moe.py:474-482`). Same pattern in the modular `TritonExperts`: `False, # mul_routed_weights` for w1 (`fused_moe.py:2095`), `not apply_router_weight_on_input,` for w2 (`:2129`). | `fused_moe.py:467-482` | — | +| F2b | `CHUNK_SIZE` value | **32768.** Note a real mismatch in `envs.py`: the type annotation says `64 * 1024` but the runtime lambda — which is what is actually read — says `"32768"`. The annotation is cosmetic; upstream has the same mismatch. | use `fused_moe.py:1626-1627`; annotation `vllm/envs.py:55`; **runtime** `vllm/envs.py:652-653` | `CHUNK_SIZE = envs.VLLM_FUSED_MOE_CHUNK_SIZE`
`M = min(num_tokens, CHUNK_SIZE)`
annotation: `VLLM_FUSED_MOE_CHUNK_SIZE: int = 64 * 1024`
runtime: `lambda: int(os.getenv("VLLM_FUSED_MOE_CHUNK_SIZE", "32768")),` | +| F2b | Workspace layout (`E, N, _ = w1.size()`, `K = w2.size(1)`, `top_k_num = topk_ids.size(1)`) | `cache13` is one flat `(M * top_k_num * max(N, K),)` allocation; **`intermediate_cache1` and `intermediate_cache3` are aliasing views into it**, while `intermediate_cache2` is a separate allocation because it is live at the same time as cache1. | alloc `fused_moe.py:1653-1662`; rationale comment `fused_moe.py:1651-1652` | `cache13 = torch.empty(M * top_k_num * max(N, K), …)`
`intermediate_cache1 = cache13[:M * top_k_num * N].view(M, top_k_num, N)`
`intermediate_cache3 = cache13[:M * top_k_num * K].view(M, top_k_num, K)`
`# This needs separate memory since it's used concurrently with cache1`
`intermediate_cache2 = torch.empty((M * top_k_num, N // 2), …)` | +| F2b | Chunking loop | Iterates `(num_tokens // CHUNK_SIZE) + 1` times over `hidden_states[begin:end]`. Only a **short trailing chunk** triggers re-slicing of the three caches and a recomputed `config`. Output goes straight into `out_hidden_states`, which **is** `hidden_states` because `inplace=True`. | loop `fused_moe.py:1685`, slice `:1689`; trailing-chunk branch `:1695`, re-slice `:1700-1703`, config `:1704`; inplace `:1674-1675`, caller `layer.py:525` | `for chunk in range((num_tokens // CHUNK_SIZE) + 1):`
`if tokens_in_chunk < CHUNK_SIZE and chunk > 0:` | +| F2c | How is `expert_map` built, and what marks "not on this rank"? | `determine_expert_map(ep_size, ep_rank, global_num_experts)` returns `(global_num_experts, None)` when `ep_size == 1`; otherwise a length-`global_num_experts` int32 tensor prefilled with **`-1`**, with this rank's contiguous slice overwritten by `0..local_num_experts-1`. **Sentinel = `-1`.** | fn `layer.py:680-721`; EP=1 short-circuit `layer.py:703-704`; build `layer.py:714-719` | `# Create a tensor of size num_experts filled with -1`
`expert_map = torch.full((global_num_experts, ), -1, dtype=torch.int32)`
`start_idx = ep_rank * base_experts + min(ep_rank, remainder)`
`expert_map[start_idx:start_idx + local_num_experts] = torch.arange(`
` 0, local_num_experts, dtype=torch.int32)` | +| F2c | Where is it applied? | **Not to `topk_ids`** — those stay global. It is applied to the **per-block `expert_ids`** produced by the sort, right after `ops.moe_align_block_size`. | `vllm/model_executor/layers/fused_moe/moe_align_block_size.py:84-85` | `if expert_map is not None:`
` expert_ids = expert_map[expert_ids]` | +| F2c | What the kernel does with `-1` | Short-circuits the block and **writes zeros**, so the downstream unweighted sum stays correct without any masking. | `fused_moe.py:379-387`; same guard `:163-171` (gptq/awq) | `off_experts = tl.load(expert_ids_ptr + pid_m).to(tl.int64)`
`if off_experts == -1:`
` # Write back zeros to the output when the expert is not`
` # in the current expert parallel rank.`
` write_zeros_to_output(...)`
` return` | +| F2c | **Simulator consequence** | Under EP, each rank still allocates the **full `[M, top_k, ...]` workspace** and still runs a **full `moe_sum` over all `top_k` slots**; off-rank slots are zero-filled locally, not dropped. Only the GEMM blocks are skipped. Cost must be modelled as "full workspace + full reduction, reduced GEMM occupancy", not "1/ep_size of everything". | `fused_moe.py:379-387` + `:1653-1657` + `:1797-1798` | — | +| F2d | Where is the local top-k output reduction? | `ops.moe_sum`, reducing `[num_tokens, topk, hidden] → [num_tokens, hidden]`. Implementation is a plain unrolled sum over the topk axis, dispatched by `topk`. Modular `TritonExperts` equivalent at `fused_moe.py:2142`. | call `fused_moe.py:1797-1798`; impl `csrc/moe/moe_align_sum_kernels.cu:110-123` (accumulate `:119`), dispatch `:277-294` | `ops.moe_sum(intermediate_cache3.view(*intermediate_cache3.size()),`
` out_hidden_states[begin_chunk_idx:end_chunk_idx])`
kernel: `x += VLLM_LDG(&input[token_idx * TOPK * d + k * d + idx]);` | +| F2d | Were topk weights already folded in? | **Yes** — by `MUL_ROUTED_WEIGHT=True` on the w2 GEMM (F2a). **`moe_sum` is an unweighted sum. A simulator must not apply routing weights again at the reduction step.** | `fused_moe.py:1785` + `:469-473` vs `:1797-1798` | — | + +### F3 — LOCAL expert aggregation vs DISTRIBUTED communication + +| # | Reduction / op | file:line | Class | +|---|---|---|---| +| R1 | Softmax normalize over experts in the topk kernel, plus the Python renormalize | `fused_moe.py:895-896` | **LOCAL** | +| R2 | w1 GEMM K-loop accumulate | `fused_moe.py:461/463`, launch `:621` | **LOCAL** (intra-GEMM) | +| R3 | w2 GEMM K-loop accumulate | same kernel, launch `fused_moe.py:1775` | **LOCAL** (intra-GEMM) | +| R4 | `MUL_ROUTED_WEIGHT` scaling | `fused_moe.py:469-473` | **LOCAL** (a scale, not a reduction) | +| R5 | `ops.moe_sum` — sum the k expert rows into 1 | `fused_moe.py:1797-1798` | **LOCAL — this *is* the expert aggregation, and it involves no network** | +| R6 | EP dispatch | `layer.py:1807-1808` | **DISTRIBUTED**, conditional | +| R7 | EP combine | `layer.py:1857` | **DISTRIBUTED**, conditional | +| R8 | TP all-reduce | `layer.py:1860` | **DISTRIBUTED**, conditional | + +**R8 — group identity, the key anti-double-count fact.** The group is **`_TP` +(`get_tp_group()`), not the EP group**, even when the profiling scope is *named* +`expert_parallel_allreduce`. `layer.py:1599-1613`: + +``` + if (self.use_pplx_kernels or self.use_deepep_ht_kernels + or self.use_deepep_ll_kernels): + return final_hidden_states + else: + op_name = ("expert_parallel_allreduce" + if self.ep_size > 1 else "moe_tensor_parallel_allreduce") + return tensor_model_parallel_all_reduce( + final_hidden_states, + record_scope_name=op_name, + ) +``` + +It bottoms out in `tp_group.all_reduce(input_)` with `tp_group = get_tp_group()` +(`vllm/distributed/communication_op.py:16, :30, :32`; `get_tp_group` → `_TP` at +`vllm/distributed/parallel_state.py:907-909`). **The fork's `record_scope_name` +only names the scope; it does not change the group.** *Do not infer the group from +the scope name.* + +**The `reduce_results` guard is NOT in `Qwen3MoeSparseMoeBlock.forward`.** In this +version `forward` (`qwen3_moe.py:176-191`) contains no reduce logic at all; the model +only sets `reduce_results=True` at construction (`qwen3_moe.py:150`). The guard lives +in `FusedMoE.forward_impl.reduce_output`, `layer.py:1859-1860`: + +``` + if self.reduce_results and (self.tp_size > 1 or self.ep_size > 1): + states = self.maybe_all_reduce_tensor_model_parallel(states) +``` + +`self.tp_size` / `self.ep_size` here are **MoE-local** values from +`FusedMoEParallelConfig.make` (`vllm/model_executor/layers/fused_moe/config.py:266-301`): +with `enable_expert_parallel=True` it returns `tp_size=1, ep_size = dp_size*tp_size` +(`config.py:293-301`), so the `or self.ep_size > 1` arm is what keeps the all-reduce +alive in EP mode — and the tensor still travels over `_TP`, which is correct because +`_EP` is constructed as the `dp × tp` flatten (`vllm/distributed/parallel_state.py:1168-1169`), +so with `DP=1` the EP and TP rank sets coincide. + +**R6/R7 condition** — `layer.py:1790-1793`: + +``` + do_naive_dispatch_combine: bool = ( + self.dp_size > 1 + and not self.moe_parallel_config.use_deepep_ht_kernels + and not self.moe_config.use_flashinfer_cutlass_kernels) +``` + +Dispatch/combine are **skipped entirely when `dp_size == 1`**. When they do run, +`get_ep_group().dispatch/.combine` (`vllm/distributed/parallel_state.py:838-860`) +forward to the CUDA communicator (`cuda_communicator.py:286-293`) and then to +`NaiveAll2AllManager`, whose traffic is on the **DP group**: + +- `dispatch` → `naive_multicast` ×2 (hidden_states, router_logits), each a loop of + per-rank broadcasts — `vllm/distributed/device_communicators/all2all.py:39-43`, + called at `:50-53`. +- `combine` → `all_hidden_states = self.dp_group.all_reduce(hidden_states)` — + `all2all.py:64`. +- `self.dp_group = get_dp_group()` — + `vllm/distributed/device_communicators/base_device_communicator.py:41`. + +**Cross-check — is any all2all used on the DEFAULT path? NO.** Two independent +source-level reasons: + +1. The default backend is `"naive"`, which **is not an all-to-all at all** — it is DP + broadcasts plus a DP all-reduce. Docstring `all2all.py:17-23` says *"It uses + all-reduce under the hood"*; code at `:39-43`, `:64`. Config: + `vllm/envs.py:159` `VLLM_ALL2ALL_BACKEND: str = "naive"`, runtime + `vllm/envs.py:1109-1110` `lambda: os.getenv("VLLM_ALL2ALL_BACKEND", "naive"),`; + selection at `vllm/distributed/device_communicators/cuda_communicator.py:85-88`. +2. The all2all *kernel* paths are gated off regardless: + `vllm/model_executor/layers/fused_moe/config.py:175-176` + `def use_all2all_kernels(self): return self.dp_size > 1 and self.use_ep`, and + `use_pplx_kernels` / `use_deepep_ht_kernels` / `use_deepep_ll_kernels` each + additionally require `envs.VLLM_ALL2ALL_BACKEND == "pplx" | "deepep_high_throughput" | + "deepep_low_latency"` (`config.py:178-191`). With the default all three are `False`, + so `forward_impl_chunked` is never taken (`layer.py:1785-1788`) and the deepep/pplx + `prepare_finalize` modules are never constructed. + +### F3 — final count: network collectives per MoE layer forward + +| Topology (default `VLLM_ALL2ALL_BACKEND=naive`) | Collectives per MoE layer | Which | +|---|---|---| +| TP=N>1, DP=1, EP **off** | **1** | 1 × all-reduce on `_TP` (size N), scope `moe_tensor_parallel_allreduce` — `layer.py:1860` → `communication_op.py:30` | +| TP=N>1, DP=1, EP **on** | **1** | 1 × all-reduce on `_TP` (size N), scope `expert_parallel_allreduce`. MoE-local `tp_size` is 1, but `ep_size=N` satisfies the guard at `layer.py:1859`. No dispatch/combine, since `dp_size == 1` (`layer.py:1791`) | +| TP=1, DP=1 (single GPU) | **0** | guard at `layer.py:1859` is false | +| TP=N, DP=M>1, EP on | **2M + 2** | dispatch = 2 × (M DP broadcasts) (`all2all.py:39-43`, ×2 for hidden_states + router_logits); combine = 1 × DP all-reduce (`all2all.py:64`); plus 1 × `_TP` all-reduce (`layer.py:1860`) | + +**Simulator guidance.** On the standard TP-and/or-EP, DP=1 deployment there is +**exactly one network collective per MoE layer**: a single hidden-size all-reduce over +the model TP group, emitted **after** `moe_sum`. Everything upstream of it — the +topk-softmax normalize, both GEMM K-loops, `MUL_ROUTED_WEIGHT`, and `moe_sum` — is +intra-GPU and must **not** be charged as communication. The gate (`ReplicatedLinear`, +`vllm/model_executor/layers/linear.py:352-360`) has no collective. Counted separately: +the attention block's `o_proj` (`RowParallelLinear`, `qwen3_moe.py:381`) contributes its +own TP all-reduce per decoder layer — that is outside the MoE block and must not be +double-counted against R8. + +### F — fork-vs-upstream for these files + +Method: `git diff -w --ignore-blank-lines`. Raw diff over the six files is 977 changed +lines; the whitespace-ignoring diff is 327 — **roughly two thirds of the diff is pure +yapf/line-wrap reformatting.** + +| File | raw / `-w` lines | Verdict | +|---|---|---| +| `qwen3_moe.py` | 227 / 185 | **Instrumentation-only.** All adds are `record_function_or_nullcontext(...)` scopes, `record_frontier_op_meta(...)` blocks, and cached attributes used solely by them (`self.rope_scaling` `:233`, `self.layer_idx` `:235`, `self.qkv_bias` `:236`, `self.rms_norm_eps` `:237`, `:444`). No math, no collective, no routing changed. One edge: the `attn_rope` metadata block can `raise RuntimeError` on missing positions meta (`:325-330`) — reachable only while recording is active. | +| `fused_moe.py` | 376 / 86 | **BEHAVIOR CHANGE** — see below. | +| `layer.py` | 214 / 40 | **Instrumentation-only.** `frontier_moe_routing_context(...)` wrappers around `quant_method.apply` (`:1694-1702`, `:1812-1820`), `record_function_or_nullcontext` scopes (`:1477`, `:1802`, `:1855`), `log_frontier_moe_routing_from_context(topk_ids)` (`:1487`, `:1578`). The only non-scope edit is `maybe_all_reduce_tensor_model_parallel` passing `record_scope_name=op_name` (`:1608-1613`) — the underlying call is still `get_tp_group().all_reduce`. | +| `deepep_ht_prepare_finalize.py` | 59 / 5 | **Instrumentation-only.** 1 import + 2 scopes around `buffer.dispatch` (`:88`) and `buffer.combine` (`:274`). | +| `deepep_ll_prepare_finalize.py` | 40 / 8 | **Instrumentation-only.** 1 import + 2 scopes around `low_latency_dispatch` (`:152`) / `low_latency_combine` (`:225`); the remaining `-`/`+` pair is an argument re-wrap. | +| `pplx_prepare_finalize.py` | 61 / 7 | **Instrumentation-only.** 1 import + 3 scopes around `a2a.dispatch` (`:206`, `:243`) and `a2a.combine` (`:322`). | + +Byte-identical to upstream and therefore safe to treat as reference: +`modular_kernel.py`, `utils.py`, `all2all.py`, `activation.py`. Instrumentation-only +and always falling through to the same underlying call: `communication_op.py` (+25), +`parallel_state.py` (+11). + +Instrumentation is inert by default: `record_function_or_nullcontext` returns +`contextlib.nullcontext()` unless `envs.VLLM_CUSTOM_SCOPES_FOR_PROFILING` +(`vllm/envs.py:188`, default `False`) **or** a Frontier op-logger contextvar is installed +and `frontier_trace.is_active()` (`vllm/v1/utils.py:1179-1200`); +`should_record_frontier_op_meta` returns `False` under the same conditions +(`vllm/v1/utils.py:1212-1216`). + +#### `fused_moe.py` — the real changes + +| # | Change | HEAD line | Gated? | Default | +|---|---|---|---|---| +| 1 | `uniform_topk()` added — deterministic round-robin routing, all weights `1.0/topk` | `fused_moe.py:908` | `VLLM_MOE_UNIFORM_ROUTING` | **OFF** (`vllm/envs.py:150` `VLLM_MOE_UNIFORM_ROUTING: bool = False`; `vllm/envs.py:1044` `lambda: bool(int(os.getenv("VLLM_MOE_UNIFORM_ROUTING", "0")))`) | +| 2 | `fused_topk` early-returns to `uniform_topk` | `fused_moe.py:980-981` | same env | OFF | +| 3 | `grouped_topk` early-returns to `uniform_topk`, bypassing all grouped logic including `e_score_correction_bias` | `fused_moe.py:1023-1026` | same env | OFF — and not on the Qwen3 path anyway (`use_grouped_topk=False`) | +| 4 | **`vllm_topk_softmax` gains a `renormalize` parameter and passes it as a 5th argument to the C++ op** | `fused_moe.py:884-898`, esp. `:893` | **NOT gated — always on** | always | + +**Item 4 is an ABI change and the one thing to flag before any run.** The Python side now +calls the op with **5** arguments (`fused_moe.py:893`; wrapper +`vllm/_custom_ops.py:1506-1510`), but the **in-tree C++ side was not changed** — +`git diff --numstat HEAD -- csrc/` returns **zero rows**, and the registered schema +is still **4** arguments (`csrc/moe/torch_bindings.cpp:6-9`: +`"topk_softmax(Tensor! topk_weights, Tensor! topk_indices, Tensor! token_expert_indices, Tensor gating_output) -> ()"`; +`csrc/moe/moe_ops.h:5-7` declares four `torch::Tensor&`). + +The fork does not build its own csrc: `vllm/_moe_C.py:14-21` is a new shim loading +`_moe_C.abi3.so` from `$VLLM_FRONTIER_COMPILED_PACKAGE` (`vllm/_C.py` is the identical +sibling for `_C`). So this call path **requires an out-of-tree prebuilt extension whose +`topk_softmax` accepts `renormalize`**. That prebuilt package could not be located on this +host, so the 5-arg binary was **not verified**. Numerically the change is a no-op either +way: `fused_moe.py:895-896` still performs +`topk_weights = topk_weights / topk_weights.sum(dim=-1, keepdim=True)` when +`renormalize`, so an in-kernel renormalize would be idempotent with it. + +--- + +## Could not determine / open items + +| # | Item | What is missing | +|---|---|---| +| 1 | Whether the 5-arg `topk_softmax` binary actually exists | `vllm/_moe_C.py:14-21` loads `_moe_C.abi3.so` from `$VLLM_FRONTIER_COMPILED_PACKAGE`, which was not set and whose canonical path (`/local/ycfeng/anaconda3/envs/frontier/lib/python3.10/site-packages/vllm`) does not exist on this host. The only `_moe_C.abi3.so` files present on this machine belong to other environments and register the **4-arg** schema, which would raise on this call path. **Resolving this needs the actual profiling environment**, not more source reading. Note the task forbade installing packages and starting servers, so this was not pursued further. | +| 2 | Empirical confirmation of the coordinator timing constants | The values `4`, `50`, `100`, `5000` are confirmed **statically**. Their *effective* cadence depends on poller wake-ups and the dedup in `core.py:1081`, which only a running DP deployment would show. No server was started, per the task constraints. | +| 3 | `min_stats_update_interval_ms` override reachability | `DPCoordinator.__init__` does not forward it into the `Process` kwargs (`coordinator.py:81-86`), so `100` appears to be unconditional in practice. I found no caller that passes a non-default value, but I did not exhaustively search external/test callers of `DPCoordinatorProc.run_coordinator`. | +| 4 | Behavior of `stats_changed` when an out-of-order report arrives | `coordinator.py:308-310` writes the counts and sets `stats_changed = True` **unconditionally**, including after the out-of-order warning at `:303-307`. Whether that is intended (counts are still the engine's latest known state) or a latent bug is a design question the source does not settle. Flagging it because a simulator that mirrors the ordering check but *not* the unconditional write would diverge. | +| 5 | Exact HEAD line for one `gpu_model_runner.py` seq-len fill | The `self.seq_lens.np[num_reqs:].fill(0)` "Fill unused with 0 for full cuda graph mode" line is cited approximately (`GMR:~1184`). The neighbouring slot-mapping fill at `GMR:1266-1268` was read directly and is exact. | +| 6 | Runtime effect of the fork's forced `torch.cuda.synchronize()` | `GMR:2428` adds a sync when Frontier tracing is active. It changes measured timing but not values. The magnitude was not measured — that requires a GPU run. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index 8d74f19a..f0df39ad 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -4,7 +4,8 @@ | Date | Change | | --- | --- | -| 2026-09-21 | Created with the pinned source snapshot. Dispositions are filled in Step 1. | +| 2026-09-21 | Created with the pinned source snapshot. | +| 2026-09-21 | Step 1 complete: candidate and vLLM audits landed, dispositions recorded, two decision checkpoints raised. | ## Pinned source snapshot @@ -15,20 +16,122 @@ | Candidate parent before evidence import | `b7f8d055461d9208b8ae57eceeb6c0246cbc8d3c` | Source comparison point (verified equal source tree to `a7b3320`). | | Merge base | `d71ad80b0800880808a0857fd30477e6d96592c6` | Verified with `git merge-base`. | | `fwyc0573/vLLM-BS` | `ea95f571e20937c7c908c6d59ddd1cd6bf9268f1` | vLLM reference, `.real-engine/vLLM-BS`. | +| Upstream `vllm-project/vllm` `v0.10.2` | `01efc7ef781391e744ed08c3292817a773d654e6` | Resolved by fetching the tag into the reference checkout. | | `fwyc0573/frontier-htsim` main gitlink | `b8518afcc310f0fe0e3ce52ba6b4f0bf57a3be04` | Current optional backend. | | Candidate gitlink | `e564935d3874d8c71b52a554ab7c9a72e5e19f68` | Not reachable on the configured remote (HTTP 422, 2026-09-21). | -## Candidate change dispositions +### vLLM reference relationship to v0.10.2 -To be completed in Step 1. Format: path, donor hunk summary, old defect, main behavior, reference behavior, disposition (`PORT`/`ADAPT`/`ALREADY_PRESENT`/`DROP`/`BLOCKED`), chosen owner, planned test. +The fork's HEAD is a direct descendant of upstream `v0.10.2`: the merge base equals the tag commit, and the fork is 40 commits ahead, 0 behind. It pulled nothing from upstream after branching. -| Path | Disposition | Notes | +Decisive for this PR: `vllm/v1/engine/core_client.py`, `vllm/v1/engine/coordinator.py` and `vllm/forward_context.py` are **byte-identical to v0.10.2**, so the DP placement and count-publication behavior read for work package W4 is upstream behavior, not fork behavior. `vllm/v1/core/sched/scheduler.py` and `vllm/v1/worker/gpu_model_runner.py` do differ; the differences are characterized per file in `reference_vllm_0_10_2.md`. + +## Supporting audit reports + +| File | Contents | +| --- | --- | +| `audit_scheduler.md` | Three-way source audit of the candidate's scheduler changes (W2, W3, W4). | +| `audit_predictor_profiling.md` | Three-way source audit of the candidate's predictor and profiling changes (W5, W6). | +| `reference_vllm_0_10_2.md` | vLLM reference-behavior tables with pinned file and line citations. | + +Claims in those reports that this PR depends on were re-verified directly against the source before being recorded here; the five spot checks are listed in `validation.md`. + +## Work package dispositions + +### W2 — Round-robin DP placement rotation + +| Item | Finding | +| --- | --- | +| Defect on main | Present. `frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py:385-386` assigns the DP lane with `dp_id = local_idx % self._replica_dp_size`, where `local_idx` enumerates only the requests handled in the current call. The persistent `_request_counter` is used for the replica index (`:372`, `:376`) but not for the lane. | +| Main already has the correct formula | Yes, for one role only. `_schedule_decode_lane_round_robin` at `:438-439` computes `replica_idx = (counter + idx) % n` and `dp_id = (counter + idx) // n % dp_size` from the same persistent counter. `schedule()` routes that method only for `ClusterType.DECODE` (`:75-76`); `_schedule_batch_mode` is the `else` branch (`:85-86`) and serves MONOLITHIC and PREFILL. | +| Disposition | **ADAPT.** The fix is an internal-consistency repair using main's own formula, not a new policy. | +| Caveat to resolve during implementation | `_schedule_batch_mode` returns results grouped per replica; the decode variant returns arrival order. The two cannot simply be merged until the consumers of that ordering are checked. Recorded as an open item. | +| Planned test | Extend `tests/unit/test_cluster_scheduler_dp_lanes.py`: same ordered request stream split across different call boundaries must give the same request-to-lane assignment; cover DP1, DP>2, multiple replicas, non-contiguous replica ids, an empty call, and continuation after it. | + +### W3 — Shared monolithic forward completion + +| Item | Finding | +| --- | --- | +| Defect on main | Present at three layers. `frontier/events/replica_stage_schedule_event.py:157-184` selects the prefill or decode sync path from the lane's own `num_prefill_tokens`; `frontier/scheduler/utils/sync_state.py:29-34` allocates two independent waiting rooms for MONOLITHIC; `frontier/scheduler/utils/forward_sync_state.py:38-41` partitions the step-id namespace into `"prefill"` and `"decode"`. A mixed-phase forward puts one required lane in each room, neither peer is idle, and both rooms stall. | +| Clean ports | The `"forward"` sync kind, and the cross-lane duplicate-request guard in `frontier/scheduler/utils/ep_wave_inputs.py`. | +| Blocked hunk | The candidate's decode final-metrics hunk calls `scheduler._create_corrected_execution_time_for_metrics(...)`, which **main deleted**; `_create_prefill_corrected_execution_time_for_metrics` also changed signature and now lives at `base_cluster_scheduler.py:1138`. Porting verbatim raises `AttributeError`. Main additionally added `metrics_store` and `ep_wave_reporting_enabled` plumbing to `ep_wave_schedule.py` and `prefill_collective.py` that the candidate lacks. | +| Disposition | **ADAPT** for the lifecycle change as one coherent unit; **BLOCKED** for the metrics hunk until it is rewritten against main's current execution-time ownership. | +| Planned test | The behavior matrix in `plan.md` §9, including at least one test that drives the real event loop with overlapping prefill and decode and injects deterministic times only at the predictor boundary. | + +### W4 — Opt-in vLLM-style DP placement + +| Item | Finding | +| --- | --- | +| New public surface in the candidate | `ClusterSchedulerType.VLLM_LOAD_BALANCING`, the CLI token `--cluster_scheduler_config_type vllm_load_balancing` (no other new flag), `VllmLoadBalancingClusterSchedulerConfig`, a new `frontier/config/cluster_scheduler_config.py`, a registry entry, `RequestLoad`, `VllmDPLoadBalancer`, `BaseClusterScheduler.schedule_at` / `on_replica_batch_end`, and `BaseReplicaScheduler.get_request_load`. | +| Reference semantics confirmed | Engine selection scores `waiting * 4 + running` in the **frontend**, `vllm/v1/engine/core_client.py:1146`. The coordinator transports unweighted `[waiting, running]` pairs. Timing constants confirmed: `min_stats_update_interval_ms = 100` (`coordinator.py:116`, a floor rather than a period), the bare literals `5000` (`coordinator.py:198`) and `50` (`coordinator.py:202`). The specification's grouping of the weight 4 with the coordinator constants is corrected here: it belongs to the frontend. | +| Request populations confirmed | Admitted-but-not-yet-scheduled requests count as **running** (`scheduler.py:812-813`, appended inside `schedule()` before the model runs); preemption moves running to waiting within the same step (`:507`, `:537-538`); requests finishing this step are in neither (`:1381-1385`, before `make_stats` at `:1413`). | +| Report suppression confirmed | Two independent stats producers exist; only the DP one feeds the coordinator, and it is suppressed whenever the count pair is unchanged (`core.py:1081`). That is why the 5 s heartbeat exists and why the step counter is a sparse tag rather than a dense counter. | +| Step-identity problem | The candidate uses `ForwardSyncState.get_step_id(batch)` as the report-order key (`vllm_load_balancing_cluster_scheduler.py:52-56`, latched at `vllm_dp_load_balancer.py:64-67` against a single scalar shared across engines). On main that identity is **invalid for dense models** (`base_replica_scheduler.py:459-466` sets `_forward_cohort_id` from a per-DP-lane creation counter that dense never promotes to a Replica-scoped counter, and the policy's guard does not require MoE), **invalid for MoE with DP>1 until W3 lands** (the kind-partitioned namespace gives the two lanes different ids), and valid for DP1. Idle participants are never reported. The id also advances per layer (`sync_entry.py:64`, `forward_sync_state.py:133-139`), so it is monotonic but is not a vLLM step counter. | +| Disposition | **BLOCKED on W3**, then ADAPT. This makes the execution order W2 (independent) then W3 then W4. | +| Decision raised | See D1 below. | + +### W5 — Routing load distribution versus routing implementation identity + +| Item | Finding | +| --- | --- | +| Defect on main | Present and reachable from the public CLI. Two independent mechanisms let one routing implementation's cost model be used for another. | +| Mechanism A | Routing distribution is settable per role (`config.py:4242-4247`, `:4257-4273`, `:4302`), so a PDD run can use `PREFILL=balanced` and `DECODE=random`, which resolve to different runtime paths (`moe_routing_runtime.py:29-32`). `trained_model_signatures` is one set shared across clusters (`shared_prediction_model_manager.py:702`, `:707`, `:778`) and `ffn_signature` (`:1377-1382`) carries **no routing term** (re-verified). The second cluster therefore returns early at `:1384-1386`, before `_validate_moe_dataset_contract` at `:1433-1439`, so there is no fail-fast, and it predicts `moe_gating_routing_topk` using the other cluster's estimator through `_models_view_for_family` (`:4358-4388`), which matches on layer identity alone. | +| Mechanism B | When both clusters do train, `_store_model_precision` keys on `(model_name, identity)` at `:4238` with a layer-shape-derived identity, so the second model overwrites the first. | +| Where identity does reach today | Only dataset row selection (`:1294-1326`, `:1235-1265`) and the per-call `moe_df_cache` key (`:1486-1505`), which is scoped to a single cluster call and therefore never prevents the collision. | +| Correction to the specification's premise | The **persistent disk cache is not the hole**. `_get_model_hash` (`:4008-4056`) hashes `df.to_json()` at `:4033`, and `_load_moe_df` filters rows without dropping columns (`:3628`, `:3712-3720`), so cached artifacts are separated incidentally. | +| Disposition | **PORT** the resolver override, the config field and its copy, the predictor helper, dataset admission, the training signature term, and the per-model training identity. **ADAPT** the registry key widening: the candidate's helper rewrite enumerates only `eager` and `kernel_only` and would break main's third `device_event` measurement family (`:4160-4162`, `:544-551`). | +| Two implementation caveats | The candidate appends `_routing{path}` to the whole `ffn_signature`, which separates every MoE FFN model rather than only the routing-topk model. That is correctness-safe but over-broad and should be narrowed. Main's standalone trainer already calls this concept `routing_runtime_path` (`frontier/training/cli.py:153`), so the candidate's `moe_gating_routing_runtime_path` would be a third spelling; one name must be chosen before any public flag exists. | + +### W6 — Legacy fused-MoE profiling arithmetic + +| Item | Finding | +| --- | --- | +| What main's legacy path omits | Exactly two steps. `_run_fused_moe_iteration` (`frontier/profiling/moe/moe_vllm_kernel.py:365-431`) takes a bare first-half slice at `:404-405`, discarding the up-projection half instead of applying gated SiLU, and performs no local top-k reduction (`moe_sum` appears nowhere in the file). FP8 activation quantization (`:407-413`) and routing weights on the second GEMM (`:423`) are already correct; only the operand is wrong. | +| No conflict with main-only work | The functional and MXFP4 branch returns at `:869`, before the legacy allocations at `:904`. | +| Required adaptations | Port hunks, not the file: a whole-file take would revert 419 lines of main-only work (the functional `fused_experts` adapter, MXFP4/AITER, `device_event` timing, profile-method platform validation). Move `from vllm import _custom_ops as ops` out of the top-level `try` so it cannot perturb main's two-branch API detection (`:116-140`). Reuse main's existing `SiluAndMul` wrapper (`frontier/profiling/common/layers/activation.py:8-34`) rather than the raw op. | +| Measured scope today | The whole `_step` body, timed either with per-iteration CUDA events and a synchronize (`:603-626`) or through `record_function("vidur_moe_grouped_gemm")` (`:629-653`). All buffers are allocated outside the timed region. | +| Disposition | **PORT** the gated-activation repair. **BLOCKED** on the local output reduction; see D2 below. | + +## Decision checkpoints for the user + +### D1 — Scope of the opt-in DP placement strategy + +`ForwardSyncState.get_step_id(batch)` is not a valid report-order key for every configuration the candidate's strategy accepts: dense models and, until W3 lands, MoE with DP>1. The options are to supply a correct Replica-scoped step identity as part of W3, or to narrow the strategy's advertised capability so that it rejects the configurations where no valid identity exists. Recommendation: land W3 first, then re-evaluate whether W3's shared forward identity is itself the correct key; narrow the capability only if it is not. This is recorded rather than blocking, because W2 and W3 proceed independently. + +### D2 — Measurement scope of the corrected legacy MoE profiling + +Adding the local top-k output reduction changes what the `moe_grouped_gemm` measurement contains. On main that label **already means two different things**: the legacy path measures GEMM1, slice and GEMM2, while the functional path (`:837-869`) measures vLLM's `fused_experts` end to end, which already includes both the activation and `moe_sum`. Adding both to the legacy path converges the two, but no existing column can separate old rows from new: `moe_grouped_gemm_backend` (`frontier/profiling/moe/moe_wrapper.py:50-59`) encodes only `{frontier_loop, vllm_fused, vllm_aiter_mxfp4}` and both vLLM paths emit `vllm_fused`; `measurement_type` records the timer, `quant_signature` the quantization, `model_architecture_profile` the model, and `typed_operator_contracts` the family and TP/EP semantics. The vLLM API version is never written to the CSV. + +The candidate's own donor proposal explicitly declined to include `moe_sum`, calling it an operator-ownership decision that must not be folded in silently, and `MOE_FAMILY` (`frontier/operators/families.py:77-133`) and `MoETime` (`frontier/entities/time_components.py:503-547`) still have no reduction term. The candidate's numerical parity test compares against `fused_experts`, whose return is already reduced, so that test presupposes the decision. + +The choice is therefore: (a) add a narrowly scoped profiling metadata column that records the measured entry point and scope, then include the reduction; (b) include only the gated-activation repair in this PR and defer the reduction; or (c) defer the whole W6 package. Recommendation: (a), because the two-meanings problem exists on main today independently of this change, and a scope column fixes it once. + +## Candidate changes dropped as experiment scaffolding + +| Path pattern | Count | Reason | | --- | --- | --- | -| `tests/e2e/issue26_*`, `tests/integration/issue26_dp_*rca*`, `tests/performance/issue26_*` | DROP (default, per `plan.md` A11) | Experiment scripts; re-evaluate only a named helper. | +| `tests/e2e/issue26_*` | 52 | Calibration experiment drivers and GPU worker shell scripts. | +| `tests/integration/issue26_*` | 5 | Root-cause-analysis drivers. `issue26_dp_coordinator_reference.py` is reviewed separately before W4. | +| `tests/performance/issue26_*` | 7 | Profiling and microbenchmark drivers. | +| `task_memory/task_2026-09-07_issue26_ttft_h200/` | ~1006 files | Main deliberately removed local task memory in `26b490a`. Read for evidence; not vendored. | +| Whole-file take of `frontier/profiling/moe/moe_vllm_kernel.py` | 1 | Would revert 419 lines of main-only work. | +| `frontier/config/cluster_scheduler_config` import block as written | 1 | The module does not exist on main; the refactor PR creates one with a different content boundary. | +| Non-routing `config.py` hunks (GDN guards, `gdn_input_file`) | several | Unrelated to any work package here. | +| `rtol=0, atol=0` bitwise parity fixture and its `SimpleNamespace` stub | 2 | The stub depends on a `getattr(self, "_cluster_type", None)` form main has since hardened. | + +Retained from the candidate's tests: `tests/unit/test_cluster_scheduler_dp_lanes.py`, `test_monolithic_mixed_forward_sync.py`, `test_vllm_dp_load_balancer.py`, `test_moe_routing_runtime.py`, `test_moe_routing_runtime_model_sharing.py`, `test_moe_fused_expert_numerical_parity.py`, `test_collective_sim_zero_payload.py`, each subject to review before adoption. + +## Defect found in the candidate itself + +The candidate deletes `VLLMv1EngineReplicaScheduler._get_num_waiting_reqs_for_decision_log` while `frontier/scheduler/replica_scheduler/sglang_style_replica_scheduler.py:65` still calls it, and `SGLangStyleReplicaScheduler` subclasses `VLLMv1EngineReplicaScheduler`. Verified: at the candidate revision the caller exists and no definition does, so the SGLang decision-log path raises `AttributeError`. This PR does not reproduce that deletion. -## Design decisions +## Open items -Pending Step 1. +1. Whether `_schedule_batch_mode`'s per-replica grouped return order is load-bearing for the consumers of `ClusterScheduleEvent`'s request mapping (W2). +2. Whether W3's shared forward identity is the correct report-order key for W4, or whether a separate Replica-scoped step identity is required (D1). +3. Which of the three existing spellings becomes the single public name for the routing implementation identity (W5). +4. Pipeline-parallel behavior of the component ledgers is untested in both trees. +5. The upstream `fused_moe.py` fork change passes a fifth `renormalize` argument to `torch.ops._moe_C.topk_softmax` while the in-tree schema still declares four; the prebuilt extension could not be inspected on this host. Numerically a no-op, but it would raise rather than degrade. Relevant only if W6 native validation runs against the fork's compiled package. ## Final code-review findings diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index 6743f2bd..eaa0365b 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -4,7 +4,8 @@ | Date | Change | | --- | --- | -| 2026-09-21 | Created. Environment recorded; baseline results are recorded in the refactor task's Step 0 report because both branches share the same base commit. | +| 2026-09-21 | Created. Environment recorded; baseline results recorded in the refactor task's Step 0 report because both branches share the same base commit. | +| 2026-09-21 | Step 1 recorded: audit spot checks and the vLLM reference identity check. | ## Environment @@ -20,4 +21,21 @@ | Step | Purpose | Source under test | Command | Outcome | Baseline comparison | Limits | | --- | --- | --- | --- | --- | --- | --- | -| 0 | Baseline for touched areas | `1f694f7` | See `task_memory/task_2026-09-21_oversized_module_split/test_report_2026-09-21_step0_baseline.md` | 84 passed / 10 failed (pre-existing or environmental, listed in the report); two dummy smokes PASS | Base itself | Shared with the refactor branch (same base). | +| 0 | Baseline for touched areas | `1f694f7` | See `task_memory/task_2026-09-21_oversized_module_split/test_report_2026-09-21_step0_baseline.md` | 84 passed / 10 failed (pre-existing or environmental); co-location and PDD dense dummy smokes PASS | Base itself | Shared with the refactor branch (same base). | +| 1 | Source audit only; no code executed | `1f694f7` vs `a7b3320`; vLLM `ea95f57` | `git diff` / `git show` at the three pinned revisions; `git fetch` of upstream `v0.10.2` into the reference checkout | Recorded in `review.md`, `audit_scheduler.md`, `audit_predictor_profiling.md`, `reference_vllm_0_10_2.md` | n/a | An audit establishes intent and current behavior from source. It does not establish runtime behavior; every disposition still needs its own test in the step that implements it. | + +## Step 1 spot checks + +Five consequential audit claims were re-verified directly against the source before being recorded as project facts. + +| # | Claim | Verification | Result | +| --- | --- | --- | --- | +| 1 | Main's `_schedule_batch_mode` restarts DP-lane assignment at zero on every call | Read `round_robin_cluster_scheduler.py:365-389`: the persistent `_request_counter` drives `replica_idx`, while `dp_id = local_idx % self._replica_dp_size` uses the per-call index | CONFIRMED | +| 2 | Main already implements the intended formula for the DECODE role only | Read `:432-444` (`dp_id = (counter + idx) // num_replicas % dp_size`) and the dispatch at `:74-86` (`ClusterType.DECODE` only) | CONFIRMED | +| 3 | The candidate removes a method the SGLang scheduler still calls | `git grep _get_num_waiting_reqs_for_decision_log` at `a7b3320` returns only the caller at `sglang_style_replica_scheduler.py:65`; at `1f694f7` it returns the caller plus the definition at `vllm_v1_engine_replica_scheduler.py:2136` | CONFIRMED | +| 4 | `ffn_signature` on main carries no routing-runtime term | Read `shared_prediction_model_manager.py:1377-1382`: device, model, TP, is-MoE, architecture profile, typed contract hash, measurement family only | CONFIRMED | +| 5 | The vLLM selection weight and the coordinator intervals | Read `core_client.py:1146` (`score = waiting * 4 + running`), `coordinator.py:116` (`min_stats_update_interval_ms: int = 100`), `:198` (`5000`), `:202` (`50`) | CONFIRMED, with the correction that the weight lives in the frontend, not the coordinator | + +## Limits of Step 1 + +The audit reads source at pinned revisions. It does not run the candidate, does not measure timing, and makes no accuracy claim. The reference checkout was fetched read-only; an `upstream` remote and the tag `upstream-v0.10.2` now exist in `.real-engine/vLLM-BS`, which is outside both PR branches. From 8d46c1e8c2c5b45a426876ae4ec39ef7d7202852 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Mon, 21 Sep 2026 23:32:30 +0800 Subject: [PATCH 03/80] docs(issue26): record that the split prerequisite is met and Step 2 is unblocked --- .../task_2026-09-21_issue26_correctness_pr/progress.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index d489c1bd..8a566786 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -12,10 +12,10 @@ | --- | --- | | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | -| Prerequisite | Refactor PR fidelity matrix PASS (see `task_memory/task_2026-09-21_oversized_module_split/progress.md`) | -| Current step | Step 1 complete (audit); Steps 2-7 wait on the refactor branch | +| Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate; the split measured 67 of 67 fidelity cases identical with no predictor cache name differences. | +| Current step | Step 1 complete; Step 2 unblocked | | Publication | PUSHED_VERIFIED (records) | -| Next action | Open the draft PR, then start Step 2 (RR DP rotation) once the refactor branch has landed its splits. Decisions D1 and D2 in `review.md` are open. | +| Next action | Step 2: preserve round-robin DP rotation across scheduling calls. Decisions D1 and D2 in `review.md` remain open and are the user's; neither blocks Step 2. | ## Step status @@ -36,3 +36,4 @@ - 2026-09-21: Baseline on the shared base recorded in the refactor task's Step 0 report; vLLM reference cloned (no tags in the fork; upstream `v0.10.2` comparison deferred to Step 1). - 2026-09-21: Draft specification analyzed; eleven facts verified against main, the candidate, the submodule remote, and the host; planning interview settled twelve decisions (see `requirements.md`). Records landed under `task_memory/`, `.gitignore` narrowed, `plan.md` carries the Amendments table. - 2026-09-21: Step 1 audit complete. Three pinned-source audits landed as `audit_scheduler.md`, `audit_predictor_profiling.md`, `reference_vllm_0_10_2.md`; dispositions and two decision checkpoints recorded in `review.md`. Execution order corrected to W2 first, then W3, then W4, because the candidate's report-order key depends on the shared forward identity. The vLLM fork was confirmed to be a direct descendant of upstream v0.10.2 with the DP placement files byte-identical to the tag. +- 2026-09-21: Merged the completed oversized-module split into this branch. Merge rather than rebase, so the published review anchors stay valid. The modules this PR edits are now `config.py` 788 with `cluster_config.py` 1888, the vLLM V1 replica scheduler 1386, the prediction model manager 722 and the MoE predictor 1557, each with named child modules that give the planned fixes a clear owner. Step 2 is unblocked. From 6ab521dcf89734a886250b1ca1370269bf382f27 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Mon, 21 Sep 2026 23:41:02 +0800 Subject: [PATCH 04/80] fix(scheduler): keep round-robin DP rotation across scheduling calls Related to #26. `_schedule_batch_mode` derived the replica index from a counter that persists across calls but the DP lane from the request's index within the current call. The lane therefore restarted at zero every time `schedule()` was entered, so an identical ordered request stream landed differently depending only on how it happened to be divided between calls. With one replica and two lanes, admitting requests one at a time put every request on lane 0. Both values now come from one persistent ordinal. This is the rotation `_schedule_decode_lane_round_robin` already applies to the unified decode role, so the change makes the monolithic and prefill paths consistent with a formula already in this file rather than introducing a policy. The per-replica grouping of the returned mapping is unchanged, and a test asserts that specifically; it passes on the pre-fix code too. Four regression tests cover call partitioning, non-contiguous replica ids, DP1 through DP4, an empty scheduling call and the return order. Their sensitivity was verified by stashing the fix and rerunning: three fail on the pre-fix code for the right reason, lanes collapsing to lane 0. test_replica_identity_contract pinned the literal expression that changed. What it guards is that a non-FFN cluster scheduler takes the lane modulo the Replica-local DP size rather than a global or expert-parallel cardinality, so it now asserts that property over every lane assignment instead of matching one expression. It still fails if the lane is ever taken modulo anything else. Unit comparison against the refactor tip over 73 files: identical failure identities, four new passes. --- .../round_robin_cluster_scheduler.py | 26 ++-- .../progress.md | 10 +- tests/unit/test_cluster_scheduler_dp_lanes.py | 119 ++++++++++++++++++ tests/unit/test_replica_identity_contract.py | 17 ++- 4 files changed, 159 insertions(+), 13 deletions(-) diff --git a/frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py b/frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py index 466f1baa..1fb048a3 100644 --- a/frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py +++ b/frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py @@ -362,28 +362,36 @@ def _schedule_batch_mode(self) -> List[Tuple[int, int, Request]]: Processes all requests in the queue at once using traditional round-robin. """ - # First, distribute requests to replicas using round-robin - replica_requests = [[] for _ in range(self._num_replicas)] + # Both the replica and the DP lane come from one ordinal that persists + # across calls, so an identical ordered request stream lands on the same + # replica and the same lane however it is divided between calls. This is + # the rotation `_schedule_decode_lane_round_robin` already applies to the + # unified decode role. + replica_requests: List[List[Tuple[int, Request]]] = [ + [] for _ in range(self._num_replicas) + ] replica_ids = list(self._cluster.replicas.keys()) request_idx = 0 while self._request_queue: request = self._request_queue.pop(0) - replica_idx = (self._request_counter + request_idx) % self._num_replicas - replica_requests[replica_idx].append(request) + ordinal = self._request_counter + request_idx + replica_idx = ordinal % self._num_replicas + dp_id = (ordinal // self._num_replicas) % self._replica_dp_size + replica_requests[replica_idx].append((dp_id, request)) request_idx += 1 self._request_counter += request_idx - # Distribute requests across logical DP lanes inside each Replica. + # Results stay grouped per replica, which is the order this method has + # always returned. request_mapping = [] - for replica_idx, requests in enumerate(replica_requests): - if not requests: + for replica_idx, lane_requests in enumerate(replica_requests): + if not lane_requests: continue replica_id = replica_ids[replica_idx] - for local_idx, request in enumerate(requests): - dp_id = local_idx % self._replica_dp_size + for dp_id, request in lane_requests: request_mapping.append((replica_id, dp_id, request)) return request_mapping diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 8a566786..3ea1b731 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -13,9 +13,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate; the split measured 67 of 67 fidelity cases identical with no predictor cache name differences. | -| Current step | Step 1 complete; Step 2 unblocked | +| Current step | Step 2 implemented; awaiting its fidelity measurement | | Publication | PUSHED_VERIFIED (records) | -| Next action | Step 2: preserve round-robin DP rotation across scheduling calls. Decisions D1 and D2 in `review.md` remain open and are the user's; neither blocks Step 2. | +| Next action | Measure Step 2 against the stated expectation, then start Step 3, the shared monolithic forward lifecycle. | ## Step status @@ -23,7 +23,7 @@ | --- | --- | --- | --- | --- | --- | | 0 | Worktree, references, baseline | PASS | PASS (baseline recorded) | PUSHED_VERIFIED | NOT_REVIEWED | | 1 | Candidate/vLLM audit | PASS | n/a (source audit) | LOCAL_ONLY | NOT_REVIEWED | -| 2 | RR DP rotation | NOT_STARTED | — | — | — | +| 2 | RR DP rotation | IN_PROGRESS | unit PASS; matrix pending | LOCAL_ONLY | NOT_REVIEWED | | 3 | Shared monolithic forward | NOT_STARTED | — | — | — | | 4 | Opt-in vLLM DP placement | NOT_STARTED | — | — | — | | 5 | Routing implementation identity | NOT_STARTED | — | — | — | @@ -37,3 +37,7 @@ - 2026-09-21: Draft specification analyzed; eleven facts verified against main, the candidate, the submodule remote, and the host; planning interview settled twelve decisions (see `requirements.md`). Records landed under `task_memory/`, `.gitignore` narrowed, `plan.md` carries the Amendments table. - 2026-09-21: Step 1 audit complete. Three pinned-source audits landed as `audit_scheduler.md`, `audit_predictor_profiling.md`, `reference_vllm_0_10_2.md`; dispositions and two decision checkpoints recorded in `review.md`. Execution order corrected to W2 first, then W3, then W4, because the candidate's report-order key depends on the shared forward identity. The vLLM fork was confirmed to be a direct descendant of upstream v0.10.2 with the DP placement files byte-identical to the tag. - 2026-09-21: Merged the completed oversized-module split into this branch. Merge rather than rebase, so the published review anchors stay valid. The modules this PR edits are now `config.py` 788 with `cluster_config.py` 1888, the vLLM V1 replica scheduler 1386, the prediction model manager 722 and the MoE predictor 1557, each with named child modules that give the planned fixes a clear owner. Step 2 is unblocked. +- 2026-09-21: Step 2 implemented. `_schedule_batch_mode` now derives both the replica index and the DP lane from one ordinal that persists across scheduling calls, which is the rotation `_schedule_decode_lane_round_robin` already applies to the unified decode role. The per-replica grouping of the returned mapping is unchanged and is asserted separately. +- 2026-09-21: Four regression tests added to `tests/unit/test_cluster_scheduler_dp_lanes.py`, covering call partitioning, non-contiguous replica ids, DP1 through DP4, an empty scheduling call, and the preserved return order. Sensitivity was verified by stashing the fix and rerunning: three of the four fail on the pre-fix code for the right reason, alternating lanes collapsing to lane 0, and the return-order test passes both ways. +- 2026-09-21: `tests/unit/test_replica_identity_contract.py` pinned the literal expression `dp_id = local_idx % self._replica_dp_size`. The property it guards is that a non-FFN cluster scheduler derives the lane from the Replica-local DP size rather than a global or expert-parallel cardinality. The check now asserts that property over every lane assignment instead of one literal, so it survives a change to the ordinal but still fails if the lane is taken modulo anything else. +- 2026-09-21: Unit comparison against the refactor tip `db15e64` over 73 files: identical failure identities, 1808 to 1812 passing, the four new tests being the difference. diff --git a/tests/unit/test_cluster_scheduler_dp_lanes.py b/tests/unit/test_cluster_scheduler_dp_lanes.py index b6b2614d..89406dbe 100644 --- a/tests/unit/test_cluster_scheduler_dp_lanes.py +++ b/tests/unit/test_cluster_scheduler_dp_lanes.py @@ -4,6 +4,9 @@ from frontier.scheduler.cluster_scheduler.lor_cluster_scheduler import ( LORClusterScheduler, ) +from frontier.scheduler.cluster_scheduler.round_robin_cluster_scheduler import ( + RoundRobinClusterScheduler, +) from frontier.scheduler.cluster_scheduler.random_cluster_scheduler import ( RandomClusterScheduler, ) @@ -179,3 +182,119 @@ def lane(*, busy: bool, empty: bool): assert events[0]._replica_id == 7 assert events[0]._stage_id == 4 assert events[0]._replica_local_id == 1 + + +def _round_robin_scheduler( + *, replica_ids: list[int], dp_size: int +) -> RoundRobinClusterScheduler: + """A round-robin scheduler over the given replicas, with an empty queue.""" + + scheduler = RoundRobinClusterScheduler.__new__(RoundRobinClusterScheduler) + scheduler._cluster_type = ClusterType.MONOLITHIC + scheduler._num_replicas = len(replica_ids) + scheduler._replica_dp_size = dp_size + scheduler._cluster = SimpleNamespace( + replicas={replica_id: object() for replica_id in replica_ids} + ) + scheduler._request_queue = [] + scheduler._request_counter = 0 + return scheduler + + +def _placements_for_call_sizes( + *, replica_ids: list[int], dp_size: int, call_sizes: list[int] +) -> list[tuple[int, int]]: + """Schedule one request stream in the given batches. + + Returns the (replica id, DP lane) of each request by its position in the + stream. The stream is the same for every call partitioning, so the result + may not depend on `call_sizes`. Comparing by stream position rather than by + request id is what makes two separately constructed runs comparable. + """ + + scheduler = _round_robin_scheduler(replica_ids=replica_ids, dp_size=dp_size) + total = sum(call_sizes) + requests = [_request() for _ in range(total)] + position_of = {request.id: index for index, request in enumerate(requests)} + placements: dict[int, tuple[int, int]] = {} + + offset = 0 + for size in call_sizes: + scheduler._request_queue = requests[offset:offset + size] + offset += size + for replica_id, dp_id, request in scheduler._schedule_batch_mode(): + position = position_of[request.id] + assert position not in placements, "a request was scheduled twice" + placements[position] = (replica_id, dp_id) + + assert len(placements) == total, "every request is scheduled exactly once" + return [placements[position] for position in range(total)] + + +def test_round_robin_dp_lane_does_not_depend_on_call_partitioning() -> None: + """The defect this covers: the DP lane restarted at zero on every call. + + With one replica and two lanes, scheduling eight requests one at a time put + every request on lane 0, while scheduling them in one call alternated. The + lane must follow the request's position in the stream, not its position + within the call that happened to carry it. + """ + + one_at_a_time = _placements_for_call_sizes( + replica_ids=[7], dp_size=2, call_sizes=[1] * 8 + ) + single_burst = _placements_for_call_sizes( + replica_ids=[7], dp_size=2, call_sizes=[8] + ) + uneven = _placements_for_call_sizes( + replica_ids=[7], dp_size=2, call_sizes=[3, 1, 4] + ) + + assert one_at_a_time == single_burst == uneven + assert one_at_a_time == [(7, 0), (7, 1)] * 4 + + +def test_round_robin_placement_is_stable_across_topologies() -> None: + """Replica ids need not be contiguous and lanes may outnumber two.""" + + cases = [ + ([7], 1), + ([7], 4), + ([3, 11], 1), + ([3, 11], 2), + ([3, 11, 42], 3), + ] + for replica_ids, dp_size in cases: + burst = _placements_for_call_sizes( + replica_ids=replica_ids, dp_size=dp_size, call_sizes=[12] + ) + incremental = _placements_for_call_sizes( + replica_ids=replica_ids, dp_size=dp_size, call_sizes=[1] * 12 + ) + assert burst == incremental, (replica_ids, dp_size) + assert {replica_id for replica_id, _ in burst} <= set(replica_ids) + assert all(0 <= dp_id < dp_size for _, dp_id in burst) + + +def test_round_robin_survives_an_empty_scheduling_call() -> None: + """An empty call must neither advance the rotation nor reset it.""" + + with_gap = _placements_for_call_sizes( + replica_ids=[3, 11], dp_size=2, call_sizes=[2, 0, 2, 0, 4] + ) + without_gap = _placements_for_call_sizes( + replica_ids=[3, 11], dp_size=2, call_sizes=[8] + ) + assert with_gap == without_gap + + +def test_round_robin_returns_results_grouped_by_replica() -> None: + """The return order groups each call's results per replica, as before.""" + + scheduler = _round_robin_scheduler(replica_ids=[3, 11], dp_size=2) + scheduler._request_queue = [_request() for _ in range(6)] + + mapping = scheduler._schedule_batch_mode() + + replica_order = [replica_id for replica_id, _, _ in mapping] + assert replica_order == [3, 3, 3, 11, 11, 11] diff --git a/tests/unit/test_replica_identity_contract.py b/tests/unit/test_replica_identity_contract.py index 57737d86..ba6081b3 100644 --- a/tests/unit/test_replica_identity_contract.py +++ b/tests/unit/test_replica_identity_contract.py @@ -16,7 +16,22 @@ def test_non_ffn_cluster_scheduler_uses_replica_local_dp_identity() -> None: non_ffn_block = source[block_start:block_end] assert "self._replica_scheduler_count = attn_dp" in non_ffn_block assert "self._replica_dp_size = attn_dp" in non_ffn_block - assert "dp_id = local_idx % self._replica_dp_size" in round_robin_source + # What this guards is the lane cardinality: a non-FFN cluster scheduler must + # derive the DP lane from the Replica-local DP size, not from a global or + # expert-parallel cardinality. Assert that property over every lane + # assignment rather than one literal expression, so the check survives a + # change to how the ordinal is computed but still fails if the lane is ever + # taken modulo something else. + lane_assignments = [ + line.strip() + for line in round_robin_source.splitlines() + if line.strip().startswith("dp_id = ") + ] + assert lane_assignments, "the round-robin scheduler must assign a DP lane" + assert all( + assignment.endswith("% self._replica_dp_size") + for assignment in lane_assignments + ), lane_assignments def test_production_scheduler_surface_has_no_retired_replica_dp_size() -> None: From c18eb2cd7abc7425e79dfac285230821eeadb42b Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Mon, 21 Sep 2026 23:50:24 +0800 Subject: [PATCH 05/80] docs(issue26): record the Step 2 measurement and scope Step 3 --- .../progress.md | 22 +++++++- .../validation.md | 55 +++++++++++++++++++ 2 files changed, 74 insertions(+), 3 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 3ea1b731..b0641d8f 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -13,9 +13,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate; the split measured 67 of 67 fidelity cases identical with no predictor cache name differences. | -| Current step | Step 2 implemented; awaiting its fidelity measurement | +| Current step | Step 2 complete; Step 3 scoped, not started | | Publication | PUSHED_VERIFIED (records) | -| Next action | Measure Step 2 against the stated expectation, then start Step 3, the shared monolithic forward lifecycle. | +| Next action | Step 3: the shared monolithic forward lifecycle. Scoped below; not started. | ## Step status @@ -23,7 +23,7 @@ | --- | --- | --- | --- | --- | --- | | 0 | Worktree, references, baseline | PASS | PASS (baseline recorded) | PUSHED_VERIFIED | NOT_REVIEWED | | 1 | Candidate/vLLM audit | PASS | n/a (source audit) | LOCAL_ONLY | NOT_REVIEWED | -| 2 | RR DP rotation | IN_PROGRESS | unit PASS; matrix pending | LOCAL_ONLY | NOT_REVIEWED | +| 2 | RR DP rotation | PASS | unit PASS; matrix PASS against a stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 3 | Shared monolithic forward | NOT_STARTED | — | — | — | | 4 | Opt-in vLLM DP placement | NOT_STARTED | — | — | — | | 5 | Routing implementation identity | NOT_STARTED | — | — | — | @@ -41,3 +41,19 @@ - 2026-09-21: Four regression tests added to `tests/unit/test_cluster_scheduler_dp_lanes.py`, covering call partitioning, non-contiguous replica ids, DP1 through DP4, an empty scheduling call, and the preserved return order. Sensitivity was verified by stashing the fix and rerunning: three of the four fail on the pre-fix code for the right reason, alternating lanes collapsing to lane 0, and the return-order test passes both ways. - 2026-09-21: `tests/unit/test_replica_identity_contract.py` pinned the literal expression `dp_id = local_idx % self._replica_dp_size`. The property it guards is that a non-FFN cluster scheduler derives the lane from the Replica-local DP size rather than a global or expert-parallel cardinality. The check now asserts that property over every lane assignment instead of one literal, so it survives a change to the ordinal but still fails if the lane is taken modulo anything else. - 2026-09-21: Unit comparison against the refactor tip `db15e64` over 73 files: identical failure identities, 1808 to 1812 passing, the four new tests being the difference. +- 2026-09-21: Step 2 measured and PASS against the stated expectation. 71 of 71 cases compared, 68 identical, and the mismatch set is exactly the three cases predicted to move. Nothing moved that was not expected to, and nothing expected to move stayed. Lane occupancy confirms the direction: collapsed onto lane zero before, evenly spread after, with the control case unchanged. Full record in `validation.md`. +- 2026-09-21: Merged the refactor tip so this branch carries the four DP placement cases in its own fidelity case table. Without it the branch's own harness still had the 67-case table, and measuring the branch with its own tooling would have exercised a table that cannot see the fix. + +## Step 3 scoping, not started + +The defect is present on main at three layers, and the fix has to change all three together or the intermediate state deadlocks differently: + +1. `frontier/events/replica_stage_schedule_event.py:157-184` picks the prefill or the decode sync path from the batch's own `num_prefill_tokens`, so two DP lanes of one forward step can take different paths. +2. `frontier/scheduler/utils/sync_state.py:29-40` allocates two independent waiting rooms for `MONOLITHIC`. +3. `frontier/scheduler/utils/forward_sync_state.py:38-41` partitions the open-step binding table by kind. + +One refinement over the audit's framing, from reading the code: `ForwardSyncState._next_step_id_by_replica` is **already** shared across kinds, keyed by replica alone. Only `_open_steps_by_kind` is partitioned. So step-id allocation is already Replica-scoped and monotonic; what is partitioned is the binding table and the waiting room. That narrows the change. + +Surface: about 3,500 lines across `replica_stage_schedule_event.py`, `sync_entry.py`, `prefill_collective.py`, `decode_collective.py`, `ep_wave_schedule.py`, `ep_wave_inputs.py` and `base_cluster_scheduler.py`, with 11 call sites of the sync-kind and sync-path selection. + +Blocked hunk carried from the audit: the candidate's decode final-metrics change calls `_create_corrected_execution_time_for_metrics`, which main deleted, so it needs rewriting against main's current execution-time ownership rather than porting. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index eaa0365b..844f80d8 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -6,6 +6,7 @@ | --- | --- | | 2026-09-21 | Created. Environment recorded; baseline results recorded in the refactor task's Step 0 report because both branches share the same base commit. | | 2026-09-21 | Step 1 recorded: audit spot checks and the vLLM reference identity check. | +| 2026-09-21 | Step 2 recorded: unit sensitivity and the fidelity measurement against a stated expectation. | ## Environment @@ -39,3 +40,57 @@ Five consequential audit claims were re-verified directly against the source bef ## Limits of Step 1 The audit reads source at pinned revisions. It does not run the candidate, does not measure timing, and makes no accuracy claim. The reference checkout was fetched read-only; an `upstream` remote and the tag `upstream-v0.10.2` now exist in `.real-engine/vLLM-BS`, which is outside both PR branches. + +## Step 2 — round-robin DP rotation + +### What the fidelity matrix can and cannot show here + +Unlike the module splits, this is a behavior fix, so the matrix is expected to report mismatches. A result of zero mismatches would mean the fix did not reach the path. The gate is therefore an expectation stated **before** measuring, not a comparison against zero. + +The expectation was derived from the arithmetic: cases with `_replica_dp_size == 1` cannot change, because the old lane expression was `local_idx % 1` and the new one is `(ordinal // num_replicas) % 1`, both always zero. Within a single scheduling call starting at counter zero the two expressions also agree, so a case that admits its whole stream in one call cannot move either. + +Four cases were added to the matrix specifically to sit in the defective regime, and their regime was verified by measuring lane occupancy on the base commit rather than by reasoning. + +| Case | Base-commit lane occupancy | Expected | +| --- | --- | --- | +| `dp_dense_online_lanes2` | all 45 records on lane 0 | must move | +| `dp_dense_online_lanes4` | all 45 records on lane 0 | must move | +| `dp_dense_online_lanes2_replicas2` | replicas split 28/28 while every record is lane 0 | must move | +| `dp_dense_offline_lanes2_replicas2` | lanes split 70/70 | must not move | +| `coloc_dense_offline_attn_dp2` | lanes split 39/39 | must not move | + +`dp_dense_online_lanes2_replicas2` is the sharpest single case: both rotations are visible in one run, and only the replica index was rotating. + +### Result + +| Field | Record | +| --- | --- | +| Source under test | `6ab521d`, measured from a detached checkout | +| Cases compared | 71 of 71 | +| Identical | 68 | +| Mismatched | exactly `dp_dense_online_lanes2`, `dp_dense_online_lanes4`, `dp_dense_online_lanes2_replicas2` | +| Expected to move but did not | none | +| Moved but was not expected to | none | +| Predictor cache names | no differences | +| Result | **PASS against the stated expectation** | + +Direction, from the stage ledger's `replica_local_id`: + +| Case | Baseline | Candidate | +| --- | --- | --- | +| `dp_dense_online_lanes2` | `{0: 45}` | `{0: 44, 1: 45}` | +| `dp_dense_online_lanes4` | `{0: 45}` | `{0: 43, 1: 43, 2: 44, 3: 45}` | +| `dp_dense_online_lanes2_replicas2` | `{0: 56}` | `{0: 55, 1: 56}` | +| `dp_dense_offline_lanes2_replicas2` (control) | `{0: 70, 1: 70}` | unchanged | + +Collapsed onto lane zero before, evenly spread after, and the control did not budge. + +### Unit evidence + +Comparison against the refactor tip `db15e64` over 73 files: identical failure identities, 1808 to 1812 passing, the four new tests being the difference. + +Sensitivity was verified by stashing the fix and rerunning, not by reasoning. Three of the four new tests fail on the pre-fix code for the right reason, alternating lanes collapsing to lane 0. The fourth, which asserts the per-replica grouping of the returned mapping, passes on both, which is what confirms the ordering was preserved. + +### Limits + +The prefill role reaches the same placement path, but **no shipped recipe can give it more than one lane**, so the matrix cannot cover that half. A dense model in a disaggregated architecture is rejected outright, and the MoE wrappers enforce `ATTN_TP == MOE_TP * MOE_EP` while the runtime enforces `attn_tp * attn_dp == moe_tp * moe_ep`, which have no common solution above one lane. Both routes were attempted and both were rejected, so this is measured rather than inferred. The unit test is the only evidence for the prefill half of this fix, and the PR says so. From 33f0d5a09c48be310a78053e24de88626c17c553 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Mon, 21 Sep 2026 23:51:36 +0800 Subject: [PATCH 06/80] docs(issue26): correct what the sync state actually partitions The audit recorded that forward_sync_state partitions the step-id namespace by synchronization kind. It does not. _next_step_id_by_replica is keyed by replica alone, so allocation is already Replica-scoped and monotonic across prefill and decode. What is partitioned is _open_steps_by_kind, the open-step binding table, and the waiting room beside it. The defect is unchanged: a mixed-phase forward still puts one required lane in each table and stalls. But the change Step 3 has to make is narrower than the audit implied, and the reason the candidate's report-order key is invalid for MoE with DP above one is different: the ids come from one counter, and the lanes never reach a shared step to key a report on. Verified against the source at c18eb2c rather than taken from the earlier reading. --- task_memory/task_2026-09-21_issue26_correctness_pr/review.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index f0df39ad..cf21b9ac 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -6,6 +6,7 @@ | --- | --- | | 2026-09-21 | Created with the pinned source snapshot. | | 2026-09-21 | Step 1 complete: candidate and vLLM audits landed, dispositions recorded, two decision checkpoints raised. | +| 2026-09-21 | Corrected the W3 and W4 rows: the step-id namespace is not partitioned by sync kind, only the open-step binding table is. Verified against `forward_sync_state.py` at `c18eb2c`. | ## Pinned source snapshot @@ -52,7 +53,7 @@ Claims in those reports that this PR depends on were re-verified directly agains | Item | Finding | | --- | --- | -| Defect on main | Present at three layers. `frontier/events/replica_stage_schedule_event.py:157-184` selects the prefill or decode sync path from the lane's own `num_prefill_tokens`; `frontier/scheduler/utils/sync_state.py:29-34` allocates two independent waiting rooms for MONOLITHIC; `frontier/scheduler/utils/forward_sync_state.py:38-41` partitions the step-id namespace into `"prefill"` and `"decode"`. A mixed-phase forward puts one required lane in each room, neither peer is idle, and both rooms stall. | +| Defect on main | Present at three layers. `frontier/events/replica_stage_schedule_event.py:157-184` selects the prefill or decode sync path from the lane's own `num_prefill_tokens`; `frontier/scheduler/utils/sync_state.py:29-34` allocates two independent waiting rooms for MONOLITHIC; `frontier/scheduler/utils/forward_sync_state.py:38` partitions the open-step binding table `_open_steps_by_kind` into `"prefill"` and `"decode"`. A mixed-phase forward puts one required lane in each room, neither peer is idle, and both rooms stall. **Correction to an earlier version of this row:** the step-id *namespace* is not partitioned. `_next_step_id_by_replica` is keyed by replica alone (`:42`, `:133`, `:139`, `:163`), so allocation is already Replica-scoped and monotonic across both kinds. What is partitioned is the binding table and the waiting room, which narrows the change W3 has to make. | | Clean ports | The `"forward"` sync kind, and the cross-lane duplicate-request guard in `frontier/scheduler/utils/ep_wave_inputs.py`. | | Blocked hunk | The candidate's decode final-metrics hunk calls `scheduler._create_corrected_execution_time_for_metrics(...)`, which **main deleted**; `_create_prefill_corrected_execution_time_for_metrics` also changed signature and now lives at `base_cluster_scheduler.py:1138`. Porting verbatim raises `AttributeError`. Main additionally added `metrics_store` and `ep_wave_reporting_enabled` plumbing to `ep_wave_schedule.py` and `prefill_collective.py` that the candidate lacks. | | Disposition | **ADAPT** for the lifecycle change as one coherent unit; **BLOCKED** for the metrics hunk until it is rewritten against main's current execution-time ownership. | @@ -66,7 +67,7 @@ Claims in those reports that this PR depends on were re-verified directly agains | Reference semantics confirmed | Engine selection scores `waiting * 4 + running` in the **frontend**, `vllm/v1/engine/core_client.py:1146`. The coordinator transports unweighted `[waiting, running]` pairs. Timing constants confirmed: `min_stats_update_interval_ms = 100` (`coordinator.py:116`, a floor rather than a period), the bare literals `5000` (`coordinator.py:198`) and `50` (`coordinator.py:202`). The specification's grouping of the weight 4 with the coordinator constants is corrected here: it belongs to the frontend. | | Request populations confirmed | Admitted-but-not-yet-scheduled requests count as **running** (`scheduler.py:812-813`, appended inside `schedule()` before the model runs); preemption moves running to waiting within the same step (`:507`, `:537-538`); requests finishing this step are in neither (`:1381-1385`, before `make_stats` at `:1413`). | | Report suppression confirmed | Two independent stats producers exist; only the DP one feeds the coordinator, and it is suppressed whenever the count pair is unchanged (`core.py:1081`). That is why the 5 s heartbeat exists and why the step counter is a sparse tag rather than a dense counter. | -| Step-identity problem | The candidate uses `ForwardSyncState.get_step_id(batch)` as the report-order key (`vllm_load_balancing_cluster_scheduler.py:52-56`, latched at `vllm_dp_load_balancer.py:64-67` against a single scalar shared across engines). On main that identity is **invalid for dense models** (`base_replica_scheduler.py:459-466` sets `_forward_cohort_id` from a per-DP-lane creation counter that dense never promotes to a Replica-scoped counter, and the policy's guard does not require MoE), **invalid for MoE with DP>1 until W3 lands** (the kind-partitioned namespace gives the two lanes different ids), and valid for DP1. Idle participants are never reported. The id also advances per layer (`sync_entry.py:64`, `forward_sync_state.py:133-139`), so it is monotonic but is not a vLLM step counter. | +| Step-identity problem | The candidate uses `ForwardSyncState.get_step_id(batch)` as the report-order key (`vllm_load_balancing_cluster_scheduler.py:52-56`, latched at `vllm_dp_load_balancer.py:64-67` against a single scalar shared across engines). On main that identity is **invalid for dense models** (`base_replica_scheduler.py:459-466` sets `_forward_cohort_id` from a per-DP-lane creation counter that dense never promotes to a Replica-scoped counter, and the policy's guard does not require MoE), **invalid for MoE with DP>1 until W3 lands**, though for a narrower reason than first recorded: the ids come from one Replica-scoped counter, but the two lanes bind into separate per-kind open-step tables, so a mixed-phase forward never reaches a single shared step to key a report on, and valid for DP1. Idle participants are never reported. The id also advances per layer (`sync_entry.py:64`, `forward_sync_state.py:133-139`), so it is monotonic but is not a vLLM step counter. | | Disposition | **BLOCKED on W3**, then ADAPT. This makes the execution order W2 (independent) then W3 then W4. | | Decision raised | See D1 below. | From d244bded7772b664fee133fedb0e28ade887571f Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 01:32:58 +0800 Subject: [PATCH 07/80] docs(issue26): record the maintainer review decisions D1 and D2 --- .../review.md | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index cf21b9ac..e194f50b 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -7,6 +7,7 @@ | 2026-09-21 | Created with the pinned source snapshot. | | 2026-09-21 | Step 1 complete: candidate and vLLM audits landed, dispositions recorded, two decision checkpoints raised. | | 2026-09-21 | Corrected the W3 and W4 rows: the step-id namespace is not partitioned by sync kind, only the open-step binding table is. Verified against `forward_sync_state.py` at `c18eb2c`. | +| 2026-09-22 | Recorded the maintainer's PR #34 / PR #35 review: D1 and D2 resolved, ten review comments dispositioned, each verified against source. | ## Pinned source snapshot @@ -137,3 +138,80 @@ The candidate deletes `VLLMv1EngineReplicaScheduler._get_num_waiting_reqs_for_de ## Final code-review findings Pending Step 8. + +## Maintainer decisions and review dispositions (2026-09-22) + +Source: `.local-draft/Frontier_PR34_PR35_Review_and_D1_D2_Decisions_2026-09-22.md`, +maintainer review of PR #34 at `5ef96b5` and PR #35 at `33f0d5a`. + +### D1 — RESOLVED: W3 first, then validate the identity at the report boundary + +Approved as written. Order: implement and validate W3; inspect the identity at +the **load-report emission boundary**; reuse it only where its equality and +ordering hold; reject unsupported configurations explicitly rather than adding a +second counter to broaden W4. Dense DP>1 is not admitted by W3 landing. DP1 is a +tested degenerate case that does not validate cross-lane behavior. + +Two source-backed refinements to the reasoning, verified against +`.real-engine/vLLM-BS` at `ea95f571`: + +1. **Why per-lane counters work in the reference.** `step_counter` is owned by + `DPEngineCoreProc` (`core.py:997`, `:1012`), so the maintainer's correction is + right: per-lane is not itself the defect. The reason those per-lane counters + are still comparable across engines is that the busy loops stay in lockstep — + every iteration calls `_has_global_unfinished_reqs`, which increments the + counter (`:1134`) and all-reduces every 32 steps (`:1135`), and every wave + boundary resets it to 0 (`:1129`). A Frontier per-lane counter with no + equivalent lockstep does not inherit that property. The conclusion (exclude + dense DP>1) is unchanged; the supporting reason is the missing lockstep, not + merely "it counts something else". +2. **What the counter counts, and where the key is read.** The increment at + `:1134` happens in busy-loop step 3, *after* `_maybe_publish_request_counts()` + at `:1099`. The key on a report is therefore the count as of the end of the + previous iteration, and it counts busy-loop iterations — including iterations + that executed a dummy batch — not completed model forwards. This is the + precise reason the §2.3 instruction "literal equality with vLLM's counters is + unnecessary if grouping and order are preserved" is correct, and the reason + Frontier must not try to make the numbers match. + +**Addition for W4 implementation.** The coordinator keeps one shared +`(last_stats_wave, last_stats_step)` pair across all engines +(`coordinator.py:156-157`). A strictly newer key preserves the prior snapshot +(`:296-300`); an **equal** key takes neither branch, which is the expected path +for peer engines reporting the same step; an out-of-order key produces a +**warning only** (`:301-307`) and the counts are still applied unconditionally +(`:308-310`). W4 must therefore not add a hard runtime assertion on report order +that the reference does not have. Key equality per shared forward is a test +invariant, not a runtime abort condition. + +### D2 — RESOLVED: include the local reduction, behind a narrow versioned scope identifier + +Approved as written, including the compatibility and cache policy in §3.5 and the +validation set in §3.6. + +The maintainer's §3.3 correction is confirmed in this tree and is a real addition +to the earlier W6 audit: + +| Claim | Verification | +| --- | --- | +| Alignment is outside the legacy timed region | `moe_align_block_size(...)` at `frontier/profiling/moe/moe_vllm_kernel.py:897`; the legacy `_step` is defined at `:920`. Confirmed outside. | +| The functional entry point aligns internally | Functional `_step` at `:837` calls `fused_experts` with top-k ids; vLLM 0.10.2 aligns inside. To be re-verified against the exact supported version before admitting measurements. | +| Double counting is a live risk, not a hypothetical | `MoETime` carries `moe_shuffling_time` as a term **separate from and additive to** `moe_grouped_gemm_time`, and `total_time()` sums both (`frontier/entities/time_components.py:505`, `:508`, `:528-536`). A functional measurement that internally aligns and shuffles, charged to `moe_grouped_gemm_time` while `moe_shuffling_time` is independently predicted, counts that work twice. | + +Consequence adopted: adding gated SiLU and `moe_sum` does not make the legacy and +functional scopes equal, and no record may claim that it does. + +### Disposition of the review comments + +| Comment | Verdict | Verification | +| --- | --- | --- | +| R34-01 false success | **ACCEPT, P1** | Both mechanisms reproduced in source. `baseline_failures` is absent from the `failed` predicate (`run_matrix.py:536-543`); `complete` tests case-ID presence only (`:432-434`). All cases failing on both sides yields `compared == 0` and exit 0, which `measure_commit.py:139` prints as `VERDICT: IDENTICAL`. Second path: `list_artifacts` returns `[]` for a missing directory (`compare.py:83-84`), so two absent directories compare equal. | +| R34-02 provenance | **ACCEPT, P1, with one refinement** | Merge-and-overwrite confirmed (`run_matrix.py:294-308`, manifest rebuilt at `:314-327`). `measure_commit.py:82-90` reuses a checkout after checking `HEAD` only. Refinement: the manifest already records `git_dirty_paths` and `cases_executed_in_last_run`; what is missing is per-case provenance and any *check* of those fields, so the fix is a stamp plus a guard, not new machinery. A live instance existed: `.worktrees/fidelity-candidate-99922d2` carried a modified `run_matrix.py` (byte-identical to `4f11386`, so harmless) while being treated as a clean detached checkout. | +| R34-03 evidence record | **ACCEPT, P1** | Confirmed and broader than stated: `cases.py` yields 71 cases, and the string "71" appears in no tracked document on the refactor branch. `progress.md` still reads "Current step: Step 0", Step 2 `IN_PROGRESS`, Step 7 `NOT_STARTED`. Relevant fact for the remedy: `db15e64..5ef96b5` is one commit touching only `cases.py`, so the production tree is unchanged between the last measured commit and the tip; but `candidate_db15e64` holds 67 records with zero DP cases, so the four DP cases have never run against the refactor tip. Remedy is a full 71-case run of `5ef96b5`, not a documentation argument. | +| R34-04 retained checks | **ACCEPT, P2** | Confirmed: PR #34 adds only the four harness files; the seven touched unit files are modifications (monkeypatch retargeting). No committed test covers the CLI flag set, the public re-exports, the mixin MRO, or loading a baseline-produced estimator cache. | +| R34-05 bounded split | **ACCEPT, P3** | Non-blocking guidance; no action beyond documenting owning class and required state at each extracted boundary. | +| R35-01 W2 tests | **ACCEPT, P2** | Confirmed: `test_replica_identity_contract.py:25-34` selects lines beginning `dp_id = ` and asserts `endswith("% self._replica_dp_size")`. That is a source-string check and is not a behavioral placement test. | +| R35-02 wrapper limit | **ACCEPT, P2** | The `cases.py` docstring is accurate as written — it scopes the limit to shipped recipes — but its remedy ("validated by unit tests") is too weak. Corrected remedy: a direct-construction integration fixture driving the real event loop, with deterministic durations injected only at the predictor boundary. | +| R35-03 SGLang consumers | **ACCEPT, P1** | Confirmed live: definition at `vllm_v1_iteration_policy.py:527`, callers at `:573` and `sglang_style_replica_scheduler.py:65`. | +| R35-04 W5 scope | **ACCEPT, P2** | Consistent with the existing W5 audit rows. | +| R35-05 gates | **ACCEPT, P1** | This section is that tracked decision log. | From 7f079e139f7fd13489f8b973c5b7f148e88d6abd Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 01:43:40 +0800 Subject: [PATCH 08/80] docs(issue26): self-review of the decision record; correct D1 mechanism and R34-03 remedy --- .../review.md | 127 ++++++++++++------ 1 file changed, 86 insertions(+), 41 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index e194f50b..2089c9ad 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -8,6 +8,7 @@ | 2026-09-21 | Step 1 complete: candidate and vLLM audits landed, dispositions recorded, two decision checkpoints raised. | | 2026-09-21 | Corrected the W3 and W4 rows: the step-id namespace is not partitioned by sync kind, only the open-step binding table is. Verified against `forward_sync_state.py` at `c18eb2c`. | | 2026-09-22 | Recorded the maintainer's PR #34 / PR #35 review: D1 and D2 resolved, ten review comments dispositioned, each verified against source. | +| 2026-09-22 | Self-review of that record: corrected the lockstep mechanism and counter semantics under D1, the line references under R34-01, the R34-03 remedy (the baseline label is itself an assembled partial run), and added the omissions listed under "Found on re-review". | ## Pinned source snapshot @@ -142,7 +143,8 @@ Pending Step 8. ## Maintainer decisions and review dispositions (2026-09-22) Source: `.local-draft/Frontier_PR34_PR35_Review_and_D1_D2_Decisions_2026-09-22.md`, -maintainer review of PR #34 at `5ef96b5` and PR #35 at `33f0d5a`. +maintainer review of PR #34 at `5ef96b5` and PR #35 at `33f0d5a`. Every line +reference below was checked against the tree named in the row. ### D1 — RESOLVED: W3 first, then validate the identity at the report boundary @@ -152,51 +154,53 @@ ordering hold; reject unsupported configurations explicitly rather than adding a second counter to broaden W4. Dense DP>1 is not admitted by W3 landing. DP1 is a tested degenerate case that does not validate cross-lane behavior. -Two source-backed refinements to the reasoning, verified against -`.real-engine/vLLM-BS` at `ea95f571`: - -1. **Why per-lane counters work in the reference.** `step_counter` is owned by - `DPEngineCoreProc` (`core.py:997`, `:1012`), so the maintainer's correction is - right: per-lane is not itself the defect. The reason those per-lane counters - are still comparable across engines is that the busy loops stay in lockstep — - every iteration calls `_has_global_unfinished_reqs`, which increments the - counter (`:1134`) and all-reduces every 32 steps (`:1135`), and every wave - boundary resets it to 0 (`:1129`). A Frontier per-lane counter with no - equivalent lockstep does not inherit that property. The conclusion (exclude - dense DP>1) is unchanged; the supporting reason is the missing lockstep, not - merely "it counts something else". -2. **What the counter counts, and where the key is read.** The increment at - `:1134` happens in busy-loop step 3, *after* `_maybe_publish_request_counts()` - at `:1099`. The key on a report is therefore the count as of the end of the - previous iteration, and it counts busy-loop iterations — including iterations - that executed a dummy batch — not completed model forwards. This is the - precise reason the §2.3 instruction "literal equality with vLLM's counters is - unnecessary if grouping and order are preserved" is correct, and the reason - Frontier must not try to make the numbers match. +Source-backed refinements, verified against `.real-engine/vLLM-BS` at `ea95f571`: + +1. **Why per-lane counters group correctly in the reference.** `step_counter` + is owned by `DPEngineCoreProc` (`core.py:997`, `:1012`), so the maintainer's + correction stands: per-lane is not itself the defect. What keeps the per-lane + counters equal across engines is that **every forward, real or dummy, + performs a DP-group all-reduce** to exchange `num_tokens_across_dp` + (`forward_context.py:72-84`). That collective, not the every-32-step + all-reduce in `_has_global_unfinished_reqs` (`core.py:1134-1135`, which only + decides wave termination), is the lockstep, and it applies to dense models as + well. A Frontier per-lane counter has no equivalent per-forward + synchronization for dense DP lanes, so it cannot inherit the property. The + conclusion (exclude dense DP>1) is unchanged; this is the reason. +2. **What the counter counts and where the key is read.** The increment at + `:1134` runs in busy-loop step 3, *after* `_maybe_publish_request_counts()` + at `:1099`. A published key is therefore the count as of the end of the + previous iteration. Iterations in which every engine is idle `continue` + before step 3 (`:1103-1105`) and do not increment; iterations that ran a + dummy forward do. So it counts forwards including dummy forwards since the + last wave reset (`:1129`), not completed real-batch forwards. This is the + precise reason §2.3's "literal equality with vLLM's counters is unnecessary + if grouping and order are preserved" is right, and why Frontier must not try + to make the numbers match. **Addition for W4 implementation.** The coordinator keeps one shared `(last_stats_wave, last_stats_step)` pair across all engines -(`coordinator.py:156-157`). A strictly newer key preserves the prior snapshot -(`:296-300`); an **equal** key takes neither branch, which is the expected path -for peer engines reporting the same step; an out-of-order key produces a -**warning only** (`:301-307`) and the counts are still applied unconditionally -(`:308-310`). W4 must therefore not add a hard runtime assertion on report order -that the reference does not have. Key equality per shared forward is a test -invariant, not a runtime abort condition. +(`coordinator.py:156-157`). A strictly newer key advances the pair and, when +unpublished changes exist (`stats_changed`), first preserves the prior counts +as a snapshot (`:296-300`); an **equal** key takes neither branch, which is the +expected path for peer engines reporting the same forward; an out-of-order key +produces a **warning only** (`:301-307`) and the counts are still applied +unconditionally (`:308-310`). W4 must not add a hard runtime assertion on +report order that the reference does not have. Key equality per shared forward +is a test invariant, not a runtime abort condition. ### D2 — RESOLVED: include the local reduction, behind a narrow versioned scope identifier Approved as written, including the compatibility and cache policy in §3.5 and the validation set in §3.6. -The maintainer's §3.3 correction is confirmed in this tree and is a real addition -to the earlier W6 audit: +The maintainer's §3.3 correction is confirmed in this tree: | Claim | Verification | | --- | --- | | Alignment is outside the legacy timed region | `moe_align_block_size(...)` at `frontier/profiling/moe/moe_vllm_kernel.py:897`; the legacy `_step` is defined at `:920`. Confirmed outside. | -| The functional entry point aligns internally | Functional `_step` at `:837` calls `fused_experts` with top-k ids; vLLM 0.10.2 aligns inside. To be re-verified against the exact supported version before admitting measurements. | -| Double counting is a live risk, not a hypothetical | `MoETime` carries `moe_shuffling_time` as a term **separate from and additive to** `moe_grouped_gemm_time`, and `total_time()` sums both (`frontier/entities/time_components.py:505`, `:508`, `:528-536`). A functional measurement that internally aligns and shuffles, charged to `moe_grouped_gemm_time` while `moe_shuffling_time` is independently predicted, counts that work twice. | +| The functional entry point aligns internally | The functional `_step` at `:837` calls `_run_functional_fused_experts_iteration`; the `_step` at `:820` is the MXFP4 branch. vLLM 0.10.2 aligns inside `fused_experts`. To be re-verified against the exact supported version before admitting measurements. | +| Double counting is structurally present | Shuffling and grouped GEMM are separate additive terms in **both** accounting paths: legacy `MoETime.total_time()` sums `moe_shuffling_time` and `moe_grouped_gemm_time` (`time_components.py:505`, `:508`, `:531`, `:534`), and the typed path computes `shuffling_time` and `grouped_gemm_time` separately and adds them (`moe_operator_times.py:129-143`). Whether the shuffling predictor is actually populated for functional-backend datasets is **not verified here**; W6 must check it before claiming or denying a live double count. | Consequence adopted: adding gated SiLU and `moe_sum` does not make the legacy and functional scopes equal, and no record may claim that it does. @@ -205,13 +209,54 @@ functional scopes equal, and no record may claim that it does. | Comment | Verdict | Verification | | --- | --- | --- | -| R34-01 false success | **ACCEPT, P1** | Both mechanisms reproduced in source. `baseline_failures` is absent from the `failed` predicate (`run_matrix.py:536-543`); `complete` tests case-ID presence only (`:432-434`). All cases failing on both sides yields `compared == 0` and exit 0, which `measure_commit.py:139` prints as `VERDICT: IDENTICAL`. Second path: `list_artifacts` returns `[]` for a missing directory (`compare.py:83-84`), so two absent directories compare equal. | -| R34-02 provenance | **ACCEPT, P1, with one refinement** | Merge-and-overwrite confirmed (`run_matrix.py:294-308`, manifest rebuilt at `:314-327`). `measure_commit.py:82-90` reuses a checkout after checking `HEAD` only. Refinement: the manifest already records `git_dirty_paths` and `cases_executed_in_last_run`; what is missing is per-case provenance and any *check* of those fields, so the fix is a stamp plus a guard, not new machinery. A live instance existed: `.worktrees/fidelity-candidate-99922d2` carried a modified `run_matrix.py` (byte-identical to `4f11386`, so harmless) while being treated as a clean detached checkout. | -| R34-03 evidence record | **ACCEPT, P1** | Confirmed and broader than stated: `cases.py` yields 71 cases, and the string "71" appears in no tracked document on the refactor branch. `progress.md` still reads "Current step: Step 0", Step 2 `IN_PROGRESS`, Step 7 `NOT_STARTED`. Relevant fact for the remedy: `db15e64..5ef96b5` is one commit touching only `cases.py`, so the production tree is unchanged between the last measured commit and the tip; but `candidate_db15e64` holds 67 records with zero DP cases, so the four DP cases have never run against the refactor tip. Remedy is a full 71-case run of `5ef96b5`, not a documentation argument. | -| R34-04 retained checks | **ACCEPT, P2** | Confirmed: PR #34 adds only the four harness files; the seven touched unit files are modifications (monkeypatch retargeting). No committed test covers the CLI flag set, the public re-exports, the mixin MRO, or loading a baseline-produced estimator cache. | -| R34-05 bounded split | **ACCEPT, P3** | Non-blocking guidance; no action beyond documenting owning class and required state at each extracted boundary. | -| R35-01 W2 tests | **ACCEPT, P2** | Confirmed: `test_replica_identity_contract.py:25-34` selects lines beginning `dp_id = ` and asserts `endswith("% self._replica_dp_size")`. That is a source-string check and is not a behavioral placement test. | -| R35-02 wrapper limit | **ACCEPT, P2** | The `cases.py` docstring is accurate as written — it scopes the limit to shipped recipes — but its remedy ("validated by unit tests") is too weak. Corrected remedy: a direct-construction integration fixture driving the real event loop, with deterministic durations injected only at the predictor boundary. | -| R35-03 SGLang consumers | **ACCEPT, P1** | Confirmed live: definition at `vllm_v1_iteration_policy.py:527`, callers at `:573` and `sglang_style_replica_scheduler.py:65`. | +| R34-01 false success | **ACCEPT, P1** | Both mechanisms reproduced. `baseline_failures` is absent from the `failed` predicate (`run_matrix.py:535`); `complete` (`:437`) tests case-ID presence only; `incomplete` (`:527`) derives from it. All cases failing on both sides yields `compared == 0` and exit 0, which `measure_commit.py:139` prints as `VERDICT: IDENTICAL`. Second path: `list_artifacts` returns `[]` for a missing directory (`compare.py:83-84`), so two absent directories compare equal. | +| R34-02 provenance | **ACCEPT, P1, with one refinement** | Merge-and-overwrite confirmed (`run_matrix.py:294-308`; manifest rebuilt at `:314-327`). `measure_commit.py:82-90` reuses a checkout after checking `HEAD` only. Refinement: the manifest already records `git_dirty_paths` and `cases_executed_in_last_run` (`:319`, `:323`); what is missing is per-case provenance and any *check* of those fields, so the fix is a stamp plus a guard. The partial-run entry points that the guard must cover are `--case-filter`, `--start` and `--limit` on both drivers. Two recorded instances exist in the scratch root; see "Found on re-review". | +| R34-03 evidence record | **ACCEPT, P1; the document's cheaper remedy is not available** | Confirmed: `cases.py` yields 71 cases and the string "71" appears in no tracked document on the refactor branch; `progress.md` still reads "Current step: Step 0", Step 2 `IN_PROGRESS`, Step 7 `NOT_STARTED`. `db15e64..5ef96b5` is one commit touching only `cases.py`, so the production tree is unchanged at the tip, but `candidate_db15e64` holds 67 records and no DP case, and the **baseline label is itself an assembled partial run** (below). Remedy: recapture **both** sides as single clean full 71-case runs after the R34-01/R34-02 fixes land, on the post-fix tip, asserting that its `frontier/` tree equals `5ef96b5`. | +| R34-04 retained checks | **ACCEPT, P2** | PR #34 adds only the four harness files; the seven touched unit files are import-path and monkeypatch-target retargeting to the new modules (37 insertions, 23 deletions). No committed test covers the CLI flag set, the public re-exports, the mixin MRO, or loading a baseline-produced estimator cache. | +| R34-05 bounded split | **ACCEPT, P3** | Non-blocking guidance. | +| R35-01 W2 tests | **ACCEPT, P2** | `test_replica_identity_contract.py:25-34` selects lines beginning `dp_id = ` and asserts `endswith("% self._replica_dp_size")`: a source-string check, not a behavioral placement test. | +| R35-02 wrapper limit | **ACCEPT, P2** | The `cases.py` docstring is accurate as written (it scopes the limit to shipped recipes), but its remedy ("validated by unit tests") is too weak. Corrected remedy: a direct-construction integration fixture driving the real event loop, with deterministic durations injected only at the predictor boundary. | +| R35-03 SGLang consumers | **ACCEPT, P1** | Definition at `vllm_v1_iteration_policy.py:527`; callers at `:573` and `sglang_style_replica_scheduler.py:65`. | | R35-04 W5 scope | **ACCEPT, P2** | Consistent with the existing W5 audit rows. | | R35-05 gates | **ACCEPT, P1** | This section is that tracked decision log. | + +### Found on re-review + +Facts read from the manifests under +`/data/ycfeng/tmp/issue26-correctness-pr/refactor-fidelity/` on 2026-09-22. + +| Label | `git_head` | dirty | `case_filter` | executed in last run | `case_count` / lines | `clean_cache` | +| --- | --- | --- | --- | --- | --- | --- | +| `baseline` | `1f694f7` | clean | `dp_` | 4 | **72 / 71** | **False** | +| `candidate_db15e64` | `db15e64` | clean | none | 67 | 67 / 67 | True | +| `candidate_6ab521d` | `6ab521d` | clean | none | 71 | 71 / 71 | True | +| `candidate` | `99922d2` | **3 tracked `frontier/` files modified, 7 untracked new scheduler modules** | none | 67 | 67 / 67 | True | + +1. **The baseline is an assembled label.** Its last run was a filtered `dp_` + run of four cases merged onto the earlier 67 without a cache clean. Every + per-case artifact is genuine and its source is clean, so the per-case + equality verdicts stand, but the label as a whole is exactly the R34-02 + pattern, its cache listing is not a clean full-matrix population, and its + `case_count` (72) disagrees with its results file (71): the merge retained a + record for a case id that no longer exists in the table. Both the refactor + comparisons and the W2 measurement in `validation.md` were made against this + label. Their per-case conclusions are not withdrawn; the "full matrix" and + cache-name claims must be re-established against a clean baseline. +2. **The contaminated `candidate` label is still on disk** and its manifest is a + recorded proof of the concurrent-edit collision. It must not be reused; it + should be removed or renamed before any further comparison (deletion needs + authorization). +3. **The W2 record is a mixed-harness measurement.** `candidate_6ab521d` ran the + `6ab521d` source under the refactor tip's case table and comparator. This was + disclosed, but the record does not state the harness revision as a field. + After Checkpoint C rebases #35 onto the fixed harness, W2 should be + re-measured with harness and source at one revision. +4. **Ownership for the checkpoints.** Checkpoints A and B touch only + `tests/e2e/refactor_fidelity/`, new tests and task records on the refactor + branch, which this session authored. Checkpoint C touches the correctness + worktree that the W3 owner also uses and must be coordinated before it + starts. +5. **PR #34 status.** It was marked ready for review earlier on 2026-09-22 at + the maintainer's instruction; the review that followed requests changes + with three P1 items. Whether it returns to draft until A and B close is the + maintainer's call. From 18edbbe678147a9476d10ab245493403288c32c9 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 02:25:00 +0800 Subject: [PATCH 09/80] docs(issue26): transcribe the contaminated label's manifest before deleting it --- .../review.md | 23 +++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index 2089c9ad..66bb65a4 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -242,10 +242,25 @@ Facts read from the manifests under comparisons and the W2 measurement in `validation.md` were made against this label. Their per-case conclusions are not withdrawn; the "full matrix" and cache-name claims must be re-established against a clean baseline. -2. **The contaminated `candidate` label is still on disk** and its manifest is a - recorded proof of the concurrent-edit collision. It must not be reused; it - should be removed or renamed before any further comparison (deletion needs - authorization). +2. **The contaminated `candidate` label.** Its manifest recorded `git_head` + `99922d2` against the shared `oversized-module-split` worktree with this + working tree state, which is the concurrent-edit collision in full: + + - `M frontier/config/cluster_config.py` + - `M frontier/config/replica_config.py` + - `M frontier/scheduler/replica_scheduler/vllm_v1_engine_replica_scheduler.py` + - `M tests/e2e/refactor_fidelity/run_matrix.py` + - `?? frontier/scheduler/replica_scheduler/vllm_v1_decision_log.py` + - `?? frontier/scheduler/replica_scheduler/vllm_v1_decode_attn_cohort.py` + - `?? frontier/scheduler/replica_scheduler/vllm_v1_iteration_policy.py` + - `?? frontier/scheduler/replica_scheduler/vllm_v1_kv_allocation.py` + - `?? frontier/scheduler/replica_scheduler/vllm_v1_mtp_wait.py` + - `?? frontier/scheduler/replica_scheduler/vllm_v1_prefix_cache.py` + - `?? frontier/scheduler/replica_scheduler/vllm_v1_role_schedules.py` + + The label was deleted on 2026-09-22 with the maintainer's authorization, + after this list was transcribed here. It must never be used as a + comparison side. 3. **The W2 record is a mixed-harness measurement.** `candidate_6ab521d` ran the `6ab521d` source under the refactor tip's case table and comparator. This was disclosed, but the record does not state the harness revision as a field. From ceac2b4771be5411526cbabaae381585540d8db7 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 08:26:18 +0800 Subject: [PATCH 10/80] test(scheduler): make the W2 placement tests state where requests land Review comment R35-01: the round-robin tests compared one run against another, which shows placement does not depend on how the stream is divided but would also hold for a wrong rule applied consistently, and the identity test reads source text, which cannot tell a correct expression from an equivalent one it does not recognise. Three topologies now have their full rotation written out by hand, past the wraparound, for the case where both the replica and the lane advance. The tests drive the public schedule() instead of _schedule_batch_mode, so the cluster-type dispatch is exercised rather than assumed, and each runs for both roles that reach batch-mode placement, MONOLITHIC and PREFILL. Requests now carry increasing arrival times, so the sort_requests() that schedule() performs first orders the queue itself rather than leaning on a stable sort over equal keys. Re-running these against the pre-fix method, taken verbatim from 6ab521d^ and installed in memory, fails 12 of 22 and leaves the grouping control and the eight unrelated lane tests passing. The failures are informative: the old code produces the expected sequence exactly when the whole stream arrives in one call, in all three topologies, and collapses to lane 0 only when requests arrive one at a time. The hand-derived expectation therefore agrees with the rotation that already existed; the fix made incremental placement match it rather than introducing a policy. The source-text guard stays, with a docstring saying it is governance only and must not be satisfied by rewording a correct expression. The fidelity case docstring claimed prefill placement "therefore has to be validated by unit tests". That understated the remedy: the wrapper limit is not a runtime limit, and full coverage needs a fixture that builds a runtime configuration directly with durations injected at the predictor boundary. It now says so, and points at the scheduler-level tests for what is covered today. --- tests/e2e/refactor_fidelity/cases.py | 12 +- tests/unit/test_cluster_scheduler_dp_lanes.py | 161 +++++++++++++++--- tests/unit/test_replica_identity_contract.py | 10 ++ 3 files changed, 159 insertions(+), 24 deletions(-) diff --git a/tests/e2e/refactor_fidelity/cases.py b/tests/e2e/refactor_fidelity/cases.py index c765ce1a..1aac3f82 100644 --- a/tests/e2e/refactor_fidelity/cases.py +++ b/tests/e2e/refactor_fidelity/cases.py @@ -521,8 +521,16 @@ def _dp_placement() -> list[FidelityCase]: disaggregated mode", and the MoE wrappers require ``ATTN_TP == MOE_TP * MOE_EP`` while the runtime requires ``attn_tp * attn_dp == moe_tp * moe_ep``, which have no common solution - above one lane. Placement changes affecting the prefill role therefore - have to be validated by unit tests, not by this matrix. + above one lane. + + That is a limit of the shipped wrappers, not of the runtime. The prefill + role's placement is covered at the scheduler level by + ``tests/unit/test_cluster_scheduler_dp_lanes.py``, which drives the public + ``schedule()`` for both the monolithic and prefill roles. Covering it + through the real event loop needs a fixture that builds a valid runtime + configuration directly instead of going through a wrapper, with + deterministic durations injected at the predictor boundary; that fixture + belongs with the mixed-lane MoE work, not here. """ rows: list[tuple[str, str, str, dict[str, str], tuple[str, ...]]] = [ diff --git a/tests/unit/test_cluster_scheduler_dp_lanes.py b/tests/unit/test_cluster_scheduler_dp_lanes.py index 89406dbe..450a1d18 100644 --- a/tests/unit/test_cluster_scheduler_dp_lanes.py +++ b/tests/unit/test_cluster_scheduler_dp_lanes.py @@ -1,5 +1,7 @@ from types import SimpleNamespace +import pytest + from frontier.entities import Request from frontier.scheduler.cluster_scheduler.lor_cluster_scheduler import ( LORClusterScheduler, @@ -37,9 +39,9 @@ def on_batch_end(self, *args, **kwargs): return None -def _request(*, session_id: int | None = None) -> Request: +def _request(*, session_id: int | None = None, arrived_at: float = 0.0) -> Request: return Request( - arrived_at=0.0, + arrived_at=arrived_at, num_prefill_tokens=4, num_decode_tokens=2, session_id=session_id, @@ -184,13 +186,22 @@ def lane(*, busy: bool, empty: bool): assert events[0]._replica_local_id == 1 +#: The cluster roles whose public ``schedule()`` reaches the fixed placement +#: helper. Both fall through to ``_schedule_batch_mode``; the point of running +#: both is that the dispatch in ``schedule()`` says so, not that this list does. +BATCH_MODE_CLUSTER_TYPES = [ClusterType.MONOLITHIC, ClusterType.PREFILL] + + def _round_robin_scheduler( - *, replica_ids: list[int], dp_size: int + *, + replica_ids: list[int], + dp_size: int, + cluster_type: ClusterType = ClusterType.MONOLITHIC, ) -> RoundRobinClusterScheduler: """A round-robin scheduler over the given replicas, with an empty queue.""" scheduler = RoundRobinClusterScheduler.__new__(RoundRobinClusterScheduler) - scheduler._cluster_type = ClusterType.MONOLITHIC + scheduler._cluster_type = cluster_type scheduler._num_replicas = len(replica_ids) scheduler._replica_dp_size = dp_size scheduler._cluster = SimpleNamespace( @@ -202,7 +213,11 @@ def _round_robin_scheduler( def _placements_for_call_sizes( - *, replica_ids: list[int], dp_size: int, call_sizes: list[int] + *, + replica_ids: list[int], + dp_size: int, + call_sizes: list[int], + cluster_type: ClusterType = ClusterType.MONOLITHIC, ) -> list[tuple[int, int]]: """Schedule one request stream in the given batches. @@ -210,11 +225,18 @@ def _placements_for_call_sizes( stream. The stream is the same for every call partitioning, so the result may not depend on `call_sizes`. Comparing by stream position rather than by request id is what makes two separately constructed runs comparable. + + This drives the public `schedule()`, so the cluster-type dispatch inside it + is part of what each case exercises. Arrival times increase along the + stream, so the `sort_requests()` that `schedule()` performs first orders the + queue on its own rather than relying on a stable sort over equal keys. """ - scheduler = _round_robin_scheduler(replica_ids=replica_ids, dp_size=dp_size) + scheduler = _round_robin_scheduler( + replica_ids=replica_ids, dp_size=dp_size, cluster_type=cluster_type + ) total = sum(call_sizes) - requests = [_request() for _ in range(total)] + requests = [_request(arrived_at=float(index)) for index in range(total)] position_of = {request.id: index for index, request in enumerate(requests)} placements: dict[int, tuple[int, int]] = {} @@ -222,7 +244,7 @@ def _placements_for_call_sizes( for size in call_sizes: scheduler._request_queue = requests[offset:offset + size] offset += size - for replica_id, dp_id, request in scheduler._schedule_batch_mode(): + for replica_id, dp_id, request in scheduler.schedule(): position = position_of[request.id] assert position not in placements, "a request was scheduled twice" placements[position] = (replica_id, dp_id) @@ -231,7 +253,80 @@ def _placements_for_call_sizes( return [placements[position] for position in range(total)] -def test_round_robin_dp_lane_does_not_depend_on_call_partitioning() -> None: +#: One full rotation, written out, for topologies where both dimensions move. +#: The replica advances on every request; the lane advances once the replica +#: rotation wraps; the pair repeats after `num_replicas * dp_size` requests. +#: These sequences are derived by hand from the intended placement rule, not +#: read back from the implementation. +EXPECTED_ROTATIONS = [ + ( + [3, 11], + 2, + [(3, 0), (11, 0), (3, 1), (11, 1)] * 2, + ), + ( + [3, 11, 42], + 3, + [ + (3, 0), (11, 0), (42, 0), + (3, 1), (11, 1), (42, 1), + (3, 2), (11, 2), (42, 2), + (3, 0), (11, 0), (42, 0), + ], + ), + ( + [5, 9], + 3, + [(5, 0), (9, 0), (5, 1), (9, 1), (5, 2), (9, 2)] * 2, + ), +] + + +@pytest.mark.parametrize("cluster_type", BATCH_MODE_CLUSTER_TYPES, ids=lambda t: t.name) +@pytest.mark.parametrize( + "replica_ids, dp_size, expected", + EXPECTED_ROTATIONS, + ids=lambda value: "x".join(map(str, value)) if isinstance(value, list) else str(value), +) +def test_round_robin_places_a_stream_on_the_expected_replica_and_lane( + replica_ids: list[int], + dp_size: int, + expected: list[tuple[int, int]], + cluster_type: ClusterType, +) -> None: + """The placement sequence itself, through the public scheduling entry. + + The other round-robin tests below compare one run against another, which + establishes that placement is independent of how the stream is divided but + would also hold for a wrong rule applied consistently. This one states where + each request must land, over a full rotation and past its wraparound, for + topologies where the replica and the lane both advance. + + Both cluster roles that reach batch-mode placement are covered, so the + dispatch inside `schedule()` is exercised rather than assumed. + """ + + burst = _placements_for_call_sizes( + replica_ids=replica_ids, + dp_size=dp_size, + call_sizes=[len(expected)], + cluster_type=cluster_type, + ) + assert burst == expected + + incremental = _placements_for_call_sizes( + replica_ids=replica_ids, + dp_size=dp_size, + call_sizes=[1] * len(expected), + cluster_type=cluster_type, + ) + assert incremental == expected + + +@pytest.mark.parametrize("cluster_type", BATCH_MODE_CLUSTER_TYPES, ids=lambda t: t.name) +def test_round_robin_dp_lane_does_not_depend_on_call_partitioning( + cluster_type: ClusterType, +) -> None: """The defect this covers: the DP lane restarted at zero on every call. With one replica and two lanes, scheduling eight requests one at a time put @@ -241,20 +336,23 @@ def test_round_robin_dp_lane_does_not_depend_on_call_partitioning() -> None: """ one_at_a_time = _placements_for_call_sizes( - replica_ids=[7], dp_size=2, call_sizes=[1] * 8 + replica_ids=[7], dp_size=2, call_sizes=[1] * 8, cluster_type=cluster_type ) single_burst = _placements_for_call_sizes( - replica_ids=[7], dp_size=2, call_sizes=[8] + replica_ids=[7], dp_size=2, call_sizes=[8], cluster_type=cluster_type ) uneven = _placements_for_call_sizes( - replica_ids=[7], dp_size=2, call_sizes=[3, 1, 4] + replica_ids=[7], dp_size=2, call_sizes=[3, 1, 4], cluster_type=cluster_type ) assert one_at_a_time == single_burst == uneven assert one_at_a_time == [(7, 0), (7, 1)] * 4 -def test_round_robin_placement_is_stable_across_topologies() -> None: +@pytest.mark.parametrize("cluster_type", BATCH_MODE_CLUSTER_TYPES, ids=lambda t: t.name) +def test_round_robin_placement_is_stable_across_topologies( + cluster_type: ClusterType, +) -> None: """Replica ids need not be contiguous and lanes may outnumber two.""" cases = [ @@ -266,35 +364,54 @@ def test_round_robin_placement_is_stable_across_topologies() -> None: ] for replica_ids, dp_size in cases: burst = _placements_for_call_sizes( - replica_ids=replica_ids, dp_size=dp_size, call_sizes=[12] + replica_ids=replica_ids, + dp_size=dp_size, + call_sizes=[12], + cluster_type=cluster_type, ) incremental = _placements_for_call_sizes( - replica_ids=replica_ids, dp_size=dp_size, call_sizes=[1] * 12 + replica_ids=replica_ids, + dp_size=dp_size, + call_sizes=[1] * 12, + cluster_type=cluster_type, ) assert burst == incremental, (replica_ids, dp_size) assert {replica_id for replica_id, _ in burst} <= set(replica_ids) assert all(0 <= dp_id < dp_size for _, dp_id in burst) -def test_round_robin_survives_an_empty_scheduling_call() -> None: +@pytest.mark.parametrize("cluster_type", BATCH_MODE_CLUSTER_TYPES, ids=lambda t: t.name) +def test_round_robin_survives_an_empty_scheduling_call( + cluster_type: ClusterType, +) -> None: """An empty call must neither advance the rotation nor reset it.""" with_gap = _placements_for_call_sizes( - replica_ids=[3, 11], dp_size=2, call_sizes=[2, 0, 2, 0, 4] + replica_ids=[3, 11], + dp_size=2, + call_sizes=[2, 0, 2, 0, 4], + cluster_type=cluster_type, ) without_gap = _placements_for_call_sizes( - replica_ids=[3, 11], dp_size=2, call_sizes=[8] + replica_ids=[3, 11], dp_size=2, call_sizes=[8], cluster_type=cluster_type ) assert with_gap == without_gap -def test_round_robin_returns_results_grouped_by_replica() -> None: +@pytest.mark.parametrize("cluster_type", BATCH_MODE_CLUSTER_TYPES, ids=lambda t: t.name) +def test_round_robin_returns_results_grouped_by_replica( + cluster_type: ClusterType, +) -> None: """The return order groups each call's results per replica, as before.""" - scheduler = _round_robin_scheduler(replica_ids=[3, 11], dp_size=2) - scheduler._request_queue = [_request() for _ in range(6)] + scheduler = _round_robin_scheduler( + replica_ids=[3, 11], dp_size=2, cluster_type=cluster_type + ) + scheduler._request_queue = [ + _request(arrived_at=float(index)) for index in range(6) + ] - mapping = scheduler._schedule_batch_mode() + mapping = scheduler.schedule() replica_order = [replica_id for replica_id, _, _ in mapping] assert replica_order == [3, 3, 3, 11, 11, 11] diff --git a/tests/unit/test_replica_identity_contract.py b/tests/unit/test_replica_identity_contract.py index ba6081b3..4290add4 100644 --- a/tests/unit/test_replica_identity_contract.py +++ b/tests/unit/test_replica_identity_contract.py @@ -2,6 +2,16 @@ def test_non_ffn_cluster_scheduler_uses_replica_local_dp_identity() -> None: + """A governance check on the source text. It does not establish behavior. + + Reading source cannot tell a correct expression from an equivalent one it + does not recognize, so this test can only say that no lane assignment takes + a modulus of something other than the Replica-local DP size. Placement + correctness is carried by `tests/unit/test_cluster_scheduler_dp_lanes.py`, + which drives the public `schedule()` and asserts where each request lands. + Do not reword a correct expression to satisfy this check. + """ + source = Path( "frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py" ).read_text(encoding="utf-8") From 3d474170a5df1e68a63c7e2b281ad2cea3dbd60f Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 08:40:12 +0800 Subject: [PATCH 11/80] docs(issue26): re-measure W2 with harness and source at one revision The first W2 measurement ran the 6ab521d source against the refactor tip's case table and comparator, with the harness revision disclosed in prose rather than recorded as a field, and it was taken with the harness that could report success without comparing anything. Neither fact impugned the result; both made it a poor record. Repeating it costs less than arguing about it. Baseline 6ef0a3c against candidate ceac2b4, both clean detached checkouts, one harness at ceac2b4, no filter, clean cache, 71 executed and 426 cache files on each side. 71 of 71 compared, 68 identical, and the mismatch set is exactly the three cases predicted to move before measuring. No provenance findings, no cache differences. Lane occupancy reproduces the first measurement number for number: collapsed onto lane zero before, spread after, both offline controls untouched. The re-measurement changed the provenance of the evidence, not the evidence. Exit code 1 is the correct outcome here. The gate reports inequality; the acceptance criterion for a behavior fix is the stated expectation. Also adds the guard that keeps the new tests' coverage claim honest. TRANS falls through the same dispatch as MONOLITHIC and PREFILL, so the role list is only complete because TRANS is declared and never constructed anywhere in frontier/; the guard fails if that changes or if a new role appears. review.md gains a remediation table, one row per accepted comment with the artifact that closes it, and the W2 negative control's more interesting result: the pre-fix code produces the expected sequence exactly when the whole stream arrives in one call. --- .../progress.md | 14 ++- .../review.md | 39 ++++++ .../validation.md | 114 ++++++++++++++++++ tests/unit/test_cluster_scheduler_dp_lanes.py | 38 +++++- 4 files changed, 199 insertions(+), 6 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index b0641d8f..811dc588 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -5,6 +5,7 @@ | Date | Change | | --- | --- | | 2026-09-21 | Step 0 started: records landed, environment created, baseline pending. | +| 2026-09-22 | Maintainer review dispositions recorded; Checkpoint C closed: parent merged, W2 tests strengthened, W2 re-measured with one harness revision. | ## Status @@ -12,10 +13,10 @@ | --- | --- | | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | -| Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate; the split measured 67 of 67 fidelity cases identical with no predictor cache name differences. | -| Current step | Step 2 complete; Step 3 scoped, not started | +| Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | +| Current step | Step 2 complete and re-measured (Checkpoint C); Step 3 scoped, not started | | Publication | PUSHED_VERIFIED (records) | -| Next action | Step 3: the shared monolithic forward lifecycle. Scoped below; not started. | +| Next action | Checkpoint D / Step 3: the shared monolithic forward lifecycle, with the direct-construction integration fixture R35-02 requires. Scoped below; not started. | ## Step status @@ -23,7 +24,7 @@ | --- | --- | --- | --- | --- | --- | | 0 | Worktree, references, baseline | PASS | PASS (baseline recorded) | PUSHED_VERIFIED | NOT_REVIEWED | | 1 | Candidate/vLLM audit | PASS | n/a (source audit) | LOCAL_ONLY | NOT_REVIEWED | -| 2 | RR DP rotation | PASS | unit PASS; matrix PASS against a stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | +| 2 | RR DP rotation | PASS | unit PASS (23 tests); matrix PASS against a stated expectation, re-measured 2026-09-22 with one harness revision | PUSHED_VERIFIED | REVIEWED (R35-01 closed) | | 3 | Shared monolithic forward | NOT_STARTED | — | — | — | | 4 | Opt-in vLLM DP placement | NOT_STARTED | — | — | — | | 5 | Routing implementation identity | NOT_STARTED | — | — | — | @@ -33,6 +34,11 @@ ## Chronological updates +- 2026-09-22: Checkpoint C. Merged the corrected parent (`bb582a4`, `7dd5982`, `6ef0a3c`) into this branch by merge rather than rebase, so the published review anchors stay valid and no commit is discarded. New base/head relationship: PR #35 head `fix/issue26-correctness-pr`, base `refactor/oversized-module-split` at `6ef0a3c`. +- 2026-09-22: R35-01 addressed. The placement tests now state where each request lands: three topologies with the full rotation written out by hand past its wraparound, driven through the public `schedule()` rather than `_schedule_batch_mode`, each run for both roles that reach batch-mode placement. A guard keeps that role list honest, since `TRANS` also falls through the dispatch but is declared and never constructed anywhere in `frontier/`. The source-text check is kept as governance only and says so. Negative control against the pre-fix method: 12 of 23 fail, controls pass, and the old code turns out to produce the expected sequence exactly for a single burst -- so the fix restored an existing rotation for incremental arrival rather than introducing a policy. +- 2026-09-22: W2 re-measured with harness and source at one revision: baseline `6ef0a3c` against candidate `ceac2b4`, both clean detached checkouts, one harness at `ceac2b4`, no filter, clean cache, 71 executed and 426 cache files each. 71 of 71 compared, 68 identical, and the mismatch set is exactly the three predicted cases, with no provenance findings and no cache differences. Lane occupancy reproduces the first measurement number for number. Full record in `validation.md`. +- 2026-09-22: The `cases.py` remedy wording was corrected. It said prefill placement "has to be validated by unit tests", which understates R35-02: the wrapper limit is not a runtime limit, and full coverage needs a fixture that builds a runtime configuration directly with durations injected at the predictor boundary. That fixture is W3 acceptance work. + - 2026-09-21: Baseline on the shared base recorded in the refactor task's Step 0 report; vLLM reference cloned (no tags in the fork; upstream `v0.10.2` comparison deferred to Step 1). - 2026-09-21: Draft specification analyzed; eleven facts verified against main, the candidate, the submodule remote, and the host; planning interview settled twelve decisions (see `requirements.md`). Records landed under `task_memory/`, `.gitignore` narrowed, `plan.md` carries the Amendments table. - 2026-09-21: Step 1 audit complete. Three pinned-source audits landed as `audit_scheduler.md`, `audit_predictor_profiling.md`, `reference_vllm_0_10_2.md`; dispositions and two decision checkpoints recorded in `review.md`. Execution order corrected to W2 first, then W3, then W4, because the candidate's report-order key depends on the shared forward identity. The vLLM fork was confirmed to be a direct descendant of upstream v0.10.2 with the DP placement files byte-identical to the tag. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index 66bb65a4..679ecd9f 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -275,3 +275,42 @@ Facts read from the manifests under the maintainer's instruction; the review that followed requests changes with three P1 items. Whether it returns to draft until A and B close is the maintainer's call. + +## Remediation record + +One row per accepted comment, with the artifact that closes it. A row is only +marked closed when the evidence for it is committed, not when the change is. + +| Comment | State | Closed by | +| --- | --- | --- | +| R34-01 false success | **CLOSED** | `tests/e2e/refactor_fidelity/{run_matrix,compare}.py`; `tests/unit/test_refactor_fidelity_gate.py` (22 tests). Before/after reproduction against the pre-fix harness: four scenarios returned exit 0 and now fail; both controls unchanged. `test_report_2026-09-22_checkpoint_a_fidelity_gate.md` section 3. | +| R34-02 provenance | **CLOSED** | Per-case `source_revision` / `source_dirty` / `harness_revision`; `check_retained_records` refuses a conflicting continuation before running anything; `case_count` counts written lines; cache names compared only for clean unfiltered full runs; `measure_commit.reuse_blocked_reason` reports and refuses a dirty checkout rather than cleaning it. | +| R34-03 evidence record | **CLOSED** | Both sides recaptured as single clean full 71-case runs, one harness revision: 71 of 71 compared, 71 identical, no failures, no missing evidence, no provenance findings, 0 cache differences. `test_report_2026-09-22_checkpoint_b_final_evidence.md`. Status header, `summary.md` and the PR #34 description synchronized. | +| R34-04 retained checks | **CLOSED** | `tests/unit/test_module_split_boundaries.py` (13 tests). Beyond the ask: all 142 baseline-produced pickled estimators load under the split code and 86 predict, which cache-name equality could not show. One pre-existing `NameError` on `ClusterConfig` annotations is pinned, not fixed, and verified to fail identically on `1f694f7`. | +| R34-05 bounded split | **CLOSED** | Guidance applied while writing the new tests; an unused import removed. | +| R35-01 W2 tests | **CLOSED** | `tests/unit/test_cluster_scheduler_dp_lanes.py`: three topologies with the full rotation written out by hand past its wraparound, driven through the public `schedule()`, each run for both MONOLITHIC and PREFILL. The source-text guard is kept with a docstring stating it is governance only. Negative control against the pre-fix method: 12 of 22 fail, controls pass. | +| R35-02 wrapper limit | **PARTIAL** | The `cases.py` remedy wording is corrected: the wrapper limit is not a runtime limit, and the fixture it calls for builds a runtime configuration directly with durations injected at the predictor boundary. The fixture itself is W3 acceptance work and is not written yet. | +| R35-03 SGLang consumers | OPEN | W4. | +| R35-04 W5 scope | OPEN | W5. | +| R35-05 gates | **CLOSED** | This document. | + +### What the W2 negative control showed beyond pass/fail + +Re-running the strengthened tests against the pre-fix `_schedule_batch_mode`, +taken verbatim from `6ab521d^`, is more informative than a failure count. In all +three topologies the old code produces the **expected sequence exactly** when the +whole stream arrives in one call, and collapses onto lane 0 only when requests +arrive one at a time: + +| Topology | Expected, and pre-fix in one burst | Pre-fix, one request at a time | +| --- | --- | --- | +| replicas `[3, 11]`, 2 lanes | `(3,0) (11,0) (3,1) (11,1)` repeating | `(3,0) (11,0)` repeating — every request on lane 0 | +| replicas `[3, 11, 42]`, 3 lanes | lane advances once per replica cycle | every request on lane 0 | +| replicas `[5, 9]`, 3 lanes | `(5,0) (9,0) (5,1) (9,1) (5,2) (9,2)` repeating | every request on lane 0 | + +So the hand-derived expectation agrees with the rotation the code already +performed for a single call. The fix did not introduce a placement policy; it +made incremental arrival reach the placement that batch arrival already had. +That is the strongest available statement that W2 is a bug fix rather than a +behavior change, and it is why the matrix is expected to move only the cases +that enter the scheduler many times. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index 844f80d8..dc425ba0 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -7,6 +7,7 @@ | 2026-09-21 | Created. Environment recorded; baseline results recorded in the refactor task's Step 0 report because both branches share the same base commit. | | 2026-09-21 | Step 1 recorded: audit spot checks and the vLLM reference identity check. | | 2026-09-21 | Step 2 recorded: unit sensitivity and the fidelity measurement against a stated expectation. | +| 2026-09-22 | Step 2 re-measured with harness and source at one revision, after the gate corrections. Same expectation, same result, recorded provenance. | ## Environment @@ -94,3 +95,116 @@ Sensitivity was verified by stashing the fix and rerunning, not by reasoning. Th ### Limits The prefill role reaches the same placement path, but **no shipped recipe can give it more than one lane**, so the matrix cannot cover that half. A dense model in a disaggregated architecture is rejected outright, and the MoE wrappers enforce `ATTN_TP == MOE_TP * MOE_EP` while the runtime enforces `attn_tp * attn_dp == moe_tp * moe_ep`, which have no common solution above one lane. Both routes were attempted and both were rejected, so this is measured rather than inferred. The unit test is the only evidence for the prefill half of this fix, and the PR says so. + +## Step 2 re-measured — harness and source at one revision (2026-09-22) + +### Why it was repeated + +The original W2 measurement ran the `6ab521d` source against the refactor tip's +case table and comparator, and it was taken with the pre-correction harness. +Two things were wrong with that as a record, neither of them a defect in the +result: the harness revision was disclosed in prose but not recorded as a field, +and the harness itself could report success without comparing anything +(review comments R34-01, R34-02). Repeating it is cheaper than arguing about it. + +### Setup + +Both sides were driven by one harness, running from the candidate checkout, with +each side's production tree supplied as `--repo-root` from its own clean +detached checkout. + +```bash +# from .worktrees/w2-candidate-ceac2b4, PYTHONPATH=$PWD +OUT=/data/ycfeng/tmp/issue26-correctness-pr/w2-remeasure +python tests/e2e/refactor_fidelity/run_matrix.py run \ + --repo-root .worktrees/w2-baseline-6ef0a3c --label w2_baseline_6ef0a3c \ + --output-root "$OUT" --jobs 6 --clean-cache --continue-on-failure +python tests/e2e/refactor_fidelity/run_matrix.py run \ + --repo-root .worktrees/w2-candidate-ceac2b4 --label w2_candidate_ceac2b4 \ + --output-root "$OUT" --jobs 6 --clean-cache --continue-on-failure +python tests/e2e/refactor_fidelity/run_matrix.py compare --output-root "$OUT" \ + --baseline-label w2_baseline_6ef0a3c --candidate-label w2_candidate_ceac2b4 +``` + +| Field | Baseline | Candidate | +| --- | --- | --- | +| Source revision | `6ef0a3c` (refactor tip, no W2) | `ceac2b4` (this branch, with W2) | +| Source working tree | clean | clean | +| Harness revision | `ceac2b4` | `ceac2b4` | +| Case filter | none | none | +| Cases executed | 71 | 71 | +| `case_count` / result lines | 71 / 71 | 71 / 71 | +| Cache cleaned first | yes | yes | +| Cache files produced | 426 | 426 | + +### The expectation, unchanged from the first measurement + +Exactly three cases move: `dp_dense_online_lanes2`, `dp_dense_online_lanes4`, +`dp_dense_online_lanes2_replicas2`. The two offline DP cases and the remaining +66 do not. A result of zero mismatches would mean the fix never reached the +path; a mismatch anywhere else would mean it reached more than the path. + +### Result + +| Measure | Expected | Actual | Result | +| --- | --- | --- | --- | +| Cases compared | 71 | **71** | PASS | +| Identical | 68 | **68** | PASS | +| Mismatched | the 3 named above | **exactly those 3** | PASS | +| Expected to move but did not | none | **none** | PASS | +| Moved but was not expected to | none | **none** | PASS | +| Baseline failures | 0 | **0** | PASS | +| Candidate-only failures | 0 | **0** | PASS | +| Missing / missing evidence / differing definitions | 0 | **0 / 0 / 0** | PASS | +| Cases not compared | 0 | **0** | PASS | +| Provenance findings | none | **none** | PASS | +| Predictor cache differences | 0 | **0**, compared cleanly | PASS | + +Comparison exit code 1, which is correct here: the gate reports inequality, and +the acceptance criterion is the stated expectation, not exit 0. + +### Direction, from the stage ledger's `replica_local_id` + +| Case | Baseline | Candidate | | +| --- | --- | --- | --- | +| `dp_dense_online_lanes2` | `{0: 45}` | `{0: 44, 1: 45}` | moved | +| `dp_dense_online_lanes4` | `{0: 45}` | `{0: 43, 1: 43, 2: 44, 3: 45}` | moved | +| `dp_dense_online_lanes2_replicas2` | `{0: 56}` | `{0: 55, 1: 56}` | moved | +| `dp_dense_offline_lanes2_replicas2` | `{0: 70, 1: 70}` | `{0: 70, 1: 70}` | control, unchanged | +| `coloc_dense_offline_attn_dp2` | `{0: 39, 1: 39}` | `{0: 39, 1: 39}` | control, unchanged | + +Every number reproduces the first measurement exactly. The re-measurement +changed the provenance of the evidence, not the evidence. + +### Relationship between the measured commit and the branch tip + +The candidate measured is `ceac2b4`. Commits after it on this branch touch only +`tests/` and `task_memory/`; `git diff --stat ceac2b4..HEAD -- frontier/` is +empty, so the production tree that produced these artifacts is the branch tip's +production tree. This is the "prove the relationship" path the review prefers to +a rerun, and here it is genuinely available because no production file changed. + +### Unit evidence for the strengthened tests + +`tests/unit/test_cluster_scheduler_dp_lanes.py`, 23 tests, all passing. Against +the pre-fix `_schedule_batch_mode` taken verbatim from `6ab521d^` and installed +in memory, 12 of 23 fail; the grouping control and the eight unrelated lane +tests still pass. What the failures show is recorded in `review.md`: the old +code produces the expected sequence exactly for a single burst and collapses to +lane 0 only for incremental arrival, so the fix restored a rotation that already +existed rather than introducing one. + +Related modules, whole files: 51 failed / 1356 passed / 19 skipped, and all 51 +failures are `test_pdaf_parity_reference_observer_bootstrap.py`, which needs the +pinned PD-AF reference checkout that is absent on this host. That count matches +the inherited-failure inventory recorded for the refactor branch. + +### Limits + +- The matrix still validates only the monolithic half of W2. The prefill role is + now covered at the scheduler level, through the public `schedule()`, for both + roles that reach batch-mode placement; that is a scheduler test, not a run of + the event loop. Full runtime coverage needs the direct-construction fixture + described under R35-02, which is W3 acceptance work. +- Lane occupancy is read from the stage ledger, which records scheduled stage + executions. It shows where work was placed, not that placement is optimal. diff --git a/tests/unit/test_cluster_scheduler_dp_lanes.py b/tests/unit/test_cluster_scheduler_dp_lanes.py index 450a1d18..1e7dc652 100644 --- a/tests/unit/test_cluster_scheduler_dp_lanes.py +++ b/tests/unit/test_cluster_scheduler_dp_lanes.py @@ -1,3 +1,4 @@ +from pathlib import Path from types import SimpleNamespace import pytest @@ -187,11 +188,44 @@ def lane(*, busy: bool, empty: bool): #: The cluster roles whose public ``schedule()`` reaches the fixed placement -#: helper. Both fall through to ``_schedule_batch_mode``; the point of running -#: both is that the dispatch in ``schedule()`` says so, not that this list does. +#: helper. ``DECODE``, ``DECODE_ATTN`` and ``DECODE_FFN`` each take their own +#: branch; ``TRANS`` also falls through, but it is declared and never +#: constructed anywhere in ``frontier/``, so these two are the reachable set. +#: `test_the_batch_mode_roles_are_the_ones_that_fall_through` keeps that true. BATCH_MODE_CLUSTER_TYPES = [ClusterType.MONOLITHIC, ClusterType.PREFILL] +def test_the_batch_mode_roles_are_the_ones_that_fall_through() -> None: + """A new role must not reach batch-mode placement untested. + + The placement tests below claim to cover every role that reaches the shared + helper. That claim is only worth as much as this check: if a role is added, + or an existing branch is removed, the role either gains its own dispatch or + it lands here and must be added to the list above. + """ + + dispatched_elsewhere = { + ClusterType.DECODE, + ClusterType.DECODE_ATTN, + ClusterType.DECODE_FFN, + } + never_constructed = {ClusterType.TRANS} + falls_through = set(ClusterType) - dispatched_elsewhere - never_constructed + assert falls_through == set(BATCH_MODE_CLUSTER_TYPES), sorted( + role.name for role in falls_through.symmetric_difference( + BATCH_MODE_CLUSTER_TYPES + ) + ) + + source = Path( + "frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py" + ).read_text(encoding="utf-8") + for role in never_constructed: + assert f"ClusterType.{role.name}" not in source, ( + f"{role.name} is now referenced; decide whether it places requests" + ) + + def _round_robin_scheduler( *, replica_ids: list[int], From 65ed8a76055e77336b00373e8fd4feb8393280a9 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 09:31:44 +0800 Subject: [PATCH 12/80] fix(scheduler): give a monolithic Replica one shared forward across phases A monolithic Replica runs prefill and decode on the same attention-DP lanes, so one forward step can hold a prefill batch on one lane and a pure-decode batch on another. The two phases were given disjoint waiting rooms and disjoint open-step namespaces, so those lanes waited in different rooms, neither reached the expected lane count, and the idle-lane filler could not rescue either one because each sibling stage was busy holding the other phase. The forward never dispatched. Reproduced end to end: with attn_tp=1, attn_dp=2, moe_tp=1, moe_ep=2 the sequential run ends with a non-empty scheduler state. The fix is one lifecycle, applied as a unit rather than as a shared room with phase-specific completion: - MONOLITHIC allocates one waiting room, bound to the forward name and to both phase names, and resolves its lanes in one "forward" step-id space. - The two near-duplicate per-layer sync entries become one `enter_layer_sync(..., mode)`, mirroring the existing `schedule_layer_wave(mode=...)`. - `forward_collective.py` completes a shared cohort once: it advances the decode-phase requests once, restores the full-stage owners once, then runs each live source through its own existing phase helper with `direct_batch`, so every lane predicts its continuation from its own batch. - The post_moe collective event class is chosen from cohort contents, not from whichever lane closed the room. `EventType` values are priorities, so a pure-prefill and a pure-decode cohort keep exactly the event type they have today; only the mixed cohort, which could not complete at all, is new. - A request may no longer occupy two non-idle source lanes of one forward. - A dense layer inside a MoE model labels each source by its own phase and keeps the prefill component ledger to prefill sources. - The shared completion makes the legacy-aggregate check once per source. The per-phase helpers only make it when they pop the room themselves, and delegation hands them `direct_batch`, so the check would otherwise be lost on exactly the path that now owns it. Coverage: 24 behavior tests over every phase pairing, both arrival orders, unequal source tokens, idle participation, duplicate ownership, successive layers, a decoding request inside a prefill batch, dense-layer transitions and a disabled metrics store; plus a direct-construction integration test that runs the real Simulator event loop over a multi-lane monolithic MoE configuration and reaches four mixed-phase cohorts. Controls, each failing for its own reason: the pre-fix tree deadlocks ("Sequential simulation ended with non-empty scheduler state"); borrowed source timing trips "one attention-DP lane cannot occupy two open sync cohorts"; a repeated layer advance trips "Decode post_moe layer counter cannot advance"; a repeated ownership restoration trips "operation_id is already queued or active". tests/unit and tests/integration show the same failures as HEAD -- 84 and 5 pre-existing, identical identities, no regressions -- with 23 and 1 new passes. --- .../base_cluster_scheduler.py | 47 +- frontier/scheduler/utils/decode_collective.py | 75 ++- frontier/scheduler/utils/ep_wave_inputs.py | 16 + frontier/scheduler/utils/ep_wave_schedule.py | 88 ++- .../scheduler/utils/forward_collective.py | 140 +++++ .../scheduler/utils/forward_sync_state.py | 27 +- .../scheduler/utils/prefill_collective.py | 75 ++- frontier/scheduler/utils/sync_entry.py | 260 ++++---- frontier/scheduler/utils/sync_state.py | 36 +- .../design.md | 137 ++++ .../test_monolithic_mixed_forward_runtime.py | 384 ++++++++++++ .../test_monolithic_mixed_forward_sync.py | 593 ++++++++++++++++++ tests/unit/test_pdaf_prefill_model_time.py | 6 +- 13 files changed, 1640 insertions(+), 244 deletions(-) create mode 100644 frontier/scheduler/utils/forward_collective.py create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/design.md create mode 100644 tests/integration/test_monolithic_mixed_forward_runtime.py create mode 100644 tests/unit/test_monolithic_mixed_forward_sync.py diff --git a/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py b/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py index 1f82c802..aeb0d847 100644 --- a/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py +++ b/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py @@ -106,7 +106,11 @@ handle_decode_attn_arrival as handle_m2n_decode_attn_arrival, handle_decode_ffn_arrival, ) -from frontier.scheduler.utils.sync_entry import enter_decode_sync, enter_prefill_sync +from frontier.scheduler.utils.sync_entry import ( + enter_decode_sync, + enter_prefill_sync, + uses_shared_forward_room, +) from frontier.scheduler.utils.pdaf_phase import ( prepare_decode_attn_batch_phase, apply_decode_attn_batch_phase, @@ -128,6 +132,7 @@ ) from frontier.scheduler.utils.prefill_collective import handle_prefill_sync_collective from frontier.scheduler.utils.decode_collective import handle_decode_sync_collective +from frontier.scheduler.utils.forward_collective import handle_forward_sync_collective from frontier.scheduler.utils.afd_metadata import aggregate_afd_metadata from frontier.scheduler.utils.request_selection import collect_active_requests from frontier.scheduler.utils.replica_schedulers import build_replica_scheduler_maps @@ -1073,6 +1078,36 @@ def _uses_shared_decode_ep_wave(self, batch: Batch, layer_id: int) -> bool: require_moe_layer=True, ) + def _on_forward_ep_wave_ready(self, *, time: float, replica_id: int, stage_id: int, batch: Batch, layer_id: int, replica_local_id: int | None = None, cohort_batches: dict[int, Batch] | None = None, metrics_store=None) -> List: + """Schedule one shared monolithic forward wave, across mixed lanes.""" + + return schedule_layer_wave( + self, + mode="forward", + time=time, + replica_id=replica_id, + stage_id=stage_id, + batch=batch, + layer_id=layer_id, + replica_local_id=replica_local_id, + cohort_batches=cohort_batches, + metrics_store=metrics_store, + ) + + def on_forward_sync_collective(self, time: float, replica_id: int, stage_id: int, batch_global_id: int, sync_stage: str, layer_id: int, metrics_store): + """Complete one shared monolithic forward through the utility handler.""" + + return handle_forward_sync_collective( + self, + time, + replica_id, + stage_id, + batch_global_id, + sync_stage, + layer_id, + metrics_store, + ) + def _uses_shared_decode_layer_path(self, batch: Batch, layer_id: int) -> bool: """Return whether a shared-domain DECODE model needs layer stepping.""" model_config = getattr(getattr(self._config, "replica_config", None), "model_config", None) @@ -1122,6 +1157,11 @@ def on_dense_layer_complete( def on_prefill_sync_collective(self, time: float, replica_id: int, stage_id: int, batch_global_id: int, sync_stage: str, layer_id: int, metrics_store, *, direct_batch: Optional[Batch] = None): """Delegate PREFILL collective completion to the utility handler.""" + if direct_batch is None and uses_shared_forward_room(self): + return self.on_forward_sync_collective( + time, replica_id, stage_id, batch_global_id, sync_stage, + layer_id, metrics_store, + ) return handle_prefill_sync_collective( self, time, @@ -1208,6 +1248,11 @@ def on_decode_sync( def on_decode_sync_collective(self, time: float, replica_id: int, stage_id: int, batch_global_id: int, sync_stage: str, layer_id: int, metrics_store, *, direct_batch: Optional[Batch] = None): """Delegate DECODE collective completion to the utility handler.""" + if direct_batch is None and uses_shared_forward_room(self): + return self.on_forward_sync_collective( + time, replica_id, stage_id, batch_global_id, sync_stage, + layer_id, metrics_store, + ) return handle_decode_sync_collective( self, time, diff --git a/frontier/scheduler/utils/decode_collective.py b/frontier/scheduler/utils/decode_collective.py index 6534be10..e1a026b9 100644 --- a/frontier/scheduler/utils/decode_collective.py +++ b/frontier/scheduler/utils/decode_collective.py @@ -26,8 +26,17 @@ def handle_decode_sync_collective( metrics_store: Any, *, direct_batch: Optional[Batch] = None, + owners_restored: Optional[bool] = None, + layer_advance_done: bool = False, ): - """Complete one DECODE layer and schedule the next stage transition.""" + """Complete one DECODE layer and schedule the next stage transition. + + `owners_restored` and `layer_advance_done` name work a wider cohort already + did. A shared monolithic forward advances its decode-phase requests once and + restores full-stage owners once for all its lanes, then enters this helper + per source; repeating either here would credit a layer twice and hand out a + second ticket for the same lane. + """ from frontier.logger import get_cluster_logger from frontier.events.batch_stage_end_event import BatchStageEndEvent @@ -78,12 +87,13 @@ def handle_decode_sync_collective( stage_scheduler = scheduler.get_replica_stage_scheduler(replica_id, stage_identity, stage_id) predictor = stage_scheduler._execution_time_predictor active_requests = collect_active_requests(dp_batches.values()) - validate_decode_layer_advance( - active_requests, - scheduler._config.replica_config.model_config.num_layers, - ) - for request in active_requests: - request.mb_on_step_layer_count_increment(num_layers_completed=1) + if not layer_advance_done: + validate_decode_layer_advance( + active_requests, + scheduler._config.replica_config.model_config.num_layers, + ) + for request in active_requests: + request.mb_on_step_layer_count_increment(num_layers_completed=1) num_layers = predictor._num_layers_per_pipeline_stage bounds_getter = getattr(scheduler, "get_pipeline_stage_layer_bounds", None) @@ -100,22 +110,21 @@ def handle_decode_sync_collective( ) stage_layer_end = (stage_id + 1) * num_layers next_layer_id = layer_id + 1 - restored_full_stage_owners = scheduler._restore_forward_step_full_stage_owners( - source_batches=dp_batches, - replica_id=replica_id, - stage_id=stage_id, - layer_id=next_layer_id, - cohort_id=batch_global_id, - operation_kind="attention" if next_layer_id < stage_layer_end else "final", - ) + if owners_restored is None: + restored_full_stage_owners = scheduler._restore_forward_step_full_stage_owners( + source_batches=dp_batches, + replica_id=replica_id, + stage_id=stage_id, + layer_id=next_layer_id, + cohort_id=batch_global_id, + operation_kind="attention" if next_layer_id < stage_layer_end else "final", + ) + else: + restored_full_stage_owners = owners_restored if next_layer_id < stage_layer_end: - next_execution = predictor.predict_stage_execution_time( - sample_batch, stage_id, scheduler._cluster_type, - num_layers=1, layer_id=next_layer_id, include_ffn=False, - ) - attention_time = attention_delay_seconds(next_execution) events = [] + last_attention_time = None for participant_id, batch in dp_batches.items(): if batch.is_idle: logger.info( @@ -124,6 +133,14 @@ def handle_decode_sync_collective( batch.id, replica_id, participant_id, layer_id, ) continue + # Each lane's next attention is predicted from its own batch: its own + # context lengths and token count, not a peer's. + next_execution = predictor.predict_stage_execution_time( + batch, stage_id, scheduler._cluster_type, + num_layers=1, layer_id=next_layer_id, include_ffn=False, + ) + attention_time = attention_delay_seconds(next_execution) + last_attention_time = attention_time transition_identity = getattr(batch, "_stage_owner_replica_local_id", None) if not restored_full_stage_owners: scheduler.transition_stage_admission_for_layer( @@ -140,16 +157,13 @@ def handle_decode_sync_collective( logger.info( "[DECODE_SYNC][COLLECTIVE] post_moe completed, incremented layer count " "for %s unique requests, scheduled next layer pre_moe sync at t=%.6fs", - len(active_requests), time + attention_time, + len(active_requests), + time + last_attention_time if last_attention_time is not None else time, ) return events - full_execution = predictor.predict_stage_execution_time( - sample_batch, stage_id, scheduler._cluster_type, - num_layers=num_layers, layer_id=stage_layer_end - num_layers, include_ffn=False, - ) - final_timing = prepare_decode_final_timing(full_execution) events = [] + last_completion_time = time for participant_id, batch in dp_batches.items(): if batch.is_idle: logger.info( @@ -158,6 +172,13 @@ def handle_decode_sync_collective( batch.id, replica_id, participant_id, layer_id, ) continue + full_execution = predictor.predict_stage_execution_time( + batch, stage_id, scheduler._cluster_type, + num_layers=num_layers, layer_id=stage_layer_end - num_layers, + include_ffn=False, + ) + final_timing = prepare_decode_final_timing(full_execution) + last_completion_time = time + final_timing.total_time scheduler._record_mtp_terminal_completion_delay( batch, final_timing.mtp_terminal_overshoot_time ) @@ -195,6 +216,6 @@ def handle_decode_sync_collective( ) logger.info( "[DECODE_SYNC][COLLECTIVE] Last layer completed, scheduled batch stage end at t=%.6fs", - time + final_timing.total_time, + last_completion_time, ) return events diff --git a/frontier/scheduler/utils/ep_wave_inputs.py b/frontier/scheduler/utils/ep_wave_inputs.py index c6988e87..dbc133a8 100644 --- a/frontier/scheduler/utils/ep_wave_inputs.py +++ b/frontier/scheduler/utils/ep_wave_inputs.py @@ -48,6 +48,22 @@ def prepare_ep_wave_inputs( non_idle = tuple(source_batch for source_batch in normalized.values() if not source_batch.is_idle) if not non_idle: raise ValueError("EP wave requires a non-idle source batch") + # One request has one source owner per forward. A monolithic cohort draws + # its lanes from independent Replica schedulers, so this is the boundary + # where two of them claiming the same request first becomes visible; every + # later step would otherwise double-count its tokens and advance it twice. + owning_lane_by_request: dict[int, int] = {} + for lane_id, source_batch in normalized.items(): + if source_batch.is_idle: + continue + for request in source_batch.requests: + previous_lane = owning_lane_by_request.setdefault(request.id, lane_id) + if previous_lane != lane_id: + raise ValueError( + "a request cannot belong to two EP source lanes: " + f"request_id={request.id}, lanes={previous_lane} and {lane_id}, " + f"forward_step={step_id}" + ) sample_batch = non_idle[0] total_tokens = sum(int(source_batch.total_num_tokens) for source_batch in non_idle) total_prefill_tokens = sum(int(source_batch.num_prefill_tokens) for source_batch in non_idle) diff --git a/frontier/scheduler/utils/ep_wave_schedule.py b/frontier/scheduler/utils/ep_wave_schedule.py index 859352a8..96b5f343 100644 --- a/frontier/scheduler/utils/ep_wave_schedule.py +++ b/frontier/scheduler/utils/ep_wave_schedule.py @@ -11,6 +11,42 @@ ) +def _source_mode(mode: str, source_batch: Any) -> str: + """Resolve the phase whose continuation one source batch must follow. + + For a disaggregated role the group mode is the source mode: those clusters + run one phase. Only a shared monolithic forward can hold lanes in different + phases, and there each source follows its own. + """ + + if mode != "forward": + return mode + from frontier.scheduler.utils.forward_sync_state import source_forward_mode + + return source_forward_mode(source_batch) + + +def _cohort_collective_event_mode(mode: str, non_idle_source_batches) -> str: + """Pick one collective event class for the whole cohort, deterministically. + + Events order by `(time, event_type, id)`, so the class carries a priority. + It must not depend on which lane happened to complete the room, and a cohort + shape that already works must keep the event type it has today. Choosing by + cohort contents satisfies both: a pure-prefill cohort stays on the prefill + event, a pure-decode cohort on the decode event, and only a mixed cohort — + which cannot complete at all before this change — is new. + """ + + if mode != "forward": + return mode + from frontier.scheduler.utils.forward_sync_state import source_forward_mode + + for source_batch in non_idle_source_batches: + if source_forward_mode(source_batch) == "prefill": + return "prefill" + return "decode" + + def schedule_layer_wave( scheduler: Any, *, @@ -31,7 +67,7 @@ def schedule_layer_wave( control flow used by the two cluster modes. """ - if mode not in ("prefill", "decode"): + if mode not in ("prefill", "decode", "forward"): raise ValueError(f"unsupported EP wave mode: {mode!r}") mode_name = mode.capitalize() if not isinstance(time, Real) or not math.isfinite(float(time)): @@ -85,13 +121,10 @@ def schedule_layer_wave( participant_ep_ids=tuple(layer_workload.participant_ep_ids), ) else: - event_cls = ( - _load_prefill_dense_event() - if mode == "prefill" - else _load_decode_dense_event() - ) + event_cls = _load_dense_layer_event() dense_events = [] for source_batch in non_idle_source_batches: + source_mode = _source_mode(mode, source_batch) execution_time = predictor.predict_stage_execution_time( source_batch, stage_id, @@ -123,7 +156,7 @@ def schedule_layer_wave( "_prefill_model_execution_components_ms_by_stage", None, ) - if mode == "prefill": + if source_mode == "prefill": if ( not isinstance(component_ledger, dict) or stage_id not in component_ledger @@ -142,7 +175,7 @@ def schedule_layer_wave( stage_id, source_batch, layer_id, - mode, + source_mode, scheduler._cluster_type, ) ) @@ -152,13 +185,13 @@ def schedule_layer_wave( raise ValueError(f"{mode_name} layer wave produced no participant timing") timing = plan.timing barrier_end_time_s = timing.wave_end_time_s - if mode == "prefill": - wave_time_ms = ( - timing.dispatch_barrier_time_ms - + timing.combine_barrier_time_ms - + timing.post_combine_barrier_time_ms - ) - for source_batch in non_idle_source_batches: + wave_time_ms = ( + timing.dispatch_barrier_time_ms + + timing.combine_barrier_time_ms + + timing.post_combine_barrier_time_ms + ) + for source_batch in non_idle_source_batches: + if _source_mode(mode, source_batch) == "prefill": component_ledger = getattr( source_batch, "_prefill_model_execution_components_ms_by_stage", @@ -177,15 +210,15 @@ def schedule_layer_wave( component_ledger[stage_id].append(wave_time_ms) source_batch._prefill_ep_wave_lane_times_ms = tuple(lane_compute_times_ms) source_batch._prefill_ep_wave_workload = layer_workload - else: - for source_batch in non_idle_source_batches: + else: source_batch._decode_ep_wave_lane_times_ms = tuple(lane_compute_times_ms) - waiting_room = ( - scheduler._prefill_sync_waiting_room - if mode == "prefill" - else scheduler._decode_sync_waiting_room - ) + if mode == "forward": + waiting_room = scheduler._forward_sync_waiting_room + elif mode == "prefill": + waiting_room = scheduler._prefill_sync_waiting_room + else: + waiting_room = scheduler._decode_sync_waiting_room sync_room = waiting_room[replica_id][stage_id][cohort_id][layer_id]["post_moe"] if sync_room["batches"]: raise ValueError( @@ -197,9 +230,10 @@ def schedule_layer_wave( sync_room["arrival_times"].update( {lane_id: barrier_end_time_s for lane_id in source_batches} ) + collective_mode = _cohort_collective_event_mode(mode, non_idle_source_batches) event_cls = ( _load_prefill_sync_event() - if mode == "prefill" + if collective_mode == "prefill" else _load_decode_sync_event() ) return [ @@ -215,13 +249,7 @@ def schedule_layer_wave( ] -def _load_prefill_dense_event(): - from frontier.events.dense_layer_complete_event import DenseLayerCompleteEvent - - return DenseLayerCompleteEvent - - -def _load_decode_dense_event(): +def _load_dense_layer_event(): from frontier.events.dense_layer_complete_event import DenseLayerCompleteEvent return DenseLayerCompleteEvent diff --git a/frontier/scheduler/utils/forward_collective.py b/frontier/scheduler/utils/forward_collective.py new file mode 100644 index 00000000..a157a260 --- /dev/null +++ b/frontier/scheduler/utils/forward_collective.py @@ -0,0 +1,140 @@ +"""Completion of one shared monolithic forward across mixed source lanes.""" + +from __future__ import annotations + +from typing import Any + +from frontier.scheduler.utils.collective_timing import validate_decode_layer_advance +from frontier.scheduler.utils.forward_sync_state import source_forward_mode +from frontier.scheduler.utils.request_selection import collect_active_requests + + +def handle_forward_sync_collective( + scheduler: Any, + time: float, + replica_id: int, + stage_id: int, + batch_global_id: int, + sync_stage: str, + layer_id: int, + metrics_store: Any, +): + """Complete one shared forward once, then continue each source locally. + + Two things belong to the forward as a whole and must happen exactly once: + the decode-phase layer counters advance, and the full-stage owners the EP + wave took over are restored. Everything after that is source-local — each + lane predicts its own next attention, keeps its own stage tail and reaches + its own completion — so it is delegated to the same per-phase helper the + disaggregated roles use, entered once per source. + """ + + if sync_stage != "post_moe": + raise ValueError( + "Forward collective completion accepts only post_moe; the canonical " + "EP_WAVE enters this method at post_moe" + ) + + replica_rooms = scheduler._forward_sync_waiting_room.get(replica_id) + stage_rooms = replica_rooms.get(stage_id) if replica_rooms is not None else None + step_rooms = stage_rooms.get(batch_global_id) if stage_rooms is not None else None + layer_rooms = step_rooms.get(layer_id) if step_rooms is not None else None + if layer_rooms is None or sync_stage not in layer_rooms: + raise RuntimeError( + "Forward collective event has no matching waiting room: " + f"replica={replica_id}, stage={stage_id}, " + f"batch_global_id={batch_global_id}, layer={layer_id}, " + f"sync_stage={sync_stage}" + ) + source_batches = layer_rooms.pop(sync_stage)["batches"] + + live_batches = [batch for batch in source_batches.values() if not batch.is_idle] + if not live_batches: + raise RuntimeError( + "Forward collective completion requires a non-idle participant batch: " + f"replica={replica_id}, stage={stage_id}, " + f"batch_global_id={batch_global_id}, layer={layer_id}" + ) + # Each per-phase helper refuses a legacy aggregate synchronization by + # checking that its batch carries the wave's lane timings, but only when it + # pops the room itself. Delegation passes `direct_batch`, so that check is + # skipped there and belongs here instead -- once per source, against the + # marker the wave writes for that source's own phase. + for source_batch in live_batches: + marker = ( + "_prefill_ep_wave_lane_times_ms" + if source_forward_mode(source_batch) == "prefill" + else "_decode_ep_wave_lane_times_ms" + ) + if not hasattr(source_batch, marker): + raise RuntimeError( + "Legacy aggregate synchronization is removed; a shared forward " + "source must carry its own EP wave lane timings: " + f"replica={replica_id}, stage={stage_id}, " + f"batch_global_id={batch_global_id}, layer={layer_id}, " + f"batch={source_batch.id}, expected={marker}" + ) + + # One completed layer advances a request's decode counter once, and only if + # that request is decoding. A prefill chunk has no decode layer to credit, + # and a request carried in a prefill batch after its own prefill finished + # does, which is the case the phase-specific paths could not express. + decoding_requests = [ + request + for request in collect_active_requests(source_batches.values()) + if request.is_prefill_complete + ] + num_layers = scheduler._config.replica_config.model_config.num_layers + validate_decode_layer_advance(decoding_requests, num_layers) + for request in decoding_requests: + request.mb_on_step_layer_count_increment(num_layers_completed=1) + + stage_layer_end = _stage_layer_end(scheduler, stage_id) + next_layer_id = layer_id + 1 + owners_restored = scheduler._restore_forward_step_full_stage_owners( + source_batches=source_batches, + replica_id=replica_id, + stage_id=stage_id, + layer_id=next_layer_id, + cohort_id=batch_global_id, + operation_kind="attention" if next_layer_id < stage_layer_end else "final", + ) + + from frontier.scheduler.utils.decode_collective import handle_decode_sync_collective + from frontier.scheduler.utils.prefill_collective import handle_prefill_sync_collective + + events = [] + for source_batch in live_batches: + is_prefill_source = source_forward_mode(source_batch) == "prefill" + handler = ( + handle_prefill_sync_collective + if is_prefill_source + else handle_decode_sync_collective + ) + # Only the decode helper advances layer counters, so only it needs to be + # told they are already advanced. + already_done = {} if is_prefill_source else {"layer_advance_done": True} + events.extend( + handler( + scheduler, + time, + replica_id, + stage_id, + batch_global_id, + sync_stage, + layer_id, + metrics_store, + direct_batch=source_batch, + owners_restored=owners_restored, + **already_done, + ) + ) + return events + + +def _stage_layer_end(scheduler: Any, stage_id: int) -> int: + """Return the half-open last layer this pipeline stage owns.""" + + num_layers = scheduler._predictor._num_layers_per_pipeline_stage + _, stage_layer_end = scheduler.get_pipeline_stage_layer_bounds(stage_id, num_layers) + return stage_layer_end diff --git a/frontier/scheduler/utils/forward_sync_state.py b/frontier/scheduler/utils/forward_sync_state.py index 14805667..ca01857d 100644 --- a/frontier/scheduler/utils/forward_sync_state.py +++ b/frontier/scheduler/utils/forward_sync_state.py @@ -31,13 +31,32 @@ def source_batches_by_lane(cohort_batches, batch): return normalized +def source_forward_mode(batch) -> str: + """Return the local phase one source batch runs in a shared forward. + + This is the same rule the stage-schedule event uses to pick a batch's sync + path on a monolithic Replica (`replica_stage_schedule_event.py`): a batch + carrying prefill tokens runs the prefill path, anything else decodes. Using + one rule in both places is what keeps a cohort's per-source continuation + consistent with how its lanes entered. + """ + + return "prefill" if int(batch.num_prefill_tokens) > 0 else "decode" + + +#: Open-step namespaces. ``prefill`` and ``decode`` belong to the disaggregated +#: roles, which run one phase each. ``forward`` belongs to a monolithic cluster, +#: where one Replica runs both phases and a cohort may mix them: its lanes have +#: to resolve to one step id, so they must share one namespace. +SYNC_KINDS = ("prefill", "decode", "forward") + + class ForwardSyncState: - """Own forward-step identity bookkeeping shared by PREFILL and DECODE.""" + """Own forward-step identity bookkeeping for every synchronizing role.""" def __init__(self) -> None: self._open_steps_by_kind: dict[str, dict[tuple, int]] = { - "prefill": {}, - "decode": {}, + kind: {} for kind in SYNC_KINDS } self._next_step_id_by_replica: dict[int, int] = {} @@ -55,7 +74,7 @@ def get_step_id(batch) -> int: @staticmethod def _validate_kind(sync_kind: str) -> None: - if sync_kind not in ("prefill", "decode"): + if sync_kind not in SYNC_KINDS: raise ValueError(f"unknown synchronization kind: {sync_kind!r}") def open_steps(self, sync_kind: str) -> dict[tuple, int]: diff --git a/frontier/scheduler/utils/prefill_collective.py b/frontier/scheduler/utils/prefill_collective.py index 027381ee..e6ac0ff1 100644 --- a/frontier/scheduler/utils/prefill_collective.py +++ b/frontier/scheduler/utils/prefill_collective.py @@ -25,8 +25,16 @@ def handle_prefill_sync_collective( metrics_store: Any, *, direct_batch: Optional[Batch] = None, + owners_restored: Optional[bool] = None, ): - """Handle completion of a canonical layer-local PREFILL EP wave.""" + """Handle completion of a canonical layer-local PREFILL EP wave. + + `owners_restored` carries the result of a full-stage restoration the caller + already performed for a wider cohort. A shared monolithic forward restores + once for all its lanes and then enters this helper per source, so restoring + again here would hand out a second ticket for the same lane. `None` keeps + the standalone behavior: this call owns the restoration. + """ # Event modules import scheduler registries, so load them after the # scheduler package has finished initialization. @@ -77,45 +85,27 @@ def handle_prefill_sync_collective( f"layer={layer_id}" ) - execution_time = scheduler._predictor.predict_stage_execution_time( - sample_batch, - stage_id, - cluster_type=scheduler._cluster_type, - num_layers=1, - layer_id=layer_id, - include_ffn=False, - ) - num_layers = scheduler._predictor._num_layers_per_pipeline_stage stage_layer_start, stage_layer_end = scheduler.get_pipeline_stage_layer_bounds( stage_id, num_layers, ) next_layer_id = layer_id + 1 - restored_full_stage_owners = scheduler._restore_forward_step_full_stage_owners( - source_batches=participant_batches, - replica_id=replica_id, - stage_id=stage_id, - layer_id=next_layer_id, - cohort_id=batch_global_id, - operation_kind=( - "attention" if next_layer_id < stage_layer_end else "final" - ), - ) - - if layer_id < stage_layer_end - 1: - next_layer_execution_time = scheduler._predictor.predict_stage_execution_time( - sample_batch, - stage_id, - cluster_type=scheduler._cluster_type, - num_layers=1, + if owners_restored is None: + restored_full_stage_owners = scheduler._restore_forward_step_full_stage_owners( + source_batches=participant_batches, + replica_id=replica_id, + stage_id=stage_id, layer_id=next_layer_id, - include_ffn=False, + cohort_id=batch_global_id, + operation_kind=( + "attention" if next_layer_id < stage_layer_end else "final" + ), ) - attention_time_ms = next_layer_execution_time.get_single_layer_attention_scope_time() - attention_time = attention_delay_seconds(next_layer_execution_time) - total_time_to_next_sync = attention_time + else: + restored_full_stage_owners = owners_restored + if layer_id < stage_layer_end - 1: for replica_local_id, batch in participant_batches.items(): if batch.is_idle: logger.info( @@ -124,6 +114,20 @@ def handle_prefill_sync_collective( f"layer={layer_id})" ) continue + # Each lane continues on its own inputs: its own context lengths and + # its own token count give its own next-attention time. Borrowing one + # lane's prediction for the others is what made a mixed cohort + # advance on a batch it does not contain. + next_layer_execution_time = scheduler._predictor.predict_stage_execution_time( + batch, + stage_id, + cluster_type=scheduler._cluster_type, + num_layers=1, + layer_id=next_layer_id, + include_ffn=False, + ) + attention_time_ms = next_layer_execution_time.get_single_layer_attention_scope_time() + total_time_to_next_sync = attention_delay_seconds(next_layer_execution_time) component_ledger = getattr( batch, "_prefill_model_execution_components_ms_by_stage", @@ -185,7 +189,14 @@ def handle_prefill_sync_collective( replica_id, stage_identity, stage_id ) is_last_stage = stage_scheduler.is_last_stage - pipeline_time = execution_time.pipeline_time * 1e-3 + execution_time = scheduler._predictor.predict_stage_execution_time( + batch, + stage_id, + cluster_type=scheduler._cluster_type, + num_layers=1, + layer_id=layer_id, + include_ffn=False, + ) if not hasattr(batch, "_prefill_stage_start_time"): raise ValueError( "missing PREFILL stage start time: " diff --git a/frontier/scheduler/utils/sync_entry.py b/frontier/scheduler/utils/sync_entry.py index abb26477..82b6e4a6 100644 --- a/frontier/scheduler/utils/sync_entry.py +++ b/frontier/scheduler/utils/sync_entry.py @@ -1,8 +1,9 @@ -"""PREFILL and DECODE forward-step synchronization entry handlers.""" +"""Per-layer forward-step synchronization entry for PREFILL, DECODE, MONOLITHIC.""" from typing import Any from frontier.entities import Batch +from frontier.types import ClusterType def _can_supply_idle_lane(scheduler, sibling_stage, replica_id, stage_id): @@ -14,7 +15,33 @@ def _can_supply_idle_lane(scheduler, sibling_stage, replica_id, stage_id): return scheduler.get_stage_execution_context(replica_id, stage_id).forward_group_sealed -def enter_prefill_sync( +def uses_shared_forward_room(scheduler: Any) -> bool: + """Return whether this cluster keeps one room for both local phases. + + A monolithic Replica runs prefill and decode on the same lanes, so one + forward can hold a prefill batch on one lane and a decode batch on another. + Those lanes must wait in one room and resolve to one step id. + """ + + # Checked room-first so a lightweight scheduler fixture that never sets up + # a shared room is answered without requiring a cluster type. + return ( + getattr(scheduler, "_forward_sync_waiting_room", None) is not None + and getattr(scheduler, "_cluster_type", None) is ClusterType.MONOLITHIC + ) + + +def _load_sync_event(mode: str): + if mode == "prefill": + from frontier.events.prefill_sync_event import PrefillSyncEvent + + return PrefillSyncEvent + from frontier.events.decode_sync_event import DecodeSyncEvent + + return DecodeSyncEvent + + +def enter_layer_sync( scheduler: Any, time: float, replica_id: int, @@ -25,32 +52,66 @@ def enter_prefill_sync( layer_id: int, stage_execution_time: float, *, + mode: str, metrics_store: Any = None, ) -> list: + """Admit one lane into its forward's pre_moe room, and dispatch when full. + + `mode` is the entering batch's own local phase. It selects the layer-path + check, the event class used to fill an idle lane, and, for a disaggregated + role, which room is used. On a monolithic cluster the room and the step-id + namespace are shared, so a cohort whose lanes disagree about their phase + still resolves to one forward. + """ + del stage_execution_time - if scheduler._prefill_sync_waiting_room is None: + if mode not in ("prefill", "decode"): + raise ValueError(f"unsupported layer synchronization mode: {mode!r}") + mode_name = mode.upper() + + shared_room = uses_shared_forward_room(scheduler) + if shared_room: + waiting_room = scheduler._forward_sync_waiting_room + sync_kind = "forward" + elif mode == "prefill": + waiting_room = scheduler._prefill_sync_waiting_room + sync_kind = "prefill" + else: + waiting_room = scheduler._decode_sync_waiting_room + sync_kind = "decode" + + if waiting_room is None: raise ValueError( - "PREFILL synchronization is unavailable for a dense model; dense execution must use the full-stage protocol" + f"{mode_name} synchronization is unavailable for a dense model; " + "dense execution must use the full-stage protocol" ) if sync_stage != "pre_moe": raise ValueError( - "PREFILL synchronization entry must start at pre_moe; post_moe completion is handled by PrefillSyncCollectiveEvent" + f"{mode_name} synchronization entry must start at pre_moe; post_moe " + "completion is handled by the collective event" ) - if not scheduler._uses_shared_prefill_layer_path(batch, layer_id): + layer_path_ok = ( + scheduler._uses_shared_prefill_layer_path(batch, layer_id) + if mode == "prefill" + else scheduler._uses_shared_decode_layer_path(batch, layer_id) + ) + if not layer_path_ok: raise RuntimeError( - "Legacy PREFILL DP synchronization is removed; the current layer must use the canonical per-layer protocol" + f"Legacy {mode_name} DP synchronization is removed; the current " + "layer must use the canonical per-layer protocol" ) if replica_local_id is None: lane_id = 0 elif type(replica_local_id) is not int or replica_local_id < 0: raise ValueError( - "PREFILL replica_local_id must be an exact non-negative int or None" + f"{mode_name} replica_local_id must be an exact non-negative int or None" ) else: lane_id = replica_local_id + step_id = scheduler._resolve_forward_step( - sync_kind="prefill", - waiting_room=scheduler._prefill_sync_waiting_room, + sync_kind=sync_kind, + waiting_room=waiting_room, replica_id=replica_id, stage_id=stage_id, batch=batch, @@ -60,7 +121,7 @@ def enter_prefill_sync( ) if step_id is None: return [] - sync_room = scheduler._prefill_sync_waiting_room[replica_id][stage_id][step_id][layer_id][sync_stage] + sync_room = waiting_room[replica_id][stage_id][step_id][layer_id][sync_stage] # resolve_step retains this binding identity while step_id advances per layer. provisional_id = batch._forward_cohort_provisional_id sync_room.setdefault("provisional_cohort_id", provisional_id) @@ -72,9 +133,12 @@ def enter_prefill_sync( expected_lanes = scheduler._replica_dp_size if type(expected_lanes) is not int or expected_lanes <= 0: - raise ValueError(f"PREFILL attention-DP lane count must be positive, got {expected_lanes}") + raise ValueError( + f"{mode_name} attention-DP lane count must be positive, got {expected_lanes}" + ) if len(sync_room["batches"]) < expected_lanes and not batch.is_idle: idle_events = [] + event_cls = _load_sync_event(mode) replica_schedulers = scheduler._replica_schedulers for missing_lane in range(expected_lanes): if missing_lane in sync_room["batches"]: @@ -101,10 +165,8 @@ def enter_prefill_sync( idle_batch._stage_owner_replica_local_id = missing_lane sync_room["batches"][missing_lane] = idle_batch sync_room["arrival_times"][missing_lane] = float(time) - from frontier.events.prefill_sync_event import PrefillSyncEvent - idle_events.append( - PrefillSyncEvent( + event_cls( time=float(time), replica_id=replica_id, stage_id=stage_id, @@ -126,11 +188,11 @@ def enter_prefill_sync( provisional_id = sync_room["provisional_cohort_id"] if type(provisional_id) is not int or provisional_id < 0: raise RuntimeError( - "PREFILL synchronization room has an invalid provisional step id: " + f"{mode_name} synchronization room has an invalid provisional step id: " f"{provisional_id!r}" ) scheduler._close_forward_step( - sync_kind="prefill", + sync_kind=sync_kind, replica_id=replica_id, stage_id=stage_id, layer_id=layer_id, @@ -140,7 +202,24 @@ def enter_prefill_sync( sync_room.pop("batches", None) sync_room.pop("arrival_times", None) sync_room.pop("provisional_cohort_id", None) - return scheduler._on_prefill_ep_wave_ready( + + if shared_room: + return scheduler._on_forward_ep_wave_ready( + time=sync_time, + replica_id=replica_id, + stage_id=stage_id, + batch=batch, + layer_id=layer_id, + replica_local_id=replica_local_id, + cohort_batches=step_batches, + metrics_store=metrics_store, + ) + ready = ( + scheduler._on_prefill_ep_wave_ready + if mode == "prefill" + else scheduler._on_decode_ep_wave_ready + ) + return ready( time=sync_time, replica_id=replica_id, stage_id=stage_id, @@ -152,7 +231,7 @@ def enter_prefill_sync( ) -def enter_decode_sync( +def enter_prefill_sync( scheduler: Any, time: float, replica_id: int, @@ -165,125 +244,32 @@ def enter_decode_sync( *, metrics_store: Any = None, ) -> list: - del stage_execution_time - if scheduler._decode_sync_waiting_room is None: - raise ValueError( - "DECODE synchronization is unavailable for a dense model; dense execution must use the full-stage protocol" - ) - if sync_stage != "pre_moe": - raise ValueError( - "DECODE synchronization entry must start at pre_moe; post_moe completion is handled by DecodeSyncCollectiveEvent" - ) - if not scheduler._uses_shared_decode_layer_path(batch, layer_id): - raise RuntimeError( - "Legacy DECODE DP synchronization is removed; the current layer must use the canonical per-layer protocol" - ) - if replica_local_id is None: - lane_id = 0 - elif type(replica_local_id) is not int or replica_local_id < 0: - raise ValueError( - "DECODE replica_local_id must be an exact non-negative int or None" - ) - else: - lane_id = replica_local_id - step_id = scheduler._resolve_forward_step( - sync_kind="decode", - waiting_room=scheduler._decode_sync_waiting_room, - replica_id=replica_id, - stage_id=stage_id, - batch=batch, - lane_id=lane_id, - layer_id=layer_id, - sync_stage=sync_stage, + """Admit a lane whose local phase is prefill.""" + + return enter_layer_sync( + scheduler, time, replica_id, stage_id, batch, replica_local_id, + sync_stage, layer_id, stage_execution_time, + mode="prefill", metrics_store=metrics_store, ) - if step_id is None: - return [] - sync_room = scheduler._decode_sync_waiting_room[replica_id][stage_id][step_id][layer_id][sync_stage] - # resolve_step retains this binding identity while step_id advances per layer. - provisional_id = batch._forward_cohort_provisional_id - sync_room.setdefault("provisional_cohort_id", provisional_id) - existing_batch = sync_room["batches"].get(lane_id) - if batch.is_idle and existing_batch is not None and not existing_batch.is_idle: - return [] - sync_room["batches"][lane_id] = batch - sync_room["arrival_times"][lane_id] = float(time) - expected_lanes = scheduler._replica_dp_size - if type(expected_lanes) is not int or expected_lanes <= 0: - raise ValueError(f"DECODE attention-DP lane count must be positive, got {expected_lanes}") - if len(sync_room["batches"]) < expected_lanes and not batch.is_idle: - idle_events = [] - replica_schedulers = scheduler._replica_schedulers - for missing_lane in range(expected_lanes): - if missing_lane in sync_room["batches"]: - continue - sibling = replica_schedulers.get((replica_id, missing_lane)) - if sibling is None: - raise RuntimeError( - "Missing Replica scheduler for expected attention-DP lane: " - f"replica_id={replica_id}, replica_local_id={missing_lane}" - ) - sibling_stage = sibling.get_replica_stage_scheduler(stage_id) - if not _can_supply_idle_lane(scheduler, sibling_stage, replica_id, stage_id): - continue - idle_batch = Batch( - replica_id=replica_id, - requests=[], - num_tokens=[], - is_idle=True, - is_moe=batch.is_moe, - ) - idle_batch.set_global_id(expected_lanes * step_id + missing_lane) - idle_batch._forward_cohort_id = step_id - idle_batch._forward_cohort_provisional_id = provisional_id - idle_batch._stage_owner_replica_local_id = missing_lane - sync_room["batches"][missing_lane] = idle_batch - sync_room["arrival_times"][missing_lane] = float(time) - from frontier.events.decode_sync_event import DecodeSyncEvent - idle_events.append( - DecodeSyncEvent( - time=float(time), - replica_id=replica_id, - stage_id=stage_id, - batch=idle_batch, - replica_local_id=missing_lane, - sync_stage=sync_stage, - layer_id=layer_id, - stage_execution_time=0.0, - cluster_type=scheduler._cluster_type, - ) - ) - if idle_events: - return idle_events - if len(sync_room["batches"]) != expected_lanes: - return [] - sync_time = max(sync_room["arrival_times"].values()) - step_batches = dict(sync_room["batches"]) - provisional_id = sync_room["provisional_cohort_id"] - if type(provisional_id) is not int or provisional_id < 0: - raise RuntimeError( - "DECODE synchronization room has an invalid provisional step id: " - f"{provisional_id!r}" - ) - scheduler._close_forward_step( - sync_kind="decode", - replica_id=replica_id, - stage_id=stage_id, - layer_id=layer_id, - sync_stage=sync_stage, - provisional_id=provisional_id, - ) - sync_room.pop("batches", None) - sync_room.pop("arrival_times", None) - sync_room.pop("provisional_cohort_id", None) - return scheduler._on_decode_ep_wave_ready( - time=sync_time, - replica_id=replica_id, - stage_id=stage_id, - batch=batch, - layer_id=layer_id, - replica_local_id=replica_local_id, - cohort_batches=step_batches, - metrics_store=metrics_store, +def enter_decode_sync( + scheduler: Any, + time: float, + replica_id: int, + stage_id: int, + batch: Batch, + replica_local_id: int | None, + sync_stage: str, + layer_id: int, + stage_execution_time: float, + *, + metrics_store: Any = None, +) -> list: + """Admit a lane whose local phase is decode.""" + + return enter_layer_sync( + scheduler, time, replica_id, stage_id, batch, replica_local_id, + sync_stage, layer_id, stage_execution_time, + mode="decode", metrics_store=metrics_store, ) diff --git a/frontier/scheduler/utils/sync_state.py b/frontier/scheduler/utils/sync_state.py index c21ff969..661fc544 100644 --- a/frontier/scheduler/utils/sync_state.py +++ b/frontier/scheduler/utils/sync_state.py @@ -19,22 +19,34 @@ def _new_sync_waiting_room(): def initialize_sync_waiting_rooms(scheduler: Any) -> None: - """Initialize layer-sync rooms for PREFILL, MONOLITHIC, or DECODE.""" + """Initialize layer-sync rooms for PREFILL, MONOLITHIC, or DECODE. + + A monolithic Replica runs prefill and decode on the same lanes, so one + forward may hold a prefill batch on one lane and a decode batch on another. + Those lanes have to wait in the *same* room or neither ever reaches the + expected lane count. The monolithic cluster therefore gets one room, bound + to `_forward_sync_waiting_room` and to both phase names that existing call + sites select by mode. A disaggregated role runs one phase and keeps its own + single room. + """ cluster_type = scheduler._cluster_type model_config = scheduler._config.replica_config.model_config model_is_moe = model_config is not None and model_config.is_moe - if cluster_type in (ClusterType.PREFILL, ClusterType.MONOLITHIC): - if model_is_moe: - scheduler._prefill_sync_waiting_room = _new_sync_waiting_room() - scheduler._decode_sync_waiting_room = ( - _new_sync_waiting_room() - if cluster_type is ClusterType.MONOLITHIC - else None - ) - else: - scheduler._prefill_sync_waiting_room = None - scheduler._decode_sync_waiting_room = None + if cluster_type is ClusterType.MONOLITHIC: + shared_room = _new_sync_waiting_room() if model_is_moe else None + scheduler._forward_sync_waiting_room = shared_room + scheduler._prefill_sync_waiting_room = shared_room + scheduler._decode_sync_waiting_room = shared_room + return + + scheduler._forward_sync_waiting_room = None + + if cluster_type is ClusterType.PREFILL: + scheduler._prefill_sync_waiting_room = ( + _new_sync_waiting_room() if model_is_moe else None + ) + scheduler._decode_sync_waiting_room = None return if cluster_type is ClusterType.DECODE: diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md new file mode 100644 index 00000000..9eaee708 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md @@ -0,0 +1,137 @@ +# W3 design — one shared monolithic forward + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-22 | Created. Source-backed design for Checkpoint D's W3 half, with the scope decisions and their evidence. | + +## The defect, restated from source + +In `MONOLITHIC`, a batch takes the prefill or the decode per-layer sync path by +one rule, at `frontier/events/replica_stage_schedule_event.py:157-167`: + +```python +is_monolithic_prefill_moe = ... and batch.num_prefill_tokens > 0 +is_monolithic_decode_moe = ... and batch.num_prefill_tokens <= 0 + and batch.num_decode_tokens > 0 +``` + +`initialize_sync_waiting_rooms` (`frontier/scheduler/utils/sync_state.py:29-34`) +then allocates **two disjoint waiting-room trees** for that one cluster, and +`ForwardSyncState` (`forward_sync_state.py:38-41`) keeps **two open-step +namespaces**, `"prefill"` and `"decode"`. + +So when lane 0 holds a batch with prefill tokens and lane 1 holds a pure-decode +batch, lane 0 waits in the prefill room and lane 1 in the decode room. Each room +requires `expected_lanes == _replica_dp_size` members. Neither reaches it. The +idle-lane filler cannot rescue either room: `_can_supply_idle_lane` +(`sync_entry.py:8-14`) refuses a sibling whose stage `is_busy`, and the sibling +is busy holding the other phase. Both entries return `[]` and the forward never +dispatches. + +This needs `attn_dp >= 2`. With one lane a cohort has one batch and therefore one +phase, which is why the defect has never been observed on the shipped recipes. + +## Reachability, and what that means for acceptance + +The public MoE wrappers enforce `ATTN_TP == MOE_TP * MOE_EP` while the runtime +enforces `attn_tp * attn_dp == moe_tp * moe_ep`; these have no common solution +above one lane. **No case in the 71-case fidelity matrix can reach a multi-lane +monolithic MoE forward at all.** That is why R35-02 requires a +direct-construction integration fixture rather than another wrapper case, and it +is also why most of this change is expected to be fidelity-neutral. + +The expectation is therefore stated in two parts, before measuring: + +- The shared-room, shared-identity, one-restoration and per-source-continuation + changes cannot move any matrix case, because every matrix case runs one lane, + where the cohort is a single batch and `sample_batch` *is* that batch. +- The decode-layer advance for decode requests carried inside a prefill batch + (I7) is reachable under chunked prefill at one lane, so it may move MoE cases. + It is the one part of this change with a real blast radius; see below. + +## Scope decisions + +| Invariant | In scope | Why | +| --- | --- | --- | +| I1 one shared step-id namespace | yes | The defect. | +| I2 one waiting room per monolithic forward | yes | The defect. | +| I3 a request may not occupy two non-idle lanes | yes | Group-formation guard the review asks to validate at the owning boundary. | +| I4 one ownership restoration per cohort | yes | "one shared completion, one ownership restoration". | +| I5 source-local continuation | yes | "Do not choose a single sample batch and apply its predicted duration to all source lanes." | +| I7 decode-layer advance inside a mixed source | yes, measured | The prefill entry path never advances a decode request's layer counter. | +| I11 per-source dense-layer label | yes | A decode source inside a prefill-mode group is mislabelled today. | +| I6, I9, I10 | already hold | I6 follows from one room; I9/I10 verified unchanged in the audit. | +| **I8 per-source decode component ledger** | **no** | Checkpoint D says "current-main metrics ownership". Introducing the donor's `_decode_model_execution_components_ms_by_stage` would replace main's decode final-timing computation and move every reachable MoE decode case. Excluded deliberately, not overlooked. | + +## Event-type determinism: the constraint that shapes the design + +Events order by `(time, event_type, id)` (`frontier/events/base_event.py:66-72`), +and `EventType` values are the priority (`PREFILL_SYNC_COLLECTIVE = 10`, +`DECODE_SYNC_COLLECTIVE = 24`). Two consequences: + +1. **A new `EventType` value must not be given to a cohort shape that already + works.** A pure-prefill monolithic MoE cohort emits + `PrefillSyncCollectiveEvent` today and is exercised by the matrix. Moving it + to a new event type would reorder it against `REPLICA_STAGE_SCHEDULE = 11` at + equal timestamps and change results. +2. **The class must not depend on which lane completed the cohort.** Choosing it + from the triggering entry would make a mixed cohort's priority depend on + arrival order, which is nondeterminism introduced by the fix. + +Resolution: the collective event class for a monolithic cohort is chosen from +the **cohort's contents**, by the same rule the entry event uses — +`PrefillSyncCollectiveEvent` if any non-idle source has `num_prefill_tokens > 0`, +`DecodeSyncCollectiveEvent` otherwise. A pure-prefill cohort and a pure-decode +cohort therefore keep exactly the event type they have today; only the mixed +cohort, which currently deadlocks, is new. No `EventType` value is added. + +## Planned edits + +| File | Change | +| --- | --- | +| `forward_sync_state.py` | Add the `"forward"` kind to the open-step namespaces and `_validate_kind`. | +| `sync_state.py` | For `MONOLITHIC` + MoE, allocate one room and bind `_forward_sync_waiting_room` plus both legacy names to it. `PREFILL`/`DECODE` keep their own single room and get `_forward_sync_waiting_room = None`. | +| `sync_entry.py` | Collapse the two ~140-line near-duplicate entries into one `enter_layer_sync(..., mode)` with thin `enter_prefill_sync` / `enter_decode_sync` wrappers, mirroring the existing `schedule_layer_wave(mode=...)`. Use `sync_kind="forward"` and the shared room when the cluster is monolithic. | +| `ep_wave_inputs.py` | Reject a request appearing in two non-idle source lanes. | +| `ep_wave_schedule.py` | Per-source mode for the dense-layer event label and the prefill-only component ledger; choose the post_moe collective class from cohort contents. | +| `forward_collective.py` (new) | Pop the shared room, advance the decode-phase subset once, restore full-stage owners once, then run each non-idle source through its own existing helper with `direct_batch`. | +| `prefill_collective.py`, `decode_collective.py` | Accept `owners_restored` and `layer_advance_done` so the shared actions are not repeated; predict each source's continuation from that source's own batch instead of `sample_batch`. | + +Routing a mixed cohort through per-source `direct_batch` calls is what makes I5 +fall out: with `direct_batch`, the helper's `sample_batch` *is* the source batch, +so each lane already predicts its own continuation. The remaining `sample_batch` +uses inside the multi-lane loops are corrected in place. + +## What the tests must distinguish + +Per the plan and R35-02, a helper-level stub is not acceptable as the only +evidence. At least one test builds a runtime configuration directly, runs the +real event loop, real admission, ownership and completion code, and injects +deterministic durations only at the predictor boundary. It must fail both on the +old mixed-phase deadlock and on a deliberately wrong source-timing +implementation. + +## Fidelity expectation, stated before measuring + +`completed_layer_count` is read on the monolithic path only by admission guards +and diagnostics. Its two arithmetic consumers, +`ClusterBatchEndEvent._get_current_layer_id_from_batch` call sites +(`cluster_batch_end_event.py:174` and `:354`), are both PD-AF +`DECODE_ATTN`/M2N paths. No monolithic predicted duration is derived from it. + +The prediction for the 71-case matrix is therefore: + +- **All 71 cases stay exactly equal.** At one attention-DP lane every cohort + holds one source batch, so the shared room, the shared identity, the single + restoration and the per-source continuation all reduce to today's + single-source behavior; `sample_batch` *is* the only batch. +- The one behavior change reachable at one lane is the I7 layer credit for an + already-decoding request carried inside a prefill-mode batch. It changes a + counter that no monolithic timing reads. +- **If a MoE co-location case does move, the I7 credit is the cause**, and it is + the approved fidelity fix that plan section 9 requires ("advance the decode + subset exactly once per completed layer"). + +Recorded before the measurement so the result can falsify it. diff --git a/tests/integration/test_monolithic_mixed_forward_runtime.py b/tests/integration/test_monolithic_mixed_forward_runtime.py new file mode 100644 index 00000000..8f7e90cb --- /dev/null +++ b/tests/integration/test_monolithic_mixed_forward_runtime.py @@ -0,0 +1,384 @@ +"""Real-runtime acceptance for one shared monolithic forward across phases. + +The 71-case fidelity matrix cannot reach this shape: the public MoE wrappers +enforce `ATTN_TP == MOE_TP * MOE_EP` while the runtime enforces +`attn_tp * attn_dp == moe_tp * moe_ep`, and those have no common solution above +one attention-DP lane. This test therefore builds a valid runtime configuration +directly -- `attn_tp=1, attn_dp=2, moe_tp=1, moe_ep=2` on a monolithic MoE +Replica -- and runs the real `Simulator` event loop over it, with real +admission, ownership, synchronization and completion code. Deterministic +durations enter only through the predictor: constant profiling targets, plus an +observer that wraps `predict_stage_execution_time` without replacing it. +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path +import subprocess +import sys + +import pytest + + +def test_shared_monolithic_forward_completes_every_request(tmp_path): + # The child must import the same checkout this test file came from, not + # whichever tree an editable install happens to point at. + repo_root = Path(__file__).resolve().parents[2] + result = subprocess.run( + [sys.executable, str(Path(__file__).resolve()), str(tmp_path)], + env={ + **os.environ, + "OMP_NUM_THREADS": "1", + "OPENBLAS_NUM_THREADS": "1", + "PYTHONPATH": os.pathsep.join( + [str(repo_root), os.environ.get("PYTHONPATH", "")] + ).rstrip(os.pathsep), + }, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + timeout=900, + ) + (tmp_path / "run.log").write_text(result.stdout) + assert result.returncode == 0, result.stdout[-15000:] + evidence = json.loads((tmp_path / "shared_forward_evidence.json").read_text()) + # The run has to reach the shape under test, or it proves nothing. + assert evidence["mixed_phase_cohorts"] > 0, evidence + + +# Requests chosen so that chunked prefill leaves one lane prefilling while the +# other has already started decoding: unequal prefill lengths, unequal decode +# budgets, all arriving at once. +REQUEST_SHAPES = ((32, 4), (16, 4), (24, 3), (16, 3)) + + +def _build_config(root, patch): + import pandas as pd + + from frontier.config import ( + BaseModelConfig, + ClusterConfig, + FixedRequestLengthGeneratorConfig, + MetricsConfig, + PoissonRequestIntervalGeneratorConfig, + RandomForrestExecutionTimePredictorConfig, + ReplicaConfig, + SimulationConfig, + SyntheticRequestGeneratorConfig, + VllmV1SchedulerConfig, + ) + from tests.integration.test_pr33_nondummy_acceptance import _model, _profiles + + model = _model("moe") + original = BaseModelConfig.create_from_name + patch.setattr( + BaseModelConfig, + "create_from_name", + classmethod( + lambda cls, name: model if name == model._model_name else original(name) + ), + ) + profiles = _profiles(root / "profiles", model, "moe") + # Chunked prefill produces true mixed batches, whose decode attention is a + # separate profiled operator. Add those rows at the same constant targets. + for filename in ("attention.csv", "attention_kernel_only.csv"): + path = root / "profiles" / filename + frame = pd.read_csv(path, keep_default_na=False) + rows = [] + for tp in (1, 2): + for chunk in (1, 4, 8, 16): + for decodes in (1, 2, 3): + row = frame.iloc[0].to_dict() + row.update( + num_tensor_parallel_workers=tp, + is_true_mixed_batch=True, + is_mixed_batch=True, + is_prefill=True, + batch_size=1 + decodes, + total_batch_size=1 + decodes, + num_prefill_seqs=1, + prefill_chunk_size=chunk, + total_prefill_tokens=chunk, + total_tokens=chunk + decodes, + decode_batch_size=decodes, + decode_avg_kv_cache_size=16, + kv_cache_size=16, + batch_composition_ratio=chunk / (chunk + decodes), + prefill_seq_lens=json.dumps([chunk]), + prefill_kv_cache_sizes=json.dumps([0]), + ) + row["time_stats.attn_prefill.median"] = 0.08 + row["time_stats.attn_decode.median"] = 0.05 + row["time_stats.attn_kv_cache_save.median"] = 0.01 + rows.append(row) + pd.concat([frame, pd.DataFrame(rows)], ignore_index=True).to_csv( + path, index=False + ) + + predictor = RandomForrestExecutionTimePredictorConfig( + enable_dummy_mode=False, + **profiles, + num_estimators=[2], + max_depth=[2], + min_samples_split=[2], + k_fold_cv_splits=2, + num_training_job_threads=1, + prediction_max_tokens_per_request=64, + prediction_max_prefill_chunk_size=32, + prediction_max_batch_size=4, + kv_cache_prediction_granularity=16, + skip_cpu_overhead_modeling=True, + ) + replica = ReplicaConfig( + model_name=model._model_name, + device="a100", + network_device="a100_pairwise_nvlink", + num_pipeline_stages=1, + attn_tensor_parallel_size=1, + attn_dp=2, + moe_tensor_parallel_size=1, + moe_expert_parallel_size=2, + total_expert_num=8, + router_topk=2, + ) + return model, predictor, replica, ( + VllmV1SchedulerConfig, + ClusterConfig, + MetricsConfig, + SimulationConfig, + SyntheticRequestGeneratorConfig, + FixedRequestLengthGeneratorConfig, + PoissonRequestIntervalGeneratorConfig, + ) + + +def _phase_of(batch) -> str: + """The rule a monolithic Replica uses to pick a batch's sync path. + + Stated here rather than imported, so the test's expectation does not move + when the implementation helper does -- and so it can be run against a + checkout that has no such helper. + """ + + return "prefill" if int(batch.num_prefill_tokens) > 0 else "decode" + + +def _drained(room) -> bool: + """Return whether every allocated waiting-room leaf is empty.""" + + for stages in room.values(): + for steps in stages.values(): + for layers in steps.values(): + for sync_stages in layers.values(): + for entry in sync_stages.values(): + if entry.get("batches"): + return False + return True + + +def run_case(root: Path, *, reporting: bool): + from frontier.entities import Request + from frontier.request_generator.synthetic_request_generator import ( + SyntheticRequestGenerator, + ) + from frontier.scheduler.utils import ep_wave_schedule + from frontier.simulator import Simulator + from frontier.types import ClusterType + + with pytest.MonkeyPatch.context() as patch: + model, predictor_config, replica, classes = _build_config(root, patch) + ( + VllmV1SchedulerConfig, + ClusterConfig, + MetricsConfig, + SimulationConfig, + SyntheticRequestGeneratorConfig, + FixedRequestLengthGeneratorConfig, + PoissonRequestIntervalGeneratorConfig, + ) = classes + cluster = ClusterConfig( + replica_config=replica, + replica_scheduler_config=VllmV1SchedulerConfig( + num_blocks=128, + block_size=16, + batch_size_cap=4, + max_tokens_in_batch=16, + enable_chunked_prefill=True, + ), + execution_time_predictor_config=predictor_config, + ) + config = SimulationConfig( + simulation_mode="offline", + sys_arch="co-location", + enable_parallel_clusters=False, + decode_cuda_graph_mode="none", + cluster_config=cluster, + metrics_config=MetricsConfig( + output_dir=str(root / "metrics"), + cache_dir=str(root / "cache"), + run_id="shared_forward", + write_metrics=True, + store_request_metrics=True, + store_batch_metrics=reporting, + store_operation_metrics=reporting, + store_utilization_metrics=reporting, + store_plots=False, + enable_chrome_trace=False, + write_json_trace=False, + ), + request_generator_config=SyntheticRequestGeneratorConfig( + num_requests=len(REQUEST_SHAPES), + length_generator_config=FixedRequestLengthGeneratorConfig( + prefill_tokens=16, decode_tokens=3 + ), + interval_generator_config=PoissonRequestIntervalGeneratorConfig( + qps=1e6 + ), + ), + ) + requests = [ + Request(0.0, prefill, decode) for prefill, decode in REQUEST_SHAPES + ] + patch.setattr( + SyntheticRequestGenerator, "generate", lambda self: list(requests) + ) + + # Cohort membership and the prediction log, both observed without + # changing what the runtime does. + cohorts: list[dict] = [] + predictions: list[int] = [] + real_wave = ep_wave_schedule.schedule_layer_wave + + def observe_wave(scheduler, *, mode, batch, layer_id, cohort_batches=None, **kw): + sources = cohort_batches if cohort_batches else {0: batch} + cohorts.append( + { + "layer_id": layer_id, + # Where this cohort starts in the prediction log, so the + # calls its own completion makes can be isolated. + "first_call": len(predictions), + "members": { + source.id: _phase_of(source) + for source in sources.values() + if not source.is_idle + }, + } + ) + return real_wave( + scheduler, + mode=mode, + batch=batch, + layer_id=layer_id, + cohort_batches=cohort_batches, + **kw, + ) + + patch.setattr(ep_wave_schedule, "schedule_layer_wave", observe_wave) + import frontier.scheduler.cluster_scheduler.base_cluster_scheduler as bcs + + patch.setattr(bcs, "schedule_layer_wave", observe_wave) + + simulator = Simulator(config) + + # The one injection point: wrap the predictor to record which batch each + # attention-scope prediction was made for. Wrapping keeps the real + # prediction; it only makes source attribution observable. + predictor = simulator._global_scheduler.get_cluster_scheduler( + ClusterType.MONOLITHIC + )._predictor + real_predict = predictor.predict_stage_execution_time + + def observe_predict(batch, stage_id, cluster_type=None, **kwargs): + if kwargs.get("include_ffn") is False: + predictions.append(int(batch.id)) + if cluster_type is None: + return real_predict(batch, stage_id, **kwargs) + return real_predict(batch, stage_id, cluster_type, **kwargs) + + patch.setattr( + predictor, "predict_stage_execution_time", observe_predict, raising=False + ) + + simulator.run() + + cluster_scheduler = simulator._global_scheduler.get_cluster_scheduler( + ClusterType.MONOLITHIC + ) + mixed = [c for c in cohorts if len(set(c["members"].values())) > 1] + evidence = { + "reporting": reporting, + "total_cohorts": len(cohorts), + "mixed_phase_cohorts": len(mixed), + "completed_requests": sum(request.completed for request in requests), + "makespan": simulator._time, + } + + # Every request finishes, exactly once, with every token accounted for. + assert all(request.completed for request in requests), evidence + for request, (prefill, decode) in zip(requests, REQUEST_SHAPES): + assert request.num_prefill_tokens == prefill + assert request.num_decode_tokens == decode + assert request.num_processed_tokens == prefill + decode + rows = _read_request_metrics(root) + assert len(rows) == len(requests) + assert len({row["Request Id"] for row in rows}) == len(requests) + assert sum(int(float(row["request_num_tokens"])) for row in rows) == sum( + prefill + decode for prefill, decode in REQUEST_SHAPES + ) + + # Each live source in a mixed cohort continued on its own prediction. + # A cohort's completion runs after its own wave and before the next + # wave is scheduled, so that slice of the prediction log belongs to it. + boundaries = [cohort["first_call"] for cohort in cohorts] + [len(predictions)] + for index, cohort in enumerate(cohorts): + if len(set(cohort["members"].values())) < 2: + continue + window = set(predictions[cohort["first_call"] : boundaries[index + 1]]) + assert set(cohort["members"]) <= window, (cohort, sorted(window)) + + # Nothing is stranded: no waiting room holds a batch and no stage + # execution context still owns or queues a ticket. + assert _drained(cluster_scheduler._forward_sync_waiting_room) + for key, context in cluster_scheduler._stage_execution_contexts.items(): + assert context.is_idle, (key, context) + assert context.queued_tickets == (), (key, context.queued_tickets) + return evidence + + +def _read_request_metrics(root: Path): + import csv + + paths = list(root.rglob("request_metrics.csv")) + assert len(paths) == 1, paths + with paths[0].open(newline="", encoding="utf-8") as stream: + return list(csv.DictReader(stream)) + + +def main(root: Path) -> None: + evidence = {} + for reporting in (False, True): + case_root = root / ("reporting_on" if reporting else "reporting_off") + case_root.mkdir(parents=True, exist_ok=True) + evidence["on" if reporting else "off"] = run_case( + case_root, reporting=reporting + ) + # Reporting is demand-driven: enabling it records more, but the simulated + # execution -- the same cohorts and the same makespan -- does not move. + assert evidence["on"]["makespan"] == evidence["off"]["makespan"], evidence + assert evidence["on"]["total_cohorts"] == evidence["off"]["total_cohorts"] + assert ( + evidence["on"]["mixed_phase_cohorts"] == evidence["off"]["mixed_phase_cohorts"] + ) + merged = dict(evidence["off"]) + merged["reporting_variants"] = evidence + (root / "shared_forward_evidence.json").write_text( + json.dumps(merged, indent=2) + "\n" + ) + print("mixed_phase_cohorts:", merged["mixed_phase_cohorts"]) + print("completed_requests:", merged["completed_requests"]) + + +if __name__ == "__main__": + main(Path(sys.argv[1])) diff --git a/tests/unit/test_monolithic_mixed_forward_sync.py b/tests/unit/test_monolithic_mixed_forward_sync.py new file mode 100644 index 00000000..1747cc0c --- /dev/null +++ b/tests/unit/test_monolithic_mixed_forward_sync.py @@ -0,0 +1,593 @@ +"""Behavior coverage for one shared monolithic forward across mixed lanes. + +A monolithic Replica runs prefill and decode on the same attention-DP lanes, so +one forward step can hold a prefill batch on one lane and a pure-decode batch on +another. These tests drive the real synchronization entry, the real EP wave, and +the real collective completion for every phase pairing, and check the properties +the shared lifecycle owes each source: one shared identity, one completion, one +ownership restoration, and a continuation predicted from the source's own batch. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest + +from frontier.entities import Batch, Request +from frontier.events.decode_sync_collective_event import DecodeSyncCollectiveEvent +from frontier.events.decode_sync_event import DecodeSyncEvent +from frontier.events.dense_layer_complete_event import DenseLayerCompleteEvent +from frontier.events.prefill_sync_collective_event import PrefillSyncCollectiveEvent +from frontier.events.prefill_sync_event import PrefillSyncEvent +from frontier.scheduler.cluster_scheduler.round_robin_cluster_scheduler import ( + RoundRobinClusterScheduler, +) +from frontier.scheduler.replica_stage_scheduler.replica_stage_schduler import ( + ReplicaStageScheduler, +) +from frontier.scheduler.replica_stage_scheduler.stage_execution_context import ( + FULL_STAGE_WORLD, + StageExecutionContext, +) +from frontier.scheduler.utils.forward_sync_state import ForwardSyncState +from frontier.scheduler.utils.sync_state import initialize_sync_waiting_rooms +from frontier.types import ClusterType + +#: Layers owned by pipeline stage 0 in this fixture. Layer 0 is the layer the +#: tests synchronize on, so the next layer (1) stays inside the stage and the +#: continuation path is the interesting one; layer 3 is the stage tail. +_GLOBAL_ID_BASE = 0 +STAGE_LAYERS = 4 +TOTAL_LAYERS = 4 +NUM_LANES = 2 + + +class _ModelConfig: + is_moe = True + num_layers = TOTAL_LAYERS + + def __init__(self, dense_layers: frozenset[int] = frozenset()) -> None: + self._dense_layers = dense_layers + + def is_moe_layer(self, layer_id: int) -> bool: + return layer_id not in self._dense_layers + + +class _ExecutionTime: + """A single-layer prediction whose value is a function of its own batch. + + Every duration is derived from one token count, so a lane that continued on + a peer's prediction rather than on its own produces a visibly wrong time + instead of an equal one. + """ + + def __init__(self, tokens: int) -> None: + self.tokens = tokens + self.pipeline_time = 0.0 + self.model_time = float(tokens) + self.total_time = float(tokens) + self.decode_draft_proposer_time = 0.0 + self.expert_parallel_communication_time = 0.0 + + def get_single_layer_attention_scope_time(self) -> float: + return float(self.tokens) + + def get_single_layer_post_attention_time(self) -> float: + # The EP decomposition is checked for conservation, so this must equal + # the sum of the five phase times below. + return float(self.tokens) + 2.0 + + def get_single_layer_moe_pre_dispatch_time(self) -> float: + return 0.0 + + def get_single_layer_moe_dispatch_time(self) -> float: + return 1.0 + + def get_single_layer_moe_post_dispatch_compute_time(self) -> float: + return float(self.tokens) + + def get_single_layer_moe_combine_time(self) -> float: + return 1.0 + + def get_single_layer_moe_post_combine_time(self) -> float: + return 0.0 + + +class _LanePredictor: + _num_layers_per_pipeline_stage = STAGE_LAYERS + + def __init__(self) -> None: + # (layer_id, the token count of the batch the call was made for). + self.calls: list[tuple[int, int]] = [] + self._monolithic_routing_details = { + 0: { + layer_id: {0: 0.25, 1: 0.25, 2: 0.25, 3: 0.25} + for layer_id in range(TOTAL_LAYERS) + } + } + + def predict_stage_execution_time( + self, + batch, + _stage_id, + cluster_type=None, + *, + num_layers, + layer_id, + include_ffn=True, + include_attention=True, + ): + del cluster_type, num_layers, include_ffn, include_attention + per_expert = getattr(batch, "per_expert_tokens", None) + tokens = ( + sum(per_expert.values()) + if per_expert is not None + else int(batch.total_num_tokens) + ) + self.calls.append((layer_id, tokens)) + return _ExecutionTime(tokens) + + +def _request(prefill_tokens: int, decode_tokens: int, *, decoding: bool) -> Request: + request = Request( + arrived_at=0.0, + num_prefill_tokens=prefill_tokens, + num_decode_tokens=decode_tokens, + ) + if decoding: + # Batch.num_prefill_tokens credits a request's tokens to prefill only + # while its prefill is unfinished, so this is what makes a lane decode. + request._is_prefill_complete = True + request._num_processed_tokens = prefill_tokens + return request + + +def _prefill_batch(tokens: int) -> Batch: + batch = Batch(0, [_request(tokens, 4, decoding=False)], [tokens], is_moe=True) + return batch + + +def _decode_batch(tokens: int) -> Batch: + requests = [_request(8, 4, decoding=True) for _ in range(tokens)] + return Batch(0, requests, [1] * tokens, is_moe=True) + + +def _mixed_batch(prefill_tokens: int, decode_requests: int) -> Batch: + """A prefill-mode batch that also carries already-decoding requests.""" + + requests = [_request(prefill_tokens, 4, decoding=False)] + requests.extend(_request(8, 4, decoding=True) for _ in range(decode_requests)) + return Batch( + 0, requests, [prefill_tokens] + [1] * decode_requests, is_moe=True + ) + + +class _MetricsStore: + def __init__(self, *, reporting: bool) -> None: + self.ep_wave_reporting_enabled = reporting + self.stage_execution_reporting_enabled = reporting + self.wave_calls = 0 + self.stage_schedules: list[tuple] = [] + + def on_ep_wave_schedule(self, *_args, **_kwargs): + self.wave_calls += 1 + + def on_replica_stage_schedule(self, *args, **_kwargs): + self.stage_schedules.append(args) + + +def _build_scheduler( + *, dense_layers: frozenset[int] = frozenset(), lane_capacity: int = NUM_LANES +): + predictor = _LanePredictor() + scheduler = object.__new__(RoundRobinClusterScheduler) + scheduler._cluster_type = ClusterType.MONOLITHIC + scheduler._config = SimpleNamespace( + replica_config=SimpleNamespace( + model_config=_ModelConfig(dense_layers), + total_expert_num=4, + moe_expert_parallel_size=2, + moe_tensor_parallel_size=1, + router_topk=1, + num_pipeline_stages=1, + ) + ) + scheduler._predictor = predictor + scheduler._forward_sync_state = ForwardSyncState() + scheduler._replica_dp_size = NUM_LANES + initialize_sync_waiting_rooms(scheduler) + + context = StageExecutionContext( + replica_id=0, stage_id=0, ep_size=2, full_stage_capacity=lane_capacity + ) + scheduler._stage_execution_contexts = {(0, 0): context} + stages = { + lane: ReplicaStageScheduler( + replica_id=0, + stage_id=0, + is_last_stage=True, + is_moe=True, + execution_time_predictor=predictor, + cluster_type=ClusterType.MONOLITHIC, + replica_local_id=lane, + stage_execution_context=context, + ) + for lane in range(NUM_LANES) + } + scheduler._replica_schedulers = { + (0, lane): SimpleNamespace( + get_replica_stage_scheduler=lambda _stage_id, stage=stage: stage + ) + for lane, stage in stages.items() + } + scheduler.get_replica_stage_scheduler = ( + lambda _replica_id, lane, _stage_id: stages[lane if lane is not None else 0] + ) + return scheduler, predictor, context, stages + + +def _admit(stages, batch: Batch, lane: int) -> None: + """Admit one lane's batch through the real stage queue. + + Going through `add_batch` / `pop_batch_if_not_busy` is what marks the lane's + stage busy and binds its forward group, so a sibling lane holding the other + phase is treated as occupied rather than as a lane that can be filled with + an idle batch. + """ + + batch.set_global_id(NUM_LANES * _GLOBAL_ID_BASE + lane) + batch._stage_owner_replica_local_id = lane + stages[lane].add_batch(batch) + assert stages[lane].pop_batch_if_not_busy() is batch + batch._prefill_model_execution_components_ms_by_stage = {0: [1.0]} + batch._prefill_stage_start_time = 0.0 + batch._decode_stage_start_time = 0.0 + + +def _global(scheduler): + """The one global-scheduler method a cluster-internal event calls.""" + + return SimpleNamespace(get_cluster_scheduler=lambda _: scheduler) + + +def _enter(scheduler, batch: Batch, lane: int, layer_id: int, metrics_store=None): + """Enter one lane through the phase its own batch is in.""" + + entry = ( + scheduler.on_prefill_sync + if batch.num_prefill_tokens > 0 + else scheduler.on_decode_sync + ) + return entry( + 0.0, 0, 0, batch, lane, "pre_moe", layer_id, 0.0, metrics_store=metrics_store + ) + + +def _run_forward( + scheduler, stages, lane_batches: dict[int, Batch], layer_id: int, metrics_store +): + """Admit every lane, then complete the resulting collective event.""" + + for lane, batch in lane_batches.items(): + _admit(stages, batch, lane) + events: list = [] + for lane, batch in lane_batches.items(): + events.extend(_enter(scheduler, batch, lane, layer_id, metrics_store)) + collective = [ + event + for event in events + if isinstance(event, (PrefillSyncCollectiveEvent, DecodeSyncCollectiveEvent)) + ] + assert len(collective) == 1, events + event = collective[0] + # Completion goes through the real event handler, so the test exercises the + # dispatch that routes a monolithic cohort into the shared forward path. + return event, event.handle_event(_global(scheduler), metrics_store) + + +PHASE_BUILDERS = { + "prefill": lambda tokens: _prefill_batch(tokens), + "decode": lambda tokens: _decode_batch(tokens), + "mixed": lambda tokens: _mixed_batch(tokens - 1, 1), +} + + +@pytest.mark.parametrize("lane_zero_phase", sorted(PHASE_BUILDERS)) +@pytest.mark.parametrize("lane_one_phase", sorted(PHASE_BUILDERS)) +def test_every_phase_pairing_reaches_one_shared_completion( + lane_zero_phase: str, lane_one_phase: str +) -> None: + scheduler, _predictor, _context, stages = _build_scheduler() + metrics_store = _MetricsStore(reporting=False) + batches = { + 0: PHASE_BUILDERS[lane_zero_phase](4), + 1: PHASE_BUILDERS[lane_one_phase](6), + } + event, follow_on = _run_forward(scheduler, stages, batches, 0, metrics_store) + + # One forward, one identity, one collective event for both lanes. + assert batches[0]._forward_cohort_id == batches[1]._forward_cohort_id + assert event._batch_global_id == batches[0]._forward_cohort_id + # Each live lane continues on its own next-layer sync event. + assert len(follow_on) == 2 + assert {event._batch.id for event in follow_on} == { + batches[0].id, + batches[1].id, + } + assert all(event._layer_id == 1 for event in follow_on) + # The room is consumed: nothing is left waiting for a second completion. + room = scheduler._forward_sync_waiting_room[0][0][event._batch_global_id][0] + assert "post_moe" not in room + + +@pytest.mark.parametrize( + "phases", [("prefill", "decode"), ("decode", "prefill"), ("mixed", "decode")] +) +def test_the_collective_event_class_follows_cohort_contents_not_arrival_order( + phases: tuple[str, str], +) -> None: + """Events order by `(time, event_type, id)`, so the class is a priority. + + It has to be a function of the cohort, not of which lane happened to close + the room, or a mixed forward's ordering would depend on arrival order. + """ + + lane_zero_phase, lane_one_phase = phases + observed = [] + for order in ((0, 1), (1, 0)): + scheduler, _predictor, _context, stages = _build_scheduler() + batches = { + 0: PHASE_BUILDERS[lane_zero_phase](4), + 1: PHASE_BUILDERS[lane_one_phase](6), + } + for lane, batch in batches.items(): + _admit(stages, batch, lane) + events: list = [] + for lane in order: + events.extend(_enter(scheduler, batches[lane], lane, 0, None)) + collective = [ + event + for event in events + if isinstance( + event, (PrefillSyncCollectiveEvent, DecodeSyncCollectiveEvent) + ) + ] + assert len(collective) == 1 + observed.append(type(collective[0])) + assert observed[0] is observed[1] + # A cohort holding any prefill token keeps the prefill collective type it + # already has today; only a cohort with none of them decodes. + assert observed[0] is PrefillSyncCollectiveEvent + + +@pytest.mark.parametrize("order", [(0, 1), (1, 0)]) +def test_reversed_arrival_order_gives_the_same_identity_and_outcome(order) -> None: + scheduler, _predictor, _context, stages = _build_scheduler() + batches = {0: _prefill_batch(4), 1: _decode_batch(6)} + for lane, batch in batches.items(): + _admit(stages, batch, lane) + events: list = [] + for lane in order: + events.extend(_enter(scheduler, batches[lane], lane, 0, None)) + collective = [ + event + for event in events + if isinstance(event, (PrefillSyncCollectiveEvent, DecodeSyncCollectiveEvent)) + ] + assert len(collective) == 1 + assert batches[0]._forward_cohort_id == batches[1]._forward_cohort_id + follow_on = collective[0].handle_event(_global(scheduler), None) + assert sorted(event._batch.id for event in follow_on) == sorted( + batch.id for batch in batches.values() + ) + + +def test_each_source_continues_on_its_own_predicted_duration() -> None: + """A deliberate unequal-token fixture detects borrowed timing. + + `_ExecutionTime` makes every duration a function of the batch it was + predicted for, so a lane that reused a peer's prediction lands at the + peer's time instead of its own. + """ + + scheduler, predictor, _context, stages = _build_scheduler() + batches = {0: _prefill_batch(4), 1: _decode_batch(6)} + event, follow_on = _run_forward(scheduler, stages, batches, 0, None) + + continuation = {event._batch.id: event for event in follow_on} + prefill_event = continuation[batches[0].id] + decode_event = continuation[batches[1].id] + assert isinstance(prefill_event, PrefillSyncEvent) + assert isinstance(decode_event, DecodeSyncEvent) + # 4 tokens -> 4 ms, 6 tokens -> 6 ms, both measured from the wave end. + assert prefill_event.time == pytest.approx(event.time + 4e-3) + assert decode_event.time == pytest.approx(event.time + 6e-3) + assert prefill_event.time != decode_event.time + # Each lane's own token count appears in its own next-layer prediction. + next_layer_calls = [call for call in predictor.calls if call[0] == 1] + assert sorted(next_layer_calls) == [(1, 4), (1, 6)] + + +def test_an_idle_participant_does_not_gain_requests_progress_or_a_continuation() -> None: + scheduler, _predictor, _context, stages = _build_scheduler() + batch = _prefill_batch(4) + _admit(stages, batch, 0) + # Lane 1 has no work at all, so the entry fills it with an idle batch. + events = _enter(scheduler, batch, 0, 0, None) + idle_entries = [event for event in events if event._batch.is_idle] + assert len(idle_entries) == 1 + idle_batch = idle_entries[0]._batch + assert idle_batch.requests == [] + assert idle_batch._forward_cohort_id == batch._forward_cohort_id + + follow_on = idle_entries[0].handle_event(_global(scheduler), None) + collective = [ + event + for event in follow_on + if isinstance(event, (PrefillSyncCollectiveEvent, DecodeSyncCollectiveEvent)) + ] + assert len(collective) == 1 + completion = collective[0].handle_event(_global(scheduler), None) + # Only the real source continues; the idle lane produces no event, no + # request and no completed work. + assert [event._batch.id for event in completion] == [batch.id] + assert idle_batch.requests == [] + assert not getattr(idle_batch, "_prefill_ep_wave_lane_times_ms", None) + + +@pytest.mark.parametrize( + "phase,marker", + [ + ("prefill", "_prefill_ep_wave_lane_times_ms"), + ("decode", "_decode_ep_wave_lane_times_ms"), + ], +) +def test_a_source_without_its_own_wave_timings_is_refused(phase, marker) -> None: + """The per-phase helpers only run this check when they pop the room. + + The shared path hands them `direct_batch`, which skips it, so the shared + completion has to make it once per source instead of losing it. + """ + + scheduler, _predictor, _context, stages = _build_scheduler() + batches = {0: _prefill_batch(4), 1: _decode_batch(6)} + for lane, batch in batches.items(): + _admit(stages, batch, lane) + events: list = [] + for lane, batch in batches.items(): + events.extend(_enter(scheduler, batch, lane, 0, None)) + collective = [ + event + for event in events + if isinstance(event, (PrefillSyncCollectiveEvent, DecodeSyncCollectiveEvent)) + ] + assert len(collective) == 1 + victim = batches[0] if phase == "prefill" else batches[1] + assert hasattr(victim, marker) + delattr(victim, marker) + with pytest.raises(RuntimeError, match="must carry its own EP wave lane timings"): + collective[0].handle_event(_global(scheduler), None) + + +def test_a_request_owned_by_two_live_lanes_fails_at_group_formation() -> None: + scheduler, _predictor, _context, stages = _build_scheduler() + shared = _request(4, 4, decoding=False) + batches = { + 0: Batch(0, [shared], [2], is_moe=True), + 1: Batch(0, [shared], [2], is_moe=True), + } + for lane, batch in batches.items(): + _admit(stages, batch, lane) + events = _enter(scheduler, batches[0], 0, 0, None) + assert events == [] + with pytest.raises(ValueError, match="cannot belong to two EP source lanes"): + _enter(scheduler, batches[1], 1, 0, None) + + +def test_successive_forwards_release_owners_rooms_and_open_step_bindings() -> None: + scheduler, _predictor, _context, stages = _build_scheduler() + batches = {0: _prefill_batch(4), 1: _decode_batch(6)} + for lane, batch in batches.items(): + _admit(stages, batch, lane) + + cohort_ids = [] + for layer_id in range(3): + events: list = [] + for lane, batch in batches.items(): + events.extend(_enter(scheduler, batch, lane, layer_id, None)) + collective = [ + event + for event in events + if isinstance( + event, (PrefillSyncCollectiveEvent, DecodeSyncCollectiveEvent) + ) + ] + assert len(collective) == 1 + cohort_ids.append(collective[0]._batch_global_id) + follow_on = collective[0].handle_event(_global(scheduler), None) + assert len(follow_on) == 2 + # Every layer consumes its own room and its own open-step binding. + room = scheduler._forward_sync_waiting_room[0][0][cohort_ids[-1]][layer_id] + assert "post_moe" not in room + assert scheduler._forward_sync_state.open_steps("forward") == {} + assert scheduler._forward_sync_state.open_steps("prefill") == {} + assert scheduler._forward_sync_state.open_steps("decode") == {} + # Ownership is restored to exactly one full-stage ticket per live lane. + assert all( + batch._stage_admission_ticket.scope is FULL_STAGE_WORLD + for batch in batches.values() + ) + assert batches[0]._forward_cohort_id == batches[1]._forward_cohort_id + # Each layer opens and closes its own binding, so the step id advances once + # per layer while the two lanes keep agreeing on it. + assert cohort_ids == sorted(set(cohort_ids)) and len(cohort_ids) == 3 + + +def test_a_decoding_request_inside_a_prefill_batch_advances_exactly_one_layer() -> None: + scheduler, _predictor, _context, stages = _build_scheduler() + mixed = _mixed_batch(4, 1) + decoding = mixed.requests[1] + prefilling = mixed.requests[0] + batches = {0: mixed, 1: _decode_batch(2)} + peer_requests = list(batches[1].requests) + _run_forward(scheduler, stages, batches, 0, None) + + assert decoding.completed_layer_count == 1 + # A request still prefilling has no decode layer to credit. + assert prefilling.completed_layer_count == 0 + assert all(request.completed_layer_count == 1 for request in peer_requests) + + +def test_a_dense_layer_labels_each_source_by_its_own_phase() -> None: + scheduler, _predictor, _context, stages = _build_scheduler( + dense_layers=frozenset({1}) + ) + batches = {0: _prefill_batch(4), 1: _decode_batch(6)} + for lane, batch in batches.items(): + _admit(stages, batch, lane) + events: list = [] + for lane, batch in batches.items(): + events.extend(_enter(scheduler, batch, lane, 1, None)) + + # A dense layer inside a MoE model produces no EP collective at all. + assert all(isinstance(event, DenseLayerCompleteEvent) for event in events) + assert len(events) == 2 + modes = {event._batch.id: event._phase for event in events} + assert modes == {batches[0].id: "prefill", batches[1].id: "decode"} + # Only the prefill source keeps a prefill component ledger entry. + assert batches[0]._prefill_model_execution_components_ms_by_stage[0] == [1.0, 6.0] + assert batches[1]._prefill_model_execution_components_ms_by_stage[0] == [1.0] + # Each dense source is timed from its own tokens: 4 + 2 and 6 + 2 ms. + assert {event._batch.id: event.time for event in events} == { + batches[0].id: pytest.approx(6e-3), + batches[1].id: pytest.approx(8e-3), + } + + +def test_a_disabled_metrics_store_costs_the_run_nothing() -> None: + """Reporting is demand-driven at this boundary. + + A store with reporting off must produce the same simulated outcome and the + same predictor calls as no store at all, and must record nothing. The + reporting-enabled comparison needs a real predictor result and is covered + by `tests/integration/test_monolithic_mixed_forward_runtime.py`. + """ + + outcomes = [] + store = _MetricsStore(reporting=False) + for metrics_store in (None, store): + scheduler, predictor, _context, stages = _build_scheduler() + batches = {0: _prefill_batch(4), 1: _decode_batch(6)} + event, follow_on = _run_forward(scheduler, stages, batches, 0, metrics_store) + outcomes.append( + ( + event.time, + sorted(item.time for item in follow_on), + sorted(predictor.calls), + ) + ) + + assert outcomes[0] == outcomes[1] + assert store.wave_calls == 0 + assert store.stage_schedules == [] diff --git a/tests/unit/test_pdaf_prefill_model_time.py b/tests/unit/test_pdaf_prefill_model_time.py index b69f7f97..470de3d3 100644 --- a/tests/unit/test_pdaf_prefill_model_time.py +++ b/tests/unit/test_pdaf_prefill_model_time.py @@ -565,7 +565,11 @@ def test_prefill_pp2_stage_one_advances_with_global_layer_ids() -> None: assert len(events) == 1 assert isinstance(events[0], PrefillSyncEvent) assert events[0]._layer_id == 3 - assert predictor.calls == [(1, 2), (1, 3)] + # Stage 1 must address layers by their global ids: stage-local numbering + # would have asked for layer 1 here, not layer 3. Only the next layer is + # predicted on this path; the completed layer's prediction is made in the + # final-layer branch, which is the only place its value is used. + assert predictor.calls == [(1, 3)] def test_prefill_stage_schedule_resets_component_ledger_for_pipeline_stage() -> None: From bdff4aa11839800f1366c4c88d5acf0b68a0a183 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 09:55:39 +0800 Subject: [PATCH 13/80] docs(review): record the W3 fidelity measurement and rebuilt controls The 71-case matrix ran from two clean detached worktrees, both driven by one harness revision: baseline 3d47417 against candidate 65ed8a7, source_dirty False on both sides, no case filter, clean cache, 71 executed and 426 cache files each. 71 of 71 compared and 71 identical, with no provenance findings and no predictor cache differences. That is exactly what design.md predicted before the run, so its conditional I7 branch was not taken. Stated plainly in the records: no matrix case can reach a multi-lane monolithic MoE forward at all, so a null result is the pass condition for "nothing else moved" and not evidence that the defect is fixed. The evidence for that is the direct-construction runtime test. The four deliberate-defect control trees were rebuilt against the final test file, because they predated the EP wave marker guard and their recorded counts no longer matched what is delivered. Each tree now carries the final source and the final tests and differs by exactly one edit, except the baseline tree which carries the pre-fix parent in full. Counts are 22, 12, 2 and 16 of 23 unit tests, and each runtime failure remains distinct. One earlier claim is corrected where it was too strong: on the pre-fix tree the same-phase pairs do not pass outright. They pass every assertion about the forward itself and fail only at the final shared-room inspection, which the pre-fix cluster has no structure for. The mixed pairs fail earlier, at an empty collective list. The gap between the two groups is the deadlock. --- .../progress.md | 21 +- .../review.md | 9 +- ...2026-09-22_w3_shared_monolithic_forward.md | 187 ++++++++++++++++++ .../validation.md | 125 ++++++++++++ 4 files changed, 334 insertions(+), 8 deletions(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w3_shared_monolithic_forward.md diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 811dc588..931563a1 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -6,6 +6,8 @@ | --- | --- | | 2026-09-21 | Step 0 started: records landed, environment created, baseline pending. | | 2026-09-22 | Maintainer review dispositions recorded; Checkpoint C closed: parent merged, W2 tests strengthened, W2 re-measured with one harness revision. | +| 2026-09-22 | Checkpoint D first half: W3, the shared monolithic forward lifecycle, implemented, tested against four deliberate-defect controls, and committed as `65ed8a7`. | +| 2026-09-22 | W3 fidelity matrix measured: 71 of 71 identical against the expectation recorded before the run. Step 3 closed. | ## Status @@ -14,9 +16,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | -| Current step | Step 2 complete and re-measured (Checkpoint C); Step 3 scoped, not started | +| Current step | Step 3 complete: implemented, measured, records written | | Publication | PUSHED_VERIFIED (records) | -| Next action | Checkpoint D / Step 3: the shared monolithic forward lifecycle, with the direct-construction integration fixture R35-02 requires. Scoped below; not started. | +| Next action | Push the Step 3 records and add the W3 section to PR #35; then Checkpoint D second half (W4, the opt-in vLLM DP placement). | ## Step status @@ -25,7 +27,7 @@ | 0 | Worktree, references, baseline | PASS | PASS (baseline recorded) | PUSHED_VERIFIED | NOT_REVIEWED | | 1 | Candidate/vLLM audit | PASS | n/a (source audit) | LOCAL_ONLY | NOT_REVIEWED | | 2 | RR DP rotation | PASS | unit PASS (23 tests); matrix PASS against a stated expectation, re-measured 2026-09-22 with one harness revision | PUSHED_VERIFIED | REVIEWED (R35-01 closed) | -| 3 | Shared monolithic forward | NOT_STARTED | — | — | — | +| 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | LOCAL_ONLY | NOT_REVIEWED | | 4 | Opt-in vLLM DP placement | NOT_STARTED | — | — | — | | 5 | Routing implementation identity | NOT_STARTED | — | — | — | | 6 | Legacy fused-MoE profiling | NOT_STARTED | — | — | — | @@ -49,8 +51,17 @@ - 2026-09-21: Unit comparison against the refactor tip `db15e64` over 73 files: identical failure identities, 1808 to 1812 passing, the four new tests being the difference. - 2026-09-21: Step 2 measured and PASS against the stated expectation. 71 of 71 cases compared, 68 identical, and the mismatch set is exactly the three cases predicted to move. Nothing moved that was not expected to, and nothing expected to move stayed. Lane occupancy confirms the direction: collapsed onto lane zero before, evenly spread after, with the control case unchanged. Full record in `validation.md`. - 2026-09-21: Merged the refactor tip so this branch carries the four DP placement cases in its own fidelity case table. Without it the branch's own harness still had the 67-case table, and measuring the branch with its own tooling would have exercised a table that cannot see the fix. +- 2026-09-22: W3 implemented and committed as one unit (`65ed8a7`). A monolithic Replica now keeps one waiting room and one open-step namespace for both local phases, completes a cohort once, restores full-stage owners once, and then continues each source on its own batch through the phase helper it already had. The post_moe collective event class is chosen from cohort contents rather than from whichever lane closed the room, so a pure-prefill and a pure-decode cohort keep exactly the `EventType` priority they have today and only the mixed cohort -- which previously could not complete at all -- is new. The two near-duplicate sync entries collapsed into one `enter_layer_sync(..., mode)`, mirroring the existing `schedule_layer_wave(mode=...)`. I3 is enforced at group formation; I8 is deliberately excluded per Checkpoint D's "current-main metrics ownership" (recorded in `design.md`). +- 2026-09-22: One guard was added that the delegation would otherwise have dropped. Each per-phase helper refuses a legacy aggregate synchronization by checking its batch for the wave's lane timings, but only when it pops the room itself; the shared path hands it `direct_batch`, which skips that branch. The check now lives in `forward_collective`, once per source, against the marker its own phase writes. +- 2026-09-22: W3 acceptance. 23 behavior tests in `tests/unit/test_monolithic_mixed_forward_sync.py` cover every phase pairing including true mixed batches, both arrival orders, unequal source tokens, idle participation, duplicate ownership, successive layers, a decoding request inside a prefill batch, dense-layer transitions inside a MoE model, a missing wave marker, and a disabled metrics store. `tests/integration/test_monolithic_mixed_forward_runtime.py` builds `attn_tp=1, attn_dp=2, moe_tp=1, moe_ep=2` directly and runs the real `Simulator`: 24 cohorts, 4 of them mixed-phase, 4/4 requests completed, identical with reporting on and off. +- 2026-09-22: W3 controls, four trees and four distinct failures. The pre-fix source deadlocks in the real event loop ("Sequential simulation ended with non-empty scheduler state"); borrowed source timing trips "one attention-DP lane cannot occupy two open sync cohorts"; a repeated layer advance trips "Decode post_moe layer counter cannot advance"; a repeated ownership restoration trips "operation_id is already queued or active". On the pre-fix tree the same-phase pairs still complete, so the suite is not failing wholesale for an unrelated reason -- the mixed pairs fail at the empty collective list, which is the deadlock itself. +- 2026-09-22: W3 regression comparison against the branch parent `3d47417` in a dedicated detached worktree. `tests/unit`: 84 failures on both sides with identical identities, 3717 vs 3694 passing. `tests/integration`: the same five pre-existing errors (the PD-AF Reference checkout is absent on this host), 12 vs 11 passing. No regressions and no accidental fixes. +- 2026-09-22: W3 controls rebuilt against the final test file and re-run, so the recorded counts match what is delivered. Each tree now carries the final `frontier/` and the final `tests/` and differs from the delivered source by exactly one edit, except the baseline tree whose `frontier/` is the pre-fix parent in full. Counts: 22, 12, 2 and 16 of 23 unit tests fail respectively, and each runtime failure is distinct. The pre-fix tree's same-phase pairs pass every assertion about the forward itself and fail only at the final shared-room inspection; the mixed pairs fail earlier at an empty collective list, which is the deadlock isolated. +- 2026-09-22: W3 fidelity matrix PASS. Baseline `3d47417` against candidate `65ed8a7`, both clean detached checkouts with `source_dirty=False` and no dirty paths, one harness at `65ed8a7`, no filter, clean cache, 71 executed and 426 cache files each. 71 of 71 compared and **71 identical**, zero mismatches, zero provenance findings, zero predictor cache differences. That is exactly the expectation `design.md` recorded before the run; the conditional I7 branch was not taken. Note explicitly: the matrix cannot reach a multi-lane monolithic MoE forward at all, so a null result is the pass condition for "nothing else moved", not evidence that the defect is fixed. Full record in `validation.md` and `test_report_2026-09-22_w3_shared_monolithic_forward.md`. -## Step 3 scoping, not started +## Step 3 scoping, as recorded before implementation + +Kept as written so the implementation can be read against the scope it started from. Step 3 is now complete; see the W3 entries above and `design.md`. The defect is present on main at three layers, and the fix has to change all three together or the intermediate state deadlocks differently: @@ -62,4 +73,4 @@ One refinement over the audit's framing, from reading the code: `ForwardSyncStat Surface: about 3,500 lines across `replica_stage_schedule_event.py`, `sync_entry.py`, `prefill_collective.py`, `decode_collective.py`, `ep_wave_schedule.py`, `ep_wave_inputs.py` and `base_cluster_scheduler.py`, with 11 call sites of the sync-kind and sync-path selection. -Blocked hunk carried from the audit: the candidate's decode final-metrics change calls `_create_corrected_execution_time_for_metrics`, which main deleted, so it needs rewriting against main's current execution-time ownership rather than porting. +Blocked hunk carried from the audit: the candidate's decode final-metrics change calls `_create_corrected_execution_time_for_metrics`, which main deleted, so it needs rewriting against main's current execution-time ownership rather than porting. Resolved by exclusion: that hunk is I8, kept out of scope under Checkpoint D's "current-main metrics ownership" and recorded in the `design.md` scope table. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index 679ecd9f..2534d87f 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -7,6 +7,7 @@ | 2026-09-21 | Created with the pinned source snapshot. | | 2026-09-21 | Step 1 complete: candidate and vLLM audits landed, dispositions recorded, two decision checkpoints raised. | | 2026-09-21 | Corrected the W3 and W4 rows: the step-id namespace is not partitioned by sync kind, only the open-step binding table is. Verified against `forward_sync_state.py` at `c18eb2c`. | +| 2026-09-22 | W3 delivered and measured; R35-02 closed; the unreleased multi-lane monolithic MoE shape recorded as an open item. | | 2026-09-22 | Recorded the maintainer's PR #34 / PR #35 review: D1 and D2 resolved, ten review comments dispositioned, each verified against source. | | 2026-09-22 | Self-review of that record: corrected the lockstep mechanism and counter semantics under D1, the line references under R34-01, the R34-03 remedy (the baseline label is itself an assembled partial run), and added the omissions listed under "Found on re-review". | @@ -60,6 +61,7 @@ Claims in those reports that this PR depends on were re-verified directly agains | Blocked hunk | The candidate's decode final-metrics hunk calls `scheduler._create_corrected_execution_time_for_metrics(...)`, which **main deleted**; `_create_prefill_corrected_execution_time_for_metrics` also changed signature and now lives at `base_cluster_scheduler.py:1138`. Porting verbatim raises `AttributeError`. Main additionally added `metrics_store` and `ep_wave_reporting_enabled` plumbing to `ep_wave_schedule.py` and `prefill_collective.py` that the candidate lacks. | | Disposition | **ADAPT** for the lifecycle change as one coherent unit; **BLOCKED** for the metrics hunk until it is rewritten against main's current execution-time ownership. | | Planned test | The behavior matrix in `plan.md` §9, including at least one test that drives the real event loop with overlapping prefill and decode and injects deterministic times only at the predictor boundary. | +| Delivered | `65ed8a7`. Lifecycle adapted as one unit; the metrics hunk (I8) stayed **BLOCKED** and is now recorded as a deliberate exclusion in `design.md` rather than an open item. 23 behavior tests plus the real-runtime fixture; four deliberate-defect controls each fail for their own reason. Fidelity matrix 71 of 71 identical, matching the expectation recorded before the run. See `test_report_2026-09-22_w3_shared_monolithic_forward.md`. | ### W4 — Opt-in vLLM-style DP placement @@ -131,10 +133,11 @@ The candidate deletes `VLLMv1EngineReplicaScheduler._get_num_waiting_reqs_for_de ## Open items 1. Whether `_schedule_batch_mode`'s per-replica grouped return order is load-bearing for the consumers of `ClusterScheduleEvent`'s request mapping (W2). -2. Whether W3's shared forward identity is the correct report-order key for W4, or whether a separate Replica-scoped step identity is required (D1). +2. Whether W3's shared forward identity is the correct report-order key for W4, or whether a separate Replica-scoped step identity is required (D1). W3 is now delivered (`65ed8a7`): a monolithic cohort resolves to one step id in a single `"forward"` namespace regardless of its lanes' phases. W4 validates that identity at the report boundary. 3. Which of the three existing spellings becomes the single public name for the routing implementation identity (W5). 4. Pipeline-parallel behavior of the component ledgers is untested in both trees. -5. The upstream `fused_moe.py` fork change passes a fifth `renormalize` argument to `torch.ops._moe_C.topk_softmax` while the in-tree schema still declares four; the prebuilt extension could not be inspected on this host. Numerically a no-op, but it would raise rather than degrade. Relevant only if W6 native validation runs against the fork's compiled package. +5. A multi-lane monolithic MoE Replica has no released wrapper: the public MoE examples enforce `ATTN_TP == MOE_TP * MOE_EP` while the runtime enforces `attn_tp * attn_dp == moe_tp * moe_ep`, and those have no common solution above one lane. Its evidence therefore has to come from a direct-construction fixture, never from the fidelity matrix. Whether the release should offer such a wrapper is a product question, not a correctness one, and is left open. +6. The upstream `fused_moe.py` fork change passes a fifth `renormalize` argument to `torch.ops._moe_C.topk_softmax` while the in-tree schema still declares four; the prebuilt extension could not be inspected on this host. Numerically a no-op, but it would raise rather than degrade. Relevant only if W6 native validation runs against the fork's compiled package. ## Final code-review findings @@ -289,7 +292,7 @@ marked closed when the evidence for it is committed, not when the change is. | R34-04 retained checks | **CLOSED** | `tests/unit/test_module_split_boundaries.py` (13 tests). Beyond the ask: all 142 baseline-produced pickled estimators load under the split code and 86 predict, which cache-name equality could not show. One pre-existing `NameError` on `ClusterConfig` annotations is pinned, not fixed, and verified to fail identically on `1f694f7`. | | R34-05 bounded split | **CLOSED** | Guidance applied while writing the new tests; an unused import removed. | | R35-01 W2 tests | **CLOSED** | `tests/unit/test_cluster_scheduler_dp_lanes.py`: three topologies with the full rotation written out by hand past its wraparound, driven through the public `schedule()`, each run for both MONOLITHIC and PREFILL. The source-text guard is kept with a docstring stating it is governance only. Negative control against the pre-fix method: 12 of 22 fail, controls pass. | -| R35-02 wrapper limit | **PARTIAL** | The `cases.py` remedy wording is corrected: the wrapper limit is not a runtime limit, and the fixture it calls for builds a runtime configuration directly with durations injected at the predictor boundary. The fixture itself is W3 acceptance work and is not written yet. | +| R35-02 wrapper limit | **CLOSED** | `tests/integration/test_monolithic_mixed_forward_runtime.py` builds `attn_tp=1, attn_dp=2, moe_tp=1, moe_ep=2` directly, runs the real `Simulator` event loop with real admission, ownership and completion, and injects deterministic durations only by wrapping the predictor. It reaches four mixed-phase cohorts, which no wrapper and no matrix case can. On the pre-fix source the same fixture ends with a non-empty scheduler state, so the fixture is shown to detect the defect it exists for. | | R35-03 SGLang consumers | OPEN | W4. | | R35-04 W5 scope | OPEN | W5. | | R35-05 gates | **CLOSED** | This document. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w3_shared_monolithic_forward.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w3_shared_monolithic_forward.md new file mode 100644 index 00000000..2bc6f581 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w3_shared_monolithic_forward.md @@ -0,0 +1,187 @@ +# Test report — W3, one shared monolithic forward (2026-09-22) + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-22 | Created. Acceptance evidence for the shared monolithic forward lifecycle, including four deliberate-defect controls. | +| 2026-09-22 | Fidelity matrix recorded (71 of 71 identical). Controls rebuilt against the final test file and re-run; counts and failure messages updated. | + +## Environment + +| Item | Value | +| --- | --- | +| Worktree | `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr` | +| Branch / commit under test | `fix/issue26-correctness-pr` @ `65ed8a7` | +| Baseline | `/data/ycfeng/Frontier/.worktrees/w3-baseline`, detached at `3d47417` (this branch's parent) | +| Python | `/data/ycfeng/envs/frontier-py310/bin/python` (3.10) | +| Invocation | `PYTHONPATH=$PWD python -m pytest … -p no:cacheprovider` from the worktree root | +| Absent optional deps | `torch`, `matplotlib` — 10 GPU/plot-only unit modules are excluded on both sides by the same `--ignore` list | + +Control trees live under the session scratchpad +(`.../scratchpad/w3-control-*`) and are driven by `run_against_tree.py`, which +removes the editable-install meta-path finder and prepends the chosen tree, so +each control imports exactly the source it names. + +## What was measured + +| # | Check | Command | Expected | Actual | Result | +| --- | --- | --- | --- | --- | --- | +| 1 | W3 behavior matrix | `pytest tests/unit/test_monolithic_mixed_forward_sync.py` | all pass | 23 passed | PASS | +| 2 | Real-runtime acceptance | `pytest tests/integration/test_monolithic_mixed_forward_runtime.py` | 1 pass, a mixed-phase cohort actually reached | 1 passed; 24 cohorts, 4 mixed-phase, 4/4 requests completed | PASS | +| 3 | Forward-sync regression set (17 files) | see list below | all pass | 342 passed | PASS | +| 4 | Full unit suite vs baseline | `pytest tests/unit` both sides | identical failure set | 84 failed on both, identical identities; 3717 vs 3694 passed (+23 new) | PASS | +| 5 | Integration suite vs baseline | `pytest tests/integration` both sides | identical error set | 5 errors on both (PD-AF Reference checkout absent); 12 vs 11 passed (+1 new) | PASS | +| 6 | Fidelity matrix | `run_matrix.py run/compare` | all 71 cases exactly equal (expectation recorded in `design.md` before measuring) | 71 of 71 compared, 71 identical, 0 mismatched | PASS | + +### Check 3 — the forward-sync regression set + +``` +tests/unit/test_decode_ep_wave_materialization.py +tests/unit/test_forward_sync_state.py +tests/unit/test_mixed_layer_decode_ffn_scheduling.py +tests/unit/test_moe_routing_conservation.py +tests/unit/test_pdaf_parity_harness_wave2_events.py +tests/unit/test_pdaf_prefill_model_time.py +tests/unit/test_pd_decode_moe_layer_accounting.py +tests/unit/test_prefill_ep_wave_materialization.py +tests/unit/test_review_comment_fixes.py +tests/unit/test_shared_ep_layer_protocol_guard.py +tests/unit/test_shared_forward_group_admission.py +tests/unit/test_stage_reporting_contract.py +tests/unit/test_cluster_scheduler_dp_lanes.py +tests/unit/test_collective_timing.py +tests/unit/test_dense_layer_complete_event.py +tests/unit/test_ep_wave_trace_context.py +tests/integration/test_online_pdd_forward_groups.py +``` + +These cover forward identity, EP-wave materialization for both phases, stage +ownership and admission, stage execution reporting, and the sequential PDD / +PD-AF layer accounting that shares the same call chain. + +## Controls — each defect fails for its own reason + +A test that cannot fail proves nothing, so the suite was run against four +trees that each carry one specific defect. + +Each tree carries the final `frontier/` and the final `tests/`, differing from +the delivered source by exactly one edit — except the baseline tree, whose +`frontier/` is the pre-fix parent in full. + +| Tree | Defect | Unit result | Integration failure | +| --- | --- | --- | --- | +| `w3-control-baseline` | none — the pre-fix source at `3d47417` | 22 of 23 fail | `RuntimeError: Sequential simulation ended with non-empty scheduler state` | +| `w3-control-borrowed-timing` | every source continues on `live_batches[0]` | 12 of 23 fail | `ValueError: one attention-DP lane cannot occupy two open sync cohorts: replica=0, stage=0, lane=1, layer=1, sync_stage=pre_moe` | +| `w3-control-double-layer-advance` | the decode helper advances layers again | 2 of 23 fail | `ValueError: Decode post_moe layer counter cannot advance: request_id=1, completed_layer_count=4, total_layers=4` | +| `w3-control-double-owner-restore` | the per-source helper restores owners again | 16 of 23 fail | `ValueError: operation_id is already queued or active in this stage context: ('shared_layer', 0, 1, 0, 1, 'attention', 'FULL_STAGE_WORLD')` | + +Two observations worth keeping: + +- On the pre-fix tree the **same-phase** pairs still complete, so the suite is + not failing wholesale for an unrelated reason. `decode-decode` and + `prefill-prefill` pass every assertion about the forward itself — one shared + identity, one collective event, two per-source continuations — and fail only + at the last line, which inspects a shared room the pre-fix cluster does not + have (`AttributeError: 'RoundRobinClusterScheduler' object has no attribute + '_forward_sync_waiting_room'`). Every **mixed** pair instead fails earlier, at + `assert len(collective) == 1` with an empty list. That gap between the two + groups is the deadlock. +- The deadlock reproduces in the **real event loop**, not only in the fixture. + With `attn_tp=1, attn_dp=2, moe_tp=1, moe_ep=2` on a monolithic MoE Replica, + the pre-fix sequential run drains its event queue with a non-empty scheduler + state. + +## The integration fixture, and why it is not a wrapper case + +The public MoE wrappers enforce `ATTN_TP == MOE_TP * MOE_EP` while the runtime +enforces `attn_tp * attn_dp == moe_tp * moe_ep`; those have no common solution +above one attention-DP lane, so no wrapper and no fidelity-matrix case can +reach a multi-lane monolithic MoE forward. `tests/integration/ +test_monolithic_mixed_forward_runtime.py` therefore assembles the +configuration directly and runs the real `Simulator`: + +- real admission, ownership restoration, synchronization and completion code; +- deterministic durations from constant profiling targets, plus an observer + that *wraps* `predict_stage_execution_time` rather than replacing it, so + source attribution is visible without changing any prediction; +- four requests of unequal prefill and decode length arriving together under + chunked prefill, which is what puts one lane in prefill while the other + decodes. + +Asserted: every request completes, exactly once, with every token accounted +for (`request_metrics.csv` has one row per request and the token sum matches); +each live member of a mixed cohort appears in the prediction log slice that +belongs to that cohort's completion; no waiting-room leaf still holds a batch; +every `StageExecutionContext` is idle with no queued ticket; and the run is +identical with reporting on and off (same makespan, same cohort counts). + +## Fidelity matrix + +Both sides were run from **clean detached worktrees** so no untracked draft in +the development worktree could dirty the recorded provenance, and both were +driven by **one harness revision**, the candidate's. + +```bash +PYTHONPATH=/data/ycfeng/Frontier/.worktrees/w3-candidate \ + python .worktrees/w3-candidate/tests/e2e/refactor_fidelity/run_matrix.py run \ + --repo-root .worktrees/w3- --label \ + --output-root /data/ycfeng/tmp/issue26-correctness-pr/w3-fidelity \ + --python-bin /data/ycfeng/envs/frontier-py310/bin/python --jobs 8 --clean-cache + +python tests/e2e/refactor_fidelity/run_matrix.py compare \ + --output-root /data/ycfeng/tmp/issue26-correctness-pr/w3-fidelity \ + --baseline-label baseline --candidate-label candidate +``` + +| Side | `source_revision` | `source_dirty` | `git_dirty_paths` | `harness_revision` | Executed | Cache files | +| --- | --- | --- | --- | --- | --- | --- | +| baseline | `3d474170a5df` (this branch's parent) | `False` | empty | `65ed8a76055e` | 71 of 71 | 426 | +| candidate | `65ed8a76055e` | `False` | empty | `65ed8a76055e` | 71 of 71 | 426 | + +No case filter, cache cleaned before each run, 0 cases dropped as stale. + +| Metric | Value | +| --- | --- | +| cases compared | 71 of 71 in the case table | +| identical | **71** | +| mismatched | 0 | +| baseline failures / candidate-only failures | 0 / 0 | +| cases missing from one side / with missing evidence / with differing definitions | 0 / 0 / 0 | +| provenance findings | 0 | +| predictor cache: baseline-only / candidate-only / findings | 0 / 0 / 0 | +| `complete_comparison`, `predictor_cache_populated_cleanly` | `True`, `True` | + +Report: `/data/ycfeng/tmp/issue26-correctness-pr/w3-fidelity/comparison.json`. + +### Judged against the expectation recorded before measuring + +`design.md` predicted, before the run: all 71 cases stay exactly equal, because +at one attention-DP lane every cohort holds one source batch, so the shared +room, the shared identity, the single restoration and the per-source +continuation all reduce to today's single-source behavior. It further predicted +that the one behavior change reachable at one lane — the I7 decode-layer credit +for an already-decoding request carried inside a prefill-mode batch — changes a +counter no monolithic timing reads, and that **if** a MoE co-location case +moved, that credit would be the cause. + +The measurement matches the primary prediction exactly: 71 identical, 0 moved. +The conditional branch was therefore not taken, which also confirms the source +reading behind it — on the monolithic path `completed_layer_count` feeds only +admission guards and diagnostics, and its two arithmetic consumers +(`cluster_batch_end_event.py:174` and `:354`) are PD-AF `DECODE_ATTN`/M2N. + +A null result is the correct outcome here and is **not** evidence that the fix +works: no matrix case can reach a multi-lane monolithic MoE forward at all (see +below). The matrix answers "did anything else move", and the answer is no. The +evidence that the defect is fixed is checks 1 and 2 with their controls. + +## Verification limits + +- The multi-lane monolithic MoE shape has no released wrapper, so its evidence + is the direct-construction integration test rather than a matrix case. +- The integration profiles are constant synthetic targets. They make timing + deterministic and source attribution checkable; they are not trained + numerical parity. +- I8, the per-source decode component ledger, is deliberately out of scope. + See the scope table in `design.md`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index dc425ba0..6a8c6c65 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -8,6 +8,7 @@ | 2026-09-21 | Step 1 recorded: audit spot checks and the vLLM reference identity check. | | 2026-09-21 | Step 2 recorded: unit sensitivity and the fidelity measurement against a stated expectation. | | 2026-09-22 | Step 2 re-measured with harness and source at one revision, after the gate corrections. Same expectation, same result, recorded provenance. | +| 2026-09-22 | Step 3 recorded: the shared monolithic forward, its direct-construction runtime evidence, four deliberate-defect controls, and a 71-of-71 identical fidelity matrix. | ## Environment @@ -208,3 +209,127 @@ the inherited-failure inventory recorded for the refactor branch. described under R35-02, which is W3 acceptance work. - Lane occupancy is read from the stage ledger, which records scheduled stage executions. It shows where work was placed, not that placement is optimal. + +## Step 3 — one shared monolithic forward (2026-09-22) + +### What the fidelity matrix can and cannot show here + +The defect needs at least two attention-DP lanes on a monolithic MoE Replica. +The public MoE wrappers enforce `ATTN_TP == MOE_TP * MOE_EP` while the runtime +enforces `attn_tp * attn_dp == moe_tp * moe_ep`; those have no common solution +above one lane. **No case in the 71-case table can reach the defect**, so the +matrix here answers only one question — did anything *else* move — and a null +result is the pass condition, not the proof of the fix. + +The proof of the fix is the direct-construction runtime test plus its controls. + +### Setup + +Both sides ran from clean detached worktrees, driven by one harness revision, +so nothing in the development worktree could dirty the recorded provenance. + +```bash +# harness = the candidate checkout, for both sides +PYTHONPATH=/data/ycfeng/Frontier/.worktrees/w3-candidate \ + python .worktrees/w3-candidate/tests/e2e/refactor_fidelity/run_matrix.py run \ + --repo-root .worktrees/w3- --label \ + --output-root /data/ycfeng/tmp/issue26-correctness-pr/w3-fidelity \ + --python-bin /data/ycfeng/envs/frontier-py310/bin/python --jobs 8 --clean-cache + +python tests/e2e/refactor_fidelity/run_matrix.py compare \ + --output-root /data/ycfeng/tmp/issue26-correctness-pr/w3-fidelity \ + --baseline-label baseline --candidate-label candidate +``` + +| Side | `source_revision` | `source_dirty` | `git_dirty_paths` | `harness_revision` | Executed | Cache files | +| --- | --- | --- | --- | --- | --- | --- | +| baseline | `3d474170a5df` (this branch's parent) | `False` | empty | `65ed8a76055e` | 71 of 71 | 426 | +| candidate | `65ed8a76055e` | `False` | empty | `65ed8a76055e` | 71 of 71 | 426 | + +### The expectation, recorded before measuring + +From `design.md`, written before the run: + +- All 71 cases stay exactly equal. At one attention-DP lane every cohort holds + one source batch, so the shared room, the shared identity, the single + ownership restoration and the per-source continuation all reduce to today's + single-source behavior; `sample_batch` *is* the only batch. +- The one behavior change reachable at one lane is the I7 decode-layer credit + for an already-decoding request carried inside a prefill-mode batch. It moves + a counter that no monolithic timing reads. +- If a MoE co-location case does move, the I7 credit is the cause, and it is the + approved fidelity fix that plan section 9 requires. + +### Result + +| Metric | Value | +| --- | --- | +| cases compared | 71 of 71 in the case table | +| identical | **71** | +| mismatched | 0 | +| baseline failures / candidate-only failures | 0 / 0 | +| missing from one side / missing evidence / differing definitions | 0 / 0 / 0 | +| provenance findings | 0 | +| predictor cache: baseline-only / candidate-only / findings | 0 / 0 / 0 | +| `complete_comparison`, `predictor_cache_populated_cleanly` | `True`, `True` | + +PASS against the stated expectation: the primary prediction held exactly and the +conditional branch was not taken. That also confirms the source reading behind +it — on the monolithic path `completed_layer_count` feeds only admission guards +and diagnostics, and its two arithmetic consumers +(`cluster_batch_end_event.py:174` and `:354`) are PD-AF `DECODE_ATTN`/M2N. + +### Runtime evidence, which the matrix cannot supply + +`tests/integration/test_monolithic_mixed_forward_runtime.py` builds +`attn_tp=1, attn_dp=2, moe_tp=1, moe_ep=2` on a monolithic MoE Replica directly +and runs the real `Simulator`: real admission, ownership restoration, +synchronization and completion, with deterministic durations injected only at +the predictor boundary and an observer that *wraps* `predict_stage_execution_time` +rather than replacing it. Four requests of unequal length arrive together under +chunked prefill, which is what puts one lane in prefill while another decodes. + +Observed: 24 cohorts, 4 of them mixed-phase, 4 of 4 requests completed, every +token accounted for, no waiting-room leaf left holding a batch, every +`StageExecutionContext` idle with no queued ticket, and an identical run with +metrics reporting on and off. On the pre-fix source the same configuration ends +with `RuntimeError: Sequential simulation ended with non-empty scheduler state`. + +### Controls — four trees, four distinct failures + +Each control tree carries the final `frontier/` and the final `tests/`, +differing from the delivered source by exactly one edit; the baseline tree's +`frontier/` is the pre-fix parent in full. They are driven by +`run_against_tree.py`, which strips the editable-install meta-path finder and +asserts the resolved source path, because an earlier attempt to select a tree +with `PYTHONPATH` alone silently kept importing the installed package. + +| Tree | Defect | Unit | Runtime failure | +| --- | --- | --- | --- | +| baseline | none — pre-fix `3d47417` | 22 of 23 fail | `RuntimeError: Sequential simulation ended with non-empty scheduler state` | +| borrowed timing | every source continues on `live_batches[0]` | 12 of 23 fail | `ValueError: one attention-DP lane cannot occupy two open sync cohorts: replica=0, stage=0, lane=1, layer=1, sync_stage=pre_moe` | +| double layer advance | the decode helper advances layers again | 2 of 23 fail | `ValueError: Decode post_moe layer counter cannot advance: request_id=1, completed_layer_count=4, total_layers=4` | +| double owner restore | the per-source helper restores owners again | 16 of 23 fail | `ValueError: operation_id is already queued or active in this stage context: ('shared_layer', 0, 1, 0, 1, 'attention', 'FULL_STAGE_WORLD')` | + +On the pre-fix tree the same-phase pairs pass every assertion about the forward +itself and fail only at the last line, which inspects a shared room the pre-fix +cluster does not have. The mixed pairs fail earlier, at an empty collective +list. That gap between the two groups is the deadlock, isolated. + +### Regression comparison + +| Suite | Baseline `3d47417` | Candidate `65ed8a7` | Verdict | +| --- | --- | --- | --- | +| `tests/unit` | 84 failed / 3694 passed | 84 failed / 3717 passed | identical failure identities; +23 are the new tests | +| `tests/integration` | 5 errors / 11 passed | 5 errors / 12 passed | identical errors (PD-AF Reference checkout absent on this host); +1 is the new test | +| forward-sync set, 17 files | — | 342 passed | no regression in the shared call chain | + +### Limits + +- The multi-lane monolithic MoE shape has no released wrapper, so its evidence + is a direct-construction test rather than a matrix case. Anything that only a + released recipe would exercise is therefore still unmeasured for this shape. +- The integration profiles are constant synthetic targets: deterministic and + source-attributable, not trained numerical parity. +- I8, the per-source decode component ledger, is deliberately out of scope; see + the scope table in `design.md`. From cdfcdf54b54565e29ca5f1cea1c93f344980bb47 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 09:57:41 +0800 Subject: [PATCH 14/80] docs(review): mark Step 3 published and record the PR 35 W3 section --- .../task_2026-09-21_issue26_correctness_pr/progress.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 931563a1..fefb7bb3 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -18,7 +18,7 @@ | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | | Current step | Step 3 complete: implemented, measured, records written | | Publication | PUSHED_VERIFIED (records) | -| Next action | Push the Step 3 records and add the W3 section to PR #35; then Checkpoint D second half (W4, the opt-in vLLM DP placement). | +| Next action | Checkpoint D second half: W4, the opt-in vLLM-style DP placement, validating W3's shared forward identity at the report boundary per decision D1. | ## Step status @@ -27,7 +27,7 @@ | 0 | Worktree, references, baseline | PASS | PASS (baseline recorded) | PUSHED_VERIFIED | NOT_REVIEWED | | 1 | Candidate/vLLM audit | PASS | n/a (source audit) | LOCAL_ONLY | NOT_REVIEWED | | 2 | RR DP rotation | PASS | unit PASS (23 tests); matrix PASS against a stated expectation, re-measured 2026-09-22 with one harness revision | PUSHED_VERIFIED | REVIEWED (R35-01 closed) | -| 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | LOCAL_ONLY | NOT_REVIEWED | +| 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 4 | Opt-in vLLM DP placement | NOT_STARTED | — | — | — | | 5 | Routing implementation identity | NOT_STARTED | — | — | — | | 6 | Legacy fused-MoE profiling | NOT_STARTED | — | — | — | @@ -58,6 +58,7 @@ - 2026-09-22: W3 regression comparison against the branch parent `3d47417` in a dedicated detached worktree. `tests/unit`: 84 failures on both sides with identical identities, 3717 vs 3694 passing. `tests/integration`: the same five pre-existing errors (the PD-AF Reference checkout is absent on this host), 12 vs 11 passing. No regressions and no accidental fixes. - 2026-09-22: W3 controls rebuilt against the final test file and re-run, so the recorded counts match what is delivered. Each tree now carries the final `frontier/` and the final `tests/` and differs from the delivered source by exactly one edit, except the baseline tree whose `frontier/` is the pre-fix parent in full. Counts: 22, 12, 2 and 16 of 23 unit tests fail respectively, and each runtime failure is distinct. The pre-fix tree's same-phase pairs pass every assertion about the forward itself and fail only at the final shared-room inspection; the mixed pairs fail earlier at an empty collective list, which is the deadlock isolated. - 2026-09-22: W3 fidelity matrix PASS. Baseline `3d47417` against candidate `65ed8a7`, both clean detached checkouts with `source_dirty=False` and no dirty paths, one harness at `65ed8a7`, no filter, clean cache, 71 executed and 426 cache files each. 71 of 71 compared and **71 identical**, zero mismatches, zero provenance findings, zero predictor cache differences. That is exactly the expectation `design.md` recorded before the run; the conditional I7 branch was not taken. Note explicitly: the matrix cannot reach a multi-lane monolithic MoE forward at all, so a null result is the pass condition for "nothing else moved", not evidence that the defect is fixed. Full record in `validation.md` and `test_report_2026-09-22_w3_shared_monolithic_forward.md`. +- 2026-09-22: Step 3 published. `bdff4aa` pushed to `origin/fix/issue26-correctness-pr`; PR #35 body gained a W3 section that states the defect, the one-lifecycle fix, the event-priority constraint, the four controls with their distinct failures, and the matrix result together with what a null result does and does not mean. PR #35 stays draft. The W3 measurement worktrees `.worktrees/w3-baseline` and `.worktrees/w3-candidate` were removed after their manifests, results and `comparison.json` were written; the matrix output root is kept as evidence. ## Step 3 scoping, as recorded before implementation From 10dd4745b9a749d02fa40399977a7d377a22b7de Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 10:33:19 +0800 Subject: [PATCH 15/80] feat(scheduler): add an opt-in vLLM-style DP request placement policy Frontier routed requests to a Replica's attention-DP lanes by round-robin, which no vLLM deployment does: a vLLM V1 frontend picks an engine from a `waiting * 4 + running` score over counts it only sees when the coordinator publishes a snapshot. Placement therefore lags the true load, and the lag itself changes which lane a request lands on. `VllmDPLoadBalancer` models the two halves of that mechanism from vLLM v0.10.2: the frontend's selection (`core_client.py:1139-1153`, including the lowest-index tie break and the local waiting reservation that spreads requests between snapshots) and the coordinator's publication schedule (`coordinator.py:195-211`, the 100 ms changed / 5000 ms idle intervals, the 50 ms first-snapshot collection wait, and the latch that publishes the previous step's counts). Out-of-order step reports warn and still apply, as the reference does; nothing here aborts a run. The object creates no events, so a drained simulation still drains. `VllmLoadBalancingClusterScheduler` wires it to one co-location Replica. It is selected by `--cluster_scheduler_config_type vllm_load_balancing` and nothing else changes when it is not selected: `round_robin` remains the default. Supported scope is enforced in the constructor rather than documented -- one Replica, one pipeline stage, the `vllm_v1` replica scheduler, and a report key whose ordering actually holds. That last guard is measured, not assumed: a MoE Replica resolves one shared forward identity across its lanes, so the key is monotonic per Replica, while a dense Replica keeps a per-lane counter whose keys interleave under staggered arrivals. Dense therefore stays at attn_dp=1. Two seams carry it. `BaseClusterScheduler.schedule_at(time)` defaults to `schedule()`, so a placement policy that depends on elapsed time receives it as an argument instead of through a mutable bridge, and `on_replica_batch_end` is an inert hook called after the lane's request-state transition, so the reported load is the post-step one. `get_request_load()` reuses the existing decision-log waiting accessor, so a load balancer and the decision log cannot disagree about what is waiting. No placement or timing equivalence with a real vLLM deployment is claimed. --- frontier/config/cluster_scheduler_config.py | 9 + frontier/config/config.py | 1 + frontier/events/cluster_schedule_event.py | 2 +- frontier/events/global_batch_end_event.py | 5 + .../base_cluster_scheduler.py | 26 + .../cluster_scheduler_registry.py | 7 + .../vllm_load_balancing_cluster_scheduler.py | 112 +++ .../base_replica_scheduler.py | 13 + .../sglang_style_replica_scheduler.py | 5 +- .../vllm_v1_iteration_policy.py | 20 +- frontier/scheduler/request_load.py | 10 + .../scheduler/utils/vllm_dp_load_balancer.py | 175 ++++ frontier/types/cluster_scheduler_type.py | 1 + .../design.md | 131 ++- .../test_vllm_dp_placement_runtime.py | 491 ++++++++++++ tests/unit/test_vllm_dp_load_balancer.py | 757 ++++++++++++++++++ 16 files changed, 1759 insertions(+), 6 deletions(-) create mode 100644 frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py create mode 100644 frontier/scheduler/request_load.py create mode 100644 frontier/scheduler/utils/vllm_dp_load_balancer.py create mode 100644 tests/integration/test_vllm_dp_placement_runtime.py create mode 100644 tests/unit/test_vllm_dp_load_balancer.py diff --git a/frontier/config/cluster_scheduler_config.py b/frontier/config/cluster_scheduler_config.py index d757a1c4..6cf4bde6 100644 --- a/frontier/config/cluster_scheduler_config.py +++ b/frontier/config/cluster_scheduler_config.py @@ -46,3 +46,12 @@ class StickyLORClusterSchedulerConfig(BaseClusterSchedulerConfig): @staticmethod def get_type(): return ClusterSchedulerType.STICKY_LOR + + +@dataclass +class VllmLoadBalancingClusterSchedulerConfig(BaseClusterSchedulerConfig): + """Select vLLM V1's internal DP placement. Selecting it is the only knob.""" + + @staticmethod + def get_type(): + return ClusterSchedulerType.VLLM_LOAD_BALANCING diff --git a/frontier/config/config.py b/frontier/config/config.py index cb2b3986..2b4fb60c 100644 --- a/frontier/config/config.py +++ b/frontier/config/config.py @@ -120,6 +120,7 @@ RoundRobinClusterSchedulerConfig, StickyLORClusterSchedulerConfig, StickyRoundRobinClusterSchedulerConfig, + VllmLoadBalancingClusterSchedulerConfig, ) from frontier.config.execution_time_predictor_config import ( BaseExecutionTimePredictorConfig, diff --git a/frontier/events/cluster_schedule_event.py b/frontier/events/cluster_schedule_event.py index b025eb9b..a58f83bc 100644 --- a/frontier/events/cluster_schedule_event.py +++ b/frontier/events/cluster_schedule_event.py @@ -39,7 +39,7 @@ def handle_event( logger.info(f"Cluster scheduling started at {self.time:.3f}s: " f"{self._cluster_type.name} cluster with {queue_size} requests in queue") - self._request_mapping = cluster_scheduler.schedule() + self._request_mapping = cluster_scheduler.schedule_at(self.time) # DEBUG: Log request mapping mapping_summary = {} diff --git a/frontier/events/global_batch_end_event.py b/frontier/events/global_batch_end_event.py index 3d9398d2..02fe46a5 100644 --- a/frontier/events/global_batch_end_event.py +++ b/frontier/events/global_batch_end_event.py @@ -178,6 +178,11 @@ def _current_request_entries() -> list[tuple[int, object]]: thinking_round_start_times=self._thinking_round_start_times, ) replica_scheduler.on_batch_end(self._batch) # decrement running batches + # After the lane's request-state transition, so a routing policy that + # reads lane populations here observes the post-step load. + cluster_scheduler.on_replica_batch_end( + self.time, self._replica_id, self._replica_local_id, self._batch + ) thinking_requeue_events: List[BaseEvent] = [] for index, request in pre_batch_request_entries: diff --git a/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py b/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py index aeb0d847..c9494f87 100644 --- a/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py +++ b/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py @@ -427,6 +427,32 @@ def _schedule_batch_mode(self) -> List[Tuple[int, int, Request]]: def add_request(self, request: Request) -> None: self._request_queue.append(request) + def schedule_at(self, time: float) -> List[Tuple[int, int, Request]]: + """Route the queue at a known simulation time. + + The default ignores the time and routes exactly as before, so every + policy that does not need it is unaffected. A policy whose placement + depends on wall-clock progress -- a delayed load snapshot, for instance + -- overrides this instead of reaching for a mutable time bridge. + """ + + return self.schedule() + + def on_replica_batch_end( + self, + time: float, + replica_id: int, + replica_local_id: int | None, + batch: Batch, + ) -> None: + """Observe one Replica-local batch completion. Inert by default. + + Called after the batch's request-state transition, so a policy that + reads lane populations here sees the post-step state. + """ + + return None + def get_replica(self, replica_id: int) -> Replica: return self._cluster.replicas[replica_id] diff --git a/frontier/scheduler/cluster_scheduler/cluster_scheduler_registry.py b/frontier/scheduler/cluster_scheduler/cluster_scheduler_registry.py index a2a9135a..8f90b25a 100644 --- a/frontier/scheduler/cluster_scheduler/cluster_scheduler_registry.py +++ b/frontier/scheduler/cluster_scheduler/cluster_scheduler_registry.py @@ -11,6 +11,9 @@ from frontier.scheduler.cluster_scheduler.sticky_round_robin_cluster_scheduler import ( StickyRoundRobinClusterScheduler, ) +from frontier.scheduler.cluster_scheduler.vllm_load_balancing_cluster_scheduler import ( + VllmLoadBalancingClusterScheduler, +) from frontier.types.cluster_scheduler_type import ClusterSchedulerType from frontier.utils.base_registry import BaseRegistry @@ -29,3 +32,7 @@ def get_key_from_str(cls, key_str: str) -> ClusterSchedulerType: StickyRoundRobinClusterScheduler, ) ClusterSchedulerRegistry.register(ClusterSchedulerType.STICKY_LOR, StickyLORClusterScheduler) +ClusterSchedulerRegistry.register( + ClusterSchedulerType.VLLM_LOAD_BALANCING, + VllmLoadBalancingClusterScheduler, +) diff --git a/frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py b/frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py new file mode 100644 index 00000000..1c059a1c --- /dev/null +++ b/frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py @@ -0,0 +1,112 @@ +"""Route one vLLM serving Replica through its internal DP load balancer.""" + +from typing import List, Tuple + +from frontier.entities import Batch, Request +from frontier.scheduler.cluster_scheduler.base_cluster_scheduler import ( + BaseClusterScheduler, +) +from frontier.scheduler.utils.forward_sync_state import ForwardSyncState +from frontier.scheduler.utils.vllm_dp_load_balancer import VllmDPLoadBalancer +from frontier.types import ClusterType, ReplicaSchedulerType + + +class VllmLoadBalancingClusterScheduler(BaseClusterScheduler): + """Model single-frontend vLLM V1 DP routing with delayed load snapshots. + + Supported scope, and nothing wider: one co-location Replica, one modeled + frontend, the `vllm_v1` replica scheduler, one pipeline stage, and a report + key whose ordering actually holds -- a MoE model, whose attention-DP lanes + share one forward step identity, or a single lane, where the question does + not arise. Everything else is rejected in the constructor. + + No placement or timing equivalence with a real vLLM deployment is claimed. + """ + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + if ( + self._cluster_type is not ClusterType.MONOLITHIC + or self._num_replicas != 1 + or self._config.replica_config.num_pipeline_stages != 1 + or self._replica_scheduler_type is not ReplicaSchedulerType.VLLM_V1 + ): + raise ValueError( + "vllm_load_balancing supports one co-location Replica with " + "vllm_v1 and PP1, got " + f"cluster_type={self._cluster_type.name}, " + f"num_replicas={self._num_replicas}, " + f"num_pipeline_stages=" + f"{self._config.replica_config.num_pipeline_stages}, " + f"replica_scheduler={self._replica_scheduler_type}" + ) + # The load report is ordered by the forward step identity. A MoE + # Replica resolves one shared identity for every lane of one forward, so + # the key is monotonic per Replica. A dense Replica has no per-forward + # collective across its attention-DP lanes, so each lane keeps its own + # creation counter and two lanes report interleaved keys against one + # shared ordering scalar. One lane is safe either way. + if not self._config.replica_config.model_config.is_moe: + if self._replica_dp_size != 1: + raise ValueError( + "vllm_load_balancing requires a forward step identity that " + "is monotonic per Replica; a dense model provides one only " + "at attn_dp=1, got " + f"attn_dp={self._replica_dp_size}" + ) + self._serving_replica_id = next(iter(self._cluster.replicas)) + self._load_balancer = VllmDPLoadBalancer(self._replica_dp_size) + + def schedule_at(self, time: float) -> List[Tuple[int, int, Request]]: + """Route every queued request at this simulation time. + + Selection depends on when it happens, because the frontend's view of + engine load is a snapshot published on a timer. The time therefore + arrives as an argument rather than through retained state. + """ + + self.sort_requests() + mapping = [ + (self._serving_replica_id, self._load_balancer.select(time), request) + for request in self._request_queue + ] + self._request_queue.clear() + return mapping + + def schedule(self) -> List[Tuple[int, int, Request]]: + """Refuse to route without a time, rather than reuse a stale one.""" + + raise RuntimeError( + "vllm_load_balancing selects from a time-dependent load snapshot; " + "route through schedule_at(time), which ClusterScheduleEvent calls" + ) + + def on_replica_batch_end( + self, + time: float, + replica_id: int, + replica_local_id: int | None, + batch: Batch, + ) -> None: + """Report the lane's post-step load, keyed by the forward identity. + + The key's numeric value is not a vLLM step counter: it advances once per + layer, so consecutive forwards are roughly `num_layers` apart. Only its + ordering and equality are used, which is all the reference coordinator + uses its `(wave, step)` pair for. Frontier has no wave reset, so a + Replica-scoped monotonic counter collapses that pair to one scalar. + """ + + if type(replica_local_id) is not int: + raise ValueError( + "vllm_load_balancing reports load per attention-DP lane and " + "needs an exact lane index, got " + f"replica_local_id={replica_local_id!r}" + ) + lane = self.get_replica_scheduler(replica_id, replica_local_id) + self._load_balancer.report( + time, + replica_local_id, + ForwardSyncState.get_step_id(batch), + lane.get_request_load(), + ) diff --git a/frontier/scheduler/replica_scheduler/base_replica_scheduler.py b/frontier/scheduler/replica_scheduler/base_replica_scheduler.py index cf5f777b..30498599 100644 --- a/frontier/scheduler/replica_scheduler/base_replica_scheduler.py +++ b/frontier/scheduler/replica_scheduler/base_replica_scheduler.py @@ -24,6 +24,7 @@ from frontier.scheduler.cluster_scheduler.base_cluster_scheduler import ( BaseClusterScheduler, ) + from frontier.scheduler.request_load import RequestLoad class BaseReplicaScheduler(ABC): @@ -509,6 +510,18 @@ def num_pending_requests(self) -> int: def peek_waiting_requests(self) -> List[Request]: return list(self._request_queue) + def get_request_load(self) -> "RequestLoad": + """Return this lane's waiting and admitted request populations. + + A serving load balancer needs both, with the same meaning the scheduler + uses internally. Only schedulers that define those populations + unambiguously implement it. + """ + + raise NotImplementedError( + f"{type(self).__name__} does not expose a serving request load" + ) + @property def replica_id(self) -> int: return self._replica_id diff --git a/frontier/scheduler/replica_scheduler/sglang_style_replica_scheduler.py b/frontier/scheduler/replica_scheduler/sglang_style_replica_scheduler.py index b96e8991..8ddc2d4b 100644 --- a/frontier/scheduler/replica_scheduler/sglang_style_replica_scheduler.py +++ b/frontier/scheduler/replica_scheduler/sglang_style_replica_scheduler.py @@ -50,6 +50,7 @@ def _emit_schedule_decision_event( available_blocks = int(self._config.num_blocks - self._num_allocated_blocks) cluster_name = self._cluster_type.name if self._cluster_type else "MONOLITHIC" + request_load = self.get_request_load() payload: Dict[str, Any] = { "event": event, "source": "frontier", @@ -61,8 +62,8 @@ def _emit_schedule_decision_event( "token_budget": int(token_budget), "available_blocks": int(available_blocks), "num_tokens": int(num_tokens), - "num_running_reqs": len(self._running_requests), - "num_waiting_reqs": self._get_num_waiting_reqs_for_decision_log(), + "num_running_reqs": request_load.running, + "num_waiting_reqs": request_load.waiting, "max_num_running_reqs": int(self._max_num_running_reqs), "max_num_scheduled_tokens": int(self._max_num_scheduled_tokens), "batch_request_ids": [str(req_id) for req_id in (batch_request_ids or [])], diff --git a/frontier/scheduler/replica_scheduler/vllm_v1_iteration_policy.py b/frontier/scheduler/replica_scheduler/vllm_v1_iteration_policy.py index 731251fa..c27bd3ec 100644 --- a/frontier/scheduler/replica_scheduler/vllm_v1_iteration_policy.py +++ b/frontier/scheduler/replica_scheduler/vllm_v1_iteration_policy.py @@ -22,6 +22,7 @@ _log_frontier_vllm_v1_schedule_decision, schedule_decision_logging_enabled, ) +from frontier.scheduler.request_load import RequestLoad from frontier.spec_decode import compute_iteration_outcome, get_planned_draft_tokens from frontier.types import ClusterType @@ -529,6 +530,20 @@ def _get_num_waiting_reqs_for_decision_log(self) -> int: return len(self._waiting_requests) return len(self._request_queue) + len(self._preempted_requests) + def get_request_load(self) -> RequestLoad: + """Report this lane's populations from the one waiting definition. + + Running counts every admitted request, including one that is admitted + but not scheduled in the current iteration. Waiting reuses the + decision-log accessor above so a load balancer and the decision log can + never disagree about what is waiting. + """ + + return RequestLoad( + self._get_num_waiting_reqs_for_decision_log(), + len(self._running_requests), + ) + def _apply_long_prefill_token_threshold( self, request: Request, num_new_tokens: int ) -> int: @@ -558,6 +573,7 @@ def _emit_schedule_decision_event( available_blocks = int(self._config.num_blocks - self._num_allocated_blocks) cluster_name = self._cluster_type.name if self._cluster_type else "MONOLITHIC" + request_load = self.get_request_load() payload: Dict[str, Any] = { "event": event, "source": "frontier", @@ -569,8 +585,8 @@ def _emit_schedule_decision_event( "token_budget": int(token_budget), "available_blocks": int(available_blocks), "num_tokens": int(num_tokens), - "num_running_reqs": len(self._running_requests), - "num_waiting_reqs": self._get_num_waiting_reqs_for_decision_log(), + "num_running_reqs": request_load.running, + "num_waiting_reqs": request_load.waiting, "max_num_running_reqs": int(self._max_num_running_reqs), "max_num_scheduled_tokens": int(self._max_num_scheduled_tokens), "batch_request_ids": [str(req_id) for req_id in (batch_request_ids or [])], diff --git a/frontier/scheduler/request_load.py b/frontier/scheduler/request_load.py new file mode 100644 index 00000000..3efd2cbf --- /dev/null +++ b/frontier/scheduler/request_load.py @@ -0,0 +1,10 @@ +"""Scheduler request populations exposed to serving load balancers.""" + +from typing import NamedTuple + + +class RequestLoad(NamedTuple): + """Count waiting and admitted requests, including unscheduled running work.""" + + waiting: int + running: int diff --git a/frontier/scheduler/utils/vllm_dp_load_balancer.py b/frontier/scheduler/utils/vllm_dp_load_balancer.py new file mode 100644 index 00000000..76e99cdf --- /dev/null +++ b/frontier/scheduler/utils/vllm_dp_load_balancer.py @@ -0,0 +1,175 @@ +"""vLLM V1 internal DP selection and coordinator count publication. + +Reference: vLLM v0.10.2, `vllm/v1/engine/core_client.py` (the frontend's engine +selection) and `vllm/v1/engine/coordinator.py` (the coordinator's count +publication). Every constant below is cited against that source. + +What is modeled: which engine a request is routed to, given counts the frontend +observes with a delay, and when the coordinator publishes a new snapshot of +those counts. What is not modeled: IPC transport latency, more than one +frontend, elastic scaling, and the coordinator's warm-start publication phase. + +Timers advance lazily. This object creates no events, so it cannot keep a +drained simulation alive; a deadline that has passed is applied the next time +the balancer is consulted. +""" + +from math import isfinite + +from frontier.logger import init_logger +from frontier.scheduler.request_load import RequestLoad + +logger = init_logger(__name__) + +# `score = waiting * 4 + running` (reference `core_client.py:1146`). +WAITING_SCORE_WEIGHT = 4 +# `wait_for = stats_update_interval_ms if stats_changed else 5000` +# (reference `coordinator.py:195-198`), whose default is 100 +# (`coordinator.py:116`, `:122`, `:130`). +CHANGED_PUBLISH_INTERVAL_MS = 100 +UNCHANGED_PUBLISH_INTERVAL_MS = 5000 +# `min_timeout = 50 if last_step_counts is None else 0` +# (reference `coordinator.py:201-203`). +SNAPSHOT_COLLECTION_WAIT_MS = 50 + + +class VllmDPLoadBalancer: + """Own one frontend's load estimates and its coordinator's report state. + + The frontend's estimate (`frontend_counts`) is a delayed copy of the + coordinator's authoritative per-engine counts (`engine_counts`). Selection + reads the estimate and reserves against it locally; a new snapshot replaces + the estimate wholesale, exactly as `core_client.py:1073-1078` assigns + `self.lb_engines = sliced_counts`. + """ + + def __init__(self, num_engines: int): + if type(num_engines) is not int or num_engines < 1: + raise ValueError("DP load balancing requires at least one engine") + self.engine_counts = [RequestLoad(0, 0) for _ in range(num_engines)] + self.frontend_counts = list(self.engine_counts) + self.last_step_counts: list[RequestLoad] | None = None + self.last_report_step = -1 + self.stats_changed = False + # The reference's coordinator starts with `last_publish_time = 0` while + # the clock reads epoch milliseconds, so `wait_for - elapsed` is deeply + # negative on the first iteration and the 50 ms collection wait decides + # the first publish. Seeding the last publish one unchanged interval in + # the past reproduces that first deadline without special-casing it. + self.last_publish_ms = -UNCHANGED_PUBLISH_INTERVAL_MS + self.next_publish_ms = SNAPSHOT_COLLECTION_WAIT_MS + self.time_ms = 0 + + def _poll_deadline(self, now_ms: int) -> int: + """Return the next publish time, as the reference's poll timeout does. + + The reference waits `max(min_timeout, wait_for - elapsed)` from now + (`coordinator.py:205-206`), which is the same instant as + `max(now + min_timeout, last_publish + wait_for)`. + """ + + interval = ( + CHANGED_PUBLISH_INTERVAL_MS + if self.stats_changed + else UNCHANGED_PUBLISH_INTERVAL_MS + ) + collection_wait = ( + SNAPSHOT_COLLECTION_WAIT_MS if self.last_step_counts is None else 0 + ) + return max(now_ms + collection_wait, self.last_publish_ms + interval) + + def _advance(self, time: float, *, report_arriving: bool = False) -> int: + """Apply every publish deadline that has passed, then adopt `time`. + + `report_arriving` breaks the tie at an exact deadline. A real poller + returns the waiting message rather than timing out, so a report that + lands on its deadline is processed before the publish it triggers. + """ + + if not isfinite(time) or time < 0: + raise ValueError("DP load balancing requires finite nonnegative time") + now_ms = int(time * 1000) + if now_ms < self.time_ms: + raise ValueError("DP load balancing time cannot move backwards") + while ( + self.next_publish_ms < now_ms + or self.next_publish_ms == now_ms + and not report_arriving + ): + published_ms = self.next_publish_ms + if self.last_step_counts is not None: + # A snapshot latched for the previous step is published first + # and consumed (reference `coordinator.py:208-211`). + self.frontend_counts = self.last_step_counts + self.last_step_counts = None + else: + self.frontend_counts = list(self.engine_counts) + self.stats_changed = False + self.last_publish_ms = published_ms + self.next_publish_ms = self._poll_deadline(published_ms) + self.time_ms = now_ms + return now_ms + + def report(self, time: float, engine: int, step: int, load: RequestLoad) -> None: + """Record one engine's counts after it completed one forward step. + + Ordering is advisory, as in the reference: a strictly newer step latches + the previous counts when there are unpublished changes, an equal step is + the expected path for a peer engine reporting the same forward, and an + out-of-order step only warns (`coordinator.py:293-310`). The counts are + applied either way; nothing here aborts a run. + """ + + if type(engine) is not int or not 0 <= engine < len(self.engine_counts): + raise ValueError( + "DP load report references an unknown engine: " + f"engine={engine!r}, num_engines={len(self.engine_counts)}" + ) + if type(step) is not int or step < 0 or min(load) < 0: + raise ValueError( + "DP load report requires a nonnegative int step and nonnegative " + f"counts, got step={step!r}, load={load!r}" + ) + if load == self.engine_counts[engine]: + # The reference engine compares against its own `last_counts` and + # emits no coordinator message when they match + # (`core.py:1080-1087`). That value never diverges from the + # coordinator's copy, which every report overwrites. + return + now_ms = self._advance(time, report_arriving=True) + if step > self.last_report_step: + if self.stats_changed: + self.last_step_counts = list(self.engine_counts) + self.last_report_step = step + elif step < self.last_report_step: + logger.warning( + "Received DP load stats for out-of-order step %d from engine %d " + "(expected >= %d); counts are still applied", + step, + engine, + self.last_report_step, + ) + self.engine_counts[engine] = load + self.stats_changed = True + self.next_publish_ms = self._poll_deadline(now_ms) + + def select(self, time: float) -> int: + """Choose the first minimum-score engine and reserve local waiting load. + + The reference scans from the frontend's own start index with a strict + `<`, so the lowest-index minimum wins, then increments that engine's + waiting estimate by the frontend count (`core_client.py:1139-1153`). + One modeled frontend means that reservation is exactly one request. + """ + + self._advance(time) + engine = min( + range(len(self.frontend_counts)), + key=lambda index: ( + WAITING_SCORE_WEIGHT * self.frontend_counts[index].waiting + + self.frontend_counts[index].running + ), + ) + load = self.frontend_counts[engine] + self.frontend_counts[engine] = RequestLoad(load.waiting + 1, load.running) + return engine diff --git a/frontier/types/cluster_scheduler_type.py b/frontier/types/cluster_scheduler_type.py index 4054133d..95d61686 100644 --- a/frontier/types/cluster_scheduler_type.py +++ b/frontier/types/cluster_scheduler_type.py @@ -7,3 +7,4 @@ class ClusterSchedulerType(BaseIntEnum): LOR = 2 STICKY_ROUND_ROBIN = 3 STICKY_LOR = 4 + VLLM_LOAD_BALANCING = 5 diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md index 9eaee708..df106635 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md @@ -1,10 +1,16 @@ -# W3 design — one shared monolithic forward +# Issue 26 Correctness PR — Design Records + +One section per work package. Each records the source-backed reasoning, the +scope decisions and the pre-measurement expectation for that package. ## Modification History | Date | Change | | --- | --- | | 2026-09-22 | Created. Source-backed design for Checkpoint D's W3 half, with the scope decisions and their evidence. | +| 2026-09-22 | Restructured into per-work-package sections and added the W4 design, with the report-key identity measured at the emission boundary. | + +# W3 — one shared monolithic forward ## The defect, restated from source @@ -135,3 +141,126 @@ The prediction for the 71-case matrix is therefore: subset exactly once per completed layer"). Recorded before the measurement so the result can falsify it. + +--- + +# W4 — opt-in vLLM-style DP request placement + +## What this adds, and what it is not + +A new **opt-in** cluster-scheduler policy, `vllm_load_balancing`, that routes +requests inside one serving Replica the way vLLM V1's frontend does: pick the +engine with the lowest `4 * waiting + running` from a **delayed** snapshot of +engine counts, reserving local waiting load between snapshots. Nothing selects +it by default; every existing policy and default is untouched. + +This models vLLM's *source-level* selection and count-publication behavior. It +does **not** claim placement or timing equivalence with a real vLLM deployment, +and it does not model IPC transport latency, multiple frontends, elastic +scaling, or warm-start publication phase. + +## Reference facts, verified in the pinned checkout + +All from `.real-engine/vLLM-BS` at `ea95f571`, read rather than assumed: + +| Fact | Location | +| --- | --- | +| `score = waiting * 4 + running`, scanned from `eng_start_index`, strict `<` so the first minimum wins | `vllm/v1/engine/core_client.py:1139-1150` | +| Local reservation after selection: `current_counts[eng_index][0] += self.client_count` — with one frontend, `waiting + 1` | `core_client.py:1151-1153` | +| A published snapshot **replaces** the frontend estimate wholesale; it is not merged into local reservations | `core_client.py:1073-1078` (`self.lb_engines = sliced_counts`) | +| Publish interval: `stats_update_interval_ms` when changed, else `5000` ms | `coordinator.py:195-198` | +| `min_stats_update_interval_ms` default `100` | `coordinator.py:116`, `:122`, `:130` | +| Minimum collection wait `50` ms while a previous-step snapshot is pending | `coordinator.py:201-203` | +| Poll timeout `max(min_timeout, wait_for - elapsed)` | `coordinator.py:205-206` | +| On timeout: publish the pending previous-step snapshot if present, else the current counts and clear `stats_changed` | `coordinator.py:207-218` | +| The order key is the pair `(wave, step)` against one shared `(last_stats_wave, last_stats_step)` | `coordinator.py:156-157`, `:293-300` | +| A strictly newer key latches the prior counts **only when** `stats_changed`; an **equal** key takes neither branch; an out-of-order key logs a **warning** and the counts are still applied | `coordinator.py:293-310` | + +So the constants are confirmed as 4, 50 ms, 100 ms and 5000 ms, and the +out-of-order case is a warning, never an abort. Per decision D1, W4 must not add +a runtime assertion on report order that the reference does not have. + +## The report key, measured at the emission boundary + +Decision D1 asks for exactly this: inspect the identity **at the load-report +emission boundary**, reuse it only where its equality and ordering hold, and +reject the rest explicitly rather than adding a second counter. + +The boundary is `GlobalBatchEndEvent`, whose `_replica_local_id` and batch reach +the new `on_replica_batch_end` hook. A probe patched that handler and recorded +`(replica_local_id, ForwardSyncState.get_step_id(batch))` for every completion +of a real `Simulator` run, in four monolithic shapes: + +| Shape | Observed `(lane, step)` sequence | Verdict | +| --- | --- | --- | +| MoE `attn_dp=2`, all arrivals at once | `(1,3) (0,3) (1,7) (0,7) (1,11) (0,11) (1,15) (0,15) (1,19) (0,19)` | **valid** — both lanes of one forward share one key; distinct forwards are strictly increasing | +| MoE `attn_dp=2`, staggered online arrivals | `(0,3) (0,7) (0,11) (1,15) (1,19) (1,23) (0,27) (0,31) (0,35) (1,39) (1,43) (1,47) (0,51) (0,55) (0,59)` | **valid** — strictly increasing across the whole run whichever lane reports | +| MoE `attn_dp=1` | `3 7 11 15 19 23 27` | valid, degenerate | +| Dense `attn_dp=1` | `0 1 2 3 4 5 6` | valid, degenerate | +| Dense `attn_dp=2`, all arrivals at once | `(1,0) (0,0) (1,1) (0,1) (1,2) (0,2) (1,3) (0,3) (1,4) (0,4)` | **incidentally** paired; nothing enforces it | +| Dense `attn_dp=2`, staggered online arrivals | `(0,0) (0,1) (0,2) (1,0) (1,1) (1,2) (0,3) (0,4) (0,5) (1,3) (1,4) (1,5) (0,6) (0,7) (0,8)` | **INVALID** — lane 1's step `0` arrives after lane 0's step `2` | + +Two things this settles that the audit could only argue: + +1. **MoE monolithic is valid because W3 landed.** The ids come from the + Replica-scoped `_next_step_id_by_replica` counter that W3's single shared + room resolves, so they are monotonic per Replica regardless of which lanes + are live. Both lanes of one forward report the same key, which is precisely + the reference's "equal key, peer engine" path. +2. **Dense with more than one lane is invalid, and observably so.** Each dense + lane emits its own `0, 1, 2, …` from a per-lane creation counter that no + shared forward ever promotes, because a dense monolithic Replica has no + per-forward DP collective to keep the lanes in lockstep. Under staggered + arrivals the counters interleave out of order against one shared + `last_report_step`, so the previous-step snapshot latch would be wrong. + +Also observed: `replica_local_id` at this boundary is **always an exact `int`**, +never the full-stage `None`, in all four shapes; and an idle batch never reaches +the hook. So the engine index needs a type/range check, not a `None` branch. + +Numeric caveat, unchanged from the audit and kept in a code comment: the value +advances per layer, so consecutive forwards are about `num_layers` apart. Only +its ordering and equality are used. The reference key is a `(wave, step)` pair; +Frontier has no wave reset, so a Replica-scoped monotonic counter collapses the +pair to a single scalar. + +## Scope decisions + +| Decision | Choice | Why | +| --- | --- | --- | +| Topology guard | `MONOLITHIC` + one Replica + `PP1` + `vllm_v1` + (**MoE or `attn_dp == 1`**) | The first four are the candidate's. The fifth is decision D1's "reject unsupported configurations explicitly", and the dense multi-lane row above is the measurement behind it. | +| Second step counter | **no** | D1 forbids broadening W4 with a new counter. W3's identity is reused where it holds. | +| Runtime order assertion | **no** | The reference warns and applies the counts. W4 mirrors the warning. Key equality per shared forward is a test invariant, not a runtime abort. | +| `waiting + 1` reservation | keep | One modeled frontend, so `client_count == 1`. The one-frontend restriction is stated in the class docstring. | +| Heartbeat events | **none** | Timers advance lazily inside `select`/`report`. The policy creates no events, so it cannot keep a drained simulation alive — asserted in a test rather than assumed. | +| `get_request_load` placement | next to the existing `_get_num_waiting_reqs_for_decision_log` in `vllm_v1_iteration_policy.py`, delegating to it | The donor deleted that helper and broke `sglang_style_replica_scheduler.py`; this branch's split already moved it into the shared policy mixin and pinned it with a boundary test. Delegating gives one definition of "waiting" for the balancer and both decision-log emitters, without churning that boundary. | +| Config module extraction | **already done** | The merged module split created `frontier/config/cluster_scheduler_config.py`. W4 only adds one dataclass to it, which is a registry entry through an unchanged mechanism. | + +## Planned edits + +| File | Change | +| --- | --- | +| `frontier/types/cluster_scheduler_type.py` | `VLLM_LOAD_BALANCING = 5`. | +| `frontier/config/cluster_scheduler_config.py` | `VllmLoadBalancingClusterSchedulerConfig`, no fields. | +| `frontier/scheduler/cluster_scheduler/cluster_scheduler_registry.py` | One registry entry. | +| `frontier/scheduler/request_load.py` (new) | `RequestLoad(NamedTuple)` with `waiting`, `running`. | +| `frontier/scheduler/utils/vllm_dp_load_balancer.py` (new) | The pure state machine: `report`, `select`, lazy timer advance, reference-mirrored warning. | +| `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py` (new) | Topology guard, `schedule_at`, `schedule`, `on_replica_batch_end`. | +| `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py` | Two inert default seams: `schedule_at(time)` delegating to `schedule()`, `on_replica_batch_end(...)` returning `None`. | +| `frontier/scheduler/replica_scheduler/base_replica_scheduler.py` | `get_request_load()` raising `NotImplementedError` with the concrete class name. | +| `frontier/scheduler/replica_scheduler/vllm_v1_iteration_policy.py` | `get_request_load()` delegating to the existing waiting helper; both decision-log payloads consume it. | +| `frontier/events/cluster_schedule_event.py` | `schedule()` → `schedule_at(self.time)`. | +| `frontier/events/global_batch_end_event.py` | Call `on_replica_batch_end` after the request-state transition in `on_batch_end`. | + +## Fidelity expectation, stated before measuring + +The policy is opt-in and no existing configuration selects it. The two seams are +inert for the five existing policies: `schedule_at` delegates to the same +`schedule()` the event called before, and `on_replica_batch_end` returns `None`. +The decision-log payloads keep identical values because `get_request_load` +delegates to the same helper they already called. + +**Prediction: all 71 fidelity cases stay exactly equal.** Anything that moves is +a defect in the seams, not an approved behavior change — unlike W2 and W3, W4 +has no reachable fidelity fix, so a single mismatch falsifies the change rather +than confirming it. diff --git a/tests/integration/test_vllm_dp_placement_runtime.py b/tests/integration/test_vllm_dp_placement_runtime.py new file mode 100644 index 00000000..6825c21e --- /dev/null +++ b/tests/integration/test_vllm_dp_placement_runtime.py @@ -0,0 +1,491 @@ +"""Real-runtime wiring for the opt-in vLLM-style DP placement policy. + +The state machine itself is covered by `tests/unit/test_vllm_dp_load_balancer.py` +against reference-derived expectations. What can only be shown by running the +simulator is the wiring: that `ClusterScheduleEvent` is what supplies the +routing time, that `GlobalBatchEndEvent` reports the lane's **post**-step load, +that the report key is ordered the way the policy's guard assumes, and that the +policy introduces no event that keeps a drained run alive. + +The child process runs two configurations that differ only in the cluster +scheduler policy, so the comparison isolates the policy. Execution time comes +from the dummy predictor: placement here is decided by the balancer, not by +latency realism, and both policies see the same durations. +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path +import subprocess +import sys + +import pytest + + +def _run_child(tmp_path: Path, case: str) -> dict: + repo_root = Path(__file__).resolve().parents[2] + result = subprocess.run( + [sys.executable, str(Path(__file__).resolve()), str(tmp_path), case], + env={ + **os.environ, + "OMP_NUM_THREADS": "1", + "OPENBLAS_NUM_THREADS": "1", + "PYTHONPATH": os.pathsep.join( + [str(repo_root), os.environ.get("PYTHONPATH", "")] + ).rstrip(os.pathsep), + }, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + timeout=900, + ) + (tmp_path / f"{case}.log").write_text(result.stdout) + assert result.returncode == 0, result.stdout[-15000:] + return json.loads((tmp_path / f"{case}_evidence.json").read_text()) + + +def test_dp_placement_runs_and_reports_post_step_load(tmp_path): + evidence = _run_child(tmp_path, "moe_dp2") + policy = evidence["vllm_load_balancing"] + baseline = evidence["round_robin"] + + # The run has to reach the shape under test. + assert policy["num_lanes"] == 2 + assert policy["completed_requests"] == policy["num_requests"] + + # 1. The routing time comes from ClusterScheduleEvent, not retained state. + # Every recorded routing time is one of the cluster schedule times, and + # the policy's own `schedule()` would have raised had it been used. + assert policy["routing_times"] + assert policy["routing_times"] == policy["cluster_schedule_times"] + + # 2. Placement, derived independently: four requests arrive together, no + # report can precede the first routing, so the estimate is empty and only + # the local reservation moves the choice. + assert policy["first_four_lanes"] == [0, 1, 0, 1] + + # 3. The report key is ordered as the capability guard assumes: never + # decreasing, and equal only for peer lanes of one shared forward. + keys = policy["report_keys"] + assert keys == sorted(keys) + for key, lanes in policy["lanes_by_key"].items(): + assert len(lanes) == len(set(lanes)), (key, lanes) + + # 4. The reported load is the post-step state. The lane's own release is + # bracketed, so pre- and post-step load are distinct values for at least + # some batches; every report matches the post-step one and the reports are + # not merely the pre-step values. + assert policy["reports_after_the_lane_released_the_batch"] == policy["num_reports"] + assert policy["reports_where_the_release_changed_the_load"] > 0 + assert policy["reports_matching_post_step"] == policy["num_reports"] + assert policy["reports_matching_pre_step"] < policy["num_reports"] + + # 5. No event type is introduced, and the run drains rather than being kept + # alive by a heartbeat. + assert set(policy["event_types"]) == set(baseline["event_types"]) + assert policy["makespan"] > 0 + assert baseline["completed_requests"] == baseline["num_requests"] + + +def test_placement_follows_published_load_where_round_robin_cannot(tmp_path): + """Spread the arrivals so snapshots land between them, and skew the load. + + This is the discriminating case: the two policies see identical arrivals, + identical durations and identical lane capacity, so any difference in + placement comes from reading the published load. + """ + + evidence = _run_child(tmp_path, "moe_dp2_online") + policy = evidence["vllm_load_balancing"] + baseline = evidence["round_robin"] + + assert policy["completed_requests"] == policy["num_requests"] == 6 + assert baseline["completed_requests"] == baseline["num_requests"] == 6 + + # 1. The routing time is the cluster schedule time, now at several distinct + # instants rather than one, so a retained or stale time would show up. + assert len(set(policy["cluster_schedule_times"])) > 1 + assert policy["routing_times"] == policy["cluster_schedule_times"] + assert baseline["routing_times"] == baseline["cluster_schedule_times"] + + # 2. Round-robin alternates because it cannot see load. The policy places + # strictly fewer requests on the lane that is still draining the one long + # request, which is the whole point of reading the snapshot. + assert baseline["placements"] == [0, 1, 0, 1, 0, 1] + assert policy["placements"] != baseline["placements"] + assert policy["placements"][:2] == [0, 1] + assert policy["placements"].count(0) < baseline["placements"].count(0) + + # 3. The report key stays ordered across a much longer run, and peer lanes of + # one shared forward remain the only source of equal keys. + keys = policy["report_keys"] + assert keys == sorted(keys) + assert len(keys) > 10 + for key, lanes in policy["lanes_by_key"].items(): + assert len(lanes) == len(set(lanes)), (key, lanes) + + # 4. Still the post-step load, and still no new event type. + assert policy["reports_after_the_lane_released_the_batch"] == policy["num_reports"] + assert policy["reports_where_the_release_changed_the_load"] > 0 + assert policy["reports_matching_post_step"] == policy["num_reports"] + assert policy["reports_matching_pre_step"] < policy["num_reports"] + assert set(policy["event_types"]) == set(baseline["event_types"]) + + +def test_a_single_lane_shape_routes_everything_to_lane_zero(tmp_path): + evidence = _run_child(tmp_path, "dense_dp1") + policy = evidence["vllm_load_balancing"] + + assert policy["num_lanes"] == 1 + assert policy["completed_requests"] == policy["num_requests"] + assert set(policy["first_four_lanes"]) == {0} + + +# --------------------------------------------------------------------------- +# Child process +# --------------------------------------------------------------------------- + +REQUEST_SHAPES = 4 + + +def _model(*, is_moe: bool): + from frontier.config import BaseModelConfig + from frontier.types import ActivationType, NormType + + model = BaseModelConfig( + num_layers=4, + num_q_heads=4, + num_kv_heads=2, + embedding_dim=256, + mlp_hidden_dim=64, + max_position_embeddings=4096, + use_gated_mlp=True, + use_bias=False, + use_qkv_bias=False, + activation=ActivationType.SILU, + norm=NormType.RMS_NORM, + post_attn_norm=True, + vocab_size=1024, + is_moe=is_moe, + num_experts=8 if is_moe else 0, + num_experts_per_tok=2 if is_moe else 0, + torch_dtype="bfloat16", + ) + model._model_name = f"w4_runtime_{'moe' if is_moe else 'dense'}" + return model + + +def _config( + root: Path, + patch, + *, + is_moe: bool, + attn_dp: int, + moe_ep: int, + policy, + trace: str | None = None, +): + from frontier.config import ( + BaseModelConfig, + ClusterConfig, + FixedRequestLengthGeneratorConfig, + MetricsConfig, + PoissonRequestIntervalGeneratorConfig, + RandomForrestExecutionTimePredictorConfig, + ReplicaConfig, + SimulationConfig, + SyntheticRequestGeneratorConfig, + TraceRequestGeneratorConfig, + VllmV1SchedulerConfig, + ) + + model = _model(is_moe=is_moe) + original = BaseModelConfig.create_from_name + patch.setattr( + BaseModelConfig, + "create_from_name", + classmethod( + lambda cls, name: model if name == model._model_name else original(name) + ), + ) + moe_fields = ( + dict( + moe_tensor_parallel_size=1, + moe_expert_parallel_size=moe_ep, + total_expert_num=8, + router_topk=2, + ) + if is_moe + else {} + ) + replica = ReplicaConfig( + model_name=model._model_name, + device="a100", + network_device="a100_pairwise_nvlink", + num_pipeline_stages=1, + attn_tensor_parallel_size=1, + attn_dp=attn_dp, + memory_margin_fraction=0.1, + **moe_fields, + ) + cluster = ClusterConfig( + replica_config=replica, + replica_scheduler_config=VllmV1SchedulerConfig( + num_blocks=128, + block_size=16, + batch_size_cap=4, + max_tokens_in_batch=16, + enable_chunked_prefill=True, + ), + cluster_scheduler_config=policy(), + # Placement here is decided by the balancer, not by latency realism, so + # the predictor only has to be deterministic and identical across the + # two policies being compared. + execution_time_predictor_config=RandomForrestExecutionTimePredictorConfig( + enable_dummy_mode=True + ), + ) + generator = ( + TraceRequestGeneratorConfig(trace_file=trace) + if trace is not None + else SyntheticRequestGeneratorConfig( + num_requests=REQUEST_SHAPES, + length_generator_config=FixedRequestLengthGeneratorConfig( + prefill_tokens=16, decode_tokens=3 + ), + interval_generator_config=PoissonRequestIntervalGeneratorConfig(qps=1e6), + ) + ) + return SimulationConfig( + simulation_mode="online" if trace is not None else "offline", + sys_arch="co-location", + enable_parallel_clusters=False, + decode_cuda_graph_mode="none", + cluster_config=cluster, + metrics_config=MetricsConfig( + output_dir=str(root / "metrics"), + cache_dir=str(root / "cache"), + run_id="dp_placement", + write_metrics=False, + store_request_metrics=False, + store_batch_metrics=False, + store_operation_metrics=False, + store_utilization_metrics=False, + store_plots=False, + enable_chrome_trace=False, + write_json_trace=False, + ), + request_generator_config=generator, + ) + + +def run_case( + root: Path, + *, + is_moe: bool, + attn_dp: int, + moe_ep: int, + policy_name: str, + trace: str | None = None, +): + """Run one configuration and return what only the event loop can show.""" + + from frontier.config import ( + RoundRobinClusterSchedulerConfig, + VllmLoadBalancingClusterSchedulerConfig, + ) + from frontier.events.cluster_schedule_event import ClusterScheduleEvent + from frontier.events.global_batch_end_event import GlobalBatchEndEvent + from frontier.scheduler.cluster_scheduler.base_cluster_scheduler import ( + BaseClusterScheduler, + ) + from frontier.scheduler.cluster_scheduler.vllm_load_balancing_cluster_scheduler import ( # noqa: E501 + VllmLoadBalancingClusterScheduler, + ) + from frontier.scheduler.replica_scheduler.vllm_v1_engine_replica_scheduler import ( # noqa: E501 + VLLMv1EngineReplicaScheduler, + ) + from frontier.scheduler.utils.forward_sync_state import ForwardSyncState + from frontier.simulator import Simulator + + policy = { + "vllm_load_balancing": VllmLoadBalancingClusterSchedulerConfig, + "round_robin": RoundRobinClusterSchedulerConfig, + }[policy_name] + + cluster_schedule_times: list[float] = [] + routing_times: list[float] = [] + placements: list[int] = [] + reports: list[dict] = [] + releases: dict[int, dict] = {} + event_types: set[str] = set() + + original_cluster_schedule = ClusterScheduleEvent.handle_event + original_batch_end = GlobalBatchEndEvent.handle_event + + def observed_cluster_schedule(self, scheduler, metrics_store): + cluster_schedule_times.append(float(self.time)) + events = original_cluster_schedule(self, scheduler, metrics_store) + event_types.update(type(event).__name__ for event in events or []) + return events + + def observing_schedule_at(original): + def observed(self, time): + routing_times.append(float(time)) + mapping = original(self, time) + placements.extend(lane for _, lane, _ in mapping) + return mapping + + return observed + + def observing_release(original): + """Bracket the lane's own release so pre- and post-step load differ.""" + + def observed(self, batch): + before = list(self.get_request_load()) + result = original(self, batch) + releases[batch.id] = { + "before": before, + "after": list(self.get_request_load()), + } + return result + + return observed + + def observing_hook(original): + def observed(self, time, replica_id, replica_local_id, batch): + result = original(self, time, replica_id, replica_local_id, batch) + balancer = getattr(self, "_load_balancer", None) + if balancer is not None and type(replica_local_id) is int: + # `None` means the lane had not yet released this batch when the + # hook ran, which is itself the ordering evidence. + release = releases.get(batch.id) + reports.append( + { + "lane": replica_local_id, + "key": ForwardSyncState.get_step_id(batch), + "released": release is not None, + "pre_step": release["before"] if release else None, + "post_step": release["after"] if release else None, + "reported": list(balancer.engine_counts[replica_local_id]), + } + ) + return result + + return observed + + def observed_batch_end(self, scheduler, metrics_store): + events = original_batch_end(self, scheduler, metrics_store) + event_types.update(type(event).__name__ for event in events or []) + return events + + with pytest.MonkeyPatch.context() as patch: + config = _config( + root, + patch, + is_moe=is_moe, + attn_dp=attn_dp, + moe_ep=moe_ep, + policy=policy, + trace=trace, + ) + patch.setattr(ClusterScheduleEvent, "handle_event", observed_cluster_schedule) + patch.setattr(GlobalBatchEndEvent, "handle_event", observed_batch_end) + patch.setattr( + VLLMv1EngineReplicaScheduler, + "on_batch_end", + observing_release(vars(VLLMv1EngineReplicaScheduler)["on_batch_end"]), + ) + # Both the inert base seam and the policy's override have to be + # wrapped: patching only the base would silently observe nothing on the + # very policy under test. + for owner in (BaseClusterScheduler, VllmLoadBalancingClusterScheduler): + if "schedule_at" in vars(owner): + patch.setattr( + owner, + "schedule_at", + observing_schedule_at(vars(owner)["schedule_at"]), + ) + if "on_replica_batch_end" in vars(owner): + patch.setattr( + owner, + "on_replica_batch_end", + observing_hook(vars(owner)["on_replica_batch_end"]), + ) + simulator = Simulator(config) + simulator.run() + requests = list(simulator._all_requests) + + lanes_by_key: dict[str, list[int]] = {} + for report in reports: + lanes_by_key.setdefault(str(report["key"]), []).append(report["lane"]) + + return { + "num_lanes": attn_dp, + "num_requests": len(requests), + "completed_requests": sum(1 for request in requests if request.completed), + "makespan": max((request.completed_at for request in requests), default=0.0), + "cluster_schedule_times": cluster_schedule_times, + "routing_times": routing_times, + "first_four_lanes": placements[:4], + "placements": placements, + "report_keys": [report["key"] for report in reports], + "lanes_by_key": lanes_by_key, + "num_reports": len(reports), + "reports_after_the_lane_released_the_batch": sum( + 1 for report in reports if report["released"] + ), + "reports_matching_post_step": sum( + 1 for report in reports if report["post_step"] == report["reported"] + ), + "reports_matching_pre_step": sum( + 1 for report in reports if report["pre_step"] == report["reported"] + ), + "reports_where_the_release_changed_the_load": sum( + 1 for report in reports if report["pre_step"] != report["post_step"] + ), + "event_types": sorted(event_types), + } + + +# One long request holds lane 0 while the rest are short, and the arrivals are +# spread far enough apart that published snapshots land between them. Load-blind +# round-robin cannot react to that asymmetry; a load-balancing policy can. +ASYMMETRIC_TRACE = """arrived_at,num_prefill_tokens,num_decode_tokens +0.0,16,40 +0.0,16,1 +0.4,16,1 +0.6,16,1 +0.8,16,1 +1.0,16,1 +""" + + +def _trace_file(root: Path) -> str: + path = root / "asymmetric_arrivals.csv" + path.write_text(ASYMMETRIC_TRACE) + return str(path) + + +if __name__ == "__main__": + root = Path(sys.argv[1]) + case = sys.argv[2] + shape = { + "moe_dp2": dict(is_moe=True, attn_dp=2, moe_ep=2), + "dense_dp1": dict(is_moe=False, attn_dp=1, moe_ep=1), + "moe_dp2_online": dict(is_moe=True, attn_dp=2, moe_ep=2, trace=True), + }[case] + evidence = {} + for policy_name in ("vllm_load_balancing", "round_robin"): + case_root = root / case / policy_name + case_root.mkdir(parents=True, exist_ok=True) + arguments = dict(shape) + if arguments.pop("trace", False): + arguments["trace"] = _trace_file(case_root) + evidence[policy_name] = run_case( + case_root, policy_name=policy_name, **arguments + ) + (root / f"{case}_evidence.json").write_text(json.dumps(evidence, indent=1)) + print(json.dumps({k: v["completed_requests"] for k, v in evidence.items()})) diff --git a/tests/unit/test_vllm_dp_load_balancer.py b/tests/unit/test_vllm_dp_load_balancer.py new file mode 100644 index 00000000..08dbcf16 --- /dev/null +++ b/tests/unit/test_vllm_dp_load_balancer.py @@ -0,0 +1,757 @@ +"""Behavior tests for the opt-in vLLM-style DP placement policy. + +Three layers are covered here: the pure selection and publication state +machine, the real `vllm_v1` load accessor it consumes, and the policy's +capability guard built through the real cluster-scheduler constructor. The +real-runtime evidence -- that `ClusterScheduleEvent` supplies the time, that +`GlobalBatchEndEvent` reports post-step load, and that the policy keeps no +simulation alive -- lives in +`tests/integration/test_vllm_dp_placement_runtime.py`. + +Expected values are derived from vLLM v0.10.2 `core_client.py` and +`coordinator.py`, cited in the module under test. They are not produced by +re-running the implementation. +""" + +from __future__ import annotations + +import sys + +import pytest + +from frontier.config import ( + BaseModelConfig, + ClusterConfig, + FixedRequestLengthGeneratorConfig, + LORClusterSchedulerConfig, + MetricsConfig, + PoissonRequestIntervalGeneratorConfig, + RandomClusterSchedulerConfig, + ReplicaConfig, + RoundRobinClusterSchedulerConfig, + SarathiSchedulerConfig, + SimulationConfig, + StickyLORClusterSchedulerConfig, + StickyRoundRobinClusterSchedulerConfig, + SyntheticRequestGeneratorConfig, + VllmLoadBalancingClusterSchedulerConfig, + VllmV1SchedulerConfig, +) +from frontier.config.cluster_scheduler_config import BaseClusterSchedulerConfig +from frontier.config.flat_dataclass import create_flat_dataclass +from frontier.config.utils import get_all_subclasses +from frontier.entities import Cluster, Request +from frontier.scheduler.cluster_scheduler.base_cluster_scheduler import ( + BaseClusterScheduler, +) +from frontier.scheduler.cluster_scheduler.cluster_scheduler_registry import ( + ClusterSchedulerRegistry, +) +from frontier.scheduler.cluster_scheduler.lor_cluster_scheduler import ( + LORClusterScheduler, +) +from frontier.scheduler.cluster_scheduler.random_cluster_scheduler import ( + RandomClusterScheduler, +) +from frontier.scheduler.cluster_scheduler.round_robin_cluster_scheduler import ( + RoundRobinClusterScheduler, +) +from frontier.scheduler.cluster_scheduler.sticky_lor_cluster_scheduler import ( + StickyLORClusterScheduler, +) +from frontier.scheduler.cluster_scheduler.sticky_round_robin_cluster_scheduler import ( + StickyRoundRobinClusterScheduler, +) +from frontier.scheduler.replica_scheduler.base_replica_scheduler import ( + BaseReplicaScheduler, +) +from frontier.scheduler.request_load import RequestLoad +from frontier.scheduler.utils.vllm_dp_load_balancer import VllmDPLoadBalancer +from frontier.types import ( + ActivationType, + ClusterSchedulerType, + ClusterType, + NormType, +) + + +# -------------------------------------------------------------------------- +# Selection: reference core_client.py:1139-1153 +# -------------------------------------------------------------------------- + + +def test_selection_weights_waiting_four_times_against_running() -> None: + balancer = VllmDPLoadBalancer(2) + # Engine 0 scores 4*1 + 0 = 4; engine 1 scores 4*0 + 3 = 3. + balancer.frontend_counts = [RequestLoad(1, 0), RequestLoad(0, 3)] + + assert balancer.select(0.0) == 1 + + +def test_selection_weight_is_exactly_four_at_the_boundary() -> None: + balancer = VllmDPLoadBalancer(2) + # 4*1 + 0 == 4*0 + 4, so the tie falls to the lowest index. + balancer.frontend_counts = [RequestLoad(1, 0), RequestLoad(0, 4)] + assert balancer.select(0.0) == 0 + + balancer = VllmDPLoadBalancer(2) + # One more running unit on engine 1 breaks the tie the other way. + balancer.frontend_counts = [RequestLoad(1, 0), RequestLoad(0, 5)] + assert balancer.select(0.0) == 0 + + balancer = VllmDPLoadBalancer(2) + balancer.frontend_counts = [RequestLoad(1, 1), RequestLoad(0, 4)] + assert balancer.select(0.0) == 1 + + +def test_equal_scores_always_choose_the_lowest_engine_index() -> None: + balancer = VllmDPLoadBalancer(4) + balancer.frontend_counts = [RequestLoad(2, 1)] * 4 + + assert balancer.select(0.0) == 0 + + +def test_local_reservations_spread_requests_between_snapshots() -> None: + balancer = VllmDPLoadBalancer(2) + + # Counts start empty, and nothing is published in between, so the local + # waiting reservation is the only thing that moves the choice. + assert [balancer.select(0.0) for _ in range(5)] == [0, 1, 0, 1, 0] + + +def test_a_published_snapshot_replaces_local_reservations() -> None: + balancer = VllmDPLoadBalancer(2) + balancer.select(0.0) + balancer.select(0.0) + assert balancer.frontend_counts == [RequestLoad(1, 0), RequestLoad(1, 0)] + + # A report makes engine 1 the loaded one, and the publish that follows + # replaces the estimate wholesale rather than adjusting the reservations. + balancer.report(0.01, 1, 0, RequestLoad(0, 6)) + assert balancer.select(1.0) == 0 + assert balancer.frontend_counts == [RequestLoad(1, 0), RequestLoad(0, 6)] + + +def test_one_engine_always_selects_lane_zero() -> None: + balancer = VllmDPLoadBalancer(1) + + assert [balancer.select(0.0), balancer.select(0.5), balancer.select(9.0)] == [0] * 3 + + +# -------------------------------------------------------------------------- +# Publication: reference coordinator.py:192-218, :293-310 +# -------------------------------------------------------------------------- + + +def test_the_first_publish_waits_only_the_collection_interval() -> None: + balancer = VllmDPLoadBalancer(2) + + # 50 ms, because the reference's `wait_for - elapsed` is deeply negative on + # its first iteration and `min_timeout` decides. + assert balancer.next_publish_ms == 50 + + +def test_changed_counts_republish_after_the_changed_interval() -> None: + balancer = VllmDPLoadBalancer(2) + balancer.report(0.0, 0, 0, RequestLoad(1, 0)) + # First publish consumes the collection wait. + balancer.select(0.06) + assert balancer.last_publish_ms == 50 + + balancer.report(0.06, 0, 1, RequestLoad(2, 0)) + + # stats_changed, so 100 ms after the last publish. + assert balancer.next_publish_ms == 150 + + +def test_unchanged_counts_republish_after_the_idle_interval() -> None: + balancer = VllmDPLoadBalancer(2) + balancer.report(0.0, 0, 0, RequestLoad(1, 0)) + balancer.select(0.06) + + # The publish cleared stats_changed, so the next deadline is the idle one. + assert balancer.stats_changed is False + assert balancer.next_publish_ms == 50 + 5000 + + +def test_a_newer_step_latches_the_previous_counts_when_changes_are_pending() -> None: + balancer = VllmDPLoadBalancer(2) + balancer.report(0.0, 0, 0, RequestLoad(1, 0)) + assert balancer.last_step_counts is None + + balancer.report(0.001, 1, 1, RequestLoad(5, 5)) + + # The step advanced with an unpublished change, so the counts as of the + # previous step are held back for the next publish. + assert balancer.last_step_counts == [RequestLoad(1, 0), RequestLoad(0, 0)] + assert balancer.engine_counts == [RequestLoad(1, 0), RequestLoad(5, 5)] + + # That snapshot, not the newer counts, is what the frontend sees first. + # Under the previous-step snapshot engine 1 is empty and wins; under the + # newer counts engine 0 would have won with 4 against 25. + assert balancer.select(0.06) == 1 + assert balancer.last_step_counts is None + # The published snapshot plus this selection's local reservation. + assert balancer.frontend_counts == [RequestLoad(1, 0), RequestLoad(1, 0)] + + +def test_a_newer_step_without_pending_changes_latches_nothing() -> None: + balancer = VllmDPLoadBalancer(2) + balancer.report(0.0, 0, 0, RequestLoad(1, 0)) + balancer.select(0.06) + assert balancer.stats_changed is False + + balancer.report(0.06, 0, 4, RequestLoad(2, 0)) + + assert balancer.last_step_counts is None + assert balancer.last_report_step == 4 + + +def test_peer_engines_reporting_one_forward_do_not_latch_a_snapshot() -> None: + """Both attention-DP lanes of one shared forward carry the same key.""" + + balancer = VllmDPLoadBalancer(2) + balancer.report(0.0, 0, 3, RequestLoad(1, 0)) + assert balancer.last_report_step == 3 + + balancer.report(0.0, 1, 3, RequestLoad(0, 2)) + + # The equal key takes neither branch, exactly as coordinator.py:293-300. + assert balancer.last_step_counts is None + assert balancer.last_report_step == 3 + assert balancer.engine_counts == [RequestLoad(1, 0), RequestLoad(0, 2)] + + +def test_an_unchanged_report_changes_nothing_at_all() -> None: + balancer = VllmDPLoadBalancer(2) + balancer.report(0.0, 0, 0, RequestLoad(1, 1)) + before = ( + list(balancer.engine_counts), + balancer.last_report_step, + balancer.next_publish_ms, + balancer.stats_changed, + ) + + # The reference engine compares against its own last counts and sends no + # message, so nothing downstream moves -- not even the step latch. + balancer.report(0.02, 0, 9, RequestLoad(1, 1)) + + assert ( + list(balancer.engine_counts), + balancer.last_report_step, + balancer.next_publish_ms, + balancer.stats_changed, + ) == before + + +def test_an_out_of_order_step_warns_and_still_applies_the_counts(caplog) -> None: + balancer = VllmDPLoadBalancer(2) + balancer.report(0.0, 0, 7, RequestLoad(1, 0)) + + with caplog.at_level("WARNING"): + balancer.report(0.0, 1, 2, RequestLoad(3, 4)) + + assert "out-of-order step 2" in caplog.text + # Applied unconditionally, as coordinator.py:308-310 does. Nothing aborts. + assert balancer.engine_counts[1] == RequestLoad(3, 4) + assert balancer.last_report_step == 7 + + +# -------------------------------------------------------------------------- +# Ordering and time validation +# -------------------------------------------------------------------------- + + +def test_a_report_on_its_deadline_is_processed_before_that_publish() -> None: + balancer = VllmDPLoadBalancer(2) + balancer.report(0.0, 0, 0, RequestLoad(1, 0)) + assert balancer.next_publish_ms == 50 + + # A real poller returns the waiting message instead of timing out, so the + # report is applied and then moves the deadline. + balancer.report(0.05, 1, 1, RequestLoad(9, 9)) + + assert balancer.last_publish_ms == -5000 + assert balancer.engine_counts[1] == RequestLoad(9, 9) + assert balancer.last_step_counts == [RequestLoad(1, 0), RequestLoad(0, 0)] + + +def test_a_selection_on_a_passed_deadline_sees_the_new_snapshot() -> None: + balancer = VllmDPLoadBalancer(2) + balancer.report(0.0, 1, 0, RequestLoad(0, 6)) + assert balancer.frontend_counts == [RequestLoad(0, 0), RequestLoad(0, 0)] + + # At the deadline itself a selection publishes first, then chooses. + assert balancer.select(0.05) == 0 + assert balancer.last_publish_ms == 50 + + +def test_several_reports_inside_one_millisecond_stay_in_one_publish_window() -> None: + balancer = VllmDPLoadBalancer(3) + for engine in range(3): + balancer.report(0.0001 * engine, engine, 0, RequestLoad(engine + 1, 0)) + + assert balancer.time_ms == 0 + assert balancer.last_publish_ms == -5000 + assert balancer.engine_counts == [ + RequestLoad(1, 0), + RequestLoad(2, 0), + RequestLoad(3, 0), + ] + + +def test_time_must_not_move_backwards() -> None: + balancer = VllmDPLoadBalancer(2) + balancer.select(1.0) + + with pytest.raises(ValueError, match="cannot move backwards"): + balancer.select(0.5) + + +@pytest.mark.parametrize("time", [-1.0, float("nan"), float("inf")]) +def test_time_must_be_finite_and_nonnegative(time) -> None: + balancer = VllmDPLoadBalancer(2) + + with pytest.raises(ValueError, match="finite nonnegative time"): + balancer.select(time) + + +@pytest.mark.parametrize("engine", [-1, 2, True, 1.0, None]) +def test_an_unknown_engine_is_rejected(engine) -> None: + balancer = VllmDPLoadBalancer(2) + + with pytest.raises(ValueError, match="unknown engine"): + balancer.report(0.0, engine, 0, RequestLoad(1, 0)) + + +@pytest.mark.parametrize( + ("step", "load"), + [(-1, RequestLoad(1, 0)), (0, RequestLoad(-1, 0)), (0, RequestLoad(0, -2))], +) +def test_negative_steps_and_counts_are_rejected(step, load) -> None: + balancer = VllmDPLoadBalancer(2) + + with pytest.raises(ValueError, match="nonnegative"): + balancer.report(0.0, 0, step, load) + + +@pytest.mark.parametrize("num_engines", [0, -1, 1.0, None]) +def test_the_engine_count_must_be_a_positive_int(num_engines) -> None: + with pytest.raises(ValueError, match="at least one engine"): + VllmDPLoadBalancer(num_engines) + + +# -------------------------------------------------------------------------- +# The load accessor, on the real vllm_v1 scheduler +# -------------------------------------------------------------------------- + + +def _model(*, is_moe: bool) -> BaseModelConfig: + model = BaseModelConfig( + num_layers=4, + num_q_heads=4, + num_kv_heads=2, + embedding_dim=256, + mlp_hidden_dim=64, + max_position_embeddings=4096, + use_gated_mlp=True, + use_bias=False, + use_qkv_bias=False, + activation=ActivationType.SILU, + norm=NormType.RMS_NORM, + post_attn_norm=True, + vocab_size=1024, + is_moe=is_moe, + num_experts=8 if is_moe else 0, + num_experts_per_tok=2 if is_moe else 0, + torch_dtype="bfloat16", + ) + model._model_name = f"w4_dp_{'moe' if is_moe else 'dense'}" + return model + + +def _policy_scheduler( + patch, + *, + is_moe: bool = True, + attn_dp: int = 2, + moe_ep: int = 2, + num_replicas: int = 1, + num_pipeline_stages: int = 1, + cluster_type: ClusterType = ClusterType.MONOLITHIC, + replica_scheduler_config=None, + cluster_scheduler_config=None, +): + """Build a real cluster scheduler through the real constructor path.""" + + model = _model(is_moe=is_moe) + original = BaseModelConfig.create_from_name + patch.setattr( + BaseModelConfig, + "create_from_name", + classmethod( + lambda cls, name: model if name == model._model_name else original(name) + ), + ) + moe_fields = ( + dict( + moe_tensor_parallel_size=1, + moe_expert_parallel_size=moe_ep, + total_expert_num=8, + router_topk=2, + ) + if is_moe + else {} + ) + replica_config = ReplicaConfig( + model_name=model._model_name, + device="a100", + network_device="a100_pairwise_nvlink", + num_pipeline_stages=num_pipeline_stages, + attn_tensor_parallel_size=1, + attn_dp=attn_dp, + memory_margin_fraction=0.1, + **moe_fields, + ) + generator_config = SyntheticRequestGeneratorConfig( + num_requests=2, + length_generator_config=FixedRequestLengthGeneratorConfig( + prefill_tokens=8, decode_tokens=2 + ), + interval_generator_config=PoissonRequestIntervalGeneratorConfig(qps=1.0), + ) + policy_config = cluster_scheduler_config or VllmLoadBalancingClusterSchedulerConfig() + cluster_config = ClusterConfig( + cluster_type=cluster_type, + num_replicas=num_replicas, + replica_config=replica_config, + replica_scheduler_config=replica_scheduler_config + or VllmV1SchedulerConfig( + num_blocks=64, + block_size=16, + batch_size_cap=4, + max_tokens_in_batch=16, + enable_chunked_prefill=True, + ), + cluster_scheduler_config=policy_config, + ) + metrics_config = MetricsConfig( + write_metrics=False, + store_plots=False, + enable_chrome_trace=False, + write_json_trace=False, + ) + cluster = Cluster(cluster_config, metrics_config, generator_config) + return ClusterSchedulerRegistry.get( + policy_config.get_type(), + config=cluster_config, + cluster=cluster, + request_generator_config=generator_config, + predictor=None, + ) + + +def _request(tokens: int = 8) -> Request: + return Request(arrived_at=0.0, num_prefill_tokens=tokens, num_decode_tokens=2) + + +def test_the_load_accessor_separates_waiting_from_admitted_running(monkeypatch) -> None: + with pytest.MonkeyPatch.context() as patch: + scheduler = _policy_scheduler(patch) + lane = scheduler.get_replica_scheduler(scheduler._serving_replica_id, 0) + + assert lane.get_request_load() == RequestLoad(0, 0) + + for _ in range(3): + lane.add_request(_request()) + # Queued but not admitted: waiting only. + assert lane.get_request_load() == RequestLoad(3, 0) + + batch = lane.on_schedule(0.0) + load = lane.get_request_load() + # Admission moves requests into running; the accessor must not count a + # request twice, and the two populations must add up. + assert load.running == len(lane._running_requests) + assert load.waiting == len(lane._request_queue) + len( + lane._preempted_requests + ) + assert load.running + load.waiting == 3 + assert batch is not None + + +def test_a_preempted_request_is_waiting_again(monkeypatch) -> None: + with pytest.MonkeyPatch.context() as patch: + scheduler = _policy_scheduler(patch) + lane = scheduler.get_replica_scheduler(scheduler._serving_replica_id, 0) + for _ in range(2): + lane.add_request(_request()) + lane.on_schedule(0.0) + admitted = lane.get_request_load() + + victim = lane._running_requests[0] + lane._preempted_requests.append(victim) + lane._running_requests.remove(victim) + + assert lane.get_request_load() == RequestLoad( + admitted.waiting + 1, admitted.running - 1 + ) + + +def test_a_scheduler_without_a_serving_load_definition_says_which_one() -> None: + class _Bare(BaseReplicaScheduler): + def _get_next_batch(self, *args, **kwargs): + return None + + def on_batch_end(self, *args, **kwargs): + return None + + bare = _Bare.__new__(_Bare) + + with pytest.raises(NotImplementedError, match="_Bare does not expose"): + bare.get_request_load() + + +# -------------------------------------------------------------------------- +# Capability guard +# -------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + ("label", "kwargs", "message"), + [ + ("two_replicas", dict(num_replicas=2), "one co-location Replica"), + ("pipeline_parallel", dict(num_pipeline_stages=2), "one co-location Replica"), + ( + "wrong_replica_scheduler", + dict(replica_scheduler_config=SarathiSchedulerConfig( + num_blocks=64, block_size=16, batch_size_cap=4, + chunk_size=16, + )), + "one co-location Replica", + ), + ( + "dense_multi_lane", + dict(is_moe=False, attn_dp=2), + "monotonic per Replica", + ), + ], +) +def test_each_unsupported_topology_is_rejected_at_construction( + label, kwargs, message +) -> None: + with pytest.MonkeyPatch.context() as patch: + with pytest.raises(ValueError, match=message): + _policy_scheduler(patch, **kwargs) + + +@pytest.mark.parametrize( + ("is_moe", "attn_dp", "moe_ep"), + [(True, 2, 2), (True, 1, 1), (False, 1, 1)], +) +def test_the_supported_shapes_construct(is_moe, attn_dp, moe_ep) -> None: + with pytest.MonkeyPatch.context() as patch: + scheduler = _policy_scheduler( + patch, is_moe=is_moe, attn_dp=attn_dp, moe_ep=moe_ep + ) + + assert scheduler._load_balancer is not None + assert len(scheduler._load_balancer.engine_counts) == attn_dp + + +def test_routing_without_a_time_is_refused_rather_than_reusing_a_stale_one() -> None: + with pytest.MonkeyPatch.context() as patch: + scheduler = _policy_scheduler(patch) + scheduler.add_request(_request()) + + with pytest.raises(RuntimeError, match="route through schedule_at"): + scheduler.schedule() + + # The queue is untouched, so the refusal loses no work. + assert len(scheduler._request_queue) == 1 + + +def test_routing_places_every_queued_request_on_the_serving_replica() -> None: + with pytest.MonkeyPatch.context() as patch: + scheduler = _policy_scheduler(patch) + requests = [_request() for _ in range(4)] + for request in requests: + scheduler.add_request(request) + + mapping = scheduler.schedule_at(0.0) + + replica_id = scheduler._serving_replica_id + assert [(rid, lane) for rid, lane, _ in mapping] == [ + (replica_id, 0), + (replica_id, 1), + (replica_id, 0), + (replica_id, 1), + ] + assert [request for _, _, request in mapping] == requests + assert scheduler._request_queue == [] + + +def test_an_unknown_lane_identity_is_rejected_at_the_report_boundary() -> None: + with pytest.MonkeyPatch.context() as patch: + scheduler = _policy_scheduler(patch) + + with pytest.raises(ValueError, match="exact lane index"): + scheduler.on_replica_batch_end(0.0, scheduler._serving_replica_id, None, None) + + +# -------------------------------------------------------------------------- +# The two base seams stay inert for every existing policy +# -------------------------------------------------------------------------- + + +def _bare_policy(scheduler_type, requests): + from types import SimpleNamespace + + scheduler = scheduler_type.__new__(scheduler_type) + # The rotation ordinal W2 made persistent, and the sticky policies' session + # table and counter, are normally set up by __init__. + scheduler._request_counter = 0 + scheduler._session_counter = 0 + scheduler._session_to_target_map = {} + scheduler._cluster_type = ClusterType.MONOLITHIC + scheduler._num_replicas = 2 + scheduler._replica_dp_size = 2 + scheduler._cluster = SimpleNamespace(replicas={0: object(), 1: object()}) + scheduler._request_queue = list(requests) + scheduler._replica_schedulers = { + (replica_id, dp_id): SimpleNamespace(num_pending_requests=0) + for replica_id in (0, 1) + for dp_id in (0, 1) + } + return scheduler + + +@pytest.mark.parametrize( + "scheduler_type", + [ + RoundRobinClusterScheduler, + LORClusterScheduler, + RandomClusterScheduler, + StickyRoundRobinClusterScheduler, + StickyLORClusterScheduler, + ], +) +def test_every_existing_policy_routes_identically_through_schedule_at( + scheduler_type, monkeypatch +) -> None: + monkeypatch.setattr( + "frontier.scheduler.cluster_scheduler.random_cluster_scheduler.randint", + lambda _low, _high: 0, + ) + # The sticky policies route by session, so every request carries one. + requests = [ + Request( + arrived_at=float(index), + num_prefill_tokens=8, + num_decode_tokens=2, + session_id=index % 3, + ) + for index in range(6) + ] + + direct = _bare_policy(scheduler_type, requests).schedule() + through_seam = _bare_policy(scheduler_type, requests).schedule_at(12.5) + + assert [(rid, lane, request.id) for rid, lane, request in through_seam] == [ + (rid, lane, request.id) for rid, lane, request in direct + ] + + +def test_the_default_batch_end_seam_is_inert() -> None: + class _Bare(BaseClusterScheduler): + def schedule(self): + return [] + + bare = _Bare.__new__(_Bare) + before = dict(vars(bare)) + + assert bare.on_replica_batch_end(1.0, 0, 0, None) is None + assert dict(vars(bare)) == before + + +# -------------------------------------------------------------------------- +# Configuration discovery +# -------------------------------------------------------------------------- + + +def test_all_cluster_policy_configs_and_registry_entries_remain_discoverable() -> None: + configs = { + subclass.get_type(): subclass + for subclass in get_all_subclasses(BaseClusterSchedulerConfig) + } + + assert set(configs) == set(ClusterSchedulerType) + for scheduler_type in ClusterSchedulerType: + assert ClusterSchedulerRegistry.get_class(scheduler_type) is not None + + +def test_the_accepted_policy_tokens_are_the_previous_five_plus_one() -> None: + tokens = { + str(subclass.get_type()) + for subclass in get_all_subclasses(BaseClusterSchedulerConfig) + } + + assert tokens == { + "round_robin", + "random", + "lor", + "sticky_round_robin", + "sticky_lor", + "vllm_load_balancing", + } + + +def test_the_generated_cli_selects_the_policy_and_keeps_the_old_default() -> None: + flat_config = create_flat_dataclass(SimulationConfig) + assert "cluster_scheduler_config_type" in flat_config.metadata_mapping + + original_argv = sys.argv + try: + sys.argv = ["frontier.main"] + default_parsed = flat_config.create_from_cli_args() + sys.argv = [ + "frontier.main", + "--cluster_scheduler_config_type", + "vllm_load_balancing", + ] + selected = flat_config.create_from_cli_args() + finally: + sys.argv = original_argv + + default_config = default_parsed.reconstruct_original_dataclass() + selected_config = selected.reconstruct_original_dataclass() + assert isinstance( + default_config.cluster_config.cluster_scheduler_config, + RoundRobinClusterSchedulerConfig, + ) + assert isinstance( + selected_config.cluster_config.cluster_scheduler_config, + VllmLoadBalancingClusterSchedulerConfig, + ) + + +@pytest.mark.parametrize( + "config_type", + [ + RoundRobinClusterSchedulerConfig, + RandomClusterSchedulerConfig, + LORClusterSchedulerConfig, + StickyRoundRobinClusterSchedulerConfig, + StickyLORClusterSchedulerConfig, + VllmLoadBalancingClusterSchedulerConfig, + ], +) +def test_every_policy_config_round_trips_through_copy_and_dict(config_type) -> None: + from copy import deepcopy + + from frontier.config.utils import dataclass_to_dict + + config = config_type() + assert deepcopy(config) == config + # The serialized identity is the selector token, which is what a config + # artifact has to round-trip through. + assert dataclass_to_dict(config)["name"] == str(config_type.get_type()) From 0fd12c4e054284f468e22bc8f6b961e71a5d6a6e Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 10:44:28 +0800 Subject: [PATCH 16/80] docs(review): record the W4 measurement, controls and fidelity result Adds the W4 test report, a Step 4 section in `validation.md`, the chronological progress entries, and the delivered-row and open-item-2 resolution in `review.md`. Also makes the W4 unit file order-independent. Running it after a MoE configuration in the same process tripped the process-global `IS_MOE` latch, because the file builds both dense and MoE shapes; it now resets the simulation globals around each test, as `tests/unit/test_config_owned_contracts.py` already does. The fidelity matrix was measured at `10dd474`, which predates this fixture, and is unaffected: no matrix case runs a unit test file. --- .../progress.md | 19 +- .../review.md | 3 +- ..._report_2026-09-22_w4_vllm_dp_placement.md | 215 ++++++++++++++++++ .../validation.md | 111 +++++++++ tests/unit/test_vllm_dp_load_balancer.py | 15 ++ 5 files changed, 358 insertions(+), 5 deletions(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w4_vllm_dp_placement.md diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index fefb7bb3..97ccff84 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -8,6 +8,7 @@ | 2026-09-22 | Maintainer review dispositions recorded; Checkpoint C closed: parent merged, W2 tests strengthened, W2 re-measured with one harness revision. | | 2026-09-22 | Checkpoint D first half: W3, the shared monolithic forward lifecycle, implemented, tested against four deliberate-defect controls, and committed as `65ed8a7`. | | 2026-09-22 | W3 fidelity matrix measured: 71 of 71 identical against the expectation recorded before the run. Step 3 closed. | +| 2026-09-22 | Checkpoint D second half: W4, the opt-in vLLM-style DP placement policy, implemented and committed as `10dd474`; measured against five deliberate-defect controls and a 71-of-71 identical fidelity matrix. Step 4 closed. | ## Status @@ -16,9 +17,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | -| Current step | Step 3 complete: implemented, measured, records written | -| Publication | PUSHED_VERIFIED (records) | -| Next action | Checkpoint D second half: W4, the opt-in vLLM-style DP placement, validating W3's shared forward identity at the report boundary per decision D1. | +| Current step | Step 4 complete: implemented, measured, records written | +| Publication | LOCAL_ONLY (W4 commits not yet pushed) | +| Next action | Push W4 and add its PR #35 section, then Checkpoint E: W5, the routing implementation identity, whose public name is still an open naming decision (open item 3). | ## Step status @@ -28,7 +29,7 @@ | 1 | Candidate/vLLM audit | PASS | n/a (source audit) | LOCAL_ONLY | NOT_REVIEWED | | 2 | RR DP rotation | PASS | unit PASS (23 tests); matrix PASS against a stated expectation, re-measured 2026-09-22 with one harness revision | PUSHED_VERIFIED | REVIEWED (R35-01 closed) | | 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | -| 4 | Opt-in vLLM DP placement | NOT_STARTED | — | — | — | +| 4 | Opt-in vLLM DP placement | PASS | unit PASS (61 new, 3778 total, failure set identical to the parent); integration PASS (3 cases in the real event loop, including a placement that diverges from round-robin); five deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | LOCAL_ONLY | NOT_REVIEWED | | 5 | Routing implementation identity | NOT_STARTED | — | — | — | | 6 | Legacy fused-MoE profiling | NOT_STARTED | — | — | — | | 7 | Optional zero-payload backend | NOT_STARTED (facts in `plan.md` A7) | — | — | — | @@ -60,6 +61,16 @@ - 2026-09-22: W3 fidelity matrix PASS. Baseline `3d47417` against candidate `65ed8a7`, both clean detached checkouts with `source_dirty=False` and no dirty paths, one harness at `65ed8a7`, no filter, clean cache, 71 executed and 426 cache files each. 71 of 71 compared and **71 identical**, zero mismatches, zero provenance findings, zero predictor cache differences. That is exactly the expectation `design.md` recorded before the run; the conditional I7 branch was not taken. Note explicitly: the matrix cannot reach a multi-lane monolithic MoE forward at all, so a null result is the pass condition for "nothing else moved", not evidence that the defect is fixed. Full record in `validation.md` and `test_report_2026-09-22_w3_shared_monolithic_forward.md`. - 2026-09-22: Step 3 published. `bdff4aa` pushed to `origin/fix/issue26-correctness-pr`; PR #35 body gained a W3 section that states the defect, the one-lifecycle fix, the event-priority constraint, the four controls with their distinct failures, and the matrix result together with what a null result does and does not mean. PR #35 stays draft. The W3 measurement worktrees `.worktrees/w3-baseline` and `.worktrees/w3-candidate` were removed after their manifests, results and `comparison.json` were written; the matrix output root is kept as evidence. +- 2026-09-22: W4 implemented and committed as one unit (`10dd474`). `VllmDPLoadBalancer` models the two halves of vLLM V1's internal DP mechanism separately: the frontend's `waiting * 4 + running` selection with its lowest-index tie break and its local waiting reservation, and the coordinator's publication schedule with the 100 ms changed / 5000 ms idle intervals, the 50 ms first-snapshot collection wait, and the latch that publishes the previous step's counts. Every constant is cited against vLLM v0.10.2. Out-of-order step reports warn and still apply, as the reference does. The object creates no events, so a drained simulation still drains. `VllmLoadBalancingClusterScheduler` is selected only by `--cluster_scheduler_config_type vllm_load_balancing`; `round_robin` stays the default. +- 2026-09-22: D1 settled empirically rather than by argument. A probe at the load-report emission boundary showed MoE dp2 report keys strictly increasing under both at-once and staggered arrivals, paired when both lanes are live, while dense dp2 under staggered online arrivals interleaves (lane 1 step 0 arriving after lane 0 step 2). The constructor therefore accepts MoE at any lane count and dense only at `attn_dp=1`, and says so in the error message. No separate Replica-scoped step identity was introduced. +- 2026-09-22: Two seams carry the policy without a mutable time bridge. `BaseClusterScheduler.schedule_at(time)` defaults to `schedule()` and `ClusterScheduleEvent` now calls it, so placement that depends on elapsed time receives the time as an argument; the policy's own `schedule()` raises rather than reusing a stale snapshot. `on_replica_batch_end` is inert by default and is called after `replica_scheduler.on_batch_end`, so a policy reading lane populations there sees the post-step state. `get_request_load()` delegates to the existing decision-log waiting accessor, which `tests/unit/test_module_split_boundaries.py` pins, so a load balancer and the decision log cannot disagree about what is waiting. +- 2026-09-22: W4 acceptance. 61 unit tests in `tests/unit/test_vllm_dp_load_balancer.py` cover selection, publication timing, ordering and validation, the real `vllm_v1` load accessor, the capability guard, seam inertness for all five existing policies, and config/CLI discovery. `tests/integration/test_vllm_dp_placement_runtime.py` runs three cases in the real event loop, each twice -- once under the policy and once under `round_robin` -- and records that routing times equal the `ClusterScheduleEvent` times at five distinct instants, that the report keys are ordered with every equal-key pair carrying two distinct lanes, that every report matches the post-step load while strictly fewer match the pre-step load, and that no event type is introduced. +- 2026-09-22: The discriminating runtime case. With identical arrivals, identical dummy-mode durations and identical lane capacity, the policy places `[0,1,1,0,1,1]` where round-robin places `[0,1,0,1,0,1]`, putting strictly fewer requests on the lane still draining one 40-token request. That is the property the policy exists for, measured rather than asserted. +- 2026-09-22: One real defect surfaced during the focused regression run and was fixed rather than explained away. Running the W4 unit file after a MoE configuration in the same process tripped the process-global `IS_MOE` latch ("already initialized to True, cannot change to False"), because the file builds both dense and MoE shapes. It now resets the simulation globals around each test with `global_vars.reset_global_vars()`, matching `tests/unit/test_config_owned_contracts.py`. Checks 1, 3, 4 and 5 were re-run afterwards and the controls were rebuilt against the final test files. +- 2026-09-22: W4 controls, five trees and five distinct failure subsets. Restoring `schedule()` in the event fails all 3 integration cases; moving the hook above `on_batch_end` fails 2 with `reports_after_the_lane_released_the_batch` at `0 == 10`; unweighting the waiting term fails exactly the weight and boundary tests; dropping the local reservation fails 4 unit and 2 integration; deleting the dense-lane guard fails exactly the `dense_multi_lane` construction case. The baseline control tree passes 64 of 64, so the harness itself is sound inside a control tree. +- 2026-09-22: W4 regression comparison against the branch parent `cdfcdf5`. `tests/unit`: 84 failures on both sides with identical identities, 3778 vs 3717 passing. `tests/integration`: the same five pre-existing errors (the PD-AF Reference checkout is absent on this host), 21 skipped on both, 15 vs 12 passing. A focused 46-file set covering cluster scheduling, the decision log and the two edited events: 51 failed / 1432 passed, every failure already in the known 84-failure baseline. +- 2026-09-22: W4 fidelity matrix PASS. Baseline `cdfcdf5` against candidate `10dd474`, both clean detached checkouts with `source_dirty=False` and no dirty paths, one harness at `10dd474`, no filter, clean cache, 71 executed and 426 cache files each. 71 of 71 compared and **71 identical**, zero mismatches, zero provenance findings, zero predictor cache differences -- exactly the expectation `design.md` recorded before the run. Note explicitly: no matrix case selects the new policy, so a null result is the pass condition for "nothing else moved", not evidence about the policy. Full record in `validation.md` and `test_report_2026-09-22_w4_vllm_dp_placement.md`. + ## Step 3 scoping, as recorded before implementation Kept as written so the implementation can be read against the scope it started from. Step 3 is now complete; see the W3 entries above and `design.md`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index 2534d87f..4e6a282a 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -74,6 +74,7 @@ Claims in those reports that this PR depends on were re-verified directly agains | Step-identity problem | The candidate uses `ForwardSyncState.get_step_id(batch)` as the report-order key (`vllm_load_balancing_cluster_scheduler.py:52-56`, latched at `vllm_dp_load_balancer.py:64-67` against a single scalar shared across engines). On main that identity is **invalid for dense models** (`base_replica_scheduler.py:459-466` sets `_forward_cohort_id` from a per-DP-lane creation counter that dense never promotes to a Replica-scoped counter, and the policy's guard does not require MoE), **invalid for MoE with DP>1 until W3 lands**, though for a narrower reason than first recorded: the ids come from one Replica-scoped counter, but the two lanes bind into separate per-kind open-step tables, so a mixed-phase forward never reaches a single shared step to key a report on, and valid for DP1. Idle participants are never reported. The id also advances per layer (`sync_entry.py:64`, `forward_sync_state.py:133-139`), so it is monotonic but is not a vLLM step counter. | | Disposition | **BLOCKED on W3**, then ADAPT. This makes the execution order W2 (independent) then W3 then W4. | | Decision raised | See D1 below. | +| Delivered | `10dd474`. The balancer models the frontend's selection and the coordinator's publication schedule separately, each constant cited against vLLM v0.10.2. The step-identity problem is resolved by measurement rather than by assumption: MoE keys are monotonic per Replica after W3, dense keys interleave under staggered arrivals, so the constructor rejects dense above one lane. 61 unit tests, 3 real-runtime cases including a placement that diverges from round-robin, five deliberate-defect controls each failing for its own reason, and a 71-of-71 identical fidelity matrix. See `test_report_2026-09-22_w4_vllm_dp_placement.md`. | ### W5 — Routing load distribution versus routing implementation identity @@ -133,7 +134,7 @@ The candidate deletes `VLLMv1EngineReplicaScheduler._get_num_waiting_reqs_for_de ## Open items 1. Whether `_schedule_batch_mode`'s per-replica grouped return order is load-bearing for the consumers of `ClusterScheduleEvent`'s request mapping (W2). -2. Whether W3's shared forward identity is the correct report-order key for W4, or whether a separate Replica-scoped step identity is required (D1). W3 is now delivered (`65ed8a7`): a monolithic cohort resolves to one step id in a single `"forward"` namespace regardless of its lanes' phases. W4 validates that identity at the report boundary. +2. RESOLVED. Whether W3's shared forward identity is the correct report-order key for W4, or whether a separate Replica-scoped step identity is required (D1). W3 delivered one step id per monolithic cohort in a single `"forward"` namespace regardless of its lanes' phases, and W4 (`10dd474`) measured that identity at the report boundary: a MoE Replica's keys are non-decreasing with every equal-key pair carrying two distinct lanes, while a dense Replica keeps a per-lane creation counter whose keys interleave under staggered online arrivals. No separate identity was introduced; the dense multi-lane shape is rejected in the constructor instead. 3. Which of the three existing spellings becomes the single public name for the routing implementation identity (W5). 4. Pipeline-parallel behavior of the component ledgers is untested in both trees. 5. A multi-lane monolithic MoE Replica has no released wrapper: the public MoE examples enforce `ATTN_TP == MOE_TP * MOE_EP` while the runtime enforces `attn_tp * attn_dp == moe_tp * moe_ep`, and those have no common solution above one lane. Its evidence therefore has to come from a direct-construction fixture, never from the fidelity matrix. Whether the release should offer such a wrapper is a product question, not a correctness one, and is left open. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w4_vllm_dp_placement.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w4_vllm_dp_placement.md new file mode 100644 index 00000000..940a65db --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w4_vllm_dp_placement.md @@ -0,0 +1,215 @@ +# Test report — W4, opt-in vLLM-style DP request placement (2026-09-22) + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-22 | Created. Acceptance evidence for the opt-in `vllm_load_balancing` cluster scheduler, including five deliberate-defect controls. | + +## Environment + +| Item | Value | +| --- | --- | +| Worktree | `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr` | +| Branch / commit under test | `fix/issue26-correctness-pr` @ `10dd474` | +| Baseline | `cdfcdf5`, this commit's parent (the published W3 state) | +| Reference read | vLLM v0.10.2 at `/data/ycfeng/Frontier/.real-engine/vLLM-BS` @ `ea95f571` | +| Python | `/data/ycfeng/envs/frontier-py310/bin/python` (3.10) | +| Invocation | `PYTHONPATH=$PWD python -m pytest … -p no:cacheprovider` from the worktree root | +| Absent optional deps | `torch`, `matplotlib` — the same 10 GPU/plot-only unit modules are excluded on both sides by one `--ignore` list | + +Control trees live under the session scratchpad (`.../scratchpad/w4-control-*`) +and are driven by `run_w4_tree.py`, which strips the editable-install +meta-path finder, prepends the chosen tree and asserts that +`frontier.scheduler.utils.vllm_dp_load_balancer` resolved inside it. `frontier` +is a namespace package here, so without that step the development worktree +would win and a control would silently test the delivered source. + +## What was measured + +| # | Check | Command | Expected | Actual | Result | +| --- | --- | --- | --- | --- | --- | +| 1 | Balancer and policy behavior | `pytest tests/unit/test_vllm_dp_load_balancer.py` | all pass | 61 passed | PASS | +| 2 | Real-runtime wiring | `pytest tests/integration/test_vllm_dp_placement_runtime.py` | all pass, and the policy must actually diverge from round-robin | 3 passed; placements `[0,1,1,0,1,1]` vs round-robin's `[0,1,0,1,0,1]` | PASS | +| 3 | Focused regression set (46 files touching cluster scheduling, the decision log, and the two edited events) | see below | no failure outside the known baseline set | 51 failed / 1432 passed; all 51 are in the 84-failure baseline | PASS | +| 4 | Full unit suite vs baseline | `pytest tests/unit` | identical failure set | 84 failed on both, identical identities; 3778 vs 3717 passed (+61 new) | PASS | +| 5 | Integration suite vs baseline | `pytest tests/integration` | identical error set | 5 errors on both (PD-AF Reference checkout absent), 21 skipped on both; 15 vs 12 passed (+3 new) | PASS | +| 6 | Fidelity matrix | `run_matrix.py run/compare` | all 71 cases exactly equal (expectation recorded in `design.md` before measuring) | see the fidelity section | PASS | + +### Check 3 — the focused regression set + +Every `tests/unit` file mentioning `cluster_scheduler`, `ClusterScheduler`, +`decision_log`, `ClusterScheduleEvent`, `GlobalBatchEndEvent` or +`get_num_waiting_reqs`, minus the 10 GPU-only modules: 46 files, recorded in +`.../scratchpad/w4_regression_set_run.txt`. + +One real defect surfaced here and was fixed rather than explained away. Running +the W4 unit file after a MoE configuration in the same process tripped the +process-global `IS_MOE` latch: + +``` +RuntimeError: IS_MOE already initialized to True, cannot change to False +``` + +The file builds both dense and MoE shapes, so it now resets the simulation +globals around each test with `global_vars.reset_global_vars()`, which is what +`tests/unit/test_config_owned_contracts.py` already does. Checks 1, 3, 4 and 5 +were re-run afterwards; the numbers above are the post-fix ones. + +## Check 2 — what the event loop showed + +Each case runs the same configuration twice, under `vllm_load_balancing` and +under `round_robin`, in one child process per case; the only difference between +the two runs is the policy. Execution time comes from the dummy predictor, so +both policies see identical durations. Recorded evidence: +`.../scratchpad/w4-evidence/_evidence.json`. + +| Evidence | `moe_dp2` (offline) | `moe_dp2_online` | `dense_dp1` (offline) | +| --- | --- | --- | --- | +| lanes / requests / completed | 2 / 4 / 4 | 2 / 6 / 6 | 1 / 4 / 4 | +| cluster schedule times | `[0.0]` | `[0.0, 0.4, 0.6, 0.8, 1.0]` | `[0.0]` | +| routing times | identical to the above | identical to the above | identical to the above | +| placements, policy | `[0,1,0,1]` | **`[0,1,1,0,1,1]`** | `[0,0,0,0]` | +| placements, round-robin | `[0,1,0,1]` | **`[0,1,0,1,0,1]`** | `[0,0,0,0]` | +| report keys | `3,3,7,7,11,11,15,15,19,19` | `3,3,7,11,…,159` (44) | `0,1,2,3,4,5,6` | +| keys non-decreasing | yes | yes | yes | +| keys with a repeated lane | none | none | none | +| reports / after the lane released the batch | 10 / 10 | 44 / 44 | 7 / 7 | +| reports matching post-step / pre-step load | 10 / 6 | 44 / 38 | 7 / 3 | +| reports where the release moved the load | 4 | 6 | 4 | +| event types produced | `ReplicaScheduleEvent` | `ReplicaScheduleEvent` | `ReplicaScheduleEvent` | +| makespan (s) | 0.31 | 2.48 | 0.448 | + +Four things in that table are load-bearing. + +- **The routing time is the event's time.** Recorded routing times equal the + recorded `ClusterScheduleEvent` times exactly, at five distinct instants in + the online case. The policy's own `schedule()` raises, so a stale or retained + time could not have been substituted silently. +- **The policy diverges from round-robin.** In the online case the two runs see + the same arrivals, the same durations and the same lane capacity, and the + policy places strictly fewer requests on lane 0 — the lane still draining the + one 40-token request. Round-robin cannot do that; it alternates. +- **The reported load is the post-step one.** The lane's own `on_batch_end` is + bracketed, so pre- and post-step load are distinct values for the batches + that released work. Every report matches the post-step value, and strictly + fewer match the pre-step value (6 of 10, 38 of 44, 3 of 7) — so the reports + are not merely the pre-step state relabelled. +- **The report key ordering the guard assumes actually holds.** MoE keys are + non-decreasing and every equal-key pair carries two distinct lanes, which is + one shared forward observed twice. Keys advance by `num_layers = 4` per + forward in the MoE cases and by 1 in the dense single-lane case, which is the + measured identity recorded in `design.md`; only ordering and equality are + used. +- **No event type is introduced and the run drains.** Both policies produce the + same event-type set and both complete every request. + +## Controls — each defect fails for its own reason + +Each tree is the delivered source and the delivered tests with exactly one +edit; `diff -r frontier/` against the delivered source reports exactly one +differing file per tree. + +| Control | The one edit | Tests that fail | Why that is the right failure | +| --- | --- | --- | --- | +| `baseline` | none | 0 of 64 | the harness itself passes in a control tree | +| `no-time-plumbing` | `ClusterScheduleEvent` calls `schedule()` again | 3 integration | the policy refuses to route without a time, so the whole real-runtime surface fails rather than routing from a stale snapshot | +| `pre-step-report` | the hook moves above `replica_scheduler.on_batch_end` | 2 integration | `reports_after_the_lane_released_the_batch` drops to `0 == 10`, and the online placements collapse onto round-robin's | +| `unweighted-waiting` | `WAITING_SCORE_WEIGHT = 1` | 2 unit | the weight and its exact boundary are the two tests that pin `waiting * 4 + running` | +| `no-local-reservation` | `select` stops reserving the chosen lane | 4 unit + 2 integration | without the reservation every request in one routing burst lands on the same lane; the snapshot-replacement and latch tests also lose their discriminating state | +| `no-dense-lane-guard` | the `is_moe or attn_dp == 1` rejection is deleted | 1 unit | only the `dense_multi_lane` construction case, which is exactly the shape whose report-key ordering was measured to interleave | + +Controls were rebuilt against the final test files after the `IS_MOE` fixture +was added and re-run; the counts above are from that run +(`.../scratchpad/w4_ctrl_*.log`). + +## Fidelity matrix + +Both sides ran from **clean detached worktrees** so no untracked draft could +dirty the recorded provenance, and both were driven by **one harness +revision**, the candidate's. + +```bash +PYTHONPATH=/data/ycfeng/Frontier/.worktrees/w4-candidate \ + python /data/ycfeng/Frontier/.worktrees/w4-candidate/tests/e2e/refactor_fidelity/run_matrix.py run \ + --repo-root /data/ycfeng/Frontier/.worktrees/w4- --label \ + --output-root /data/ycfeng/tmp/issue26-correctness-pr/w4-fidelity \ + --python-bin /data/ycfeng/envs/frontier-py310/bin/python --jobs 8 --clean-cache + +python tests/e2e/refactor_fidelity/run_matrix.py compare \ + --output-root /data/ycfeng/tmp/issue26-correctness-pr/w4-fidelity \ + --baseline-label baseline --candidate-label candidate +``` + +| Side | `source_revision` | `source_dirty` | `git_dirty_paths` | `harness_revision` | Executed | Cache files | +| --- | --- | --- | --- | --- | --- | --- | +| baseline | `cdfcdf54b545` (this commit's parent) | `False` | empty | `10dd4745b9a7` | 71 of 71 | 426 | +| candidate | `10dd4745b9a7` | `False` | empty | `10dd4745b9a7` | 71 of 71 | 426 | + +No case filter, cache cleaned before each run, 0 cases dropped as stale. + +| Metric | Value | +| --- | --- | +| cases compared | 71 of 71 in the case table | +| identical | **71** | +| mismatched | 0 | +| baseline failures / candidate-only failures | 0 / 0 | +| cases missing from one side / with missing evidence / with differing definitions | 0 / 0 / 0 | +| provenance findings | 0 | +| predictor cache: baseline-only / candidate-only / findings | 0 / 0 / 0 | +| `complete_comparison`, `predictor_cache_populated_cleanly` | `True`, `True` | + +Report: `/data/ycfeng/tmp/issue26-correctness-pr/w4-fidelity/comparison.json`. + +### Judged against the expectation recorded before measuring + +`design.md` recorded, before the matrix ran: + +> **Prediction: all 71 fidelity cases stay exactly equal.** … W4 has no +> reachable fidelity fix, so a single mismatch falsifies the change rather than +> confirming it. + +71 of 71 identical, so the prediction held. What that does and does not +establish: + +- It **does** establish that nothing in the 71 cases moved. All three of the + reachable edits are inert on those paths: `schedule_at` defaults to + `schedule()`, `on_replica_batch_end` returns `None` on every policy the + matrix selects, and `get_request_load()` recomputes the decision-log payload + from the same two accessors it replaced. +- It **does not** establish anything about the new policy. No matrix case + selects `vllm_load_balancing`; the matrix cases all use the default + `round_robin`. The policy's own evidence is checks 1 and 2 and the controls. + +## Verification limits + +- **No vLLM equivalence is claimed or measured.** The selection rule, the tie + break, the reservation, the publication intervals and the collection wait are + each cited against vLLM v0.10.2 source, and the unit tests pin them against + values derived from that source by hand. Nothing here was compared against a + running vLLM deployment, and the object deliberately omits IPC latency, more + than one frontend, elastic scaling and the coordinator's warm-start phase. +- **The report key is not a vLLM step counter.** It is + `ForwardSyncState.get_step_id(batch)`, which advances once per layer, so + consecutive forwards are roughly `num_layers` apart. Only its ordering and + equality are used, which is all the reference coordinator uses its + `(wave, step)` pair for. Frontier has no wave reset, so the pair collapses to + one scalar. +- **The dense restriction is measured at one shape, not proved in general.** + The `attn_dp=1` requirement for dense models comes from a probe showing dense + dp2 keys interleaving under staggered online arrivals while MoE dp2 keys stay + ordered. That is one measurement per shape, recorded in `design.md`, not an + exhaustive argument; the constructor rejects the shape rather than relying on + the observation holding everywhere. +- **Integration timing is dummy-mode timing.** The three real-runtime cases use + the dummy predictor, so the placements they produce are not latency-realistic. + They are still load-sensitive and deterministic, and both policies in a case + see identical durations, which is what those checks depend on. +- **The online divergence is one scenario.** `[0,1,1,0,1,1]` versus + `[0,1,0,1,0,1]` shows the policy reads published load; it does not + characterise the policy's behavior across arrival patterns, and the test + asserts the direction (fewer requests on the busy lane) rather than the exact + sequence. +- **Five controls, not a proof.** Each control shows that one specific piece of + the change is load-bearing for a specific test. They do not exclude a defect + that no control models. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index 6a8c6c65..d3366c3e 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -9,6 +9,7 @@ | 2026-09-21 | Step 2 recorded: unit sensitivity and the fidelity measurement against a stated expectation. | | 2026-09-22 | Step 2 re-measured with harness and source at one revision, after the gate corrections. Same expectation, same result, recorded provenance. | | 2026-09-22 | Step 3 recorded: the shared monolithic forward, its direct-construction runtime evidence, four deliberate-defect controls, and a 71-of-71 identical fidelity matrix. | +| 2026-09-22 | Step 4 recorded: the opt-in vLLM-style DP placement policy, its real-runtime wiring evidence including a divergence from round-robin, five deliberate-defect controls, and a 71-of-71 identical fidelity matrix. | ## Environment @@ -333,3 +334,113 @@ list. That gap between the two groups is the deadlock, isolated. source-attributable, not trained numerical parity. - I8, the per-source decode component ledger, is deliberately out of scope; see the scope table in `design.md`. + +## Step 4 — opt-in vLLM-style DP request placement (2026-09-22) + +### What the fidelity matrix can and cannot show here + +The policy is opt-in and no case in the 71-case table selects it; every case +uses the default `round_robin`. **The matrix therefore answers one question +only — did anything else move — and a null result is the pass condition, not +evidence about the policy.** The policy's own evidence is the balancer unit +suite, the real-runtime integration cases, and five deliberate-defect controls. + +Three edits are reachable from the existing matrix paths, and all three are +inert there: `BaseClusterScheduler.schedule_at` defaults to `schedule()`, +`on_replica_batch_end` returns `None` on every policy the matrix selects, and +`get_request_load()` rebuilds the decision-log payload from the same two +accessors it replaced. + +### Setup + +Both sides ran from clean detached worktrees, driven by one harness revision. + +```bash +# harness = the candidate checkout, for both sides +PYTHONPATH=/data/ycfeng/Frontier/.worktrees/w4-candidate \ + python /data/ycfeng/Frontier/.worktrees/w4-candidate/tests/e2e/refactor_fidelity/run_matrix.py run \ + --repo-root /data/ycfeng/Frontier/.worktrees/w4- --label \ + --output-root /data/ycfeng/tmp/issue26-correctness-pr/w4-fidelity \ + --python-bin /data/ycfeng/envs/frontier-py310/bin/python --jobs 8 --clean-cache + +python tests/e2e/refactor_fidelity/run_matrix.py compare \ + --output-root /data/ycfeng/tmp/issue26-correctness-pr/w4-fidelity \ + --baseline-label baseline --candidate-label candidate +``` + +| Side | `source_revision` | `source_dirty` | `git_dirty_paths` | `harness_revision` | Executed | Cache files | +| --- | --- | --- | --- | --- | --- | --- | +| baseline | `cdfcdf54b545` (this commit's parent) | `False` | empty | `10dd4745b9a7` | 71 of 71 | 426 | +| candidate | `10dd4745b9a7` | `False` | empty | `10dd4745b9a7` | 71 of 71 | 426 | + +### The expectation, recorded in `design.md` before measuring + +> **Prediction: all 71 fidelity cases stay exactly equal.** … W4 has no +> reachable fidelity fix, so a single mismatch falsifies the change rather than +> confirming it. + +| Metric | Value | +| --- | --- | +| cases compared / identical / mismatched | 71 of 71 / **71** / 0 | +| baseline failures / candidate-only failures | 0 / 0 | +| missing from one side / missing evidence / differing definitions | 0 / 0 / 0 | +| provenance findings / predictor-cache findings | 0 / 0 | +| `complete_comparison`, `predictor_cache_populated_cleanly` | `True`, `True` | + +Prediction held. Report: +`/data/ycfeng/tmp/issue26-correctness-pr/w4-fidelity/comparison.json`. + +### Runtime evidence for the policy itself + +Each case runs one configuration twice — under `vllm_load_balancing` and under +`round_robin` — in a single child process, with the dummy predictor so both +policies see identical durations. + +| Evidence | `moe_dp2` | `moe_dp2_online` | `dense_dp1` | +| --- | --- | --- | --- | +| routing times == cluster schedule times | `[0.0]` | `[0.0, 0.4, 0.6, 0.8, 1.0]` | `[0.0]` | +| placements, policy vs round-robin | `[0,1,0,1]` / `[0,1,0,1]` | **`[0,1,1,0,1,1]` / `[0,1,0,1,0,1]`** | `[0,0,0,0]` / `[0,0,0,0]` | +| report keys | `3,3,7,7,…,19,19` | `3,3,7,…,159` (44) | `0,1,2,3,4,5,6` | +| ordered / no repeated lane per key | yes / yes | yes / yes | yes / yes | +| reports after the lane's release | 10 of 10 | 44 of 44 | 7 of 7 | +| matching post-step / pre-step load | 10 / 6 | 44 / 38 | 7 / 3 | +| event types vs round-robin | equal | equal | equal | + +The online row is the discriminating one: identical arrivals, durations and +lane capacity, and the policy still places strictly fewer requests on the lane +draining the one long request. Round-robin cannot, because it cannot see load. + +### Controls + +Each tree is the delivered source and tests with exactly one edit. + +| Control | Fails | Where | +| --- | --- | --- | +| `baseline` | 0 of 64 | — | +| `no-time-plumbing` (`schedule()` restored in the event) | 3 | integration | +| `pre-step-report` (hook moved above `on_batch_end`) | 2 | integration; `reports_after_the_lane_released_the_batch` is `0 == 10` | +| `unweighted-waiting` (`WAITING_SCORE_WEIGHT = 1`) | 2 | unit | +| `no-local-reservation` (`select` stops reserving) | 6 | 4 unit + 2 integration | +| `no-dense-lane-guard` (guard deleted) | 1 | unit, the `dense_multi_lane` case | + +### Regression comparison + +| Suite | Baseline `cdfcdf5` | Candidate `10dd474` | Verdict | +| --- | --- | --- | --- | +| `tests/unit` | 84 failed / 3717 passed | 84 failed / 3778 passed | identical failure identities; +61 are the new tests | +| `tests/integration` | 5 errors / 12 passed | 5 errors / 15 passed | identical errors (PD-AF Reference checkout absent on this host); +3 are the new tests | +| focused set, 46 files | — | 51 failed / 1432 passed | all 51 are in the known 84-failure baseline | + +### Limits + +- No vLLM equivalence is claimed or measured. Every constant is cited against + vLLM v0.10.2 source; nothing was compared against a running deployment. IPC + latency, multiple frontends, elastic scaling and the coordinator's warm-start + phase are deliberately absent. +- The dense `attn_dp=1` restriction rests on one probe per shape (recorded in + `design.md`), which is why the constructor rejects the shape instead of the + code relying on the observation holding everywhere. +- Integration placements are dummy-mode placements: deterministic and + load-sensitive, not latency-realistic. +- The online divergence is one arrival pattern. The test asserts the direction + — fewer requests on the busy lane — not the exact sequence. diff --git a/tests/unit/test_vllm_dp_load_balancer.py b/tests/unit/test_vllm_dp_load_balancer.py index 08dbcf16..e8c6c682 100644 --- a/tests/unit/test_vllm_dp_load_balancer.py +++ b/tests/unit/test_vllm_dp_load_balancer.py @@ -37,6 +37,7 @@ VllmLoadBalancingClusterSchedulerConfig, VllmV1SchedulerConfig, ) +from frontier.config import global_vars from frontier.config.cluster_scheduler_config import BaseClusterSchedulerConfig from frontier.config.flat_dataclass import create_flat_dataclass from frontier.config.utils import get_all_subclasses @@ -75,6 +76,20 @@ ) +@pytest.fixture(autouse=True) +def reset_simulation_globals(): + """Keep the process-global model latch out of it. + + Building a `SimulationConfig` latches `IS_MOE` for the whole process, and + this file builds both dense and MoE shapes. Resetting on both sides is what + `tests/unit/test_config_owned_contracts.py` already does. + """ + + global_vars.reset_global_vars() + yield + global_vars.reset_global_vars() + + # -------------------------------------------------------------------------- # Selection: reference core_client.py:1139-1153 # -------------------------------------------------------------------------- From cb8ba1420e615f53cca7e281661c250388372d6d Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 10:46:03 +0800 Subject: [PATCH 17/80] docs(review): mark Step 4 published and record the PR 35 W4 section --- .../task_2026-09-21_issue26_correctness_pr/progress.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 97ccff84..023a3ef8 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -18,8 +18,8 @@ | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | | Current step | Step 4 complete: implemented, measured, records written | -| Publication | LOCAL_ONLY (W4 commits not yet pushed) | -| Next action | Push W4 and add its PR #35 section, then Checkpoint E: W5, the routing implementation identity, whose public name is still an open naming decision (open item 3). | +| Publication | PUSHED_VERIFIED (records) | +| Next action | Checkpoint E: W5, the routing implementation identity, whose public name is still an open naming decision (open item 3), then the D2-scoped W6. | ## Step status @@ -29,7 +29,7 @@ | 1 | Candidate/vLLM audit | PASS | n/a (source audit) | LOCAL_ONLY | NOT_REVIEWED | | 2 | RR DP rotation | PASS | unit PASS (23 tests); matrix PASS against a stated expectation, re-measured 2026-09-22 with one harness revision | PUSHED_VERIFIED | REVIEWED (R35-01 closed) | | 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | -| 4 | Opt-in vLLM DP placement | PASS | unit PASS (61 new, 3778 total, failure set identical to the parent); integration PASS (3 cases in the real event loop, including a placement that diverges from round-robin); five deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | LOCAL_ONLY | NOT_REVIEWED | +| 4 | Opt-in vLLM DP placement | PASS | unit PASS (61 new, 3778 total, failure set identical to the parent); integration PASS (3 cases in the real event loop, including a placement that diverges from round-robin); five deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 5 | Routing implementation identity | NOT_STARTED | — | — | — | | 6 | Legacy fused-MoE profiling | NOT_STARTED | — | — | — | | 7 | Optional zero-payload backend | NOT_STARTED (facts in `plan.md` A7) | — | — | — | @@ -70,6 +70,7 @@ - 2026-09-22: W4 controls, five trees and five distinct failure subsets. Restoring `schedule()` in the event fails all 3 integration cases; moving the hook above `on_batch_end` fails 2 with `reports_after_the_lane_released_the_batch` at `0 == 10`; unweighting the waiting term fails exactly the weight and boundary tests; dropping the local reservation fails 4 unit and 2 integration; deleting the dense-lane guard fails exactly the `dense_multi_lane` construction case. The baseline control tree passes 64 of 64, so the harness itself is sound inside a control tree. - 2026-09-22: W4 regression comparison against the branch parent `cdfcdf5`. `tests/unit`: 84 failures on both sides with identical identities, 3778 vs 3717 passing. `tests/integration`: the same five pre-existing errors (the PD-AF Reference checkout is absent on this host), 21 skipped on both, 15 vs 12 passing. A focused 46-file set covering cluster scheduling, the decision log and the two edited events: 51 failed / 1432 passed, every failure already in the known 84-failure baseline. - 2026-09-22: W4 fidelity matrix PASS. Baseline `cdfcdf5` against candidate `10dd474`, both clean detached checkouts with `source_dirty=False` and no dirty paths, one harness at `10dd474`, no filter, clean cache, 71 executed and 426 cache files each. 71 of 71 compared and **71 identical**, zero mismatches, zero provenance findings, zero predictor cache differences -- exactly the expectation `design.md` recorded before the run. Note explicitly: no matrix case selects the new policy, so a null result is the pass condition for "nothing else moved", not evidence about the policy. Full record in `validation.md` and `test_report_2026-09-22_w4_vllm_dp_placement.md`. +- 2026-09-22: Step 4 published. `0fd12c4` pushed to `origin/fix/issue26-correctness-pr`; PR #35 body gained a W4 section that states the mechanism with its citations, the measured report-key table behind the dense restriction, the two seams, the placement divergence from round-robin, the five controls with their distinct failure subsets, and the matrix result together with the fact that no matrix case selects the policy. PR #35 stays draft. The W4 measurement worktrees `.worktrees/w4-baseline` and `.worktrees/w4-candidate` were removed after their manifests, results and `comparison.json` were written; the matrix output root is kept as evidence. ## Step 3 scoping, as recorded before implementation From de2bee894d9392d73916641b490507657e1cfa24 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 11:29:24 +0800 Subject: [PATCH 18/80] docs(review): close W5 as not ported and record why The W5 collision needs two clusters in one run to resolve different MoE routing implementations. They cannot: `moe_routing_distribution_type` is declared once on `ReplicaConfig`, `ClusterConfig` declares no per-role override for it, so `get_field_value` always falls back to the global field, and the generated CLI exposes exactly one flag for it. Both recorded mechanisms are therefore latent code, not reachable behavior. Making them reachable requires adding the per-role override first, which is the configuration surface this step would then have to defend with a registry routing axis. Keeping the current contract instead: one global distribution, one derived routing path per run. Records what the module actually decides (which `moe.csv` rows train `moe_gating_routing_topk`) and the measured cost of picking the wrong path on the three datasets that carry both: 3.1% median, 7.3-7.8% and 4.2-5.4% of the summed per-layer operator medians. That is why the existing distribution-to-path mapping stays; it is not a gain from the reverted override, which changed nothing while unset. No source change. The drafted implementation is archived as a patch. --- .../plan.md | 2 + .../progress.md | 13 +- .../requirements.md | 8 + .../review.md | 9 +- ...w5_reverted_moe_routing_runtime_path.patch | 745 ++++++++++++++++++ 5 files changed, 770 insertions(+), 7 deletions(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/w5_reverted_moe_routing_runtime_path.patch diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md index 614ab7ba..3e9182a5 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md @@ -501,6 +501,8 @@ The selection and publication rules are source-backed; actual Frontier events ex ## 11. Step 5 — Separate routing load distribution from implementation identity +> **Closed 2026-09-22 without source changes (user decision; see `review.md` W5 "Final disposition" and `requirements.md`).** The premise check found the collision this step fixes unreachable on main: the routing distribution has one global field and no per-role override, so every cluster resolves one routing path per run. The override that would have made it reachable was judged not worth its configuration surface, and the drafted implementation is archived as `w5_reverted_moe_routing_runtime_path.patch`. The specification below is kept as written for the record. + ### Source scope ```text diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 023a3ef8..db1002ab 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -9,6 +9,7 @@ | 2026-09-22 | Checkpoint D first half: W3, the shared monolithic forward lifecycle, implemented, tested against four deliberate-defect controls, and committed as `65ed8a7`. | | 2026-09-22 | W3 fidelity matrix measured: 71 of 71 identical against the expectation recorded before the run. Step 3 closed. | | 2026-09-22 | Checkpoint D second half: W4, the opt-in vLLM-style DP placement policy, implemented and committed as `10dd474`; measured against five deliberate-defect controls and a 71-of-71 identical fidelity matrix. Step 4 closed. | +| 2026-09-22 | Checkpoint E first half: W5 closed as NOT PORTED by user decision after the premise check showed the collision unreachable on main; the drafted implementation was reverted before commit and archived as a patch. | ## Status @@ -17,9 +18,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | -| Current step | Step 4 complete: implemented, measured, records written | -| Publication | PUSHED_VERIFIED (records) | -| Next action | Checkpoint E: W5, the routing implementation identity, whose public name is still an open naming decision (open item 3), then the D2-scoped W6. | +| Current step | Step 5 closed without source changes (NOT PORTED, user decision); records written | +| Publication | LOCAL_ONLY (this record) | +| Next action | Checkpoint E second half: the D2-scoped W6 legacy fused-MoE profiling arithmetic (`frontier/profiling/moe/moe_vllm_kernel.py`), then Checkpoint F's conditional W7. | ## Step status @@ -30,7 +31,7 @@ | 2 | RR DP rotation | PASS | unit PASS (23 tests); matrix PASS against a stated expectation, re-measured 2026-09-22 with one harness revision | PUSHED_VERIFIED | REVIEWED (R35-01 closed) | | 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 4 | Opt-in vLLM DP placement | PASS | unit PASS (61 new, 3778 total, failure set identical to the parent); integration PASS (3 cases in the real event loop, including a placement that diverges from round-robin); five deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | -| 5 | Routing implementation identity | NOT_STARTED | — | — | — | +| 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | records only | REVIEWED (user chose to keep the single global field) | | 6 | Legacy fused-MoE profiling | NOT_STARTED | — | — | — | | 7 | Optional zero-payload backend | NOT_STARTED (facts in `plan.md` A7) | — | — | — | | 8 | Combined regression, PR hand-off | NOT_STARTED | — | — | — | @@ -72,6 +73,10 @@ - 2026-09-22: W4 fidelity matrix PASS. Baseline `cdfcdf5` against candidate `10dd474`, both clean detached checkouts with `source_dirty=False` and no dirty paths, one harness at `10dd474`, no filter, clean cache, 71 executed and 426 cache files each. 71 of 71 compared and **71 identical**, zero mismatches, zero provenance findings, zero predictor cache differences -- exactly the expectation `design.md` recorded before the run. Note explicitly: no matrix case selects the new policy, so a null result is the pass condition for "nothing else moved", not evidence about the policy. Full record in `validation.md` and `test_report_2026-09-22_w4_vllm_dp_placement.md`. - 2026-09-22: Step 4 published. `0fd12c4` pushed to `origin/fix/issue26-correctness-pr`; PR #35 body gained a W4 section that states the mechanism with its citations, the measured report-key table behind the dense restriction, the two seams, the placement divergence from round-robin, the five controls with their distinct failure subsets, and the matrix result together with the fact that no matrix case selects the policy. PR #35 stays draft. The W4 measurement worktrees `.worktrees/w4-baseline` and `.worktrees/w4-candidate` were removed after their manifests, results and `comparison.json` were written; the matrix output root is kept as evidence. +- 2026-09-22: W5 premise check before implementation. The review's Mechanism A rests on the routing distribution being settable per role. It is not: `ClusterConfig` declares no per-role `moe_routing_distribution_type` override, `get_field_value` always misses to the global `ReplicaConfig` field, and the generated CLI has one flag for it. Verified on this branch and on `main@1f694f7` with `git grep` and `python -m frontier.main --help`. Every cluster in a run resolves one routing path, so neither W5 mechanism can fire today; it could fire only after adding a per-role override. Put to the user with three options (global field only, global + per-role, port as specified); the user first chose global + per-role, and the config surface, single-owner resolver, routing-aware training signature and family gate, and a three-part registry key were drafted (uncommitted). +- 2026-09-22: W5 magnitude, measured on request. On the three checked-in `moe.csv` files carrying both routing paths, the matched `moe_gating_routing_topk` cost differs by 3.1% (median; up to 24% at the smallest token counts), 7.3-7.8% and 4.2-5.4% of the summed per-layer operator medians (a800 qwen3-a3b-30b-moe, h800 Phi-tiny-MoE, h800 step-moe-noquant-small). This is the cost of the *existing* mapping picking the wrong kernel for a scenario, and the reason the mapping stays; the drafted override itself changed nothing while unset. +- 2026-09-22: W5 closed as NOT PORTED. After the explanation of the collision case, of what `uniform_topk` is (the profiler's round-robin routing path, `moe_impl.py:75-103`), and of the three options, the user decided to keep the current contract: one global `moe_routing_distribution_type`, one derived path per run. The twelve drafted source/test files were restored with `git checkout` after saving the diff to `w5_reverted_moe_routing_runtime_path.patch` (745 lines). `review.md` W5 rows corrected and closed; `requirements.md` records the decision verbatim; `plan.md` section 11 is kept as specified with a closure note. No `frontier/` change, so no fidelity matrix run for this step. + ## Step 3 scoping, as recorded before implementation Kept as written so the implementation can be read against the scope it started from. Step 3 is now complete; see the W3 entries above and `design.md`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md index f1b63387..1bf6ec58 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md @@ -26,3 +26,11 @@ The draft specification is landed verbatim in `plan.md` together with an Amendme | Q5 Publication | Authorized: push both branches to `origin`, create/update draft PRs. Not authorized: merge, force-push, history rewrite, closing Issue 26. | | Q6 Stop point | Stop after the Step 0 push for user review. | | vLLM reference | Clone into `.real-engine/` (local exclude), pinned to `ea95f57`. | + +## Decisions taken during execution + +| Date | Question | Decision | +| --- | --- | --- | +| 2026-09-22 | W5 open item 3: the single public name for the routing implementation identity, asked under the AGENTS.md naming gate with three materially plausible spellings. | **`moe_routing_runtime_path`**, a `ReplicaConfig` field, CLI `--replica_config_moe_routing_runtime_path`. It shares the prefix of the neighbouring `moe_routing_seed`, `moe_routing_trace_path` and `moe_routing_distribution_type`. The profiling CSV column stays `routing_runtime_path` and the standalone trainer flag stays `--routing_runtime_path`; neither is renamed. | +| 2026-09-22 | W5 scope, asked after the naming decision: global field only, or global plus per-role overrides. | Global + per-role overrides (`prefill`, `decode`, `decode_ffn`). Superseded the same day by the row below. | +| 2026-09-22 | `[Original Request]` "我在rethink添加 frontier.moe_routing_runtime的必要性。我的concern是，该部分的align是否对模拟准确度意义重大？如果对fidelity的影响不到0.5%,我认为完全无需引入如此复杂的变量和setting,这会使得可用性和可读性变得很困难" -> after the explanation of what the module decides and the measured 3-8% per-layer cost of a wrong path versus the 0% change of the unset override: "我认为采用 当前默认的 两个 cluster 都只能读同一个全局 moe_routing_distribution_type，永远解析出同一个 path ；进行回退" | **W5 not ported.** Keep the existing contract: one global `moe_routing_distribution_type`, one derived routing path per run, no override field, no registry routing axis. The drafted implementation was reverted before any commit and archived as `w5_reverted_moe_routing_runtime_path.patch`. Both earlier W5 decisions above are void. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index 4e6a282a..13869632 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -10,6 +10,7 @@ | 2026-09-22 | W3 delivered and measured; R35-02 closed; the unreleased multi-lane monolithic MoE shape recorded as an open item. | | 2026-09-22 | Recorded the maintainer's PR #34 / PR #35 review: D1 and D2 resolved, ten review comments dispositioned, each verified against source. | | 2026-09-22 | Self-review of that record: corrected the lockstep mechanism and counter semantics under D1, the line references under R34-01, the R34-03 remedy (the baseline label is itself an assembled partial run), and added the omissions listed under "Found on re-review". | +| 2026-09-22 | W5 closed without source changes. Corrected the Mechanism A premise: the routing distribution has no per-role override on main, so both W5 mechanisms are unreachable from any released configuration. The user chose to keep the single global field; the drafted implementation was reverted and archived as a patch. | ## Pinned source snapshot @@ -80,13 +81,15 @@ Claims in those reports that this PR depends on were re-verified directly agains | Item | Finding | | --- | --- | -| Defect on main | Present and reachable from the public CLI. Two independent mechanisms let one routing implementation's cost model be used for another. | -| Mechanism A | Routing distribution is settable per role (`config.py:4242-4247`, `:4257-4273`, `:4302`), so a PDD run can use `PREFILL=balanced` and `DECODE=random`, which resolve to different runtime paths (`moe_routing_runtime.py:29-32`). `trained_model_signatures` is one set shared across clusters (`shared_prediction_model_manager.py:702`, `:707`, `:778`) and `ffn_signature` (`:1377-1382`) carries **no routing term** (re-verified). The second cluster therefore returns early at `:1384-1386`, before `_validate_moe_dataset_contract` at `:1433-1439`, so there is no fail-fast, and it predicts `moe_gating_routing_topk` using the other cluster's estimator through `_models_view_for_family` (`:4358-4388`), which matches on layer identity alone. | +| Defect on main | **Corrected 2026-09-22: present as latent code, not reachable.** Both mechanisms below need two clusters that resolve different routing paths in one run. On main and on this branch, `moe_routing_distribution_type` is declared once, on `ReplicaConfig`; `ClusterConfig` declares no `{prefill,decode,decode_ffn}_replica_config_moe_routing_distribution_type` field, so `get_field_value("moe_routing_distribution_type")` (`cluster_role_config.py:58-63`, `:73-75`, `:87-89`) always takes its `getattr(..., None)` miss and returns the global value, and `python -m frontier.main --help` generates exactly one flag, `--replica_config_moe_routing_distribution_type`. Every cluster in a run therefore resolves the same path and neither mechanism can fire. The original row, kept below for the record, took the `get_field_value` lookup as evidence of a declared override. | +| Mechanism A (as originally recorded; premise corrected above) | Routing distribution is settable per role (`config.py:4242-4247`, `:4257-4273`, `:4302`), so a PDD run can use `PREFILL=balanced` and `DECODE=random`, which resolve to different runtime paths (`moe_routing_runtime.py:29-32`). `trained_model_signatures` is one set shared across clusters (`shared_prediction_model_manager.py:702`, `:707`, `:778`) and `ffn_signature` (`:1377-1382`) carries **no routing term** (re-verified). The second cluster therefore returns early at `:1384-1386`, before `_validate_moe_dataset_contract` at `:1433-1439`, so there is no fail-fast, and it predicts `moe_gating_routing_topk` using the other cluster's estimator through `_models_view_for_family` (`:4358-4388`), which matches on layer identity alone. | | Mechanism B | When both clusters do train, `_store_model_precision` keys on `(model_name, identity)` at `:4238` with a layer-shape-derived identity, so the second model overwrites the first. | | Where identity does reach today | Only dataset row selection (`:1294-1326`, `:1235-1265`) and the per-call `moe_df_cache` key (`:1486-1505`), which is scoped to a single cluster call and therefore never prevents the collision. | | Correction to the specification's premise | The **persistent disk cache is not the hole**. `_get_model_hash` (`:4008-4056`) hashes `df.to_json()` at `:4033`, and `_load_moe_df` filters rows without dropping columns (`:3628`, `:3712-3720`), so cached artifacts are separated incidentally. | | Disposition | **PORT** the resolver override, the config field and its copy, the predictor helper, dataset admission, the training signature term, and the per-model training identity. **ADAPT** the registry key widening: the candidate's helper rewrite enumerates only `eager` and `kernel_only` and would break main's third `device_event` measurement family (`:4160-4162`, `:544-551`). | | Two implementation caveats | The candidate appends `_routing{path}` to the whole `ffn_signature`, which separates every MoE FFN model rather than only the routing-topk model. That is correctness-safe but over-broad and should be narrowed. Main's standalone trainer already calls this concept `routing_runtime_path` (`frontier/training/cli.py:153`), so the candidate's `moe_gating_routing_runtime_path` would be a third spelling; one name must be chosen before any public flag exists. | +| What the override would have decided | One thing only: which `moe.csv` rows train `moe_gating_routing_topk`, i.e. whether that operator's predicted cost is the fused-topk kernel's or the profiler's uniform round-robin path's (`frontier/profiling/moe/moe_impl.py:75-103`, `:195-225`). It never touched expert load, grouped GEMM, EP synchronization or scheduling. Measured on the three checked-in `moe.csv` datasets that carry both routing paths (`data/profiling/compute/a800/qwen3-a3b-30b-moe`, `h800/Phi-tiny-MoE-instruct`, `h800/step-moe-noquant-small`): matched-feature `moe_gating_routing_topk` medians are 0.073 / 0.189 ms, 0.050 / 0.082 ms and 0.041 / 0.072 ms (standard / uniform), and the difference is 3.1% (median; 24% at the smallest token counts), 7.3-7.8% and 4.2-5.4% of the summed per-layer operator medians. That is the cost of choosing the wrong path, and it is why the existing distribution-to-path mapping stays. It is not a fidelity change attributable to the override: with the field unset, every configuration resolved exactly as before. | +| Final disposition (2026-09-22) | **NOT PORTED.** The user decided to keep the current contract -- every cluster reads the one global `moe_routing_distribution_type` and resolves one routing path per run -- rather than add the override that would make the collision reachable and then the registry axis that would defend against it. The drafted implementation (global field, three per-role overrides, four CLI flags, single-owner resolver, routing-aware training signature and family gate, a `(model_name, identity, routing_runtime_path)` registry key with legacy-conflict rejection; 745 patch lines, never committed) is archived at `w5_reverted_moe_routing_runtime_path.patch` for reference. Nothing in `frontier/` changed for W5. The retained candidate tests `test_moe_routing_runtime.py` / `test_moe_routing_runtime_model_sharing.py` listed under "Retained from the candidate's tests" are therefore not adopted either. | ### W6 — Legacy fused-MoE profiling arithmetic @@ -135,7 +138,7 @@ The candidate deletes `VLLMv1EngineReplicaScheduler._get_num_waiting_reqs_for_de 1. Whether `_schedule_batch_mode`'s per-replica grouped return order is load-bearing for the consumers of `ClusterScheduleEvent`'s request mapping (W2). 2. RESOLVED. Whether W3's shared forward identity is the correct report-order key for W4, or whether a separate Replica-scoped step identity is required (D1). W3 delivered one step id per monolithic cohort in a single `"forward"` namespace regardless of its lanes' phases, and W4 (`10dd474`) measured that identity at the report boundary: a MoE Replica's keys are non-decreasing with every equal-key pair carrying two distinct lanes, while a dense Replica keeps a per-lane creation counter whose keys interleave under staggered online arrivals. No separate identity was introduced; the dense multi-lane shape is rejected in the constructor instead. -3. Which of the three existing spellings becomes the single public name for the routing implementation identity (W5). +3. CLOSED 2026-09-22. The naming question was first answered (`moe_routing_runtime_path`, a `ReplicaConfig` field), then made moot: the user decided W5 is not ported at all, because the collision it fixes is unreachable while the routing distribution has a single global field, and the override that would make it reachable was judged not worth its configuration surface. See the W5 "Final disposition" row. No public name exists; the CSV column `routing_runtime_path`, the trainer flag `--routing_runtime_path` and main's internal spellings are unchanged. 4. Pipeline-parallel behavior of the component ledgers is untested in both trees. 5. A multi-lane monolithic MoE Replica has no released wrapper: the public MoE examples enforce `ATTN_TP == MOE_TP * MOE_EP` while the runtime enforces `attn_tp * attn_dp == moe_tp * moe_ep`, and those have no common solution above one lane. Its evidence therefore has to come from a direct-construction fixture, never from the fidelity matrix. Whether the release should offer such a wrapper is a product question, not a correctness one, and is left open. 6. The upstream `fused_moe.py` fork change passes a fifth `renormalize` argument to `torch.ops._moe_C.topk_softmax` while the in-tree schema still declares four; the prebuilt extension could not be inspected on this host. Numerically a no-op, but it would raise rather than degrade. Relevant only if W6 native validation runs against the fork's compiled package. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w5_reverted_moe_routing_runtime_path.patch b/task_memory/task_2026-09-21_issue26_correctness_pr/w5_reverted_moe_routing_runtime_path.patch new file mode 100644 index 00000000..3300684d --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w5_reverted_moe_routing_runtime_path.patch @@ -0,0 +1,745 @@ +diff --git a/frontier/config/cluster_config.py b/frontier/config/cluster_config.py +index 52bc253..fb199ab 100644 +--- a/frontier/config/cluster_config.py ++++ b/frontier/config/cluster_config.py +@@ -197,6 +197,13 @@ class ClusterConfig(ClusterRoleConfigBuilder, ClusterTopologySummary): + "mode_dependency": "pd-af-disaggregation", + }, + ) ++ prefill_replica_config_moe_routing_runtime_path: Optional[str] = field( ++ default=None, ++ metadata={ ++ "help": "MoE gating routing implementation used for timing prediction in the prefill cluster. Valid values: 'standard_fused_topk' or 'uniform_topk'. Leave unset to inherit replica_config.", ++ "mode_dependency": "pd-af-disaggregation", ++ }, ++ ) + decode_attn_replica_config_memory_margin_fraction: Optional[float] = field( + default=None, + metadata={ +@@ -302,6 +309,13 @@ class ClusterConfig(ClusterRoleConfigBuilder, ClusterTopologySummary): + "mode_dependency": "pd-af-disaggregation", + }, + ) ++ decode_ffn_replica_config_moe_routing_runtime_path: Optional[str] = field( ++ default=None, ++ metadata={ ++ "help": "MoE gating routing implementation used for timing prediction in the decode FFN cluster. Valid values: 'standard_fused_topk' or 'uniform_topk'. Leave unset to inherit replica_config.", ++ "mode_dependency": "pd-af-disaggregation", ++ }, ++ ) + + # === PD-Disaggregation Mode: Unified DECODE Cluster Configuration === + decode_replica_config_memory_margin_fraction: Optional[float] = field( +@@ -374,6 +388,13 @@ class ClusterConfig(ClusterRoleConfigBuilder, ClusterTopologySummary): + "mode_dependency": "pd-disaggregation", + }, + ) ++ decode_replica_config_moe_routing_runtime_path: Optional[str] = field( ++ default=None, ++ metadata={ ++ "help": "MoE gating routing implementation used for timing prediction in the unified decode cluster. Valid values: 'standard_fused_topk' or 'uniform_topk'. Leave unset to inherit replica_config.", ++ "mode_dependency": "pd-disaggregation", ++ }, ++ ) + decode_replica_config_network_device: Optional[str] = field( + default=None, + metadata={ +diff --git a/frontier/config/cluster_role_config.py b/frontier/config/cluster_role_config.py +index 2393949..3dbfece 100644 +--- a/frontier/config/cluster_role_config.py ++++ b/frontier/config/cluster_role_config.py +@@ -73,6 +73,7 @@ class ClusterRoleConfigBuilder: + moe_routing_distribution_type = get_field_value( + "moe_routing_distribution_type" + ) ++ moe_routing_runtime_path = get_field_value("moe_routing_runtime_path") + attn_tensor_parallel_size = get_field_value("attn_tensor_parallel_size") + attn_dp = get_field_value("attn_dp") + else: +@@ -87,6 +88,7 @@ class ClusterRoleConfigBuilder: + moe_routing_distribution_type = get_field_value( + "moe_routing_distribution_type" + ) ++ moe_routing_runtime_path = get_field_value("moe_routing_runtime_path") + if cluster_prefix == "decode_ffn": + num_pipeline_stages = 1 + else: +@@ -116,6 +118,7 @@ class ClusterRoleConfigBuilder: + "decode_attn_steady_state_measurement_report_path" + ), + moe_routing_distribution_type=moe_routing_distribution_type, ++ moe_routing_runtime_path=moe_routing_runtime_path, + device=get_field_value("device"), + network_device=get_field_value("network_device"), + cluster_prefix=cluster_prefix, +@@ -587,6 +590,7 @@ class ClusterRoleConfigBuilder: + router_topk=original_config.router_topk, + moe_routing_seed=original_config.moe_routing_seed, + moe_routing_distribution_type=original_config.moe_routing_distribution_type, ++ moe_routing_runtime_path=original_config.moe_routing_runtime_path, + moe_routing_trace_path=original_config.moe_routing_trace_path, + decode_attn_initial_lane_trace_path=( + original_config.decode_attn_initial_lane_trace_path +diff --git a/frontier/config/replica_config.py b/frontier/config/replica_config.py +index 6514f9d..2c5eaf1 100644 +--- a/frontier/config/replica_config.py ++++ b/frontier/config/replica_config.py +@@ -3,6 +3,7 @@ + from __future__ import annotations + + from dataclasses import dataclass, field ++from typing import Optional + + from frontier.config.device_sku_config import BaseDeviceSKUConfig + from frontier.config.model_config import BaseModelConfig +@@ -10,6 +11,9 @@ from frontier.config.node_sku_config import BaseNodeSKUConfig + from frontier.config.speculative_decoding_config import ( + SpeculativeDecodingConfig, + ) ++from frontier.moe_routing_runtime import ( ++ validate_moe_gating_routing_runtime_path, ++) + + + @dataclass +@@ -95,6 +99,16 @@ class ReplicaConfig: + "token-to-expert load skew without changing router_topk/model semantics." + }, + ) ++ moe_routing_runtime_path: Optional[str] = field( ++ default=None, ++ metadata={ ++ "help": "MoE gating routing implementation used for timing prediction. " ++ "Valid values: 'standard_fused_topk' or 'uniform_topk'. This is a " ++ "separate dimension from moe_routing_distribution_type, which controls " ++ "expert-load skew. Leave unset to keep deriving the implementation " ++ "from the distribution." ++ }, ++ ) + device: str = field( + default="a100", + metadata={"help": "Device."}, +@@ -171,6 +185,15 @@ class ReplicaConfig: + f"got {self.moe_routing_distribution_type!r}" + ) + ++ # `None` keeps deriving the implementation from the distribution, which ++ # is what every configuration did before this override existed. An ++ # explicit value is validated here so an unsupported one fails at ++ # configuration time rather than at dataset admission. ++ if self.moe_routing_runtime_path is not None: ++ self.moe_routing_runtime_path = validate_moe_gating_routing_runtime_path( ++ self.moe_routing_runtime_path ++ ) ++ + # Validate pipeline parallelism configuration early + if self.model_config.num_layers % self.num_pipeline_stages != 0: + raise ValueError( +diff --git a/frontier/execution_time_predictor/layer_contract_resolution.py b/frontier/execution_time_predictor/layer_contract_resolution.py +index 8306f09..78eea94 100644 +--- a/frontier/execution_time_predictor/layer_contract_resolution.py ++++ b/frontier/execution_time_predictor/layer_contract_resolution.py +@@ -21,7 +21,7 @@ from frontier.execution_time_predictor.prediction_model_identity import ( + from frontier.model_architectures import LayerKind, ResolvedLayerContract + from frontier.moe_routing_runtime import ( + filter_moe_gating_routing_topk_rows, +- resolve_moe_gating_routing_runtime_path, ++ resolve_replica_moe_routing_runtime_path, + ) + from frontier.operators.binding import resolve_operator_query_tp_mode + from frontier.operators.families import ( +@@ -448,8 +448,8 @@ class LayerContractResolution: + ].drop_duplicates().itertuples(index=False, name=None) + } + ) +- requested_routing_runtime_path = resolve_moe_gating_routing_runtime_path( +- getattr(replica_config, "moe_routing_distribution_type", "balanced") ++ requested_routing_runtime_path = ( ++ resolve_replica_moe_routing_runtime_path(replica_config) + ) + + missing_requirements: List[str] = [] +diff --git a/frontier/execution_time_predictor/moe_routing_workload.py b/frontier/execution_time_predictor/moe_routing_workload.py +index 0f6bc13..09826f4 100644 +--- a/frontier/execution_time_predictor/moe_routing_workload.py ++++ b/frontier/execution_time_predictor/moe_routing_workload.py +@@ -22,7 +22,6 @@ from frontier.moe_ep_workload import ( + resolve_ep_lane_workload, + resolve_routing_details, + ) +-from frontier.moe_routing_runtime import resolve_moe_gating_routing_runtime_path + from frontier.types import ClusterType + from typing import Dict, List, Mapping, Optional + +@@ -53,9 +52,14 @@ class MoeRoutingWorkload: + ) + + def _get_requested_moe_gating_routing_runtime_path(self) -> str: +- return resolve_moe_gating_routing_runtime_path( +- getattr(self, "_moe_routing_distribution_type", "balanced") +- ) ++ """Return the routing implementation this predictor was built for. ++ ++ Resolved once in the predictor's constructor, so dataset admission and ++ the training identity cannot disagree with the runtime query about ++ which routing implementation is in force. ++ """ ++ ++ return self._moe_gating_routing_runtime_path + + @staticmethod + +diff --git a/frontier/execution_time_predictor/prediction_family_trainers.py b/frontier/execution_time_predictor/prediction_family_trainers.py +index da19f7d..02f9bdc 100644 +--- a/frontier/execution_time_predictor/prediction_family_trainers.py ++++ b/frontier/execution_time_predictor/prediction_family_trainers.py +@@ -31,6 +31,7 @@ from frontier.execution_time_predictor.prediction_model_identity import ( + _get_prefill_hot_moe_gating_model_names, + _is_moe_gating_family_model_name, + _layer_contract_kwargs, ++ _routing_runtime_kwargs, + _normalize_layer_contract_context, + _resolve_model_architecture_profile, + _resolve_model_architecture_profile_id, +@@ -49,7 +50,9 @@ from frontier.moe_gating_runtime import ( + ) + from frontier.moe_routing_runtime import ( + filter_moe_gating_routing_topk_rows, +- resolve_moe_gating_routing_runtime_path, ++ MOE_GATING_ROUTING_TOPK_MODEL_NAME, ++ is_moe_routing_runtime_sensitive_model, ++ resolve_replica_moe_routing_runtime_path, + ) + from frontier.operators.families import ( + FFN_FAMILY, +@@ -113,7 +116,38 @@ class PredictionFamilyTrainers: + f"_family{self._measurement_family_name(active_measurement_type)}" + ) + +- if ffn_signature in trained_model_signatures: ++ # Only the gating top-k model's measured cost depends on the routing ++ # implementation, so the implementation separates that one model's ++ # signature rather than the whole FFN family's. The family gate below ++ # therefore also has to see this cluster's routing model, or a second ++ # cluster asking for a different implementation would be skipped here ++ # and would silently reuse the first cluster's estimator. ++ requested_routing_runtime_path = ( ++ resolve_replica_moe_routing_runtime_path(replica_config) ++ if is_moe_model ++ else None ++ ) ++ ++ def _routing_runtime_path_for(model_name: str) -> Optional[str]: ++ """Return the routing implementation one model's cost depends on.""" ++ ++ if is_moe_routing_runtime_sensitive_model(model_name): ++ return requested_routing_runtime_path ++ return None ++ ++ def _model_signature(model_name: str) -> str: ++ routing_runtime_path = _routing_runtime_path_for(model_name) ++ if routing_runtime_path is not None: ++ return ( ++ f"{model_name}_routing{routing_runtime_path}_{ffn_signature}" ++ ) ++ return f"{model_name}_{ffn_signature}" ++ ++ if ffn_signature in trained_model_signatures and ( ++ requested_routing_runtime_path is None ++ or _model_signature(MOE_GATING_ROUTING_TOPK_MODEL_NAME) ++ in trained_model_signatures ++ ): + logger.info(f"Skipping FFN models training for {cluster_type} - already trained with signature {ffn_signature}") + return models + +@@ -169,9 +203,6 @@ class PredictionFamilyTrainers: + cluster_type, + **_layer_contract_kwargs(primary_contract), + ) +- requested_routing_runtime_path = resolve_moe_gating_routing_runtime_path( +- getattr(replica_config, "moe_routing_distribution_type", "balanced") +- ) + + moe_df_cache: Dict[ + Tuple[ +@@ -215,9 +246,7 @@ class PredictionFamilyTrainers: + else: + ep_key = replica_config.moe_expert_parallel_size + +- runtime_path_key: Optional[str] = None +- if base_model_name == "moe_gating_routing_topk": +- runtime_path_key = requested_routing_runtime_path ++ runtime_path_key = _routing_runtime_path_for(base_model_name) + + gating_context_key: Optional[str] = None + if _is_moe_gating_family_model_name(base_model_name): +@@ -271,7 +300,7 @@ class PredictionFamilyTrainers: + return moe_df_cache[cache_key], tp_key, ep_key, op_layer_contract + + for model_name in moe_model_names: +- model_signature = f"{model_name}_{ffn_signature}" ++ model_signature = _model_signature(model_name) + if model_signature not in trained_model_signatures: + try: + ( +@@ -368,6 +397,11 @@ class PredictionFamilyTrainers: + training_context=op_training_context, + ) + train_kwargs.update(_layer_contract_kwargs(op_layer_contract)) ++ # The routing implementation is part of this model's ++ # identity, not only of its training-row selection. ++ train_kwargs.update( ++ _routing_runtime_kwargs(_routing_runtime_path_for(model_name)) ++ ) + models[model_name] = self._train_single_model( + **train_kwargs, + ) +@@ -1505,8 +1539,15 @@ class PredictionFamilyTrainers: + training_context: Optional[Dict[str, Any]] = None, + persist_exact_lookup: bool = True, + layer_contract: Optional[ResolvedLayerContract] = None, ++ routing_runtime_path: Optional[str] = None, + ) -> BaseEstimator: +- """Train a single model with given data and configuration.""" ++ """Train a single model with given data and configuration. ++ ++ `routing_runtime_path` is the MoE routing implementation this model was ++ trained on. It is set only for the models whose measured cost depends on ++ it, and it separates their registry identity from an otherwise identical ++ model trained on the other implementation. ++ """ + layer_contract, training_context = _normalize_layer_contract_context( + training_context, + explicit_layer_contract=layer_contract, +@@ -1591,6 +1632,7 @@ class PredictionFamilyTrainers: + model_name, + profiling_precision, + cached_model, ++ **_routing_runtime_kwargs(routing_runtime_path), + **_layer_contract_kwargs(layer_contract), + ) + return cached_model +@@ -1695,6 +1737,7 @@ class PredictionFamilyTrainers: + model_name, + profiling_precision, + best_estimator, ++ **_routing_runtime_kwargs(routing_runtime_path), + **_layer_contract_kwargs(layer_contract), + ) + return best_estimator +diff --git a/frontier/execution_time_predictor/prediction_model_identity.py b/frontier/execution_time_predictor/prediction_model_identity.py +index 7153053..77c1429 100644 +--- a/frontier/execution_time_predictor/prediction_model_identity.py ++++ b/frontier/execution_time_predictor/prediction_model_identity.py +@@ -303,6 +303,12 @@ def _layer_contract_kwargs( + if operator_name is not None: + kwargs["operator_name"] = operator_name + return kwargs ++def _routing_runtime_kwargs(routing_runtime_path: Optional[str]) -> Dict[str, Any]: ++ """Return the routing axis only for the models whose cost depends on it.""" ++ ++ if routing_runtime_path is None: ++ return {} ++ return {"routing_runtime_path": routing_runtime_path} + def _is_moe_gating_family_model_name(model_name: str) -> bool: + base_model_name = get_moe_gating_base_model_name(model_name) + return _get_moe_family_operator_by_model_name( +diff --git a/frontier/execution_time_predictor/prediction_model_registry.py b/frontier/execution_time_predictor/prediction_model_registry.py +index bc250b8..e971df1 100644 +--- a/frontier/execution_time_predictor/prediction_model_registry.py ++++ b/frontier/execution_time_predictor/prediction_model_registry.py +@@ -23,6 +23,10 @@ from frontier.execution_time_predictor.prediction_model_identity import ( + from frontier.logger import init_logger + from frontier.model_architectures import LayerKind, ResolvedLayerContract + from frontier.moe_gating_runtime import get_moe_gating_base_model_name ++from frontier.moe_routing_runtime import ( ++ is_moe_routing_runtime_sensitive_model, ++ resolve_replica_moe_routing_runtime_path, ++) + from frontier.types import ClusterType, MeasurementType + from sklearn.base import BaseEstimator + from typing import Any, Dict, List, Mapping, Optional, Tuple +@@ -254,9 +258,41 @@ class PredictionModelRegistry: + setattr(model, "_frontier_layer_cache_identity", identity) + return identity + ++ @staticmethod ++ def _model_routing_runtime_identity( ++ model: BaseEstimator, ++ routing_runtime_path: Optional[str], ++ ) -> Optional[str]: ++ """Attach and return the routing implementation a model was trained on. ++ ++ Mirrors `_model_contract_identity`: a conflict between what the caller ++ asks for and what the estimator already carries is an error, never a ++ relabeling. ++ """ ++ ++ attached = getattr(model, "_frontier_routing_runtime_path", None) ++ if attached is not None and not isinstance(attached, str): ++ raise TypeError( ++ "_frontier_routing_runtime_path must be a string when present" ++ ) ++ if ( ++ routing_runtime_path is not None ++ and attached is not None ++ and routing_runtime_path != attached ++ ): ++ raise ValueError( ++ "model routing implementation conflicts with the requested " ++ f"routing_runtime_path: attached={attached!r}, " ++ f"requested={routing_runtime_path!r}" ++ ) ++ resolved = routing_runtime_path or attached ++ if resolved is not None: ++ setattr(model, "_frontier_routing_runtime_path", resolved) ++ return resolved ++ + def _contract_model_registry( + self, family_name: str +- ) -> Dict[Tuple[str, Optional[str]], BaseEstimator]: ++ ) -> Dict[Tuple[str, Optional[str], Optional[str]], BaseEstimator]: + registry_attr = { + "eager": "_trained_models_eager_by_contract", + "device_event": "_trained_models_device_event_by_contract", +@@ -268,7 +304,7 @@ class PredictionModelRegistry: + + def _contract_precision_registry( + self, family_name: str +- ) -> Dict[str, Dict[Tuple[str, Optional[str]], BaseEstimator]]: ++ ) -> Dict[str, Dict[Tuple[str, Optional[str], Optional[str]], BaseEstimator]]: + registry_attr = { + "eager": "_models_by_precision_eager_by_contract", + "device_event": "_models_by_precision_device_event_by_contract", +@@ -323,20 +359,46 @@ class PredictionModelRegistry: + precision: str, + model: BaseEstimator, + layer_contract: Optional[ResolvedLayerContract] = None, ++ routing_runtime_path: Optional[str] = None, + ) -> None: ++ """Register one trained estimator under its selected semantic domain. ++ ++ ``routing_runtime_path`` is a second identity axis, carried only by the ++ models whose measured cost depends on the MoE routing implementation. ++ Two implementations produce the same layer shape, so without it one ++ cluster's estimator would answer another cluster's query. ++ """ ++ + if not isinstance(precision, str) or not precision.strip(): + raise ValueError(f"precision must be a non-empty string, got {precision!r}") + precision_key = precision.upper() + family_name = self._measurement_family_name(self._active_measurement_type) + identity = self._model_contract_identity(model, layer_contract) ++ routing_key = self._model_routing_runtime_identity(model, routing_runtime_path) + if identity is None: ++ # The legacy registry is keyed by name alone, so it cannot hold two ++ # routing implementations of one model. Reject the collision rather ++ # than overwrite the first estimator with the second. ++ stored = self._legacy_model_registry(family_name).get(model_name) ++ if stored is not None: ++ stored_routing = getattr( ++ stored, "_frontier_routing_runtime_path", None ++ ) ++ if stored_routing != routing_key: ++ raise ValueError( ++ f"Model {model_name!r} is already registered without a " ++ "typed layer contract for routing_runtime_path=" ++ f"{stored_routing!r} and cannot also hold " ++ f"{routing_key!r}. A typed architecture profile is " ++ "required to separate two routing implementations." ++ ) + self._legacy_model_registry(family_name)[model_name] = model + self._legacy_precision_registry(family_name).setdefault( + precision_key, {} + )[model_name] = model + return + +- model_key = (model_name, identity) ++ model_key = (model_name, identity, routing_key) + self._contract_model_registry(family_name)[model_key] = model + self._contract_precision_registry(family_name).setdefault( + precision_key, {} +@@ -349,6 +411,7 @@ class PredictionModelRegistry: + *, + precision_key: Optional[str] = None, + requested_identity: Optional[str] = None, ++ routing_runtime_path: Optional[str] = None, + ) -> Optional[BaseEstimator]: + if precision_key is not None: + precision_key = precision_key.upper() +@@ -358,10 +421,32 @@ class PredictionModelRegistry: + else: + source = self._contract_model_registry(family_name) + +- typed_candidates = { +- identity: candidate +- for (candidate_name, identity), candidate in source.items() ++ # A routing request selects among the estimators registered for one ++ # model name. Without one, a single registered routing implementation ++ # is unambiguous and several are not, which mirrors how this method ++ # already treats an absent layer-contract request. ++ matched = { ++ (identity, candidate_routing): candidate ++ for (candidate_name, identity, candidate_routing), candidate in ( ++ source.items() ++ ) + if candidate_name == model_name ++ and ( ++ routing_runtime_path is None ++ or candidate_routing == routing_runtime_path ++ ) ++ } ++ if routing_runtime_path is None: ++ registered_routing_paths = {routing for _identity, routing in matched} ++ if len(registered_routing_paths) > 1: ++ raise ValueError( ++ f"Model '{model_name}' is trained for multiple MoE routing " ++ "implementations; provide routing_runtime_path. Available " ++ "implementations: " ++ f"{sorted(str(path) for path in registered_routing_paths)}" ++ ) ++ typed_candidates = { ++ identity: candidate for (identity, _routing), candidate in matched.items() + } + legacy = ( + self._legacy_precision_bucket(family_name, precision_key).get(model_name) +@@ -373,6 +458,14 @@ class PredictionModelRegistry: + if legacy is not None + else None + ) ++ if ( ++ legacy is not None ++ and routing_runtime_path is not None ++ and getattr(legacy, "_frontier_routing_runtime_path", None) ++ != routing_runtime_path ++ ): ++ legacy = None ++ legacy_identity = None + + if requested_identity is not None: + model = typed_candidates.get(requested_identity) +@@ -436,6 +529,30 @@ class PredictionModelRegistry: + is_moe_model=layer_kind is not LayerKind.DENSE, + ) + ++ def _resolve_cluster_routing_runtime_path( ++ self, ++ cluster_type: Optional[ClusterType], ++ model_name: str, ++ ) -> Optional[str]: ++ """Resolve the MoE routing implementation one cluster view asks for. ++ ++ Only the gating top-k model's measured cost depends on the routing ++ implementation, so every other model keeps a `None` axis and its ++ lookups are unchanged. ++ """ ++ ++ if cluster_type is None or not is_moe_routing_runtime_sensitive_model( ++ model_name ++ ): ++ return None ++ cluster_configs = getattr(self, "_cluster_configs", None) or {} ++ replica_config = getattr( ++ cluster_configs.get(cluster_type), "replica_config", None ++ ) ++ if replica_config is None: ++ return None ++ return resolve_replica_moe_routing_runtime_path(replica_config) ++ + def _is_ffn_typed_model_for_cluster( + self, + cluster_type: Optional[ClusterType], +@@ -464,25 +581,40 @@ class PredictionModelRegistry: + """Project one measurement family's canonical registries to model names.""" + + names = set(self._legacy_model_registry(family_name)) +- names.update( +- model_name +- for model_name, _identity in self._contract_model_registry(family_name) +- ) ++ registered_routing_paths: Dict[str, set] = {} ++ for model_name, _identity, routing in self._contract_model_registry( ++ family_name ++ ): ++ names.add(model_name) ++ registered_routing_paths.setdefault(model_name, set()).add(routing) + models: Dict[str, BaseEstimator] = {} + for model_name in sorted(names): + if self._is_ffn_typed_model_for_cluster(cluster_type, model_name): + continue + contract = self._resolve_cluster_model_contract(cluster_type, model_name) + identity = _serialize_selected_layer_cache_identity(contract) ++ # Only ask for a routing implementation when the registry actually ++ # holds more than one, so a configuration that trains a single ++ # implementation keeps resolving exactly the model it always did. ++ routing_runtime_path = None ++ if len(registered_routing_paths.get(model_name, ())) > 1: ++ routing_runtime_path = self._resolve_cluster_routing_runtime_path( ++ cluster_type, model_name ++ ) + model = self._get_family_model( + family_name, + model_name, + requested_identity=identity, ++ routing_runtime_path=routing_runtime_path, + ) +- if identity is not None and model is None: ++ if model is None and ( ++ identity is not None or routing_runtime_path is not None ++ ): + raise ValueError( +- f"No trained model for {model_name!r} matches the typed contract " +- f"requested by cluster {cluster_type!r}: {identity}" ++ f"No trained model for {model_name!r} matches the domain " ++ f"requested by cluster {cluster_type!r}: " ++ f"layer_contract={identity}, " ++ f"routing_runtime_path={routing_runtime_path!r}" + ) + if model is not None: + models[model_name] = model +@@ -493,8 +625,9 @@ class PredictionModelRegistry: + model_name: str, + precision: Optional[str] = None, + layer_contract: Optional[ResolvedLayerContract] = None, ++ routing_runtime_path: Optional[str] = None, + ) -> Optional[BaseEstimator]: +- """Get a model by name, precision, and optional typed contract.""" ++ """Get a model by name, precision, and optional typed domain.""" + + if self._all_dummy_mode: + return None +@@ -506,6 +639,7 @@ class PredictionModelRegistry: + model_name, + precision_key=precision_key, + requested_identity=requested_identity, ++ routing_runtime_path=routing_runtime_path, + ) + if model is not None: + return model +diff --git a/frontier/execution_time_predictor/shared_prediction_model_manager.py b/frontier/execution_time_predictor/shared_prediction_model_manager.py +index c432721..e4cb9ff 100644 +--- a/frontier/execution_time_predictor/shared_prediction_model_manager.py ++++ b/frontier/execution_time_predictor/shared_prediction_model_manager.py +@@ -74,7 +74,6 @@ from frontier.moe_gating_runtime import ( + ) + from frontier.moe_routing_runtime import ( + filter_moe_gating_routing_topk_rows, +- resolve_moe_gating_routing_runtime_path, + ) + from frontier.operators.families import ( + FFN_FAMILY, +diff --git a/frontier/execution_time_predictor/sklearn_moe_execution_time_predictor.py b/frontier/execution_time_predictor/sklearn_moe_execution_time_predictor.py +index 92c8d21..d78be0d 100644 +--- a/frontier/execution_time_predictor/sklearn_moe_execution_time_predictor.py ++++ b/frontier/execution_time_predictor/sklearn_moe_execution_time_predictor.py +@@ -40,7 +40,7 @@ from frontier.moe_gating_runtime import ( + ) + from frontier.moe_routing_runtime import ( + filter_moe_gating_routing_topk_rows, +- resolve_moe_gating_routing_runtime_path, ++ resolve_replica_moe_routing_runtime_path, + ) + from frontier.moe_ep_workload import ( + EPLaneWorkload, +@@ -419,10 +419,11 @@ class SklearnMoEExecutionTimePredictor( + "moe_routing_seed must be an exact non-negative int, " + f"got {self._moe_routing_seed}." + ) ++ # One owner for the routing implementation identity. `None` on the ++ # configuration keeps deriving it from the distribution; an explicit ++ # value overrides it. Every consumer reads this attribute. + self._moe_gating_routing_runtime_path = ( +- resolve_moe_gating_routing_runtime_path( +- self._moe_routing_distribution_type +- ) ++ resolve_replica_moe_routing_runtime_path(replica_config) + ) + + super().__init__( +diff --git a/frontier/moe_routing_runtime.py b/frontier/moe_routing_runtime.py +index 70b1463..00412f8 100644 +--- a/frontier/moe_routing_runtime.py ++++ b/frontier/moe_routing_runtime.py +@@ -1,7 +1,11 @@ + from __future__ import annotations + ++from typing import Optional ++ + import pandas as pd + ++from frontier.moe_gating_runtime import get_moe_gating_base_model_name ++ + + STANDARD_MOE_GATING_ROUTING_RUNTIME_PATH = "standard_fused_topk" + UNIFORM_MOE_GATING_ROUTING_RUNTIME_PATH = "uniform_topk" +@@ -10,6 +14,19 @@ SUPPORTED_MOE_GATING_ROUTING_RUNTIME_PATHS = { + UNIFORM_MOE_GATING_ROUTING_RUNTIME_PATH, + } + ++# The routing implementation changes the measured cost of selecting and ++# normalizing the top-k experts, and of nothing else in the MoE FFN. ++MOE_GATING_ROUTING_TOPK_MODEL_NAME = "moe_gating_routing_topk" ++ ++ ++def is_moe_routing_runtime_sensitive_model(model_name: str) -> bool: ++ """Report whether one trained model's cost depends on the routing path.""" ++ ++ return ( ++ get_moe_gating_base_model_name(model_name) ++ == MOE_GATING_ROUTING_TOPK_MODEL_NAME ++ ) ++ + + def validate_moe_gating_routing_runtime_path(requested_runtime_path: str) -> str: + normalized_path = str(requested_runtime_path).strip() +@@ -36,6 +53,37 @@ def resolve_moe_gating_routing_runtime_path( + ) + + ++def select_moe_gating_routing_runtime_path( ++ moe_routing_distribution_type: str, ++ moe_routing_runtime_path: Optional[str] = None, ++) -> str: ++ """Resolve the routing runtime path one configuration asks for. ++ ++ ``None`` means "follow the distribution", which is what every configuration ++ did before the override existed. An implicit default and an explicit equal ++ value therefore resolve to the same string, so two configurations that ++ differ only in how they spell the same runtime keep sharing one model. ++ """ ++ ++ if moe_routing_runtime_path is None: ++ return resolve_moe_gating_routing_runtime_path(moe_routing_distribution_type) ++ return validate_moe_gating_routing_runtime_path(moe_routing_runtime_path) ++ ++ ++def resolve_replica_moe_routing_runtime_path(replica_config) -> str: ++ """Resolve one Replica configuration's routing runtime path, once. ++ ++ Every consumer -- dataset admission, training identity, the trained-model ++ registry and the runtime query -- reads the path from here, so none of them ++ re-derives it from the distribution on its own. ++ """ ++ ++ return select_moe_gating_routing_runtime_path( ++ replica_config.moe_routing_distribution_type, ++ replica_config.moe_routing_runtime_path, ++ ) ++ ++ + def filter_moe_gating_routing_topk_rows( + df: pd.DataFrame, + *, +diff --git a/tests/unit/test_profiling_governance_minimal_red.py b/tests/unit/test_profiling_governance_minimal_red.py +index a6ed0ff..747aa1b 100644 +--- a/tests/unit/test_profiling_governance_minimal_red.py ++++ b/tests/unit/test_profiling_governance_minimal_red.py +@@ -1392,6 +1392,8 @@ def test_routed_and_shared_training_carry_distinct_contracts(tmp_path) -> None: + attn_tensor_parallel_size=8, + moe_tensor_parallel_size=1, + moe_expert_parallel_size=8, ++ moe_routing_distribution_type="balanced", ++ moe_routing_runtime_path=None, + speculative_decoding_config=None, + ) + manager = object.__new__(ExecutionTimePredictionModelManager) From bbbfcaa3d849b6cb2173bf09c6b31d5a4ffdfa80 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 11:30:41 +0800 Subject: [PATCH 19/80] docs(review): mark the W5 closure published and record the PR 35 section --- .../task_2026-09-21_issue26_correctness_pr/progress.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index db1002ab..936d1d30 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -19,7 +19,7 @@ | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | | Current step | Step 5 closed without source changes (NOT PORTED, user decision); records written | -| Publication | LOCAL_ONLY (this record) | +| Publication | PUSHED_VERIFIED (records) | | Next action | Checkpoint E second half: the D2-scoped W6 legacy fused-MoE profiling arithmetic (`frontier/profiling/moe/moe_vllm_kernel.py`), then Checkpoint F's conditional W7. | ## Step status @@ -31,7 +31,7 @@ | 2 | RR DP rotation | PASS | unit PASS (23 tests); matrix PASS against a stated expectation, re-measured 2026-09-22 with one harness revision | PUSHED_VERIFIED | REVIEWED (R35-01 closed) | | 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 4 | Opt-in vLLM DP placement | PASS | unit PASS (61 new, 3778 total, failure set identical to the parent); integration PASS (3 cases in the real event loop, including a placement that diverges from round-robin); five deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | -| 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | records only | REVIEWED (user chose to keep the single global field) | +| 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | PUSHED_VERIFIED (records + PR 35 section) | REVIEWED (user chose to keep the single global field) | | 6 | Legacy fused-MoE profiling | NOT_STARTED | — | — | — | | 7 | Optional zero-payload backend | NOT_STARTED (facts in `plan.md` A7) | — | — | — | | 8 | Combined regression, PR hand-off | NOT_STARTED | — | — | — | @@ -77,6 +77,8 @@ - 2026-09-22: W5 magnitude, measured on request. On the three checked-in `moe.csv` files carrying both routing paths, the matched `moe_gating_routing_topk` cost differs by 3.1% (median; up to 24% at the smallest token counts), 7.3-7.8% and 4.2-5.4% of the summed per-layer operator medians (a800 qwen3-a3b-30b-moe, h800 Phi-tiny-MoE, h800 step-moe-noquant-small). This is the cost of the *existing* mapping picking the wrong kernel for a scenario, and the reason the mapping stays; the drafted override itself changed nothing while unset. - 2026-09-22: W5 closed as NOT PORTED. After the explanation of the collision case, of what `uniform_topk` is (the profiler's round-robin routing path, `moe_impl.py:75-103`), and of the three options, the user decided to keep the current contract: one global `moe_routing_distribution_type`, one derived path per run. The twelve drafted source/test files were restored with `git checkout` after saving the diff to `w5_reverted_moe_routing_runtime_path.patch` (745 lines). `review.md` W5 rows corrected and closed; `requirements.md` records the decision verbatim; `plan.md` section 11 is kept as specified with a closure note. No `frontier/` change, so no fidelity matrix run for this step. +- 2026-09-22: W5 closure published. `de2bee8` pushed to `origin/fix/issue26-correctness-pr`; PR #35 gained a "W5, which is not in this PR, and why" section carrying the reachability correction with its `cluster_role_config.py:58-63` citation, what the module decides (`moe_impl.py:75-103`, `:195-225`), the three-dataset magnitude table, and the explicit statement that the reverted override's own fidelity effect was zero by construction. The Status line now reads "W6 onward is still to come". PR #35 stays draft. Verified the published body matches what was sent (the one-byte difference is the trailing newline `gh --jq` adds). + ## Step 3 scoping, as recorded before implementation Kept as written so the implementation can be read against the scope it started from. Step 3 is now complete; see the W3 entries above and `design.md`. From 7269bac885c91910f0e1e74500fd752c41fab4a8 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 13:32:20 +0800 Subject: [PATCH 20/80] fix(profiling): complete the legacy fused-MoE expert computation `_run_fused_moe_iteration` fed the second expert GEMM the gate half of the first projection, unactivated, and never reduced the per-expert outputs. vLLM's own `fused_experts_impl` runs GEMM1, a gated activation, the optional activation quantization, GEMM2 with the routing weights, and a local top-k reduction. Two operations were missing: `silu_and_mul` and `moe_sum`. The path is live. It is selected whenever vLLM exposes the low-level API, which the pinned reference v0.10.2 does and `environment_profiling.yml` pins with `vllm>=0.10,<0.11`. Estimated from the checked-in datasets, the omitted kernels are 16.5% of the corrected `moe_grouped_gemm` time at 4096 tokens on a800/qwen3-a3b-30b-moe, 6.8% median over its rows. Both new buffers are allocated once at the profiling site, outside the timed step, and the activation writes into a preallocated tensor rather than through the `SiluAndMul` wrapper, whose `forward` allocates on every call and would land inside the measured region. `moe_grouped_gemm` now means the complete local expert computation. That is already what the functional backend measured, since `fused_experts` returns reduced hidden states, so this removes a scope disagreement between the two backends rather than creating one. The reduction is a sum over one token's own experts; no communication cost is added or removed. No gated/non-gated branch: the profiler only ever materializes `w1` with `2 * E` rows, so a conditional would be unreachable. 7 CPU tests cover the arithmetic against a written-out reference, the comparison that distinguishes it from the old slice, call order and operand provenance, routing-weight placement, the FP8 quantizer input, workspace reuse, and allocation-site dimensions. Native GPU parity is NOT_RUN: no local environment selects this path. --- frontier/profiling/moe/moe_vllm_kernel.py | 57 ++- .../progress.md | 12 +- .../review.md | 8 + ...t_2026-09-22_w6_fused_expert_arithmetic.md | 208 +++++++++ .../validation.md | 16 + .../unit/test_moe_fused_expert_arithmetic.py | 396 ++++++++++++++++++ 6 files changed, 691 insertions(+), 6 deletions(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md create mode 100644 tests/unit/test_moe_fused_expert_arithmetic.py diff --git a/frontier/profiling/moe/moe_vllm_kernel.py b/frontier/profiling/moe/moe_vllm_kernel.py index c89b97f9..b7d35aee 100644 --- a/frontier/profiling/moe/moe_vllm_kernel.py +++ b/frontier/profiling/moe/moe_vllm_kernel.py @@ -37,6 +37,8 @@ VLLM_API_VERSION = None FP8_QUANT_AVAILABLE = False _functional_fused_experts = None +# Bound only by the low-level import below, like `_functional_fused_experts`. +_vllm_custom_ops = None _FUNCTIONAL_MXFP4_STATES: Dict[Tuple[Any, ...], Dict[str, Any]] = {} @@ -126,6 +128,11 @@ def plan_mxfp4_weight_layout( try_get_optimal_moe_config, get_config_dtype_str, ) + # The gated activation and the local top-k reduction belong to the same + # low-level API as the two kernel invocations. Importing them here means + # a build without them selects the functional path instead of running an + # incomplete expert computation. + from vllm import _custom_ops as _vllm_custom_ops VLLM_API_VERSION = "0.10.x" except ImportError: @@ -368,6 +375,8 @@ def _run_fused_moe_iteration( w2: torch.Tensor, intermediate_cache1: torch.Tensor, intermediate_cache2: torch.Tensor, + intermediate_cache3: torch.Tensor, + out_hidden_states: torch.Tensor, topk_weights: torch.Tensor, sorted_token_ids: torch.Tensor, expert_ids: torch.Tensor, @@ -383,6 +392,24 @@ def _run_fused_moe_iteration( per_channel_quant: bool = False, block_shape: Optional[List[int]] = None, ) -> None: + """Run one complete local expert computation, as vLLM's own path does. + + Reference: `fused_experts_impl` in vLLM 0.10.x `fused_moe.py`, which runs + the first expert GEMM, a gated activation, the optional activation + quantization, the second expert GEMM with the routing weights, and a local + reduction of the top-k expert outputs. The buffers follow the same naming: + + ```text + intermediate_cache1 (M, top_k, 2 * E) first GEMM output, gate | up + intermediate_cache2 (M * top_k, E) gated activation output + intermediate_cache3 (M, top_k, H) second GEMM output, per expert + out_hidden_states (M, H) local top-k reduction + ``` + + The reduction is a local sum over the `top_k` expert outputs of one token. + It is not a collective and adds no communication cost. + """ + _invoke_kernel( A=A.contiguous(), B=w1.contiguous(), @@ -401,8 +428,15 @@ def _run_fused_moe_iteration( block_shape=block_shape, ) - intermediate_cache1_flat = intermediate_cache1.view(-1, intermediate_cache1.shape[-1]) - intermediate_cache2_input = intermediate_cache1_flat[:, :expert_hidden_dim_per_partition].contiguous() + # Gated SiLU over the two halves of the first projection. Taking the gate + # half alone would skip this kernel and feed the wrong operand to the second + # GEMM. The caller always materializes `w1` with `2 * E` rows, so the gated + # layout is the only one this path can produce. + torch.ops._C.silu_and_mul( + intermediate_cache2, + intermediate_cache1.view(-1, intermediate_cache1.shape[-1]), + ) + intermediate_cache2_input = intermediate_cache2 intermediate_A_scale = None if use_fp8: @@ -415,7 +449,7 @@ def _run_fused_moe_iteration( _invoke_kernel( A=intermediate_cache2_input, B=w2.contiguous(), - C=intermediate_cache2.contiguous(), + C=intermediate_cache3.contiguous(), topk_weights=topk_weights.contiguous(), sorted_token_ids=sorted_token_ids.contiguous(), expert_ids=expert_ids.contiguous(), @@ -430,6 +464,9 @@ def _run_fused_moe_iteration( block_shape=block_shape, ) + # Local reduction of one token's top-k expert outputs. + _vllm_custom_ops.moe_sum(intermediate_cache3, out_hidden_states) + def _run_functional_fused_experts_iteration( A: torch.Tensor, @@ -910,12 +947,24 @@ def _step() -> None: dtype=output_dtype, ) intermediate_cache2 = torch.empty( + num_tokens * top_k, + expert_hidden_dim_per_partition, + device=device, + dtype=output_dtype, + ) + intermediate_cache3 = torch.empty( num_tokens, top_k, hidden_dim, device=device, dtype=output_dtype, ) + out_hidden_states = torch.empty( + num_tokens, + hidden_dim, + device=device, + dtype=output_dtype, + ) def _step() -> None: _run_fused_moe_iteration( @@ -924,6 +973,8 @@ def _step() -> None: w2=w2, intermediate_cache1=intermediate_cache1, intermediate_cache2=intermediate_cache2, + intermediate_cache3=intermediate_cache3, + out_hidden_states=out_hidden_states, topk_weights=topk_weights, sorted_token_ids=sorted_token_ids, expert_ids=expert_ids, diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 936d1d30..a45c3666 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -9,6 +9,7 @@ | 2026-09-22 | Checkpoint D first half: W3, the shared monolithic forward lifecycle, implemented, tested against four deliberate-defect controls, and committed as `65ed8a7`. | | 2026-09-22 | W3 fidelity matrix measured: 71 of 71 identical against the expectation recorded before the run. Step 3 closed. | | 2026-09-22 | Checkpoint D second half: W4, the opt-in vLLM-style DP placement policy, implemented and committed as `10dd474`; measured against five deliberate-defect controls and a 71-of-71 identical fidelity matrix. Step 4 closed. | +| 2026-09-22 | Checkpoint E second half: W6 arithmetic repaired and CPU-validated; measurement ownership decided; native GPU validation and artifact identity left open. | | 2026-09-22 | Checkpoint E first half: W5 closed as NOT PORTED by user decision after the premise check showed the collision unreachable on main; the drafted implementation was reverted before commit and archived as a patch. | ## Status @@ -18,9 +19,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | -| Current step | Step 5 closed without source changes (NOT PORTED, user decision); records written | +| Current step | Step 6 source and CPU validation complete; native GPU validation NOT_RUN and artifact identity OPEN | | Publication | PUSHED_VERIFIED (records) | -| Next action | Checkpoint E second half: the D2-scoped W6 legacy fused-MoE profiling arithmetic (`frontier/profiling/moe/moe_vllm_kernel.py`), then Checkpoint F's conditional W7. | +| Next action | Two user decisions on W6 (artifact identity metadata; whether to run the native GPU parity matrix), then Checkpoint F's conditional W7. | ## Step status @@ -32,7 +33,7 @@ | 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 4 | Opt-in vLLM DP placement | PASS | unit PASS (61 new, 3778 total, failure set identical to the parent); integration PASS (3 cases in the real event loop, including a placement that diverges from round-robin); five deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | PUSHED_VERIFIED (records + PR 35 section) | REVIEWED (user chose to keep the single global field) | -| 6 | Legacy fused-MoE profiling | NOT_STARTED | — | — | — | +| 6 | Legacy fused-MoE profiling | PARTIAL: arithmetic and measurement ownership done; native validation NOT_RUN, artifact identity OPEN | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778) | pending | — | | 7 | Optional zero-payload backend | NOT_STARTED (facts in `plan.md` A7) | — | — | — | | 8 | Combined regression, PR hand-off | NOT_STARTED | — | — | — | @@ -79,6 +80,11 @@ - 2026-09-22: W5 closure published. `de2bee8` pushed to `origin/fix/issue26-correctness-pr`; PR #35 gained a "W5, which is not in this PR, and why" section carrying the reachability correction with its `cluster_role_config.py:58-63` citation, what the module decides (`moe_impl.py:75-103`, `:195-225`), the three-dataset magnitude table, and the explicit statement that the reverted override's own fidelity effect was zero by construction. The Status line now reads "W6 onward is still to come". PR #35 stays draft. Verified the published body matches what was sent (the one-byte difference is the trailing newline `gh --jq` adds). +- 2026-09-22: W6 reachability and magnitude, checked before implementing, following the rule the W5 revert established. The repaired path is live: `moe_vllm_kernel` selects `_run_fused_moe_iteration` whenever vLLM exposes the low-level API, and all five imported names resolve in the pinned reference v0.10.2, which is what `environment_profiling.yml` pins. Magnitude estimated analytically from the checked-in datasets: 16.5% of the corrected `moe_grouped_gemm` time at 4096 tokens on `a800/qwen3-a3b-30b-moe` (6.8% median, 26.3% max), 1.1-1.4% on the two h800 datasets that stop at 64 tokens. Well above the 0.5% bar, and the repair adds no configuration surface, so it was implemented. +- 2026-09-22: W6 implemented. `silu_and_mul` into a preallocated activation buffer, then `moe_sum` into a preallocated output, matching vLLM's `fused_experts_impl` operand for operand. Two buffers added at the allocation site, outside the timed step. No gated/non-gated branch: `profile_fused_moe_kernel` only ever materializes `w1` with `2 * E` rows, so a conditional would be unreachable. Two recorded adaptations were deliberately not followed — the `_custom_ops` import sits inside the low-level `try` (it cannot fail where that API exists, and this way a build lacking it takes the functional path), and main's `SiluAndMul` wrapper was not reused because its `forward` allocates on every call, which would land inside the timed region. Both deviations and their reasons are in `review.md`. +- 2026-09-22: W6 CPU validation. 7 tests in `tests/unit/test_moe_fused_expert_arithmetic.py` under the Torch environment, covering the arithmetic against a written-out reference, the discriminating comparison with the old slice, call order and operand provenance, routing-weight placement, the local reduction, FP8 quantizer input, workspace reuse across steps, and allocation-site dimensions under TP=2. Regression against a detached `HEAD` worktree: same single environment-dependent failure both sides. Default-environment suite unchanged at 84 failed / 3778 passed. No fidelity matrix: the simulator cannot import the changed module. +- 2026-09-22: W6 left two items open rather than deciding them unilaterally. Artifact identity — `resolve_grouped_gemm_backend` labels both the legacy and the functional path `vllm_fused`, so no checked-in row can be classified as a complete or an incomplete measurement, and `profiling_patch_tag` turns out to exist only in one CSV and nowhere in the source. Native GPU parity — the repaired path is selected only under `vllm>=0.10,<0.11`, and both local Torch environments carry newer vLLM. + ## Step 3 scoping, as recorded before implementation Kept as written so the implementation can be read against the scope it started from. Step 3 is now complete; see the W3 entries above and `design.md`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index 13869632..e3340b46 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -10,6 +10,7 @@ | 2026-09-22 | W3 delivered and measured; R35-02 closed; the unreleased multi-lane monolithic MoE shape recorded as an open item. | | 2026-09-22 | Recorded the maintainer's PR #34 / PR #35 review: D1 and D2 resolved, ten review comments dispositioned, each verified against source. | | 2026-09-22 | Self-review of that record: corrected the lockstep mechanism and counter semantics under D1, the line references under R34-01, the R34-03 remedy (the baseline label is itself an assembled partial run), and added the omissions listed under "Found on re-review". | +| 2026-09-22 | W6 arithmetic delivered with CPU validation; two deviations from the recorded adaptations justified; artifact identity raised as an open decision; native GPU validation NOT_RUN. | | 2026-09-22 | W5 closed without source changes. Corrected the Mechanism A premise: the routing distribution has no per-role override on main, so both W5 mechanisms are unreachable from any released configuration. The user chose to keep the single global field; the drafted implementation was reverted and archived as a patch. | ## Pinned source snapshot @@ -98,6 +99,13 @@ Claims in those reports that this PR depends on were re-verified directly agains | What main's legacy path omits | Exactly two steps. `_run_fused_moe_iteration` (`frontier/profiling/moe/moe_vllm_kernel.py:365-431`) takes a bare first-half slice at `:404-405`, discarding the up-projection half instead of applying gated SiLU, and performs no local top-k reduction (`moe_sum` appears nowhere in the file). FP8 activation quantization (`:407-413`) and routing weights on the second GEMM (`:423`) are already correct; only the operand is wrong. | | No conflict with main-only work | The functional and MXFP4 branch returns at `:869`, before the legacy allocations at `:904`. | | Required adaptations | Port hunks, not the file: a whole-file take would revert 419 lines of main-only work (the functional `fused_experts` adapter, MXFP4/AITER, `device_event` timing, profile-method platform validation). Move `from vllm import _custom_ops as ops` out of the top-level `try` so it cannot perturb main's two-branch API detection (`:116-140`). Reuse main's existing `SiluAndMul` wrapper (`frontier/profiling/common/layers/activation.py:8-34`) rather than the raw op. | +| Delivered 2026-09-22 | Both omitted steps restored in `_run_fused_moe_iteration`, with the two buffers the reference uses, and `moe_grouped_gemm` redefined as the complete local expert computation. 7 new CPU tests in `tests/unit/test_moe_fused_expert_arithmetic.py`, all passing; regression against `HEAD` shows the same single environment-dependent failure on both sides. Full record in `test_report_2026-09-22_w6_fused_expert_arithmetic.md`. | +| Deviation 1 from the adaptations above | The `_custom_ops` import went **inside** the low-level `try`, not outside it. Reason: `fused_moe.py` itself imports `_custom_ops` at its top, so the import cannot fail where the low-level API succeeds, and it therefore cannot perturb the two-branch detection. Placing it there additionally means a build lacking it selects the functional path rather than running an incomplete computation. | +| Deviation 2 from the adaptations above | Main's `SiluAndMul` wrapper was **not** reused. Reason: its `forward` calls `torch.empty` on every invocation (`activation.py:29-33`), which would put an allocation inside the timed profiling step and change what `moe_grouped_gemm` measures. The repair calls `torch.ops._C.silu_and_mul` into a preallocated buffer, exactly as vLLM's `fused_experts_impl` does, and a test asserts one workspace is shared across every profiled step. | +| Measurement ownership, decided | `moe_grouped_gemm` = GEMM1, gated activation, optional activation quantization, GEMM2, local top-k reduction. This is already what the functional backend measured (`fused_experts` returns reduced hidden states), so the repair removes a scope disagreement between the two backends. `MOE_FAMILY` (`frontier/operators/families.py:77`) has no operator for the reduction, so counting it here counts it exactly once without a fifth operator, a new column or a new trained model. The reference's own `record_function("moe_grouped_gemm")` excludes `moe_sum`; that finer split is diagnostics, and the difference is recorded rather than adopted. | +| Magnitude, estimated before implementing | On `a800/qwen3-a3b-30b-moe` the two missing kernels are an estimated 16.5% of the corrected `moe_grouped_gemm` time at 4096 tokens (6.8% median over all rows, 26.3% max). On the two 64-token h800 datasets, 1.1-1.4%. Analytical estimate at 80% of peak HBM, not a measurement. | +| Open: artifact identity | `resolve_grouped_gemm_backend` (`moe_wrapper.py:50-59`) returns `vllm_fused` for both the legacy and the functional path, so no column distinguishes an incomplete legacy measurement from a complete one, and the checked-in datasets cannot be classified. `profiling_patch_tag` appears in one CSV but nowhere in the source, so it is not a live mechanism. Nothing in `frontier/` reads `moe_grouped_gemm_backend`; only `tests/unit/test_moe_native_admission.py:93` asserts the mxfp4 label. Options: (1) record the producing implementation in a new column, leaving old rows empty and adding no gate; (2) option 1 plus a trainer admission gate, which would immediately reject every checked-in dataset including the `examples/profiling` smokes; (3) defer and document. Recommended: (1). Raised for a user decision, as the plan requires. | +| Open: native validation | NOT_RUN. The repaired path is selected only under `vllm>=0.10,<0.11`; the two Torch environments on this host carry vLLM 0.11.0 and 0.28.0 and both take the functional path. A GPU worker with the pinned profiling environment is required, and the affected FP8 path belongs in the same matrix. | | Measured scope today | The whole `_step` body, timed either with per-iteration CUDA events and a synchronize (`:603-626`) or through `record_function("vidur_moe_grouped_gemm")` (`:629-653`). All buffers are allocated outside the timed region. | | Disposition | **PORT** the gated-activation repair. **BLOCKED** on the local output reduction; see D2 below. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md new file mode 100644 index 00000000..6499d299 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md @@ -0,0 +1,208 @@ +# W6 — Legacy fused-MoE expert arithmetic + +Date: 2026-09-22. Branch `fix/issue26-correctness-pr`, worktree +`/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`. + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-22 | Created: reachability check, magnitude estimate, source repair, CPU validation. Native GPU validation NOT_RUN. | + +## 1. Is the defect reachable + +Yes, on the supported profiling environment, and only there. + +`frontier/profiling/moe/moe_vllm_kernel.py` selects one of two expert paths at +import time. When vLLM exposes the low-level API (`fused_moe_kernel`, +`invoke_fused_moe_kernel`, `moe_align_block_size`, `try_get_optimal_moe_config`, +`get_config_dtype_str`) it sets `VLLM_API_VERSION = "0.10.x"` and profiles +through Frontier's own `_run_fused_moe_iteration`. Otherwise it falls back to +`VLLM_API_VERSION = "functional_fused_experts"` and calls vLLM's complete +`fused_experts`, which is correct by construction. + +All five names resolve in the pinned reference vLLM v0.10.2 +(`/data/ycfeng/Frontier/.real-engine/vLLM-BS` @ `ea95f571`, verified by +`grep` against `vllm/model_executor/layers/fused_moe/fused_moe.py`), and +`environment_profiling.yml:25` pins `vllm>=0.10,<0.11`. The documented profiling +environment therefore takes the legacy path. + +Observed on this host, for the record: neither locally available Torch +environment reproduces it. `openmopd-py312` carries vLLM 0.11.0 and +`oneshot-opd-py312` carries vLLM 0.28.0; both report +`VLLM_API_VERSION = functional_fused_experts`. No environment on this machine +selects the legacy path, which is why native validation cannot run here. + +## 2. What was wrong + +vLLM's own `fused_experts_impl` runs: + +```text +GEMM1 -> gated activation -> (activation quantization) -> GEMM2 with routing weights -> local top-k reduction +``` + +`_run_fused_moe_iteration` ran GEMM1, then took `intermediate_cache1[:, :E]` — +the gate half of the `gate | up` projection, unactivated — as the operand of +GEMM2, and never reduced the per-expert outputs. Two operations were missing: +`silu_and_mul` and `moe_sum`. + +The second GEMM's `mul_routed_weight=True, top_k=1` already matched the +reference, as did the FP8 activation quantization; only the operand was wrong. + +## 3. Magnitude, estimated before implementing + +The two missing kernels are memory-bound elementwise work. Their cost was +estimated as `bytes / (peak HBM x 0.80)` and compared against the measured +`moe_grouped_gemm` median in the checked-in datasets: + +```text +silu_and_mul : read 2E, write E, per routed row -> 2 * 3E * M * top_k bytes +moe_sum : read top_k * H, write H, per token -> 2 * H * M * (top_k + 1) bytes +``` + +| Dataset | shape | at the largest profiled token count | over all rows (median / max) | +| --- | --- | --- | --- | +| `a800/qwen3-a3b-30b-moe` | H=2048, E=768, top_k=8 | 4096 tokens: measured 0.934 ms, estimated missing 0.185 ms → **16.5%** | 6.8% / 26.3% | +| `h800/Qwen3-30B-A3B-tiny` | H=2048, E=768, top_k=8 | 64 tokens: 0.130 ms, 0.0018 ms → 1.4% | 0.2% / 1.4% | +| `h800/step-moe-noquant-small` | H=7168, E=5120, top_k=3 | 64 tokens: 0.337 ms, 0.0036 ms → 1.1% | 0.1% / 2.2% | + +The error scales with token count: negligible at decode batch sizes, +material at prefill and chunked-prefill sizes. The two tiny h800 datasets only +reach 64 tokens, which is why their share stays small. + +This is an analytical estimate with a stated bandwidth assumption, not a +measurement. It was used to decide whether the repair was worth making, not to +claim a corrected number. + +## 4. The repair + +`frontier/profiling/moe/moe_vllm_kernel.py`: + +- The low-level import block also imports `vllm._custom_ops` as + `_vllm_custom_ops`, so a build without it selects the functional path instead + of running an incomplete computation. `_vllm_custom_ops = None` is declared at + module level beside the existing `_functional_fused_experts = None`. +- `_run_fused_moe_iteration` now runs `torch.ops._C.silu_and_mul` into a + dedicated activation buffer, feeds that buffer to GEMM2 (and to the FP8 + quantizer when enabled), and finishes with + `_vllm_custom_ops.moe_sum(intermediate_cache3, out_hidden_states)`. +- `profile_fused_moe_kernel` allocates the two additional buffers once, outside + the timed step: `intermediate_cache2` as `(M * top_k, E)` and + `out_hidden_states` as `(M, H)`. The former GEMM2 output buffer is renamed + `intermediate_cache3`, matching the reference's naming. + +**No gated/non-gated branch was added.** `profile_fused_moe_kernel` always +materializes `w1` with `2 * expert_hidden_dim_per_partition` rows +(`moe_vllm_kernel.py`, weight construction), and `plan_mxfp4_weight_layout` is +called with `use_gated=True`. The gated layout is the only one this entry point +can produce, so a conditional would be unreachable code. Profiling a non-gated +expert would first require a different weight layout, which this function does +not support today. + +## 5. Measurement ownership + +The corrected `moe_grouped_gemm` target is the **complete local expert +computation**: GEMM1, gated activation, optional activation quantization, GEMM2, +and the local top-k reduction. Both profile methods time the whole step — +`_collect_cuda_event_stats` brackets `step_fn`, and +`_collect_record_function_stats` wraps it in one `vidur_moe_grouped_gemm` scope. + +This makes the legacy path measure the same scope the functional path already +measured: vLLM's `fused_experts` returns reduced hidden states, so `moe_sum` was +always inside the functional measurement. The repair removes a disagreement +between the two backends rather than creating one. + +Frontier's `MOE_FAMILY` (`frontier/operators/families.py:77`) has exactly four +operators — `moe_gating_linear`, `moe_gating_routing_topk`, `moe_shuffling`, +`moe_grouped_gemm` — and none of them represents the reduction. Counting it +inside `moe_grouped_gemm` counts it exactly once without adding a fifth +operator, a new profiling column, or a new trained model. + +Noted for the record, because it differs: the Frontier-instrumented reference +vLLM puts `moe_sum` *outside* its `record_function("moe_grouped_gemm")` scope +(added by reference commit `139cba30b`). That split is finer-grained +diagnostics; Frontier's operator of the same name means the whole local expert +computation. + +The local reduction is a sum over one token's own top-k expert outputs. It is +not a collective, and no DP/TP/EP communication cost was added or removed. + +## 6. CPU validation + +Environment: `/data/ycfeng/envs/openmopd-py312/bin/python` (Python 3.12, Torch +2.8.0+cu128, vLLM 0.11.0), `PYTHONPATH=$PWD`, from the worktree root. This +environment is used because the profiling-boundary tests import Torch; no GPU +kernel executes. + +New file `tests/unit/test_moe_fused_expert_arithmetic.py`, 7 tests, all passing. +It replaces `_invoke_kernel`, `torch.ops._C.silu_and_mul` and +`_vllm_custom_ops.moe_sum` with plain-Torch references, so it validates the +composition, not native numerics. The file's docstring says so. + +| Test | What it settles | +| --- | --- | +| `test_the_iteration_computes_the_gated_expert_output` | The full path equals a directly written gated-SwiGLU MoE reference. | +| `test_a_gated_activation_is_not_the_first_half_of_the_projection` | The repair is observable: the old slice-only arithmetic gives a different answer, so the test above cannot pass against the unrepaired path. | +| `test_the_second_gemm_consumes_the_activation_buffer` | Call order `GEMM1 -> silu_and_mul -> GEMM2 -> moe_sum`; GEMM1 has `mul_routed_weight=False, top_k=top_k`; the activation reads `(M*top_k, 2E)` and writes the buffer GEMM2 reads; GEMM2 has `mul_routed_weight=True, top_k=1`; the reduction reads `cache3` and writes `out`. | +| `test_the_local_reduction_sums_the_top_k_expert_outputs` | The reduction is local and per token; output shapes `(M, top_k, H) -> (M, H)`. | +| `test_the_activation_buffer_is_quantized_rather_than_the_raw_projection` | Under FP8 the quantizer receives the gated activation, with the block-derived group size. | +| `test_repeated_iterations_do_not_leak_a_previous_result` | Workspace reuse across profiling steps does not carry a stale output; the second result is finite and matches its own reference. | +| `test_the_profiler_allocates_the_four_buffers_the_computation_needs` | Allocation-site dimensions for all four buffers under TP=2, and one workspace shared by every profiled step. | + +### Regression comparison + +Same command, same environment, against a detached worktree at `HEAD` +(`bbbfcaa`), removed afterwards: + +| Selection | At `HEAD` | With the repair | +| --- | --- | --- | +| `test_moe_mxfp4_increment10.py`, `test_moe_fused_event_contract.py`, `test_moe_native_admission.py`, `test_native_profiling_model_type_policy.py` | 1 failed, 136 passed | 1 failed, 143 passed (7 new) | + +The single failure is the same on both sides: +`test_functional_vllm_kernel_exposes_mxfp4_switch_without_importing_vllm` +asserts `check_vllm_available() is False`, which needs an environment with Torch +but *without* vLLM. Neither local environment provides that combination. It is +environment-dependent, not a regression. + +Default environment (`/data/ycfeng/envs/frontier-py310/bin/python`, no Torch), +`pytest tests/unit --continue-on-collection-errors`: **84 failed, 3778 passed, +49 skipped, 11 errors**, matching the W4 baseline of 84 failures and 3778 +passing. Collection errors go from 10 to 11: the new file imports Torch at +module level, exactly as the seven existing profiling test files in that list do +(`test_moe_fused_event_contract.py`, `test_moe_native_admission.py`, +`test_native_profiling_model_type_policy.py`, `test_mla_native_profiling_wrapper.py`, +`test_moe_gating_constructor_boundary.py`, `test_moe_load_distribution_contract.py`, +`test_moe_routing_input_contract.py`). Verified by re-collecting with the new +file ignored: 10 errors. + +## 7. Simulator fidelity + +No fidelity matrix run. The change is confined to +`frontier/profiling/moe/moe_vllm_kernel.py`, which the simulator cannot import: +it requires Torch, and the simulator runs in an environment without it. The +evidence is the unchanged default-environment suite result above — the +simulator's own 3778 passing tests are byte-identical in outcome, and the +matrix's inputs are checked-in CSVs, not freshly profiled data. + +Stated plainly: this repair changes what a *future* profiling run measures. It +changes nothing about a simulation run from existing data. + +## 8. Not done + +- **Native numerical validation: NOT_RUN.** The plan requires a GPU local-expert + parity test against the pinned vLLM reference, including more than one local + expert partition and a boundary case with uneven expert occupancy. It needs a + worker with `vllm>=0.10,<0.11`, because that is the only configuration that + selects the repaired path. No such environment exists on this host. Pending a + decision on the GPU run. +- **Artifact identity: OPEN, raised for decision.** Existing rows record + `moe_grouped_gemm_backend='vllm_fused'` for both the legacy and the functional + path (`resolve_grouped_gemm_backend`, `moe_wrapper.py:50-59`), so the column + cannot distinguish an incomplete legacy measurement from a complete one. + `profiling_patch_tag` exists in one checked-in CSV but appears nowhere in the + source, so it is not a live mechanism. No consumer in `frontier/` reads + `moe_grouped_gemm_backend`; only `tests/unit/test_moe_native_admission.py:93` + asserts the mxfp4 label. Options and a recommendation are in `review.md`. +- **FP8 path**: the repair changes what the FP8 quantizer receives, so the FP8 + path is an affected native path. It is not proven by the CPU shape test and is + included in the pending GPU matrix. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index d3366c3e..3fc05366 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -444,3 +444,19 @@ Each tree is the delivered source and tests with exactly one edit. load-sensitive, not latency-realistic. - The online divergence is one arrival pattern. The test asserts the direction — fewer requests on the busy lane — not the exact sequence. + +## Step 6 — legacy fused-MoE expert arithmetic (2026-09-22) + +Full record: `test_report_2026-09-22_w6_fused_expert_arithmetic.md`. + +| Item | Result | +| --- | --- | +| Reachability | Confirmed. The repaired path is selected when vLLM exposes the low-level API, which the pinned reference v0.10.2 and the `environment_profiling.yml` pin `vllm>=0.10,<0.11` both do. Neither Torch environment on this host reproduces it (vLLM 0.11.0 and 0.28.0 both select the functional path). | +| Magnitude, stated before implementing | Estimated 16.5% of the corrected `moe_grouped_gemm` time at 4096 tokens on `a800/qwen3-a3b-30b-moe`; 6.8% median, 26.3% max over its rows; 1.1-1.4% on the two 64-token h800 datasets. Analytical, at 80% of peak HBM. | +| CPU tests | `tests/unit/test_moe_fused_expert_arithmetic.py`, 7 new tests, all passing under `/data/ycfeng/envs/openmopd-py312/bin/python` (Torch 2.8.0, vLLM 0.11.0). Composition only: the native calls are replaced by plain-Torch references, and the file says so. | +| Discriminating check | `test_a_gated_activation_is_not_the_first_half_of_the_projection` computes the old slice-only arithmetic and asserts the repaired result differs, so the reference-equality test cannot pass against the unrepaired path. | +| Regression, Torch environment | Against a detached worktree at `HEAD` (`bbbfcaa`), same four existing files: 1 failed / 136 passed at HEAD, 1 failed / 143 passed with the repair. Same failure identity on both sides — `test_functional_vllm_kernel_exposes_mxfp4_switch_without_importing_vllm` needs Torch without vLLM, which no local environment provides. | +| Regression, default environment | `pytest tests/unit --continue-on-collection-errors` under `frontier-py310`: 84 failed, 3778 passed, 49 skipped, 11 errors — the W4 baseline of 84 failures and 3778 passing, unchanged. Collection errors 10 -> 11 because the new file imports Torch at module level, as the seven existing profiling test files in that list already do; verified by re-collecting with it ignored. | +| Fidelity matrix | Not run, deliberately. The change is confined to a module the simulator cannot import (it requires Torch, absent from the simulator environment), and the matrix consumes checked-in CSVs rather than fresh profiling. The unchanged default-environment suite is the evidence. The repair changes what a future profiling run measures, not any simulation from existing data. | +| Native GPU parity | **NOT_RUN.** Requires a worker with `vllm>=0.10,<0.11`. The FP8 path is affected by the repair and belongs in the same matrix. | +| Artifact identity | **OPEN**, raised for decision; see `review.md`. Existing rows cannot be classified as complete or incomplete from their metadata. | diff --git a/tests/unit/test_moe_fused_expert_arithmetic.py b/tests/unit/test_moe_fused_expert_arithmetic.py new file mode 100644 index 00000000..bc09e3bf --- /dev/null +++ b/tests/unit/test_moe_fused_expert_arithmetic.py @@ -0,0 +1,396 @@ +"""CPU boundary tests for the legacy fused-MoE expert computation. + +No native kernel runs here. `_invoke_kernel`, the gated activation and the local +top-k reduction are replaced by plain-Torch references, so what these tests +check is the *composition*: which operand reaches the second expert GEMM, where +the routing weights are applied, and whether the top-k expert outputs are +reduced. That is exactly what the repaired path changes. + +This is boundary validation, not native numerical parity. Parity against vLLM's +own kernels needs a GPU and is recorded separately. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from frontier.profiling.moe import moe_vllm_kernel as kernel + + +NUM_TOKENS = 5 +TOP_K = 2 +HIDDEN_DIM = 6 +EXPERT_HIDDEN_DIM = 4 +NUM_EXPERTS = 3 + + +@pytest.fixture +def expert_problem(): + """One small gated MoE problem with fixed routing, entirely on CPU.""" + + generator = torch.Generator().manual_seed(7) + activations = torch.randn(NUM_TOKENS, HIDDEN_DIM, generator=generator) + w1 = torch.randn( + NUM_EXPERTS, 2 * EXPERT_HIDDEN_DIM, HIDDEN_DIM, generator=generator + ) + w2 = torch.randn(NUM_EXPERTS, HIDDEN_DIM, EXPERT_HIDDEN_DIM, generator=generator) + topk_ids = torch.tensor( + [[0, 1], [1, 2], [2, 0], [0, 2], [1, 1]], dtype=torch.int32 + ) + topk_weights = torch.rand(NUM_TOKENS, TOP_K, generator=generator) + return SimpleNamespace( + A=activations, w1=w1, w2=w2, topk_ids=topk_ids, topk_weights=topk_weights + ) + + +def _expected_output(problem) -> torch.Tensor: + """The gated expert computation, written out directly.""" + + out = torch.zeros(NUM_TOKENS, HIDDEN_DIM) + for token in range(NUM_TOKENS): + for slot in range(TOP_K): + expert = int(problem.topk_ids[token, slot]) + projected = problem.A[token] @ problem.w1[expert].T + gate, up = projected[:EXPERT_HIDDEN_DIM], projected[EXPERT_HIDDEN_DIM:] + activated = torch.nn.functional.silu(gate) * up + expert_out = activated @ problem.w2[expert].T + out[token] += problem.topk_weights[token, slot] * expert_out + return out + + +def _sliced_output(problem) -> torch.Tensor: + """What the path produced before the repair: the gate half, unactivated.""" + + out = torch.zeros(NUM_TOKENS, HIDDEN_DIM) + for token in range(NUM_TOKENS): + for slot in range(TOP_K): + expert = int(problem.topk_ids[token, slot]) + projected = problem.A[token] @ problem.w1[expert].T + expert_out = projected[:EXPERT_HIDDEN_DIM] @ problem.w2[expert].T + out[token] += problem.topk_weights[token, slot] * expert_out + return out + + +@pytest.fixture +def native_stubs(monkeypatch, expert_problem): + """Replace the three native calls with plain-Torch references. + + Each stub also records its call, so a test can assert the call order and the + operands without reaching into the profiling return contract. + """ + + calls = [] + topk_ids = expert_problem.topk_ids + + def fake_invoke_kernel(*, A, B, C, topk_weights, mul_routed_weight, top_k, **_): + calls.append( + SimpleNamespace( + name="invoke_kernel", + A=A, + mul_routed_weight=mul_routed_weight, + top_k=top_k, + ) + ) + rows = C.view(NUM_TOKENS, TOP_K, C.shape[-1]) + for token in range(NUM_TOKENS): + for slot in range(TOP_K): + expert = int(topk_ids[token, slot]) + if top_k == 1: + source = A.view(NUM_TOKENS, TOP_K, A.shape[-1])[token, slot] + else: + source = A[token] + value = source @ B[expert].T + if mul_routed_weight: + value = value * topk_weights[token, slot] + rows[token, slot] = value + + def fake_silu_and_mul(out, inp): + calls.append(SimpleNamespace(name="silu_and_mul", out=out, inp=inp)) + half = inp.shape[-1] // 2 + out.copy_(torch.nn.functional.silu(inp[:, :half]) * inp[:, half:]) + + def fake_moe_sum(inp, out): + calls.append(SimpleNamespace(name="moe_sum", inp=inp, out=out)) + out.copy_(inp.sum(dim=1)) + + monkeypatch.setattr(kernel, "_invoke_kernel", fake_invoke_kernel) + monkeypatch.setattr( + kernel, + "torch", + SimpleNamespace(ops=SimpleNamespace(_C=SimpleNamespace(silu_and_mul=fake_silu_and_mul))), + ) + monkeypatch.setattr( + kernel, "_vllm_custom_ops", SimpleNamespace(moe_sum=fake_moe_sum) + ) + return calls + + +def _run(problem): + cache1 = torch.empty(NUM_TOKENS, TOP_K, 2 * EXPERT_HIDDEN_DIM) + cache2 = torch.empty(NUM_TOKENS * TOP_K, EXPERT_HIDDEN_DIM) + cache3 = torch.empty(NUM_TOKENS, TOP_K, HIDDEN_DIM) + out = torch.empty(NUM_TOKENS, HIDDEN_DIM) + kernel._run_fused_moe_iteration( + A=problem.A, + w1=problem.w1, + w2=problem.w2, + intermediate_cache1=cache1, + intermediate_cache2=cache2, + intermediate_cache3=cache3, + out_hidden_states=out, + topk_weights=problem.topk_weights, + sorted_token_ids=torch.zeros(1, dtype=torch.int32), + expert_ids=torch.zeros(1, dtype=torch.int32), + num_tokens_post_padded=torch.zeros(1, dtype=torch.int32), + top_k=TOP_K, + config={"BLOCK_SIZE_M": 16}, + expert_hidden_dim_per_partition=EXPERT_HIDDEN_DIM, + block_dims=None, + ) + return SimpleNamespace(cache1=cache1, cache2=cache2, cache3=cache3, out=out) + + +def test_the_iteration_computes_the_gated_expert_output(expert_problem, native_stubs): + """The full local computation matches a directly written reference.""" + + result = _run(expert_problem) + + torch.testing.assert_close(result.out, _expected_output(expert_problem)) + + +def test_a_gated_activation_is_not_the_first_half_of_the_projection( + expert_problem, native_stubs +): + """The repair is observable: slicing the gate half gives a different answer. + + Without this the test above would pass against the unrepaired path whenever + the reference happened to agree, which for a gated layout it never does. + """ + + result = _run(expert_problem) + sliced = _sliced_output(expert_problem) + + assert not torch.allclose(result.out, sliced, atol=1e-4) + assert (result.out - sliced).abs().max() > 1e-3 + + +def test_the_second_gemm_consumes_the_activation_buffer(expert_problem, native_stubs): + """Operand provenance, which is what the defect got wrong.""" + + result = _run(expert_problem) + + names = [call.name for call in native_stubs] + assert names == ["invoke_kernel", "silu_and_mul", "invoke_kernel", "moe_sum"] + + first_gemm, activation, second_gemm, reduction = native_stubs + assert first_gemm.A is expert_problem.A + assert first_gemm.mul_routed_weight is False + assert first_gemm.top_k == TOP_K + + # The activation reads the whole `gate | up` projection and writes the + # buffer the second GEMM then reads. + assert activation.inp.shape == (NUM_TOKENS * TOP_K, 2 * EXPERT_HIDDEN_DIM) + assert activation.out is result.cache2 + assert second_gemm.A is result.cache2 + + # Routing weights are applied once, on the second GEMM, over one expert. + assert second_gemm.mul_routed_weight is True + assert second_gemm.top_k == 1 + + assert reduction.inp is result.cache3 + assert reduction.out is result.out + + +def test_the_local_reduction_sums_the_top_k_expert_outputs( + expert_problem, native_stubs +): + """The reduction is local: one token's k outputs, no cross-token mixing.""" + + result = _run(expert_problem) + + torch.testing.assert_close(result.out, result.cache3.sum(dim=1)) + assert result.cache3.shape == (NUM_TOKENS, TOP_K, HIDDEN_DIM) + assert result.out.shape == (NUM_TOKENS, HIDDEN_DIM) + + +def test_the_activation_buffer_is_quantized_rather_than_the_raw_projection( + monkeypatch, expert_problem, native_stubs +): + """Under FP8 the quantizer must receive the gated activation.""" + + seen = {} + + def fake_quantize(tensor, *, group_size): + seen["tensor"] = tensor + seen["group_size"] = group_size + return tensor, torch.ones(1) + + monkeypatch.setattr(kernel, "quantize_activations_to_fp8", fake_quantize) + + cache1 = torch.empty(NUM_TOKENS, TOP_K, 2 * EXPERT_HIDDEN_DIM) + cache2 = torch.empty(NUM_TOKENS * TOP_K, EXPERT_HIDDEN_DIM) + cache3 = torch.empty(NUM_TOKENS, TOP_K, HIDDEN_DIM) + out = torch.empty(NUM_TOKENS, HIDDEN_DIM) + kernel._run_fused_moe_iteration( + A=expert_problem.A, + w1=expert_problem.w1, + w2=expert_problem.w2, + intermediate_cache1=cache1, + intermediate_cache2=cache2, + intermediate_cache3=cache3, + out_hidden_states=out, + topk_weights=expert_problem.topk_weights, + sorted_token_ids=torch.zeros(1, dtype=torch.int32), + expert_ids=torch.zeros(1, dtype=torch.int32), + num_tokens_post_padded=torch.zeros(1, dtype=torch.int32), + top_k=TOP_K, + config={"BLOCK_SIZE_M": 16}, + expert_hidden_dim_per_partition=EXPERT_HIDDEN_DIM, + block_dims=(128, 64), + use_fp8=True, + ) + + assert seen["tensor"] is cache2 + assert seen["group_size"] == 64 + + +def test_repeated_iterations_do_not_leak_a_previous_result( + expert_problem, native_stubs +): + """Workspace reuse across profiling steps must not carry stale outputs.""" + + cache1 = torch.empty(NUM_TOKENS, TOP_K, 2 * EXPERT_HIDDEN_DIM) + cache2 = torch.empty(NUM_TOKENS * TOP_K, EXPERT_HIDDEN_DIM) + cache3 = torch.empty(NUM_TOKENS, TOP_K, HIDDEN_DIM) + out = torch.empty(NUM_TOKENS, HIDDEN_DIM) + + def run_with(activations): + kernel._run_fused_moe_iteration( + A=activations, + w1=expert_problem.w1, + w2=expert_problem.w2, + intermediate_cache1=cache1, + intermediate_cache2=cache2, + intermediate_cache3=cache3, + out_hidden_states=out, + topk_weights=expert_problem.topk_weights, + sorted_token_ids=torch.zeros(1, dtype=torch.int32), + expert_ids=torch.zeros(1, dtype=torch.int32), + num_tokens_post_padded=torch.zeros(1, dtype=torch.int32), + top_k=TOP_K, + config={"BLOCK_SIZE_M": 16}, + expert_hidden_dim_per_partition=EXPERT_HIDDEN_DIM, + block_dims=None, + ) + return out.clone() + + first = run_with(expert_problem.A) + second = run_with(expert_problem.A * 2.0) + + assert not torch.allclose(first, second) + expert_problem.A = expert_problem.A * 2.0 + torch.testing.assert_close(second, _expected_output(expert_problem)) + assert torch.isfinite(second).all() + + +def test_the_profiler_allocates_the_four_buffers_the_computation_needs(monkeypatch): + """Allocation-site dimensions, including the two buffers the repair adds. + + `intermediate_cache2` holds one activated row per routed token, and + `out_hidden_states` holds one reduced row per token. Getting either shape + wrong would silently change what the second GEMM and the reduction see. + """ + + from contextlib import nullcontext + from unittest.mock import Mock + + num_tokens, top_k = 3, 2 + hidden_dim, expert_hidden_dim, tensor_parallel_size = 8, 16, 2 + expert_hidden_per_partition = expert_hidden_dim // tensor_parallel_size + + class Event: + def __init__(self, **kwargs): + assert kwargs == {"enable_timing": True} + + def record(self): + pass + + def elapsed_time(self, other): + return 1.0 + + proxy = SimpleNamespace( + version=SimpleNamespace(hip=None, cuda="12.8"), + cuda=SimpleNamespace(Event=Event, synchronize=Mock()), + randn=lambda *shape, **kwargs: torch.zeros(*shape, dtype=kwargs["dtype"]), + empty=lambda *shape, **kwargs: torch.zeros(*shape, dtype=kwargs["dtype"]), + bfloat16=torch.bfloat16, + float16=torch.float16, + float32=torch.float32, + int32=torch.int32, + ) + monkeypatch.setattr(kernel, "torch", proxy) + monkeypatch.setattr(kernel, "VLLM_AVAILABLE", True) + monkeypatch.setattr(kernel, "VLLM_API_VERSION", "0.10.x") + monkeypatch.setattr( + kernel, "get_config_dtype_str", lambda dtype: "float16", raising=False + ) + monkeypatch.setattr( + kernel, + "try_get_optimal_moe_config", + lambda **kwargs: {"BLOCK_SIZE_M": 16}, + raising=False, + ) + monkeypatch.setattr( + kernel, + "moe_align_block_size", + lambda *args, **kwargs: (None, None, None), + raising=False, + ) + monkeypatch.setattr( + "frontier.profiling.common.vllm_compat.vllm_config_context", + lambda **kwargs: nullcontext(), + ) + + seen = [] + monkeypatch.setattr( + kernel, "_run_fused_moe_iteration", lambda **kwargs: seen.append(kwargs) + ) + + routing = Mock() + routing.to.return_value = routing + routing.contiguous.return_value = routing + kernel.profile_fused_moe_kernel( + num_tokens=num_tokens, + num_experts=2, + hidden_dim=hidden_dim, + expert_hidden_dim=expert_hidden_dim, + top_k=top_k, + topk_weights=routing, + topk_ids=routing, + tensor_parallel_size=tensor_parallel_size, + warmup_steps=1, + active_steps=2, + profile_method="cuda_event", + ) + + assert seen, "the legacy path did not run" + call = seen[0] + assert call["intermediate_cache1"].shape == ( + num_tokens, + top_k, + 2 * expert_hidden_per_partition, + ) + assert call["intermediate_cache2"].shape == ( + num_tokens * top_k, + expert_hidden_per_partition, + ) + assert call["intermediate_cache3"].shape == (num_tokens, top_k, hidden_dim) + assert call["out_hidden_states"].shape == (num_tokens, hidden_dim) + + # Every profiled step reuses one workspace, so the measured cost is the + # computation and not repeated allocation. + assert all( + later["intermediate_cache2"] is call["intermediate_cache2"] for later in seen + ) + assert all(later["out_hidden_states"] is call["out_hidden_states"] for later in seen) From 697f219bd0b65cb48d7e45eaa2ff4129383417f4 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 14:01:13 +0800 Subject: [PATCH 21/80] test(profiling): add native fused-MoE expert parity against vLLM The CPU boundary tests validate the composition of the repaired expert iteration with stubbed kernels. They cannot show that the real kernels agree with vLLM's own path, which is what the repair claims. Add a GPU parity test that drives `_run_fused_moe_iteration` with the same buffer shapes, kernel config and alignment that `profile_fused_moe_kernel` uses, and compares its output tensor against `fused_experts` at zero tolerance. It covers the production Qwen3-A3B-30B shape at 4096 and 4097 tokens on two expert-parallel shards, a smaller shape with a different top-k and experts that receive no tokens, repeated invocation, and the FP8 path as a structural check. Production shapes are read from the checked-in model config so they stay tied to the model Frontier profiles. The test runs only under CUDA with a vLLM build that exposes the low-level fused-MoE API, because that is the only configuration selecting the repaired path; every other environment skips. Also drop `expert_hidden_dim_per_partition` from the iteration signature. The gated activation reads the whole first projection, so the parameter no longer selects anything. --- frontier/profiling/moe/moe_vllm_kernel.py | 2 - .../test_moe_fused_expert_numerical_parity.py | 389 ++++++++++++++++++ .../unit/test_moe_fused_expert_arithmetic.py | 3 - 3 files changed, 389 insertions(+), 5 deletions(-) create mode 100644 tests/integration/test_moe_fused_expert_numerical_parity.py diff --git a/frontier/profiling/moe/moe_vllm_kernel.py b/frontier/profiling/moe/moe_vllm_kernel.py index b7d35aee..83254110 100644 --- a/frontier/profiling/moe/moe_vllm_kernel.py +++ b/frontier/profiling/moe/moe_vllm_kernel.py @@ -383,7 +383,6 @@ def _run_fused_moe_iteration( num_tokens_post_padded: torch.Tensor, top_k: int, config: Dict, - expert_hidden_dim_per_partition: int, block_dims: Optional[Tuple[int, int]], A_scale: Optional[torch.Tensor] = None, w1_scale: Optional[torch.Tensor] = None, @@ -981,7 +980,6 @@ def _step() -> None: num_tokens_post_padded=num_tokens_post_padded, top_k=top_k, config=config, - expert_hidden_dim_per_partition=expert_hidden_dim_per_partition, block_dims=block_dims, A_scale=A_scale, w1_scale=w1_scale, diff --git a/tests/integration/test_moe_fused_expert_numerical_parity.py b/tests/integration/test_moe_fused_expert_numerical_parity.py new file mode 100644 index 00000000..9ed53711 --- /dev/null +++ b/tests/integration/test_moe_fused_expert_numerical_parity.py @@ -0,0 +1,389 @@ +"""Native parity for the legacy fused-MoE expert path against vLLM's own. + +Both sides run on the GPU with the same activations, weights, routing weights, +routing ids, expert map, kernel config and dtype. The comparison is the final +output tensor, including the local top-k reduction, so it covers exactly what +the repaired `_run_fused_moe_iteration` changed. + +This runs only where the repaired path is selected: CUDA plus a vLLM build that +exposes the low-level fused-MoE API (`vllm>=0.10,<0.11`). Any other environment +skips, because a newer vLLM routes profiling through `fused_experts` directly +and there is nothing here to compare. + +Run on an already authorized worker. This module never provisions workers and +never substitutes a simulated kernel for a native one. +""" + +import json +import os + +import pytest + +torch = pytest.importorskip("torch", reason="the vLLM expert kernels require PyTorch") +pytest.importorskip("vllm", reason="the reference implementation lives in vLLM") + +from frontier.profiling.common.vllm_compat import vllm_config_context +from frontier.profiling.moe import moe_vllm_kernel as kernel + + +pytestmark = [ + pytest.mark.skipif( + not torch.cuda.is_available(), + reason="the vLLM expert kernels require a CUDA device", + ), + pytest.mark.skipif( + kernel.VLLM_API_VERSION != "0.10.x", + reason=( + "the repaired path is selected only by the low-level vLLM API; this " + f"build reports VLLM_API_VERSION={kernel.VLLM_API_VERSION!r}" + ), + ), +] + +# The production case is taken from the checked-in model config rather than +# retyped here, so the shapes stay tied to the model Frontier actually profiles. +PRODUCTION_MODEL = "qwen3-a3b-30b-moe" + + +def _production_shape(): + config_path = os.path.join( + "data", "config", "models", f"{PRODUCTION_MODEL}.json" + ) + if not os.path.exists(config_path): + pytest.skip(f"run from the repository root; {config_path} not found") + with open(config_path, "r", encoding="utf-8") as handle: + config = json.load(handle) + return { + "hidden": config["hidden_size"], + "width": config["moe_intermediate_size"], + "top_k": config["num_experts_per_tok"], + "num_experts": config["num_experts"], + } + + +def _uniform_routing(num_tokens, num_experts, top_k, dtype): + """The routing Frontier's `random` distribution resolves to.""" + + from frontier.profiling.moe.moe_impl import uniform_topk + + hidden_proxy = torch.empty(num_tokens, 1, device="cuda", dtype=dtype) + logits = torch.empty(num_tokens, num_experts, device="cuda", dtype=dtype) + topk_weights, topk_ids, _ = uniform_topk(hidden_proxy, logits, top_k) + return topk_weights, topk_ids + + +def _skewed_routing(num_tokens, num_experts, top_k, dtype): + """Popularity-weighted routing that leaves two experts with no tokens. + + An even split hides a shard that mishandles an expert with an empty block + range, or one whose block count differs from its neighbours. The last two + global experts are excluded by construction so the assertion that they stay + empty does not depend on sampling luck. + """ + + generator = torch.Generator().manual_seed(29) + allowed = torch.arange(num_experts - 2) + popularity = 1.0 / (torch.arange(allowed.numel(), dtype=torch.float64) + 1.0) + probabilities = (popularity / popularity.sum()).repeat(num_tokens, 1) + picks = torch.multinomial( + probabilities, top_k, replacement=False, generator=generator + ) + topk_ids = allowed[picks].to(torch.int32) + weights = torch.rand(num_tokens, top_k, generator=generator) + weights = weights / weights.sum(dim=-1, keepdim=True) + return ( + weights.to(device="cuda", dtype=torch.float32), + topk_ids.to("cuda"), + ) + + +def _build_case( + *, + num_tokens, + hidden, + width, + top_k, + num_experts, + expert_parallel_size, + ep_rank, + seed, + routing=_uniform_routing, + dtype=torch.bfloat16, +): + """Materialize one expert-parallel shard and its routing, on the device.""" + + if num_experts % expert_parallel_size: + raise ValueError("num_experts must divide evenly across the EP ranks") + local_experts = num_experts // expert_parallel_size + + torch.manual_seed(seed) + activations = torch.randn(num_tokens, hidden, device="cuda", dtype=dtype) * 0.1 + # `w1` carries the gate and up projections stacked, which is the only layout + # `profile_fused_moe_kernel` produces. + w1 = torch.randn( + local_experts, 2 * width, hidden, device="cuda", dtype=dtype + ) * 0.1 + w2 = torch.randn(local_experts, hidden, width, device="cuda", dtype=dtype) * 0.1 + + topk_weights, topk_ids = routing(num_tokens, num_experts, top_k, dtype) + + # Only this rank's experts are local; every other global id maps to -1, + # exactly as an EP deployment presents its shard to the kernel. + expert_map = torch.full((num_experts,), -1, device="cuda", dtype=torch.int32) + start = ep_rank * local_experts + expert_map[start : start + local_experts] = torch.arange( + local_experts, device="cuda", dtype=torch.int32 + ) + return activations, w1, w2, topk_weights, topk_ids, expert_map + + +def _run_repaired_path(activations, w1, w2, topk_weights, topk_ids, expert_map): + """Drive `_run_fused_moe_iteration` exactly as the profiler drives it. + + The buffer shapes and the config/alignment calls are copied from + `profile_fused_moe_kernel`, so a change that breaks the profiler's own setup + breaks this test too. + """ + + num_tokens, hidden = activations.shape + top_k = topk_ids.shape[1] + width = w1.shape[1] // 2 + global_num_experts = expert_map.numel() + + config = kernel.try_get_optimal_moe_config( + w1_shape=w1.shape, + w2_shape=w2.shape, + top_k=top_k, + dtype=kernel.get_config_dtype_str(activations.dtype), + M=num_tokens, + block_shape=None, + ) + sorted_token_ids, expert_ids, num_tokens_post_padded = kernel.moe_align_block_size( + topk_ids, + config["BLOCK_SIZE_M"], + global_num_experts, + expert_map=expert_map, + ) + + intermediate_cache1 = torch.empty( + num_tokens, top_k, w1.shape[1], device="cuda", dtype=activations.dtype + ) + intermediate_cache2 = torch.empty( + num_tokens * top_k, width, device="cuda", dtype=activations.dtype + ) + intermediate_cache3 = torch.empty( + num_tokens, top_k, hidden, device="cuda", dtype=activations.dtype + ) + out_hidden_states = torch.empty( + num_tokens, hidden, device="cuda", dtype=activations.dtype + ) + + kernel._run_fused_moe_iteration( + A=activations, + w1=w1, + w2=w2, + intermediate_cache1=intermediate_cache1, + intermediate_cache2=intermediate_cache2, + intermediate_cache3=intermediate_cache3, + out_hidden_states=out_hidden_states, + topk_weights=topk_weights, + sorted_token_ids=sorted_token_ids, + expert_ids=expert_ids, + num_tokens_post_padded=num_tokens_post_padded, + top_k=top_k, + config=config, + block_dims=None, + ) + return out_hidden_states + + +def _run_vllm_reference(activations, w1, w2, topk_weights, topk_ids, expert_map): + from vllm.model_executor.layers.fused_moe.fused_moe import fused_experts + + return fused_experts( + activations, + w1, + w2, + topk_weights, + topk_ids, + inplace=False, + activation="silu", + global_num_experts=expert_map.numel(), + expert_map=expert_map, + ) + + +def _assert_matches_reference(case): + with vllm_config_context(): + actual = _run_repaired_path(*case) + expected = _run_vllm_reference(*case) + torch.cuda.synchronize() + + assert torch.isfinite(actual).all(), "the repaired path produced a non-finite value" + torch.testing.assert_close(actual, expected, rtol=0, atol=0) + return actual + + +@pytest.mark.parametrize("num_tokens", [4096, 4097]) +@pytest.mark.parametrize("ep_rank", [0, 1]) +def test_production_shaped_expert_output_matches_vllm(num_tokens, ep_rank): + """Production shapes at a prefill-sized token count, on two local shards. + + 4097 is not a multiple of any kernel block size, so it exercises the padded + tail. Two ranks cover the global-to-local expert id remapping. This is the + token range where the previously missing kernels cost the most, so it is the + case that most needs to be right. + """ + + case = _build_case( + num_tokens=num_tokens, + expert_parallel_size=8, + ep_rank=ep_rank, + seed=13, + **_production_shape(), + ) + _assert_matches_reference(case) + + +@pytest.mark.parametrize("top_k", [2, 4]) +def test_uneven_expert_occupancy_matches_vllm(top_k): + """A smaller shape, a different top-k, and experts that receive no tokens.""" + + num_experts = 16 + case = _build_case( + num_tokens=257, + hidden=512, + width=256, + top_k=top_k, + num_experts=num_experts, + expert_parallel_size=2, + ep_rank=1, + seed=5, + routing=_skewed_routing, + ) + topk_ids, expert_map = case[4], case[5] + local_ids = torch.nonzero(expert_map >= 0).flatten() + routed_here = torch.isin(topk_ids.to(torch.int64), local_ids) + assert routed_here.any(), "the case must route some tokens to this shard" + assert not routed_here.all(), "the case must also route tokens away from it" + for empty_expert in (num_experts - 2, num_experts - 1): + assert empty_expert in local_ids, "the empty experts must belong to this shard" + assert not (topk_ids == empty_expert).any(), "this expert must stay empty" + + _assert_matches_reference(case) + + +def test_repeated_invocations_do_not_reuse_a_stale_result(): + """Two different inputs through the same code path give two right answers. + + The profiler calls the iteration many times over reused buffers. A result + that survived into the next call would still look plausible on its own. + """ + + shape = _production_shape() + first = _build_case( + num_tokens=512, expert_parallel_size=8, ep_rank=0, seed=101, **shape + ) + second = _build_case( + num_tokens=512, expert_parallel_size=8, ep_rank=0, seed=202, **shape + ) + + first_actual = _assert_matches_reference(first) + second_actual = _assert_matches_reference(second) + + assert not torch.equal(first_actual, second_actual) + + +def test_fp8_path_runs_on_the_gated_activation(): + """The FP8 path quantizes the gated activation and still produces output. + + This is a structural check on native kernels, not FP8 parity. Frontier + quantizes weights and activations with its own helpers, so a bit-exact + comparison against `fused_experts` would first require matching those + schemes. What it does settle is that the quantizer receives the gated + activation buffer rather than a raw slice of the first projection, and that + the real kernels accept that operand and return finite values. + """ + + if not kernel.check_fp8_available(): + pytest.skip("FP8 quantization utilities are unavailable in this build") + + num_tokens, hidden, width, top_k, num_experts = 256, 512, 256, 2, 16 + activations, w1, w2, topk_weights, topk_ids, expert_map = _build_case( + num_tokens=num_tokens, + hidden=hidden, + width=width, + top_k=top_k, + num_experts=num_experts, + expert_parallel_size=2, + ep_rank=0, + seed=17, + ) + + block_shape = [128, 128] + quantized_w1, w1_scale = kernel.quantize_weights_to_fp8(w1, block_shape=block_shape) + quantized_w2, w2_scale = kernel.quantize_weights_to_fp8(w2, block_shape=block_shape) + quantized_a, a_scale = kernel.quantize_activations_to_fp8( + activations, group_size=block_shape[1] + ) + + with vllm_config_context(): + config = kernel.try_get_optimal_moe_config( + w1_shape=quantized_w1.shape, + w2_shape=quantized_w2.shape, + top_k=top_k, + dtype=kernel.get_config_dtype_str(activations.dtype), + M=num_tokens, + block_shape=block_shape, + ) + sorted_token_ids, expert_ids, padded = kernel.moe_align_block_size( + topk_ids, config["BLOCK_SIZE_M"], num_experts, expert_map=expert_map + ) + + observed = {} + original_quantize = kernel.quantize_activations_to_fp8 + + def observing_quantize(tensor, *, group_size): + observed["shape"] = tuple(tensor.shape) + return original_quantize(tensor, group_size=group_size) + + kernel.quantize_activations_to_fp8 = observing_quantize + try: + out_hidden_states = torch.empty( + num_tokens, hidden, device="cuda", dtype=activations.dtype + ) + kernel._run_fused_moe_iteration( + A=quantized_a, + w1=quantized_w1, + w2=quantized_w2, + intermediate_cache1=torch.empty( + num_tokens, top_k, 2 * width, device="cuda", dtype=activations.dtype + ), + intermediate_cache2=torch.empty( + num_tokens * top_k, width, device="cuda", dtype=activations.dtype + ), + intermediate_cache3=torch.empty( + num_tokens, top_k, hidden, device="cuda", dtype=activations.dtype + ), + out_hidden_states=out_hidden_states, + topk_weights=topk_weights, + sorted_token_ids=sorted_token_ids, + expert_ids=expert_ids, + num_tokens_post_padded=padded, + top_k=top_k, + config=config, + block_dims=(block_shape[0], block_shape[1]), + A_scale=a_scale, + w1_scale=w1_scale, + w2_scale=w2_scale, + use_fp8=True, + ) + finally: + kernel.quantize_activations_to_fp8 = original_quantize + torch.cuda.synchronize() + + # The gated activation is `(M * top_k, width)`; the raw first projection + # would be twice as wide. + assert observed["shape"] == (num_tokens * top_k, width) + assert out_hidden_states.shape == (num_tokens, hidden) + assert torch.isfinite(out_hidden_states).all() diff --git a/tests/unit/test_moe_fused_expert_arithmetic.py b/tests/unit/test_moe_fused_expert_arithmetic.py index bc09e3bf..125dc9df 100644 --- a/tests/unit/test_moe_fused_expert_arithmetic.py +++ b/tests/unit/test_moe_fused_expert_arithmetic.py @@ -145,7 +145,6 @@ def _run(problem): num_tokens_post_padded=torch.zeros(1, dtype=torch.int32), top_k=TOP_K, config={"BLOCK_SIZE_M": 16}, - expert_hidden_dim_per_partition=EXPERT_HIDDEN_DIM, block_dims=None, ) return SimpleNamespace(cache1=cache1, cache2=cache2, cache3=cache3, out=out) @@ -246,7 +245,6 @@ def fake_quantize(tensor, *, group_size): num_tokens_post_padded=torch.zeros(1, dtype=torch.int32), top_k=TOP_K, config={"BLOCK_SIZE_M": 16}, - expert_hidden_dim_per_partition=EXPERT_HIDDEN_DIM, block_dims=(128, 64), use_fp8=True, ) @@ -280,7 +278,6 @@ def run_with(activations): num_tokens_post_padded=torch.zeros(1, dtype=torch.int32), top_k=TOP_K, config={"BLOCK_SIZE_M": 16}, - expert_hidden_dim_per_partition=EXPERT_HIDDEN_DIM, block_dims=None, ) return out.clone() From 79f599aac10bc1e2224120236671fe6825725a82 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 14:08:08 +0800 Subject: [PATCH 22/80] docs(profiling): record the grouped-GEMM scope and its identity limits `moe_grouped_gemm` now means the complete local expert computation on both vLLM backends, but nothing in a profiled row says which version produced it. `moe_grouped_gemm_backend` returns `vllm_fused` for the low-level and the functional path alike, and `profiling_patch_tag` appears only in one historical CSV header with nothing in the source writing it. Per the maintainer's decision, leave the metadata alone and state the limit in the profiling guide instead: what the operator measures, how large the pre-repair gap was, and that a row cannot be checked for completeness from its own columns, so the remedy is to re-profile. Record the decision and its evidence in the task review and requirements. --- docs/profiling/README.md | 31 +++++++++++++++++++ .../requirements.md | 3 ++ .../review.md | 5 +-- 3 files changed, 37 insertions(+), 2 deletions(-) diff --git a/docs/profiling/README.md b/docs/profiling/README.md index f305ee98..a947a277 100644 --- a/docs/profiling/README.md +++ b/docs/profiling/README.md @@ -4,6 +4,7 @@ | Date | Summary of Changes | | ---------- | ------------------ | +| 2026-09-22 | Recorded the `moe_grouped_gemm` measurement scope and the limits of the backend identity columns. | | 2026-09-17 | Corrected TP8 GDN launch to use eight distributed processes. | | 2026-09-14 | Documented standard ROCm/GDN output contracts and the experimental SGLang boundary. | @@ -207,6 +208,36 @@ TP domain independently of these MoE EP values. At runtime, Step3 shared-expert work uses `attn_tp` in co-location/PREFILL/unified DECODE and the role-local `moe_tp` in DECODE_FFN. +#### What `moe_grouped_gemm` measures + +`moe_grouped_gemm` is the complete local expert computation: the first expert +GEMM, the gated activation, the optional activation quantization, the second +expert GEMM with the routing weights applied, and the local reduction over one +token's top-k expert outputs. That reduction is a per-token sum, not a +collective, so it carries no communication cost. `MOE_FAMILY` has no separate +operator for it, and the vLLM functional backend has always included it, so +counting it here counts it exactly once on both backends. + +Before 2026-09-22 the vLLM 0.10.x low-level profiling path omitted the gated +activation and the reduction. Rows produced by that path under-measure +`moe_grouped_gemm`, and the gap grows with token count: on the checked-in +`a800/qwen3-a3b-30b-moe` dataset the two missing kernels are an estimated 16.5% +of the corrected value at 4096 tokens, against 6.8% at the median row. Rows +produced by the functional backend, or by any path after that date, are +complete. + +#### Backend identity columns cannot date a row + +`moe_grouped_gemm_backend` records `vllm_fused` for both the vLLM 0.10.x +low-level path and the current functional path, so it does not distinguish a +row measured before that fix from one measured after it. +`profiling_patch_tag` appears in one historical CSV header, but nothing in the +source writes it, so it is not a live mechanism either. + +Re-profile rather than infer. If you need corrected `moe_grouped_gemm` timings +from a `vllm>=0.10,<0.11` environment, re-run the producer; an existing row +cannot be checked for completeness from its own metadata. + ### Standard GDN on ROCm The standard GDN producer uses the vLLM Qwen3.5 module path and writes a diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md index 1bf6ec58..6ad55631 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md @@ -5,6 +5,7 @@ | Date | Change | | --- | --- | | 2026-09-21 | Recorded the original request, the specification hand-off, and the decisions from the planning interview. | +| 2026-09-22 | Recorded the W6 artifact-identity decision and the native GPU validation instruction. | ## [Original Request] 2026-09-21 @@ -34,3 +35,5 @@ The draft specification is landed verbatim in `plan.md` together with an Amendme | 2026-09-22 | W5 open item 3: the single public name for the routing implementation identity, asked under the AGENTS.md naming gate with three materially plausible spellings. | **`moe_routing_runtime_path`**, a `ReplicaConfig` field, CLI `--replica_config_moe_routing_runtime_path`. It shares the prefix of the neighbouring `moe_routing_seed`, `moe_routing_trace_path` and `moe_routing_distribution_type`. The profiling CSV column stays `routing_runtime_path` and the standalone trainer flag stays `--routing_runtime_path`; neither is renamed. | | 2026-09-22 | W5 scope, asked after the naming decision: global field only, or global plus per-role overrides. | Global + per-role overrides (`prefill`, `decode`, `decode_ffn`). Superseded the same day by the row below. | | 2026-09-22 | `[Original Request]` "我在rethink添加 frontier.moe_routing_runtime的必要性。我的concern是，该部分的align是否对模拟准确度意义重大？如果对fidelity的影响不到0.5%,我认为完全无需引入如此复杂的变量和setting,这会使得可用性和可读性变得很困难" -> after the explanation of what the module decides and the measured 3-8% per-layer cost of a wrong path versus the 0% change of the unset override: "我认为采用 当前默认的 两个 cluster 都只能读同一个全局 moe_routing_distribution_type，永远解析出同一个 path ；进行回退" | **W5 not ported.** Keep the existing contract: one global `moe_routing_distribution_type`, one derived routing path per run, no override field, no registry routing axis. The drafted implementation was reverted before any commit and archived as `w5_reverted_moe_routing_runtime_path.patch`. Both earlier W5 decisions above are void. | +| 2026-09-22 | W6 artifact identity: should the profiling metadata gain a column that separates an incomplete legacy `moe_grouped_gemm` measurement from a complete one? | **"不改 metadata，只记录限制"** — do not change the profiling metadata; record the limitation only. Written into `docs/profiling/README.md` under the MoE producer. No code change, no new column, no admission gate. | +| 2026-09-22 | W6 native validation: how should the GPU parity check be run? | **"请从 dockerhub 中找到 v0.10.2 的官方镜像（如果没有，fallback 到 >=0.10, <0.11），然后参考 hand-book 中对 docker 的使用在 gpu worker 上使用该镜像。如果你需要使用原来的 benchmark ... 中的测试 suits 和插桩，你需要 mount 该 repo 到 gpu worker。如果需要进行对比验证，则 follow skill：/home/brainpp/.claude/skills/frontier-calibration"** — `vllm/vllm-openai:v0.10.2` exists on Docker Hub, so no fallback was needed. It is pulled through the company docker.io proxy as `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`. The instrumented benchmark repository is not mounted: the parity check compares tensors from vLLM's own `fused_experts` inside one process and needs no serving instrumentation. The `frontier-calibration` skill is not invoked for the same reason — its workflow is E2E simulator-versus-served-vLLM calibration, not a kernel-level tensor comparison. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index e3340b46..f9004462 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -11,6 +11,7 @@ | 2026-09-22 | Recorded the maintainer's PR #34 / PR #35 review: D1 and D2 resolved, ten review comments dispositioned, each verified against source. | | 2026-09-22 | Self-review of that record: corrected the lockstep mechanism and counter semantics under D1, the line references under R34-01, the R34-03 remedy (the baseline label is itself an assembled partial run), and added the omissions listed under "Found on re-review". | | 2026-09-22 | W6 arithmetic delivered with CPU validation; two deviations from the recorded adaptations justified; artifact identity raised as an open decision; native GPU validation NOT_RUN. | +| 2026-09-22 | W6 artifact identity decided as document-only; native parity test added and submitted to an H800 worker. | | 2026-09-22 | W5 closed without source changes. Corrected the Mechanism A premise: the routing distribution has no per-role override on main, so both W5 mechanisms are unreachable from any released configuration. The user chose to keep the single global field; the drafted implementation was reverted and archived as a patch. | ## Pinned source snapshot @@ -104,8 +105,8 @@ Claims in those reports that this PR depends on were re-verified directly agains | Deviation 2 from the adaptations above | Main's `SiluAndMul` wrapper was **not** reused. Reason: its `forward` calls `torch.empty` on every invocation (`activation.py:29-33`), which would put an allocation inside the timed profiling step and change what `moe_grouped_gemm` measures. The repair calls `torch.ops._C.silu_and_mul` into a preallocated buffer, exactly as vLLM's `fused_experts_impl` does, and a test asserts one workspace is shared across every profiled step. | | Measurement ownership, decided | `moe_grouped_gemm` = GEMM1, gated activation, optional activation quantization, GEMM2, local top-k reduction. This is already what the functional backend measured (`fused_experts` returns reduced hidden states), so the repair removes a scope disagreement between the two backends. `MOE_FAMILY` (`frontier/operators/families.py:77`) has no operator for the reduction, so counting it here counts it exactly once without a fifth operator, a new column or a new trained model. The reference's own `record_function("moe_grouped_gemm")` excludes `moe_sum`; that finer split is diagnostics, and the difference is recorded rather than adopted. | | Magnitude, estimated before implementing | On `a800/qwen3-a3b-30b-moe` the two missing kernels are an estimated 16.5% of the corrected `moe_grouped_gemm` time at 4096 tokens (6.8% median over all rows, 26.3% max). On the two 64-token h800 datasets, 1.1-1.4%. Analytical estimate at 80% of peak HBM, not a measurement. | -| Open: artifact identity | `resolve_grouped_gemm_backend` (`moe_wrapper.py:50-59`) returns `vllm_fused` for both the legacy and the functional path, so no column distinguishes an incomplete legacy measurement from a complete one, and the checked-in datasets cannot be classified. `profiling_patch_tag` appears in one CSV but nowhere in the source, so it is not a live mechanism. Nothing in `frontier/` reads `moe_grouped_gemm_backend`; only `tests/unit/test_moe_native_admission.py:93` asserts the mxfp4 label. Options: (1) record the producing implementation in a new column, leaving old rows empty and adding no gate; (2) option 1 plus a trainer admission gate, which would immediately reject every checked-in dataset including the `examples/profiling` smokes; (3) defer and document. Recommended: (1). Raised for a user decision, as the plan requires. | -| Open: native validation | NOT_RUN. The repaired path is selected only under `vllm>=0.10,<0.11`; the two Torch environments on this host carry vLLM 0.11.0 and 0.28.0 and both take the functional path. A GPU worker with the pinned profiling environment is required, and the affected FP8 path belongs in the same matrix. | +| Decided: artifact identity | **Document the limitation, change no metadata.** User decision, 2026-09-22: 不改 metadata，只记录限制. Evidence behind it is unchanged: `resolve_grouped_gemm_backend` (`moe_wrapper.py:50-59`) returns `vllm_fused` for both the low-level and the functional vLLM path, so no column separates an incomplete legacy row from a complete one; `profiling_patch_tag` carries three historical free-text values in `a800/qwen3-a3b-30b-moe/moe.csv` but nothing in the source writes it; nothing in `frontier/` reads `moe_grouped_gemm_backend`, and only `tests/unit/test_moe_native_admission.py:93` asserts the mxfp4 label. Recorded in `docs/profiling/README.md` under the MoE producer: what `moe_grouped_gemm` measures, the size of the pre-fix gap, and that the identity columns cannot date a row, so the remedy is to re-profile rather than infer. | +| Native validation | Test added: `tests/integration/test_moe_fused_expert_numerical_parity.py`, 8 cases, skipping unless CUDA is present and `VLLM_API_VERSION == "0.10.x"`. It drives `_run_fused_moe_iteration` with the buffer shapes, kernel config and alignment `profile_fused_moe_kernel` uses and compares the output tensor against `fused_experts` at `rtol=0, atol=0`. Neither Torch environment on this host selects the repaired path (vLLM 0.11.0 and 0.28.0), so it runs on an H800 worker under the official `vllm/vllm-openai:v0.10.2` image. Result recorded in `test_report_2026-09-22_w6_fused_expert_arithmetic.md`. | | Measured scope today | The whole `_step` body, timed either with per-iteration CUDA events and a synchronize (`:603-626`) or through `record_function("vidur_moe_grouped_gemm")` (`:629-653`). All buffers are allocated outside the timed region. | | Disposition | **PORT** the gated-activation repair. **BLOCKED** on the local output reduction; see D2 below. | From d695679a8a04eb432caa7f91f12cec02889b3d09 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 14:10:23 +0800 Subject: [PATCH 23/80] docs(review): record the W6 native parity submission and close artifact identity --- .../progress.md | 11 +- ...t_2026-09-22_w6_fused_expert_arithmetic.md | 106 ++++++++++++++---- .../validation.md | 5 +- 3 files changed, 98 insertions(+), 24 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index a45c3666..dab9ce82 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -11,6 +11,7 @@ | 2026-09-22 | Checkpoint D second half: W4, the opt-in vLLM-style DP placement policy, implemented and committed as `10dd474`; measured against five deliberate-defect controls and a 71-of-71 identical fidelity matrix. Step 4 closed. | | 2026-09-22 | Checkpoint E second half: W6 arithmetic repaired and CPU-validated; measurement ownership decided; native GPU validation and artifact identity left open. | | 2026-09-22 | Checkpoint E first half: W5 closed as NOT PORTED by user decision after the premise check showed the collision unreachable on main; the drafted implementation was reverted before commit and archived as a patch. | +| 2026-09-22 | W6 artifact identity decided as document-only and written into the profiling guide; native parity test added and submitted to an H800 worker as `exp-0922-140423-075005`. | ## Status @@ -19,9 +20,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | -| Current step | Step 6 source and CPU validation complete; native GPU validation NOT_RUN and artifact identity OPEN | +| Current step | Step 6 source, CPU validation and artifact-identity documentation complete; native GPU parity job `exp-0922-140423-075005` submitted, result pending | | Publication | PUSHED_VERIFIED (records) | -| Next action | Two user decisions on W6 (artifact identity metadata; whether to run the native GPU parity matrix), then Checkpoint F's conditional W7. | +| Next action | Record the native parity result, then Checkpoint F's conditional W7. | ## Step status @@ -33,7 +34,7 @@ | 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 4 | Opt-in vLLM DP placement | PASS | unit PASS (61 new, 3778 total, failure set identical to the parent); integration PASS (3 cases in the real event loop, including a placement that diverges from round-robin); five deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | PUSHED_VERIFIED (records + PR 35 section) | REVIEWED (user chose to keep the single global field) | -| 6 | Legacy fused-MoE profiling | PARTIAL: arithmetic and measurement ownership done; native validation NOT_RUN, artifact identity OPEN | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778) | pending | — | +| 6 | Legacy fused-MoE profiling | PARTIAL: arithmetic, measurement ownership and artifact-identity documentation done; native parity job submitted, result pending | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778). Native: 8 cases in `tests/integration/test_moe_fused_expert_numerical_parity.py`, submitted as `exp-0922-140423-075005` | pending | — | | 7 | Optional zero-payload backend | NOT_STARTED (facts in `plan.md` A7) | — | — | — | | 8 | Combined regression, PR hand-off | NOT_STARTED | — | — | — | @@ -85,6 +86,10 @@ - 2026-09-22: W6 CPU validation. 7 tests in `tests/unit/test_moe_fused_expert_arithmetic.py` under the Torch environment, covering the arithmetic against a written-out reference, the discriminating comparison with the old slice, call order and operand provenance, routing-weight placement, the local reduction, FP8 quantizer input, workspace reuse across steps, and allocation-site dimensions under TP=2. Regression against a detached `HEAD` worktree: same single environment-dependent failure both sides. Default-environment suite unchanged at 84 failed / 3778 passed. No fidelity matrix: the simulator cannot import the changed module. - 2026-09-22: W6 left two items open rather than deciding them unilaterally. Artifact identity — `resolve_grouped_gemm_backend` labels both the legacy and the functional path `vllm_fused`, so no checked-in row can be classified as a complete or an incomplete measurement, and `profiling_patch_tag` turns out to exist only in one CSV and nowhere in the source. Native GPU parity — the repaired path is selected only under `vllm>=0.10,<0.11`, and both local Torch environments carry newer vLLM. +- 2026-09-22: W6 artifact identity decided by the user: do not change the profiling metadata, record the limitation only. `docs/profiling/README.md` now states what `moe_grouped_gemm` measures, the size of the pre-repair gap, and that `moe_grouped_gemm_backend` and `profiling_patch_tag` cannot date a row, so the remedy is to re-profile. Confirmed while writing it that `profiling_patch_tag` holds three historical free-text values in one CSV and is written nowhere in the source. No column added, no admission gate. +- 2026-09-22: W6 native parity test added. `tests/integration/test_moe_fused_expert_numerical_parity.py`, 8 cases, adapted from the donor and extended to the plan's required matrix: Qwen3-A3B-30B shapes read from the checked-in model config at 4096 and 4097 tokens on EP ranks 0 and 1; a 257-token, 16-expert case at top-k 2 and 4 with popularity-weighted routing that leaves two local experts empty; repeated invocation with different inputs; and the FP8 path as a structural check. It drives `_run_fused_moe_iteration` with the same buffers, config and alignment `profile_fused_moe_kernel` uses and compares the output tensor against `fused_experts` at `rtol=0, atol=0`. Skips unless CUDA is present and `VLLM_API_VERSION == "0.10.x"`; verified to collect and skip cleanly locally. `expert_hidden_dim_per_partition` was dropped from the iteration signature in the same commit, since the gated activation reads the whole projection and the parameter selected nothing. +- 2026-09-22: W6 native parity submitted. StepMind Python `RJobBackend` from the local host, job `exp-0922-140423-075005`, creator `i-fengyicheng`, `steptron_ci` / `H800`, 1 GPU, image `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2` (the official Docker Hub build through the company docker.io proxy), NFS source `100.96.128.195:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`. The instrumented benchmark repo was not mounted and the `frontier-calibration` skill was not invoked: this compares tensors from vLLM's own `fused_experts` inside one process, which needs neither serving instrumentation nor an E2E calibration workflow. + ## Step 3 scoping, as recorded before implementation Kept as written so the implementation can be read against the scope it started from. Step 3 is now complete; see the W3 entries above and `design.md`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md index 6499d299..3fc0ca58 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md @@ -8,6 +8,7 @@ Date: 2026-09-22. Branch `fix/issue26-correctness-pr`, worktree | Date | Change | | --- | --- | | 2026-09-22 | Created: reachability check, magnitude estimate, source repair, CPU validation. Native GPU validation NOT_RUN. | +| 2026-09-22 | Artifact identity decided as document-only. Native parity test added and submitted to an H800 worker as `exp-0922-140423-075005`; result pending. | ## 1. Is the defect reachable @@ -187,22 +188,89 @@ matrix's inputs are checked-in CSVs, not freshly profiled data. Stated plainly: this repair changes what a *future* profiling run measures. It changes nothing about a simulation run from existing data. -## 8. Not done - -- **Native numerical validation: NOT_RUN.** The plan requires a GPU local-expert - parity test against the pinned vLLM reference, including more than one local - expert partition and a boundary case with uneven expert occupancy. It needs a - worker with `vllm>=0.10,<0.11`, because that is the only configuration that - selects the repaired path. No such environment exists on this host. Pending a - decision on the GPU run. -- **Artifact identity: OPEN, raised for decision.** Existing rows record - `moe_grouped_gemm_backend='vllm_fused'` for both the legacy and the functional - path (`resolve_grouped_gemm_backend`, `moe_wrapper.py:50-59`), so the column - cannot distinguish an incomplete legacy measurement from a complete one. - `profiling_patch_tag` exists in one checked-in CSV but appears nowhere in the - source, so it is not a live mechanism. No consumer in `frontier/` reads - `moe_grouped_gemm_backend`; only `tests/unit/test_moe_native_admission.py:93` - asserts the mxfp4 label. Options and a recommendation are in `review.md`. -- **FP8 path**: the repair changes what the FP8 quantizer receives, so the FP8 - path is an affected native path. It is not proven by the CPU shape test and is - included in the pending GPU matrix. +## 8. Native numerical validation + +### The test + +`tests/integration/test_moe_fused_expert_numerical_parity.py`, 8 cases. It +drives `_run_fused_moe_iteration` with the buffer shapes, kernel config and +block alignment that `profile_fused_moe_kernel` uses, then compares the output +tensor against vLLM's own `fused_experts` over the same activations, weights, +routing weights, routing ids and expert map. Both sides run in one process on +one GPU. + +| Case | What it covers | Plan requirement | +| --- | --- | --- | +| `test_production_shaped_expert_output_matches_vllm[0-4096]`, `[0-4097]`, `[1-4096]`, `[1-4097]` | Qwen3-A3B-30B shapes read from `data/config/models/qwen3-a3b-30b-moe.json`, EP 8, ranks 0 and 1. 4097 exercises the padded tail. | Qwen-shaped 4096/4097 with the checked-in config; more than one local expert partition. | +| `test_uneven_expert_occupancy_matches_vllm[2]`, `[4]` | 257 tokens, hidden 512, width 256, 16 experts on EP 2 rank 1, popularity-weighted routing that leaves the last two global experts empty. The test asserts the shard both receives and misses tokens, and that those two experts stay empty. | A smaller boundary case with a different valid top-k and uneven expert occupancy. | +| `test_repeated_invocations_do_not_reuse_a_stale_result` | Two different inputs through the same path, each matched against its own reference, and the two results asserted different. | Repeated invocation so workspace reuse cannot leak a stale result. | +| `test_fp8_path_runs_on_the_gated_activation` | The FP8 path on native kernels: the quantizer receives `(M * top_k, width)`, not the raw `2 * width` projection, and the output is finite. | The affected FP8 path, as a structural check. | + +Comparison tolerance is `rtol=0, atol=0`. Both sides call the same Triton +kernel with the same config, the same `torch.ops._C.silu_and_mul` and the same +`moe_sum`, so any difference is a real difference. Every case also asserts a +finite output. + +Gating: the module skips unless CUDA is present and +`VLLM_API_VERSION == "0.10.x"`, which is the only configuration that selects the +repaired path. It collects and skips cleanly in both local environments. + +### Why it runs on a worker + +Neither Torch environment on this host selects the repaired path: +`openmopd-py312` carries vLLM 0.11.0 and `oneshot-opd-py312` carries 0.28.0, +and both report `functional_fused_experts`. The pinned profiling range is +`vllm>=0.10,<0.11`. + +### Run + +| Field | Value | +| --- | --- | +| Submission host | `kun-workspace-vgen2` (local), StepMind Python `RJobBackend`, `STEPMIND_BACKEND=rjob` | +| Job name | `exp-0922-140423-075005` | +| Creator | `i-fengyicheng` | +| Charged group / tag | `steptron_ci` / `H800` | +| Shape | 1 GPU, 8 CPU, 64000Mi | +| Image | `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`, the official Docker Hub `vllm/vllm-openai:v0.10.2` through the company docker.io proxy | +| NFS mount | `100.96.128.195:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr` | +| Command | `python3 -m pytest -v -rA -p no:cacheprovider --no-header tests/integration/test_moe_fused_expert_numerical_parity.py`, after printing `nvidia-smi`, the Torch/vLLM/`fused_moe` source paths and `VLLM_API_VERSION` | +| Status | **PENDING** at the time of writing. Result and per-case outcomes are appended below when the job reaches a terminal state. | + +The instrumented benchmark repository was not mounted. This test compares +tensors from vLLM's own `fused_experts` inside one process, so it needs no +serving instrumentation. For the same reason the `frontier-calibration` skill +was not invoked: its workflow is E2E simulator-versus-served-vLLM calibration, +not a kernel-level tensor comparison. + +The image is the official upstream build, not the instrumented fork, so the +`topk_softmax` arity concern recorded as open item 6 in `review.md` does not +apply to this run. + +## 9. Artifact identity + +Decided by the user on 2026-09-22: **do not change the profiling metadata, +record the limitation only.** + +`resolve_grouped_gemm_backend` (`moe_wrapper.py:50-59`) returns `vllm_fused` +for both the low-level and the functional vLLM path, so no column separates an +incomplete legacy row from a complete one. `profiling_patch_tag` carries three +historical free-text values in `a800/qwen3-a3b-30b-moe/moe.csv`, but nothing in +the source writes it. Nothing in `frontier/` reads +`moe_grouped_gemm_backend`; only `tests/unit/test_moe_native_admission.py:93` +asserts the mxfp4 label. + +`docs/profiling/README.md` now records what `moe_grouped_gemm` measures, the +size of the pre-repair gap, and that a row cannot be checked for completeness +from its own metadata, so the remedy is to re-profile. No column was added and +no admission gate was introduced. + +## 10. Still unproven + +- **FP8 arithmetic.** The repair changes what the FP8 quantizer receives. The + native case above checks that the real kernels accept the gated activation and + return finite values; it does not check FP8 numerics against a reference. + Frontier quantizes weights and activations with its own helpers, so a bit-exact + comparison against `fused_experts` would first require matching those schemes. + Recorded as unproven rather than claimed. +- **MXFP4.** Untouched by this repair; it returns through the functional branch + before the legacy allocations. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index 3fc05366..49b5ee75 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -10,6 +10,7 @@ | 2026-09-22 | Step 2 re-measured with harness and source at one revision, after the gate corrections. Same expectation, same result, recorded provenance. | | 2026-09-22 | Step 3 recorded: the shared monolithic forward, its direct-construction runtime evidence, four deliberate-defect controls, and a 71-of-71 identical fidelity matrix. | | 2026-09-22 | Step 4 recorded: the opt-in vLLM-style DP placement policy, its real-runtime wiring evidence including a divergence from round-robin, five deliberate-defect controls, and a 71-of-71 identical fidelity matrix. | +| 2026-09-22 | Step 6 native parity test recorded and submitted as `exp-0922-140423-075005`; artifact identity closed as document-only. | ## Environment @@ -458,5 +459,5 @@ Full record: `test_report_2026-09-22_w6_fused_expert_arithmetic.md`. | Regression, Torch environment | Against a detached worktree at `HEAD` (`bbbfcaa`), same four existing files: 1 failed / 136 passed at HEAD, 1 failed / 143 passed with the repair. Same failure identity on both sides — `test_functional_vllm_kernel_exposes_mxfp4_switch_without_importing_vllm` needs Torch without vLLM, which no local environment provides. | | Regression, default environment | `pytest tests/unit --continue-on-collection-errors` under `frontier-py310`: 84 failed, 3778 passed, 49 skipped, 11 errors — the W4 baseline of 84 failures and 3778 passing, unchanged. Collection errors 10 -> 11 because the new file imports Torch at module level, as the seven existing profiling test files in that list already do; verified by re-collecting with it ignored. | | Fidelity matrix | Not run, deliberately. The change is confined to a module the simulator cannot import (it requires Torch, absent from the simulator environment), and the matrix consumes checked-in CSVs rather than fresh profiling. The unchanged default-environment suite is the evidence. The repair changes what a future profiling run measures, not any simulation from existing data. | -| Native GPU parity | **NOT_RUN.** Requires a worker with `vllm>=0.10,<0.11`. The FP8 path is affected by the repair and belongs in the same matrix. | -| Artifact identity | **OPEN**, raised for decision; see `review.md`. Existing rows cannot be classified as complete or incomplete from their metadata. | +| Native GPU parity | **SUBMITTED, result pending.** `tests/integration/test_moe_fused_expert_numerical_parity.py`, 8 cases at `rtol=0, atol=0` against `fused_experts`: Qwen3-A3B-30B shapes from the checked-in model config at 4096 and 4097 tokens on EP ranks 0 and 1; a 257-token, 16-expert case at top-k 2 and 4 whose routing leaves two local experts empty; repeated invocation with different inputs; and the FP8 path as a structural check. Job `exp-0922-140423-075005`, creator `i-fengyicheng`, `steptron_ci` / `H800`, 1 GPU, image `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`, NFS source `100.96.128.195:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`. | +| Artifact identity | **CLOSED as document-only** by user decision. The finding stands: `resolve_grouped_gemm_backend` labels both vLLM paths `vllm_fused`, and `profiling_patch_tag` holds three historical free-text values in `a800/qwen3-a3b-30b-moe/moe.csv` while nothing in the source writes it. No column was added. `docs/profiling/README.md` records the operator's scope, the size of the pre-repair gap, and that a row cannot be checked for completeness from its own metadata. | From cad3afda83a0808846fa6749b069d1326f0e22c3 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 14:15:08 +0800 Subject: [PATCH 24/80] docs(review): record the verified W7 facts and its companion-repository blocker --- .../progress.md | 7 +++++-- .../review.md | 18 ++++++++++++++++++ 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index dab9ce82..3d5afbd1 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -12,6 +12,7 @@ | 2026-09-22 | Checkpoint E second half: W6 arithmetic repaired and CPU-validated; measurement ownership decided; native GPU validation and artifact identity left open. | | 2026-09-22 | Checkpoint E first half: W5 closed as NOT PORTED by user decision after the premise check showed the collision unreachable on main; the drafted implementation was reverted before commit and archived as a patch. | | 2026-09-22 | W6 artifact identity decided as document-only and written into the profiling guide; native parity test added and submitted to an H800 worker as `exp-0922-140423-075005`. | +| 2026-09-22 | W7 investigated while the GPU job queued: candidate gitlink unpublished, three payload defects confirmed by execution, companion-repository decision pending. | ## Status @@ -22,7 +23,7 @@ | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | | Current step | Step 6 source, CPU validation and artifact-identity documentation complete; native GPU parity job `exp-0922-140423-075005` submitted, result pending | | Publication | PUSHED_VERIFIED (records) | -| Next action | Record the native parity result, then Checkpoint F's conditional W7. | +| Next action | Record the native parity result. W7 needs one user decision: authorize a companion-repository branch in `fwyc0573/frontier-htsim`, or record W7 as `EXCLUDED`. | ## Step status @@ -35,7 +36,7 @@ | 4 | Opt-in vLLM DP placement | PASS | unit PASS (61 new, 3778 total, failure set identical to the parent); integration PASS (3 cases in the real event loop, including a placement that diverges from round-robin); five deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | PUSHED_VERIFIED (records + PR 35 section) | REVIEWED (user chose to keep the single global field) | | 6 | Legacy fused-MoE profiling | PARTIAL: arithmetic, measurement ownership and artifact-identity documentation done; native parity job submitted, result pending | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778). Native: 8 cases in `tests/integration/test_moe_fused_expert_numerical_parity.py`, submitted as `exp-0922-140423-075005` | pending | — | -| 7 | Optional zero-payload backend | NOT_STARTED (facts in `plan.md` A7) | — | — | — | +| 7 | Optional zero-payload backend | INVESTIGATED, BLOCKED on a companion-repository decision. Candidate gitlink `e564935d` is unpublished, not inaccessible; the three payload defects are confirmed by execution against the published backend; the fix is companion-side only | Published runner exercised directly: explicit zero and missing field give the same `exit=2`, `-1` passes validation, CLI `0` loses to a positive spec value | — | — | | 8 | Combined regression, PR hand-off | NOT_STARTED | — | — | — | ## Chronological updates @@ -90,6 +91,8 @@ - 2026-09-22: W6 native parity test added. `tests/integration/test_moe_fused_expert_numerical_parity.py`, 8 cases, adapted from the donor and extended to the plan's required matrix: Qwen3-A3B-30B shapes read from the checked-in model config at 4096 and 4097 tokens on EP ranks 0 and 1; a 257-token, 16-expert case at top-k 2 and 4 with popularity-weighted routing that leaves two local experts empty; repeated invocation with different inputs; and the FP8 path as a structural check. It drives `_run_fused_moe_iteration` with the same buffers, config and alignment `profile_fused_moe_kernel` uses and compares the output tensor against `fused_experts` at `rtol=0, atol=0`. Skips unless CUDA is present and `VLLM_API_VERSION == "0.10.x"`; verified to collect and skip cleanly locally. `expert_hidden_dim_per_partition` was dropped from the iteration signature in the same commit, since the gated activation reads the whole projection and the parameter selected nothing. - 2026-09-22: W6 native parity submitted. StepMind Python `RJobBackend` from the local host, job `exp-0922-140423-075005`, creator `i-fengyicheng`, `steptron_ci` / `H800`, 1 GPU, image `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2` (the official Docker Hub build through the company docker.io proxy), NFS source `100.96.128.195:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`. The instrumented benchmark repo was not mounted and the `frontier-calibration` skill was not invoked: this compares tensors from vLLM's own `fused_experts` inside one process, which needs neither serving instrumentation nor an E2E calibration workflow. +- 2026-09-22: W7 investigated while the parity job queued. `fwyc0573/frontier-htsim` is public and readable, its only branch `main` is exactly the gitlink Frontier already pins, and the candidate's `e564935d` returns HTTP 422 with no local object store on this host containing it — an unpublished commit, not an access problem. The fix is entirely companion-side. Confirmed all three payload defects by running the published runner rather than reading it: an explicit `tensor_bytes = 0` and a deleted field produce the identical `missing required fields: ['tensor_bytes']` exit 2; `-1` passes validation and reaches the simulator invocation; and `set_if_none_or_zero` replaces an explicit CLI `0` with a positive spec value. Reachable from Frontier because `_validate_data_size` accepts zero and `moe_operator_times.py:512` computes the EP all-to-all payload as `embedding_dim * 2 * routed_tokens`, which is zero for an empty local lane. Frontier cannot fix it alone: a zero-byte collective still costs the intra-server latency, so short-circuiting would change the backend's semantics. Blocked on the user's decision about the second repository; nothing was created or pushed there. + ## Step 3 scoping, as recorded before implementation Kept as written so the implementation can be read against the scope it started from. Step 3 is now complete; see the W3 entries above and `design.md`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index f9004462..3228b352 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -12,6 +12,7 @@ | 2026-09-22 | Self-review of that record: corrected the lockstep mechanism and counter semantics under D1, the line references under R34-01, the R34-03 remedy (the baseline label is itself an assembled partial run), and added the omissions listed under "Found on re-review". | | 2026-09-22 | W6 arithmetic delivered with CPU validation; two deviations from the recorded adaptations justified; artifact identity raised as an open decision; native GPU validation NOT_RUN. | | 2026-09-22 | W6 artifact identity decided as document-only; native parity test added and submitted to an H800 worker. | +| 2026-09-22 | W7 facts re-verified: the candidate gitlink is unpublished, the three payload defects are confirmed by execution against the published backend, and the fix needs companion-repository authorization. | | 2026-09-22 | W5 closed without source changes. Corrected the Mechanism A premise: the routing distribution has no per-role override on main, so both W5 mechanisms are unreachable from any released configuration. The user chose to keep the single global field; the drafted implementation was reverted and archived as a patch. | ## Pinned source snapshot @@ -110,6 +111,23 @@ Claims in those reports that this PR depends on were re-verified directly agains | Measured scope today | The whole `_step` body, timed either with per-iteration CUDA events and a synchronize (`:603-626`) or through `record_function("vidur_moe_grouped_gemm")` (`:629-653`). All buffers are allocated outside the timed region. | | Disposition | **PORT** the gated-activation repair. **BLOCKED** on the local output reduction; see D2 below. | +### W7 — Optional zero-payload collective-sim backend + +Facts re-verified 2026-09-22, superseding the specification-time record in `plan.md` A7. + +| Item | Finding | +| --- | --- | +| Repository access | Not a problem. `fwyc0573/frontier-htsim` is public and readable, `pushed_at` 2026-06-08, one branch `main` at `b8518afcc310f0fe0e3ce52ba6b4f0bf57a3be04`, which is exactly the gitlink this branch and main already pin. | +| Candidate commit | **Unpublished, not inaccessible.** `e564935d3874d8c71b52a554ab7c9a72e5e19f68` returns HTTP 422 `No commit found for SHA`. No local object store on this host contains it: the submodule directory is empty in every checkout, and `git cat-file -t` fails in the main repository. The candidate's own branch still records it as its gitlink. | +| Where the fix lives | Entirely in the companion repository. The donor test drives `collective_sim_core.predictor.predict_collective_time` and `htsim_runner.py`, both inside the submodule. Frontier's side of this package is the gitlink and nothing else. | +| Defect 1, confirmed by execution | An explicit zero payload is rejected as a missing field. `htsim_runner.py:2349` tests `getattr(args, k) in (None, "", 0)` over a required-field list that includes `tensor_bytes`. Running the published `main` runner with `tensor_bytes = 0` and with the field deleted produces the identical `exit=2, Error: missing required fields: ['tensor_bytes']`. | +| Defect 2, confirmed by execution | A negative payload is accepted. `--tensor-bytes` is a bare `type=int` with no lower bound and no schema check, so `-1` passes validation and reaches the simulator invocation. | +| Defect 3, confirmed by execution | An explicit CLI zero loses to a positive spec value. `set_if_none_or_zero` (`htsim_runner.py:1398-1406`) overwrites the CLI value when it is `0`, so `--tensor-bytes 0` against a spec of 32768 yields 32768. | +| Reachability from Frontier | Real. `base_cc_backend._validate_data_size` rejects only negative sizes, so Frontier passes zero through. `moe_operator_times.py:512` computes `data_size_bytes = embedding_dim * 2 * routed_tokens` and hands it to `predict_all_to_all`; an EP lane with no routed tokens in a step makes that zero. `predict_reduce_scatter` additionally floor-divides by the device count. A MoE EP run under `--cc_backend_config_type collective_sim` therefore aborts on a legitimate empty collective. | +| Why Frontier cannot fix it alone | A zero-byte collective is not a zero-cost collective. The donor test asserts the intra-server latency term survives at payload 0 (`7 x 0.5 us`, `network_ms == 0`), which is also the plan's requirement. Short-circuiting to `0.0` in Frontier would change the backend's synchronization semantics rather than accept the input. | +| Cost of the fix | Three small edits in the companion repository: drop `tensor_bytes` from the zero-means-missing list while keeping it required, add a `>= 0` check, and make the CLI precedence distinguish "unset" from "explicitly zero". No Frontier source change; Frontier moves its gitlink and gains the donor's CPU test. | +| Blocker | Publishing to a second repository. The plan requires the user's decision on companion-repository scope before creating or pushing anything there, and `EXCLUDED` is the alternative. Neither option was taken unilaterally. | + ## Decision checkpoints for the user ### D1 — Scope of the opt-in DP placement strategy From e0e265c3ef9755f659e169aab51d7ae44e02ae3b Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 14:19:05 +0800 Subject: [PATCH 25/80] docs(review): record the PR 35 W6 and W7 sections and the re-run baseline --- .../task_2026-09-21_issue26_correctness_pr/progress.md | 8 +++++--- .../task_2026-09-21_issue26_correctness_pr/validation.md | 1 + 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 3d5afbd1..db2ad92b 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -22,7 +22,7 @@ | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | | Current step | Step 6 source, CPU validation and artifact-identity documentation complete; native GPU parity job `exp-0922-140423-075005` submitted, result pending | -| Publication | PUSHED_VERIFIED (records) | +| Publication | PUSHED_VERIFIED. Draft PR 35 body carries the W6 and W7 sections; still draft. | | Next action | Record the native parity result. W7 needs one user decision: authorize a companion-repository branch in `fwyc0573/frontier-htsim`, or record W7 as `EXCLUDED`. | ## Step status @@ -35,8 +35,8 @@ | 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 4 | Opt-in vLLM DP placement | PASS | unit PASS (61 new, 3778 total, failure set identical to the parent); integration PASS (3 cases in the real event loop, including a placement that diverges from round-robin); five deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | PUSHED_VERIFIED (records + PR 35 section) | REVIEWED (user chose to keep the single global field) | -| 6 | Legacy fused-MoE profiling | PARTIAL: arithmetic, measurement ownership and artifact-identity documentation done; native parity job submitted, result pending | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778). Native: 8 cases in `tests/integration/test_moe_fused_expert_numerical_parity.py`, submitted as `exp-0922-140423-075005` | pending | — | -| 7 | Optional zero-payload backend | INVESTIGATED, BLOCKED on a companion-repository decision. Candidate gitlink `e564935d` is unpublished, not inaccessible; the three payload defects are confirmed by execution against the published backend; the fix is companion-side only | Published runner exercised directly: explicit zero and missing field give the same `exit=2`, `-1` passes validation, CLI `0` loses to a positive spec value | — | — | +| 6 | Legacy fused-MoE profiling | PARTIAL: arithmetic, measurement ownership and artifact-identity documentation done; native parity job submitted, result pending | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778). Native: 8 cases in `tests/integration/test_moe_fused_expert_numerical_parity.py`, submitted as `exp-0922-140423-075005` | PUSHED_VERIFIED (source, tests, docs, records, PR 35 section) | — | +| 7 | Optional zero-payload backend | INVESTIGATED, BLOCKED on a companion-repository decision. Candidate gitlink `e564935d` is unpublished, not inaccessible; the three payload defects are confirmed by execution against the published backend; the fix is companion-side only | Published runner exercised directly: explicit zero and missing field give the same `exit=2`, `-1` passes validation, CLI `0` loses to a positive spec value | PUSHED_VERIFIED (PR 35 section) | — | | 8 | Combined regression, PR hand-off | NOT_STARTED | — | — | — | ## Chronological updates @@ -93,6 +93,8 @@ - 2026-09-22: W7 investigated while the parity job queued. `fwyc0573/frontier-htsim` is public and readable, its only branch `main` is exactly the gitlink Frontier already pins, and the candidate's `e564935d` returns HTTP 422 with no local object store on this host containing it — an unpublished commit, not an access problem. The fix is entirely companion-side. Confirmed all three payload defects by running the published runner rather than reading it: an explicit `tensor_bytes = 0` and a deleted field produce the identical `missing required fields: ['tensor_bytes']` exit 2; `-1` passes validation and reaches the simulator invocation; and `set_if_none_or_zero` replaces an explicit CLI `0` with a positive spec value. Reachable from Frontier because `_validate_data_size` accepts zero and `moe_operator_times.py:512` computes the EP all-to-all payload as `embedding_dim * 2 * routed_tokens`, which is zero for an empty local lane. Frontier cannot fix it alone: a zero-byte collective still costs the intra-server latency, so short-circuiting would change the backend's semantics. Blocked on the user's decision about the second repository; nothing was created or pushed there. +- 2026-09-22: W6 and W7 published. Branch pushed through `cad3afd`; draft PR 35 body now carries a W6 section (what was missing, the reachability and magnitude checks made before implementing, the repair and its two recorded departures, measurement ownership, the artifact-identity decision, CPU validation and its stated limits, and the native parity matrix) and a W7 section (unpublished gitlink, the three executed confirmations, Frontier-side reachability, and the companion-repository blocker). PR remains draft. Default-environment suite re-run at `cad3afd`: 84 failed / 3778 passed / 49 skipped / 11 errors, identical to the W4 baseline. + ## Step 3 scoping, as recorded before implementation Kept as written so the implementation can be read against the scope it started from. Step 3 is now complete; see the W3 entries above and `design.md`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index 49b5ee75..352c3af8 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -461,3 +461,4 @@ Full record: `test_report_2026-09-22_w6_fused_expert_arithmetic.md`. | Fidelity matrix | Not run, deliberately. The change is confined to a module the simulator cannot import (it requires Torch, absent from the simulator environment), and the matrix consumes checked-in CSVs rather than fresh profiling. The unchanged default-environment suite is the evidence. The repair changes what a future profiling run measures, not any simulation from existing data. | | Native GPU parity | **SUBMITTED, result pending.** `tests/integration/test_moe_fused_expert_numerical_parity.py`, 8 cases at `rtol=0, atol=0` against `fused_experts`: Qwen3-A3B-30B shapes from the checked-in model config at 4096 and 4097 tokens on EP ranks 0 and 1; a 257-token, 16-expert case at top-k 2 and 4 whose routing leaves two local experts empty; repeated invocation with different inputs; and the FP8 path as a structural check. Job `exp-0922-140423-075005`, creator `i-fengyicheng`, `steptron_ci` / `H800`, 1 GPU, image `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`, NFS source `100.96.128.195:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`. | | Artifact identity | **CLOSED as document-only** by user decision. The finding stands: `resolve_grouped_gemm_backend` labels both vLLM paths `vllm_fused`, and `profiling_patch_tag` holds three historical free-text values in `a800/qwen3-a3b-30b-moe/moe.csv` while nothing in the source writes it. No column was added. `docs/profiling/README.md` records the operator's scope, the size of the pre-repair gap, and that a row cannot be checked for completeness from its own metadata. | +| Suite re-run after the documentation and record commits | `pytest tests/unit -q --continue-on-collection-errors` under `frontier-py310` at `cad3afd`: 84 failed, 3778 passed, 49 skipped, 11 errors. Identical to the W4 baseline and to the earlier W6 measurement. | From 1b95187462ff8fb1d6975eeea31ba77510196fa6 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 14:45:58 +0800 Subject: [PATCH 26/80] fix(cc_backend): accept an empty collective through the collective-sim backend An expert-parallel lane that routes no token in a step asks for an empty transfer: moe_operator_times computes data_size_bytes = embedding_dim * 2 * routed_tokens and hands it to predict_all_to_all, and predict_reduce_scatter floor-divides by the device count and reaches zero the same way. _validate_data_size accepts zero, so the request reaches the collective-sim runner, which rejected it as a missing field and aborted the run. The repair is entirely in the submodule. htsim_runner.py read zero as "field not provided" in two places, and neither it nor the scenario schema rejected a negative payload. This moves the gitlink from b8518af to eb7bc4f (fwyc0573/frontier-htsim, branch fix/zero-payload-input-handling, companion draft PR #1). No Frontier source changes. tests/unit/test_collective_sim_zero_payload.py covers the Frontier call path on the canonical TP=4 x DP=2, EP=8 pod: an empty all-to-all and an empty reduce-scatter each keep the 7 x 0.5 us NVLink synchronization latency, a populated all-to-all still costs more, and a negative payload is still rejected by Frontier's own guard. Against gitlink b8518af, 3 of the 4 fail with "Error: missing required fields: ['tensor_bytes']". The module skips when the optional submodule is not initialized and built, which is the default. --- frontier/cc_backend/backends/collective-sim | 2 +- .../unit/test_collective_sim_zero_payload.py | 127 ++++++++++++++++++ 2 files changed, 128 insertions(+), 1 deletion(-) create mode 100644 tests/unit/test_collective_sim_zero_payload.py diff --git a/frontier/cc_backend/backends/collective-sim b/frontier/cc_backend/backends/collective-sim index b8518afc..eb7bc4fd 160000 --- a/frontier/cc_backend/backends/collective-sim +++ b/frontier/cc_backend/backends/collective-sim @@ -1 +1 @@ -Subproject commit b8518afcc310f0fe0e3ce52ba6b4f0bf57a3be04 +Subproject commit eb7bc4fda9744aa117063972eb0fbabbef91c453 diff --git a/tests/unit/test_collective_sim_zero_payload.py b/tests/unit/test_collective_sim_zero_payload.py new file mode 100644 index 00000000..1ce988f3 --- /dev/null +++ b/tests/unit/test_collective_sim_zero_payload.py @@ -0,0 +1,127 @@ +"""Hold the zero-payload boundary between Frontier and the collective-sim backend. + +`moe_operator_times` computes `data_size_bytes = embedding_dim * 2 * routed_tokens` +and hands the result to `predict_all_to_all`, so an expert-parallel lane that +routes no token in a step asks for an empty transfer. `predict_reduce_scatter` +floor-divides by the device count and reaches zero the same way. An empty +transfer is still a synchronization point, and `_validate_data_size` accepts it, +so the request reaches the collective-sim runner and must come back as a latency +rather than a backend failure. + +The runner-side and schema-side repairs live in the collective-sim submodule +(`fwyc0573/frontier-htsim`); this module covers the Frontier call path that +depends on them. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from frontier.cc_backend.cc_backend_config import CollectiveSimCCBackendConfig +from frontier.types import ClusterType + + +BACKEND_ROOT = ( + Path(__file__).resolve().parents[2] + / "frontier/cc_backend/backends/collective-sim" +) +if not (BACKEND_ROOT / "sim/datacenter/htsim_ndp").is_file(): + pytest.skip( + "The optional collective-sim backend must be initialized and built: " + "git submodule update --init frontier/cc_backend/backends/collective-sim " + "&& make -C frontier/cc_backend/backends/collective-sim/sim", + allow_module_level=True, + ) + +# The canonical mapping from AGENTS.md: one complete pod, TP=4 x DP=2 for +# attention and a single EP=8 collective for the routed experts. +GPUS_PER_SERVER = 8 +ATTN_TENSOR_PARALLEL_SIZE = 4 +ATTN_DATA_PARALLEL_SIZE = 2 +EXPERT_PARALLEL_SIZE = 8 + +# Eight NVLink-connected ranks exchange over seven hops, each costing +# nvlink_latency_us, and that term does not depend on the payload. +NVLINK_LATENCY_US = 0.5 +SYNCHRONIZATION_MS = (EXPERT_PARALLEL_SIZE - 1) * NVLINK_LATENCY_US / 1000.0 + + +@pytest.fixture +def backend(tmp_path): + """A single-server MoE pod priced with the analytic NVLink intra-server model.""" + from frontier.cc_backend.backends.collective_sim_cc_backend import ( + CollectiveSimCCBackend, + ) + + config = CollectiveSimCCBackendConfig( + cluster_servers=1, + cluster_gpus_per_server=GPUS_PER_SERVER, + parallel_tp=ATTN_TENSOR_PARALLEL_SIZE, + parallel_cp=1, + parallel_dp=ATTN_DATA_PARALLEL_SIZE, + parallel_ep=1, + runtime_num_replicas=1, + runtime_num_pipeline_stages=1, + runtime_attn_tensor_parallel_size=ATTN_TENSOR_PARALLEL_SIZE, + runtime_attn_dp=ATTN_DATA_PARALLEL_SIZE, + runtime_moe_tensor_parallel_size=1, + runtime_moe_expert_parallel_size=EXPERT_PARALLEL_SIZE, + intra_server_model="nvlink_analytic", + nvlink_latency_us=NVLINK_LATENCY_US, + runner_out_dir=str(tmp_path / "runner"), + ) + return CollectiveSimCCBackend( + config=config, + cluster_type=ClusterType.MONOLITHIC, + device_type="h100_dgx", + network_device="h100_dgx", + num_devices=GPUS_PER_SERVER, + ) + + +def test_an_empty_all_to_all_keeps_its_synchronization_latency(backend): + predicted_ms = backend.predict_all_to_all( + data_size_bytes=0, + num_devices=EXPERT_PARALLEL_SIZE, + comm_domain="EP", + ) + + assert predicted_ms == pytest.approx(SYNCHRONIZATION_MS) + + +def test_an_empty_reduce_scatter_keeps_its_synchronization_latency(backend): + # A payload smaller than the device count floor-divides to zero, which is how + # this collective reaches an empty transfer. + predicted_ms = backend.predict_reduce_scatter( + data_size_bytes=EXPERT_PARALLEL_SIZE - 1, + num_devices=EXPERT_PARALLEL_SIZE, + comm_domain="EP", + ) + + assert predicted_ms == pytest.approx(SYNCHRONIZATION_MS) + + +def test_a_populated_all_to_all_costs_more_than_an_empty_one(backend): + empty_ms = backend.predict_all_to_all( + data_size_bytes=0, + num_devices=EXPERT_PARALLEL_SIZE, + comm_domain="EP", + ) + populated_ms = backend.predict_all_to_all( + data_size_bytes=1 << 20, + num_devices=EXPERT_PARALLEL_SIZE, + comm_domain="EP", + ) + + assert populated_ms > empty_ms + + +def test_a_negative_payload_is_still_rejected(backend): + with pytest.raises(ValueError, match="data_size_bytes must be non-negative"): + backend.predict_all_to_all( + data_size_bytes=-1, + num_devices=EXPERT_PARALLEL_SIZE, + comm_domain="EP", + ) From beded3c4f47145c95a6aaeaed80ec4c84ddb6964 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 14:52:53 +0800 Subject: [PATCH 27/80] test(governance): scan only the sources Frontier owns under frontier/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three governance tests walk (repo_root / "frontier").rglob("*.py"). Once the optional collective-sim submodule is initialized, that walk reaches 38 vendored files, one of which does not parse under Python 3: tests/unit/test_model_architecture_registry.py::test_raw_model_profile_resolution_callsites_are_allowlisted IndentationError: expected an indented block after 'with' statement on line 24 frontier/cc_backend/backends/collective-sim/sim/EXAMPLES/in_and_out/process_data.py The other two tolerated it — one catches SyntaxError, the other only searches text — but both silently measured vendored files as if Frontier owned them, so the scan result depended on whether a developer had run git submodule update. tests/frontier_sources.iter_frontier_sources() yields the 413 files Frontier owns and skips the vendored subtree; all three scans go through it. With the submodule initialized and built, tests/unit returns to its 84-failure baseline with 3782 passing, the four extra being tests/unit/test_collective_sim_zero_payload.py. --- tests/frontier_sources.py | 26 +++++++++++++++++++ .../unit/test_model_architecture_registry.py | 3 ++- tests/unit/test_module_split_boundaries.py | 3 ++- tests/unit/test_spec_decode_mtp_registry.py | 4 ++- 4 files changed, 33 insertions(+), 3 deletions(-) create mode 100644 tests/frontier_sources.py diff --git a/tests/frontier_sources.py b/tests/frontier_sources.py new file mode 100644 index 00000000..3af51cd8 --- /dev/null +++ b/tests/frontier_sources.py @@ -0,0 +1,26 @@ +"""Enumerate the Python sources Frontier owns, for repository-governance tests. + +`frontier/cc_backend/backends/collective-sim` is an optional vendored submodule. +A default checkout leaves it empty and a developer populates it only to run +`--cc_backend_config_type collective_sim`, so any scan that walks `frontier/` +sees a different file set on the two sides, and one vendored file does not parse +under Python 3 at all. These scans govern Frontier's own code, so they skip it. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Iterator + + +VENDORED_SUBTREES = ("frontier/cc_backend/backends/collective-sim",) + + +def iter_frontier_sources(repo_root: Path) -> Iterator[Path]: + """Yield every Python file under `frontier/` that Frontier itself owns.""" + vendored = tuple((repo_root / subtree).resolve() for subtree in VENDORED_SUBTREES) + for path in (repo_root / "frontier").rglob("*.py"): + resolved = path.resolve() + if any(resolved.is_relative_to(root) for root in vendored): + continue + yield path diff --git a/tests/unit/test_model_architecture_registry.py b/tests/unit/test_model_architecture_registry.py index d018e86e..db6c04b5 100644 --- a/tests/unit/test_model_architecture_registry.py +++ b/tests/unit/test_model_architecture_registry.py @@ -29,6 +29,7 @@ from frontier.profiling.common.model_config import ModelConfig as ProfilingModelConfig from frontier.profiling.linear_op.profiling_plan import build_profiling_plan from frontier.types import ActivationType, ClusterType, NormType +from tests.frontier_sources import iter_frontier_sources class _LogRecordCollector(logging.Handler): @@ -572,7 +573,7 @@ def test_raw_model_profile_resolution_callsites_are_allowlisted() -> None: } observed_call_counts: dict[tuple[str, str, str], int] = {} - for path in (repo_root / "frontier").rglob("*.py"): + for path in iter_frontier_sources(repo_root): tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) function_aliases: set[str] = set() module_aliases: set[str] = set() diff --git a/tests/unit/test_module_split_boundaries.py b/tests/unit/test_module_split_boundaries.py index f548d3a6..14a271f3 100644 --- a/tests/unit/test_module_split_boundaries.py +++ b/tests/unit/test_module_split_boundaries.py @@ -26,6 +26,7 @@ import frontier.config from frontier.config.config import SimulationConfig from frontier.config.flat_dataclass import create_flat_dataclass +from tests.frontier_sources import iter_frontier_sources # --- annotation resolution in each defining module -------------------------- @@ -103,7 +104,7 @@ def _names_imported_from_frontier_config(repo_root: Path) -> dict[str, set[str]] import ast wanted: dict[str, set[str]] = {"frontier.config": set(), "frontier.config.config": set()} - for path in sorted((repo_root / "frontier").rglob("*.py")): + for path in sorted(iter_frontier_sources(repo_root)): try: tree = ast.parse(path.read_text(encoding="utf-8")) except SyntaxError: diff --git a/tests/unit/test_spec_decode_mtp_registry.py b/tests/unit/test_spec_decode_mtp_registry.py index 2762bf96..a50ae808 100644 --- a/tests/unit/test_spec_decode_mtp_registry.py +++ b/tests/unit/test_spec_decode_mtp_registry.py @@ -5,6 +5,8 @@ import pytest +from tests.frontier_sources import iter_frontier_sources + def test_target_embedded_mtp_methods_are_registry_source_for_runtime_helpers() -> None: import frontier.spec_decode.mtp_registry as registry @@ -166,7 +168,7 @@ def test_static_mtp_contract_helper_is_not_used_by_production_paths() -> None: } production_hits = [] - for path in (repo_root / "frontier").rglob("*.py"): + for path in iter_frontier_sources(repo_root): if path in allowed_paths: continue source = path.read_text(encoding="utf-8") From ff2ed0db159d3cd36e661c5709da0e34e6b0f560 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 14:56:08 +0800 Subject: [PATCH 28/80] docs(review): record the W6 native parity result and the W7 delivery W6 closes: exp-0922-145047-660565 on an H800 under codesign returned 8 of 8 at rtol=0, atol=0 against vLLM 0.10.2's own fused_experts. The report also records the three earlier attempts and their causes, since two of them are image facts the next native run needs: vllm/vllm-openai:v0.10.2 ships neither pytest nor nvidia-smi and keeps its injected driver at /usr/local/nvidia/lib64 off the loader path, the httpproxy recipe returns 407 for pip so worker packages come from the internal mirror, and logs_rjob returns empty for these jobs while get_rjob_infos plus logs_replica returns the container lines. W7 closes: the user authorized the companion-repository option, the backend fix is published as fwyc0573/frontier-htsim eb7bc4f with draft PR 1, Frontier moved its gitlink, and both negative controls hold. A new report covers the defect, the repair, both controls, the clean-checkout validation, and the governance scan the bump exposed. --- .../progress.md | 13 +- .../requirements.md | 2 + .../review.md | 14 +- ...t_2026-09-22_w6_fused_expert_arithmetic.md | 52 ++++- ...26-09-22_w7_collective_sim_zero_payload.md | 201 ++++++++++++++++++ .../validation.md | 21 +- 6 files changed, 293 insertions(+), 10 deletions(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w7_collective_sim_zero_payload.md diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index db2ad92b..214a098d 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -13,6 +13,8 @@ | 2026-09-22 | Checkpoint E first half: W5 closed as NOT PORTED by user decision after the premise check showed the collision unreachable on main; the drafted implementation was reverted before commit and archived as a patch. | | 2026-09-22 | W6 artifact identity decided as document-only and written into the profiling guide; native parity test added and submitted to an H800 worker as `exp-0922-140423-075005`. | | 2026-09-22 | W7 investigated while the GPU job queued: candidate gitlink unpublished, three payload defects confirmed by execution, companion-repository decision pending. | +| 2026-09-22 | W6 native parity PASS on H800: `exp-0922-145047-660565`, 8 of 8 at `rtol=0, atol=0`. Step 6 closed. | +| 2026-09-22 | W7 authorized and delivered: companion fix published as `eb7bc4f` with draft PR 1, Frontier gitlink moved, Frontier-side test added, and the governance scans narrowed to Frontier-owned sources. Step 7 closed. | ## Status @@ -21,9 +23,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | -| Current step | Step 6 source, CPU validation and artifact-identity documentation complete; native GPU parity job `exp-0922-140423-075005` submitted, result pending | +| Current step | Steps 6 and 7 closed. Step 8 (combined regression and final PR review) is next. | | Publication | PUSHED_VERIFIED. Draft PR 35 body carries the W6 and W7 sections; still draft. | -| Next action | Record the native parity result. W7 needs one user decision: authorize a companion-repository branch in `fwyc0573/frontier-htsim`, or record W7 as `EXCLUDED`. | +| Next action | Step 8: combined regression across the whole branch and a final read of PR 35. One follow-up carried forward: Frontier's gitlink points at a commit on the companion branch, so it must be re-pointed at `main` once `fwyc0573/frontier-htsim` PR 1 merges. | ## Step status @@ -35,12 +37,15 @@ | 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 4 | Opt-in vLLM DP placement | PASS | unit PASS (61 new, 3778 total, failure set identical to the parent); integration PASS (3 cases in the real event loop, including a placement that diverges from round-robin); five deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | PUSHED_VERIFIED (records + PR 35 section) | REVIEWED (user chose to keep the single global field) | -| 6 | Legacy fused-MoE profiling | PARTIAL: arithmetic, measurement ownership and artifact-identity documentation done; native parity job submitted, result pending | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778). Native: 8 cases in `tests/integration/test_moe_fused_expert_numerical_parity.py`, submitted as `exp-0922-140423-075005` | PUSHED_VERIFIED (source, tests, docs, records, PR 35 section) | — | -| 7 | Optional zero-payload backend | INVESTIGATED, BLOCKED on a companion-repository decision. Candidate gitlink `e564935d` is unpublished, not inaccessible; the three payload defects are confirmed by execution against the published backend; the fix is companion-side only | Published runner exercised directly: explicit zero and missing field give the same `exit=2`, `-1` passes validation, CLI `0` loses to a positive spec value | PUSHED_VERIFIED (PR 35 section) | — | +| 6 | Legacy fused-MoE profiling | PASS | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778). Native PASS: 8 of 8 in `tests/integration/test_moe_fused_expert_numerical_parity.py` at `rtol=0, atol=0` on H800, job `exp-0922-145047-660565` under `codesign` | PUSHED_VERIFIED (source, tests, docs, records, PR 35 section) | — | +| 7 | Optional zero-payload backend | PASS. Companion fix published as `fwyc0573/frontier-htsim` `eb7bc4f` with draft PR 1; Frontier gitlink moved from `b8518af`; no Frontier source change | Companion 9 passed, negative control 6 of 9 fail on pristine sources. Frontier 4 passed, negative control 3 of 4 fail at the old gitlink. Clean checkout resolves `eb7bc4f` from the published remote, builds, and passes. Suite back to the 84-failure baseline with 3782 passing after narrowing three governance scans to Frontier-owned sources | PUSHED_VERIFIED | — | | 8 | Combined regression, PR hand-off | NOT_STARTED | — | — | — | ## Chronological updates +- 2026-09-22: W6 native parity closed. `exp-0922-145047-660565` on `gpu-h800-0110` under `codesign`, image `vllm/vllm-openai:v0.10.2` through the company proxy: **8 passed in 13.70 s**, every case at `rtol=0, atol=0` against vLLM's own `fused_experts`. Three earlier attempts failed for environment reasons, each now recorded: the `steptron_ci` pool had no capacity; the image ships neither `pytest` nor `nvidia-smi` and the `httpproxy` recipe returns 407 for pip; its injected driver sits at `/usr/local/nvidia/lib64` off the loader path; and `frontier/profiling/common/utils.py` imports `pandas`, which the image also lacks. Worker packages install from `http://mirrors.i.basemind.com/pypi/simple/`. `logs_rjob` returns empty for these jobs; `get_rjob_infos` then `logs_replica` returns the container lines. +- 2026-09-22: W7 delivered under the user's authorization. Companion repository `fwyc0573/frontier-htsim` gained branch `fix/zero-payload-input-handling` at `eb7bc4f` and draft PR 1: `tensor_bytes` now merges through a helper that treats only `None` and `""` as unset, the required-field check became a table carrying per field whether zero is legal, and a negative payload is rejected in the runner and in `Scenario.validate()`. Its `.gitignore` was narrowed from `tests/` to `tests/*` plus the published test, since a bare directory rule stops git descending and makes any negation unreachable. Frontier moved its gitlink and gained `tests/unit/test_collective_sim_zero_payload.py`; no Frontier source file changed. Both negative controls hold: 6 of 9 companion tests and 3 of 4 Frontier tests fail against the pre-fix sources, each reporting `missing required fields: ['tensor_bytes']`. +- 2026-09-22: The gitlink bump exposed a latent defect in the repository governance tests. Three of them walk `frontier/` recursively; with the optional submodule initialized that walk reaches 38 vendored files, one of which does not parse under Python 3, so `test_raw_model_profile_resolution_callsites_are_allowlisted` failed and the other two silently measured vendored code. `tests/frontier_sources.iter_frontier_sources()` now yields the 413 Frontier-owned files and all three scans use it. This would have hit any developer who followed the AGENTS.md instruction to initialize the submodule, with or without this branch. - 2026-09-22: Checkpoint C. Merged the corrected parent (`bb582a4`, `7dd5982`, `6ef0a3c`) into this branch by merge rather than rebase, so the published review anchors stay valid and no commit is discarded. New base/head relationship: PR #35 head `fix/issue26-correctness-pr`, base `refactor/oversized-module-split` at `6ef0a3c`. - 2026-09-22: R35-01 addressed. The placement tests now state where each request lands: three topologies with the full rotation written out by hand past its wraparound, driven through the public `schedule()` rather than `_schedule_batch_mode`, each run for both roles that reach batch-mode placement. A guard keeps that role list honest, since `TRANS` also falls through the dispatch but is declared and never constructed anywhere in `frontier/`. The source-text check is kept as governance only and says so. Negative control against the pre-fix method: 12 of 23 fail, controls pass, and the old code turns out to produce the expected sequence exactly for a single burst -- so the fix restored an existing rotation for incremental arrival rather than introducing a policy. - 2026-09-22: W2 re-measured with harness and source at one revision: baseline `6ef0a3c` against candidate `ceac2b4`, both clean detached checkouts, one harness at `ceac2b4`, no filter, clean cache, 71 executed and 426 cache files each. 71 of 71 compared, 68 identical, and the mismatch set is exactly the three predicted cases, with no provenance findings and no cache differences. Lane occupancy reproduces the first measurement number for number. Full record in `validation.md`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md index 6ad55631..c1b3b9c7 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md @@ -37,3 +37,5 @@ The draft specification is landed verbatim in `plan.md` together with an Amendme | 2026-09-22 | `[Original Request]` "我在rethink添加 frontier.moe_routing_runtime的必要性。我的concern是，该部分的align是否对模拟准确度意义重大？如果对fidelity的影响不到0.5%,我认为完全无需引入如此复杂的变量和setting,这会使得可用性和可读性变得很困难" -> after the explanation of what the module decides and the measured 3-8% per-layer cost of a wrong path versus the 0% change of the unset override: "我认为采用 当前默认的 两个 cluster 都只能读同一个全局 moe_routing_distribution_type，永远解析出同一个 path ；进行回退" | **W5 not ported.** Keep the existing contract: one global `moe_routing_distribution_type`, one derived routing path per run, no override field, no registry routing axis. The drafted implementation was reverted before any commit and archived as `w5_reverted_moe_routing_runtime_path.patch`. Both earlier W5 decisions above are void. | | 2026-09-22 | W6 artifact identity: should the profiling metadata gain a column that separates an incomplete legacy `moe_grouped_gemm` measurement from a complete one? | **"不改 metadata，只记录限制"** — do not change the profiling metadata; record the limitation only. Written into `docs/profiling/README.md` under the MoE producer. No code change, no new column, no admission gate. | | 2026-09-22 | W6 native validation: how should the GPU parity check be run? | **"请从 dockerhub 中找到 v0.10.2 的官方镜像（如果没有，fallback 到 >=0.10, <0.11），然后参考 hand-book 中对 docker 的使用在 gpu worker 上使用该镜像。如果你需要使用原来的 benchmark ... 中的测试 suits 和插桩，你需要 mount 该 repo 到 gpu worker。如果需要进行对比验证，则 follow skill：/home/brainpp/.claude/skills/frontier-calibration"** — `vllm/vllm-openai:v0.10.2` exists on Docker Hub, so no fallback was needed. It is pulled through the company docker.io proxy as `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`. The instrumented benchmark repository is not mounted: the parity check compares tensors from vLLM's own `fused_experts` inside one process and needs no serving instrumentation. The `frontier-calibration` skill is not invoked for the same reason — its workflow is E2E simulator-versus-served-vLLM calibration, not a kernel-level tensor comparison. | +| 2026-09-22 | W7: authorize a companion-repository branch in `fwyc0573/frontier-htsim`, or record W7 as `EXCLUDED`. Asked with the option spelled out as "授权我在那个仓库建分支、提交测试和源码、推送并开 companion draft PR，然后再动 Frontier 的 gitlink。按计划这是 backend 先发布、Frontier 后跟进的顺序。" | **"1.授权"** — companion-repository work authorized. Delivered in that order: branch `fix/zero-payload-input-handling` pushed to `fwyc0573/frontier-htsim` at `eb7bc4f`, companion **draft** PR 1 opened, then Frontier's gitlink moved. PR 35 stays draft; nothing was merged. | +| 2026-09-22 | `[Original Request]` "对于当前 task 的 gpu worker 集群应该使用 codesign 而不是 step_main" | All GPU submissions for this task use `charged_group="codesign"`. Applied from `exp-0922-142415-796404` onward; the first attempt `exp-0922-140423-075005` had used `steptron_ci`, found no capacity after ~20 minutes Pending, and was stopped. The successful native parity run `exp-0922-145047-660565` is on `codesign`. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index 3228b352..48c4e92a 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -12,6 +12,7 @@ | 2026-09-22 | Self-review of that record: corrected the lockstep mechanism and counter semantics under D1, the line references under R34-01, the R34-03 remedy (the baseline label is itself an assembled partial run), and added the omissions listed under "Found on re-review". | | 2026-09-22 | W6 arithmetic delivered with CPU validation; two deviations from the recorded adaptations justified; artifact identity raised as an open decision; native GPU validation NOT_RUN. | | 2026-09-22 | W6 artifact identity decided as document-only; native parity test added and submitted to an H800 worker. | +| 2026-09-22 | W6 native parity PASS: 8 of 8 at `rtol=0, atol=0` on H800. W7 authorized and delivered; its blocker row closed and the delivery recorded. | | 2026-09-22 | W7 facts re-verified: the candidate gitlink is unpublished, the three payload defects are confirmed by execution against the published backend, and the fix needs companion-repository authorization. | | 2026-09-22 | W5 closed without source changes. Corrected the Mechanism A premise: the routing distribution has no per-role override on main, so both W5 mechanisms are unreachable from any released configuration. The user chose to keep the single global field; the drafted implementation was reverted and archived as a patch. | @@ -126,7 +127,18 @@ Facts re-verified 2026-09-22, superseding the specification-time record in `plan | Reachability from Frontier | Real. `base_cc_backend._validate_data_size` rejects only negative sizes, so Frontier passes zero through. `moe_operator_times.py:512` computes `data_size_bytes = embedding_dim * 2 * routed_tokens` and hands it to `predict_all_to_all`; an EP lane with no routed tokens in a step makes that zero. `predict_reduce_scatter` additionally floor-divides by the device count. A MoE EP run under `--cc_backend_config_type collective_sim` therefore aborts on a legitimate empty collective. | | Why Frontier cannot fix it alone | A zero-byte collective is not a zero-cost collective. The donor test asserts the intra-server latency term survives at payload 0 (`7 x 0.5 us`, `network_ms == 0`), which is also the plan's requirement. Short-circuiting to `0.0` in Frontier would change the backend's synchronization semantics rather than accept the input. | | Cost of the fix | Three small edits in the companion repository: drop `tensor_bytes` from the zero-means-missing list while keeping it required, add a `>= 0` check, and make the CLI precedence distinguish "unset" from "explicitly zero". No Frontier source change; Frontier moves its gitlink and gains the donor's CPU test. | -| Blocker | Publishing to a second repository. The plan requires the user's decision on companion-repository scope before creating or pushing anything there, and `EXCLUDED` is the alternative. Neither option was taken unilaterally. | +| Blocker | **CLEARED 2026-09-22.** The user authorized the companion-repository option ("1.授权"). Publication order as proposed: the backend commit first, Frontier's gitlink after. | + +### W7 delivery + +| Item | Outcome | +| --- | --- | +| Companion commit | `fwyc0573/frontier-htsim` branch `fix/zero-payload-input-handling`, `eb7bc4f`, companion **draft** PR 1. Three source edits plus a narrowed `.gitignore`; no change to flow generation, topology modelling or latency arithmetic. | +| Design choice worth flagging | The required-field check became a table of `(field, zero_is_valid)` rather than a special case for `tensor_bytes`, per the AGENTS.md rule that a growing category gets a table. A first draft added a fourth merge helper; it was collapsed into the existing `set_if_none_or_empty`, whose semantics are already exactly right for this field, rather than left as a near-duplicate. | +| Frontier change | Gitlink `b8518af` -> `eb7bc4f` and one new test module. No Frontier source file changed, which matches the candidate branch: its own `collective_sim_cc_backend.py` is byte-identical to this branch's. | +| Frontier test scope | The donor test re-drove the runner CLI and the submodule predictor. That is now the companion repository's own coverage, so the Frontier module tests the Frontier boundary instead: `CollectiveSimCCBackend.predict_all_to_all` and `predict_reduce_scatter` on the canonical `TP=4 x DP=2, EP=8` pod. | +| Defect found while validating | The gitlink bump made three repository-governance scans read 38 vendored files as Frontier's own, and one of them stopped parsing. Repaired with `tests/frontier_sources.iter_frontier_sources()`. Pre-existing and reachable by anyone who initializes the optional submodule; not caused by the backend fix. | +| Carried forward | Frontier's gitlink points at a commit on an unmerged companion branch. `git submodule update --init`, the documented command, resolves it; `git submodule update --remote` would follow `.gitmodules`' `branch = main` and drop the fix. Re-point the gitlink at `main` once companion PR 1 merges. | ## Decision checkpoints for the user diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md index 3fc0ca58..4b77b716 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md @@ -9,6 +9,7 @@ Date: 2026-09-22. Branch `fix/issue26-correctness-pr`, worktree | --- | --- | | 2026-09-22 | Created: reachability check, magnitude estimate, source repair, CPU validation. Native GPU validation NOT_RUN. | | 2026-09-22 | Artifact identity decided as document-only. Native parity test added and submitted to an H800 worker as `exp-0922-140423-075005`; result pending. | +| 2026-09-22 | Native parity PASS on H800 under `codesign`: `exp-0922-145047-660565`, 8 of 8 at `rtol=0, atol=0`. Three earlier attempts and their causes recorded in section 8. | ## 1. Is the defect reachable @@ -227,14 +228,57 @@ and both report `functional_fused_experts`. The pinned profiling range is | Field | Value | | --- | --- | | Submission host | `kun-workspace-vgen2` (local), StepMind Python `RJobBackend`, `STEPMIND_BACKEND=rjob` | -| Job name | `exp-0922-140423-075005` | +| Job name | `exp-0922-145047-660565` (the run that produced the result; see the attempt log below) | | Creator | `i-fengyicheng` | -| Charged group / tag | `steptron_ci` / `H800` | +| Charged group / tag | `codesign` / `H800`, per the user's instruction on 2026-09-22 | +| Node | `gpu-h800-0110.host.platform.shaipower.com`, `NVIDIA H800` | | Shape | 1 GPU, 8 CPU, 64000Mi | | Image | `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`, the official Docker Hub `vllm/vllm-openai:v0.10.2` through the company docker.io proxy | | NFS mount | `100.96.128.195:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr` | -| Command | `python3 -m pytest -v -rA -p no:cacheprovider --no-header tests/integration/test_moe_fused_expert_numerical_parity.py`, after printing `nvidia-smi`, the Torch/vLLM/`fused_moe` source paths and `VLLM_API_VERSION` | -| Status | **PENDING** at the time of writing. Result and per-case outcomes are appended below when the job reaches a terminal state. | +| Worker environment | Python 3.12.11, torch 2.8.0+cu128 (`cuda_available=True`), vLLM 0.10.2, `VLLM_API_VERSION=0.10.x`, `FP8_AVAILABLE=True`, pytest 9.1.1 and pandas 3.0.6 installed on the worker from the internal PyPI mirror | +| Command | `python3 -m pytest -q -rA -p no:cacheprovider --no-header tests/integration/test_moe_fused_expert_numerical_parity.py` | +| Status | **PASS. 8 passed in 13.70 s.** | + +### Result + +| Case | Outcome | +| --- | --- | +| `test_production_shaped_expert_output_matches_vllm[0-4096]` | PASSED | +| `test_production_shaped_expert_output_matches_vllm[0-4097]` | PASSED | +| `test_production_shaped_expert_output_matches_vllm[1-4096]` | PASSED | +| `test_production_shaped_expert_output_matches_vllm[1-4097]` | PASSED | +| `test_uneven_expert_occupancy_matches_vllm[2]` | PASSED | +| `test_uneven_expert_occupancy_matches_vllm[4]` | PASSED | +| `test_repeated_invocations_do_not_reuse_a_stale_result` | PASSED | +| `test_fp8_path_runs_on_the_gated_activation` | PASSED | + +All eight compare at `rtol=0, atol=0`, so the repaired profiling path reproduces +vLLM 0.10.2's `fused_experts` output bit for bit on the production Qwen-A3B-30B +shapes at 4096 and 4097 tokens for two EP ranks, on the uneven-occupancy +boundary case at two top-k values, across repeated invocations, and on the FP8 +path. + +### Attempts + +| Job | Charged group | Outcome | Cause | +| --- | --- | --- | --- | +| `exp-0922-140423-075005` | `steptron_ci` | stopped after ~20 min Pending | no free capacity in that pool; the user then directed this task to `codesign` | +| `exp-0922-142415-796404` | `codesign` | Failed | `pytest` absent from the image, and the `deploy.i.shaipower.com/httpproxy` recipe returns `407 Proxy Authentication Required` for pip; separately `libcuda.so.1` was not on the loader path, so vLLM fell back to `UnspecifiedPlatform` | +| `exp-0922-144336-056798` | `codesign` | Failed | pytest installed from the internal PyPI mirror and `LD_LIBRARY_PATH` repaired from `/usr/local/nvidia/lib64`, so torch saw the H800; 2 of 8 passed and 6 failed on `ModuleNotFoundError: No module named 'pandas'`, which `frontier/profiling/common/utils.py` imports at module scope | +| `exp-0922-145047-660565` | `codesign` | **Succeeded** | `pandas` added to the worker install | + +Two image facts worth recording for the next native run: the +`vllm/vllm-openai:v0.10.2` image ships neither `pytest` nor `nvidia-smi`, and +its injected driver lives at `/usr/local/nvidia/lib64` without being on the +loader path, so a worker command must prepend that directory to +`LD_LIBRARY_PATH` before importing Torch. Worker packages install from +`http://mirrors.i.basemind.com/pypi/simple/` with +`--extra-index-url http://pypi.i.basemind.com/brain/dev/+simple`; the +`httpproxy` recipe does not authenticate for pip. + +The platform retains only a tail of the worker log, and `logs_rjob` returns an +empty string for these jobs. Use `get_rjob_infos(job)` to get the replica name +and `logs_replica(replica)`, whose JSON rows carry the container lines. The instrumented benchmark repository was not mounted. This test compares tensors from vLLM's own `fused_experts` inside one process, so it needs no diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w7_collective_sim_zero_payload.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w7_collective_sim_zero_payload.md new file mode 100644 index 00000000..cee816ab --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w7_collective_sim_zero_payload.md @@ -0,0 +1,201 @@ +# W7 Test Report — Zero-Payload Collective Through the collective-sim Backend + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-22 | First issue: companion-repository fix, Frontier gitlink bump, both negative controls, clean-checkout validation, and the governance-scan repair the bump exposed. | + +## 1. Scope + +W7 is the zero-payload defect recorded in `plan.md` A7 and re-verified in +`review.md`. The user authorized the companion-repository option on +2026-09-22 ("1.授权"), so the work is: repair the backend in +`fwyc0573/frontier-htsim`, publish it, move Frontier's gitlink onto the +published commit, and add a Frontier-side regression test. + +No Frontier source file changes. The Frontier diff is the gitlink, one new test +module, and one repair to three governance scans that the gitlink bump exposed. + +## 2. What was broken, and that it is reachable + +`moe_operator_times.py:512` computes `data_size_bytes = embedding_dim * 2 * +routed_tokens` and hands it to `predict_all_to_all`. An expert-parallel lane +that routes no token in a step makes that zero. `predict_reduce_scatter` +floor-divides by the device count and reaches zero the same way for any payload +smaller than the device count. `base_cc_backend._validate_data_size` rejects +only negative sizes, so zero reaches the collective-sim runner. + +The published runner then rejected it. Three defects, each confirmed by +execution against the published `main` (`b8518af`) before any edit: + +| # | Defect | Evidence | +| --- | --- | --- | +| 1 | An explicit zero is read as a missing field | `htsim_runner.py` tested `getattr(args, k) in (None, "", 0)` over a required-field list containing `tensor_bytes`. Running with `tensor_bytes = 0` and with the field deleted produced the identical `exit=2, Error: missing required fields: ['tensor_bytes']`. | +| 2 | A negative payload is accepted | `--tensor-bytes` is a bare `type=int` with no lower bound and no schema check, so `-1` passed validation and reached flow generation. | +| 3 | An explicit CLI zero loses to a positive spec value | `set_if_none_or_zero` reads `0` as missing on both sides, so `--tensor-bytes 0` against a spec of 32768 yielded 32768. | + +A zero-byte collective is not a zero-cost collective, so Frontier cannot +short-circuit to `0.0` without changing the backend's synchronization +semantics. The repair belongs in the runner. + +## 3. The companion-repository change + +Repository `fwyc0573/frontier-htsim`, branch `fix/zero-payload-input-handling`, +commit `eb7bc4f`, companion draft PR +. + +| File | Change | +| --- | --- | +| `htsim_runner.py` | `tensor_bytes` merges through `set_if_none_or_empty`, which treats only `None` and `""` as unset, instead of `set_if_none_or_zero`. The argparse default for `--tensor-bytes` is already `None`, so scenario-file merging is unaffected. | +| `htsim_runner.py` | The required-field check became a table carrying, per field, whether zero is a legal value. Zero stays "missing" for `collective_type`, `domain_dims`, `topology`, `nodes`, `gpus_per_server` and `tp`. | +| `htsim_runner.py` | A negative payload returns `2` with `Error: tensor_bytes must be >= 0.` before flow generation. | +| `python/collective_sim_core/schema.py` | `Scenario.validate()` raises `collective.tensor_bytes must be >= 0`, so a serialized scenario cannot smuggle one past the schema. | +| `.gitignore` | `tests/` narrowed to `tests/*` plus an explicit entry for the published test, so git descends into the directory. Private working material under `tests/` stays ignored. | + +No change to flow generation, topology modelling, or any latency arithmetic. + +## 4. Companion-side validation + +Command, from the companion checkout: + +```bash +/data/ycfeng/envs/frontier-py310/bin/python -m pytest tests/test_zero_payload_input.py -q -p no:cacheprovider --no-header +``` + +| Run | Sources | Expected | Actual | Verdict | +| --- | --- | --- | --- | --- | +| Repaired | `eb7bc4f` | all pass | **9 passed** | PASS | +| Negative control | `htsim_runner.py` and `schema.py` restored to `HEAD` | the zero-payload cases fail | **6 failed, 3 passed**, the failures reporting `Error: missing required fields: ['tensor_bytes']` | PASS | + +The three that pass on pristine sources assert behavior that already worked: a +genuinely missing payload is an error, the scenario file still supplies an unset +payload, and the 32768-byte control arm. The two end-to-end cases skip when +`sim/datacenter/htsim_ndp` is absent. + +The negative control was taken by copying the two edited files aside, +`git checkout HEAD --` on them, running, then restoring — deliberately not +`git stash`, whose stack is shared with other worktrees on this host. + +## 5. Frontier-side validation + +New module `tests/unit/test_collective_sim_zero_payload.py`, four tests on the +canonical `TP=4 x DP=2, EP=8` single-server pod from AGENTS.md, priced with +`intra_server_model=nvlink_analytic` and `nvlink_latency_us=0.5`. + +Eight NVLink-connected ranks exchange over seven hops, so the payload-independent +synchronization term is `7 x 0.5 us = 0.0035 ms`. + +| Test | Expectation | Measured | +| --- | --- | --- | +| `test_an_empty_all_to_all_keeps_its_synchronization_latency` | `0.0035 ms` | `0.0035` | +| `test_an_empty_reduce_scatter_keeps_its_synchronization_latency` | `0.0035 ms` for a 7-byte payload that floor-divides to zero | `0.0035` | +| `test_a_populated_all_to_all_costs_more_than_an_empty_one` | strictly greater | `0.0060486222 > 0.0035` | +| `test_a_negative_payload_is_still_rejected` | `ValueError` from Frontier's own guard | raised | + +Command: + +```bash +PYTHONPATH=$PWD /data/ycfeng/envs/frontier-py310/bin/python -m pytest \ + tests/unit/test_collective_sim_zero_payload.py -q -rA -p no:cacheprovider --no-header +``` + +| Run | Gitlink | Expected | Actual | Verdict | +| --- | --- | --- | --- | --- | +| Repaired | `eb7bc4f` | all pass | **4 passed** | PASS | +| Negative control | `b8518af`, the pre-fix commit both this branch and main pinned | the three zero-payload cases fail | **3 failed, 1 passed**, each failure reporting `Error: missing required fields: ['tensor_bytes']` | PASS | + +The module skips at import when the optional submodule is not initialized and +built, which is the default state and the state AGENTS.md documents. + +## 6. Clean-checkout validation + +The point of this check is that the gitlink resolves from the published remote, +not from anything local to this host. + +```bash +git clone --branch fix/issue26-correctness-pr /data/ycfeng/tmp/w7_clean_checkout +cd /data/ycfeng/tmp/w7_clean_checkout +pytest tests/unit/test_collective_sim_zero_payload.py # submodule absent +git submodule update --init frontier/cc_backend/backends/collective-sim +make -C frontier/cc_backend/backends/collective-sim/sim -j"$(nproc)" +pytest tests/unit/test_collective_sim_zero_payload.py # submodule present +``` + +| Stage | Expected | Actual | Verdict | +| --- | --- | --- | --- | +| Submodule absent | the module skips | **1 skipped** | PASS | +| `git submodule update --init` | checks out `eb7bc4f` from `https://github.com/fwyc0573/frontier-htsim.git` | `Submodule path '...': checked out 'eb7bc4fda9744aa117063972eb0fbabbef91c453'` | PASS | +| `make -j` | builds | `build_exit=0` | PASS | +| Submodule present | all four pass | **4 passed** | PASS | + +`.gitmodules` still records `branch = main`, which is correct for the state after +the companion PR merges. `git submodule update --init`, the command AGENTS.md +documents, uses the recorded gitlink and lands on `eb7bc4f`; +`git submodule update --remote` would follow `main` instead and drop the fix +until the companion PR merges. + +## 7. Regression, and a governance scan the bump exposed + +```bash +PYTHONPATH=$PWD WANDB_DISABLED=true VIDUR_DISABLE_WANDB=1 \ + /data/ycfeng/envs/frontier-py310/bin/python -m pytest tests/unit -q \ + -p no:cacheprovider --no-header --continue-on-collection-errors +``` + +First run after the bump: **85 failed, 3781 passed, 49 skipped, 11 errors** +against the standing baseline of 84 failed / 3778 passed / 49 skipped / +11 errors. Diffing the two `FAILED` lists named exactly one new failure: + +``` +FAILED tests/unit/test_model_architecture_registry.py::test_raw_model_profile_resolution_callsites_are_allowlisted +``` + +Cause, confirmed by running it alone: the scan walks +`(repo_root / "frontier").rglob("*.py")` and `ast.parse`s every hit. Once the +optional submodule is initialized, that walk reaches 38 vendored files, one of +which — `sim/EXAMPLES/in_and_out/process_data.py` — raises +`IndentationError: expected an indented block after 'with' statement on line 24`. + +This is a latent defect in the governance test, not in the backend: any +developer who follows the AGENTS.md instruction to initialize the submodule +would hit it. Three test modules walk that tree: + +| Module | Behavior before | Behavior after the bump | +| --- | --- | --- | +| `test_model_architecture_registry.py:575` | `ast.parse` with no guard | fails on the unparsable vendored file | +| `test_module_split_boundaries.py:106` | `ast.parse` inside `except SyntaxError: continue` | tolerated, but silently measures vendored files | +| `test_spec_decode_mtp_registry.py:169` | substring search, no parsing | tolerated, but silently measures vendored files | + +Repair: `tests/frontier_sources.py` exposes `iter_frontier_sources(repo_root)`, +which yields the Python files under `frontier/` excluding the vendored subtree, +and all three scans call it. 413 of the 451 files under `frontier/` are +Frontier's own; the 38 excluded are the submodule's. + +| Check | Expected | Actual | Verdict | +| --- | --- | --- | --- | +| The three governance modules | all pass | **88 passed** | PASS | +| Full `tests/unit` after the repair | back to the 84-failure baseline, plus the four new tests | **84 failed, 3782 passed, 49 skipped, 11 errors**; diffing the `FAILED` lists against the baseline gives an empty set in both directions | PASS | + +## 8. Publication + +| Artifact | State | +| --- | --- | +| `fwyc0573/frontier-htsim` branch `fix/zero-payload-input-handling` | pushed, `eb7bc4f` | +| Companion PR | open, **draft** | +| Frontier commit `1b95187` | gitlink `b8518af` -> `eb7bc4f`, plus the new test module | +| PR 35 | remains **draft** | + +## 9. Limits of this validation + +- The synchronization figure `7 x 0.5 us` is the analytic NVLink intra-server + model's own arithmetic, not a measurement against hardware. These tests hold + the boundary and the monotonicity; they do not validate the model. +- With the default `intra_server_model=legacy_fabric` and + `collective_exclude_intra_server=True`, a single-server pod prices both an + empty and a 1 MiB all-to-all at `0.0`, because all traffic is intra-server and + excluded. That is the configuration's own semantics and is unrelated to the + payload; the tests therefore use the analytic model, where the payload term is + observable. +- The companion PR is draft and unmerged, so the gitlink points at a branch + commit. It must be re-pointed at `main` once that PR merges. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index 352c3af8..c42a9da2 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -11,6 +11,8 @@ | 2026-09-22 | Step 3 recorded: the shared monolithic forward, its direct-construction runtime evidence, four deliberate-defect controls, and a 71-of-71 identical fidelity matrix. | | 2026-09-22 | Step 4 recorded: the opt-in vLLM-style DP placement policy, its real-runtime wiring evidence including a divergence from round-robin, five deliberate-defect controls, and a 71-of-71 identical fidelity matrix. | | 2026-09-22 | Step 6 native parity test recorded and submitted as `exp-0922-140423-075005`; artifact identity closed as document-only. | +| 2026-09-22 | Step 6 native GPU parity recorded: 8 of 8 at `rtol=0, atol=0` on H800 under `codesign`. | +| 2026-09-22 | Step 7 recorded: the companion backend fix, both negative controls, clean-checkout validation, and the governance-scan repair the gitlink bump exposed. | ## Environment @@ -459,6 +461,23 @@ Full record: `test_report_2026-09-22_w6_fused_expert_arithmetic.md`. | Regression, Torch environment | Against a detached worktree at `HEAD` (`bbbfcaa`), same four existing files: 1 failed / 136 passed at HEAD, 1 failed / 143 passed with the repair. Same failure identity on both sides — `test_functional_vllm_kernel_exposes_mxfp4_switch_without_importing_vllm` needs Torch without vLLM, which no local environment provides. | | Regression, default environment | `pytest tests/unit --continue-on-collection-errors` under `frontier-py310`: 84 failed, 3778 passed, 49 skipped, 11 errors — the W4 baseline of 84 failures and 3778 passing, unchanged. Collection errors 10 -> 11 because the new file imports Torch at module level, as the seven existing profiling test files in that list already do; verified by re-collecting with it ignored. | | Fidelity matrix | Not run, deliberately. The change is confined to a module the simulator cannot import (it requires Torch, absent from the simulator environment), and the matrix consumes checked-in CSVs rather than fresh profiling. The unchanged default-environment suite is the evidence. The repair changes what a future profiling run measures, not any simulation from existing data. | -| Native GPU parity | **SUBMITTED, result pending.** `tests/integration/test_moe_fused_expert_numerical_parity.py`, 8 cases at `rtol=0, atol=0` against `fused_experts`: Qwen3-A3B-30B shapes from the checked-in model config at 4096 and 4097 tokens on EP ranks 0 and 1; a 257-token, 16-expert case at top-k 2 and 4 whose routing leaves two local experts empty; repeated invocation with different inputs; and the FP8 path as a structural check. Job `exp-0922-140423-075005`, creator `i-fengyicheng`, `steptron_ci` / `H800`, 1 GPU, image `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`, NFS source `100.96.128.195:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`. | +| Native GPU parity | **PASS. 8 passed in 13.70 s** on `NVIDIA H800` (`gpu-h800-0110`), job `exp-0922-145047-660565`, creator `i-fengyicheng`, charged group `codesign` per the user's 2026-09-22 instruction, 1 GPU, image `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`, NFS source `100.96.128.195:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`. Worker: Python 3.12.11, torch 2.8.0+cu128, vLLM 0.10.2, `VLLM_API_VERSION=0.10.x`, `FP8_AVAILABLE=True`. All 8 cases of `tests/integration/test_moe_fused_expert_numerical_parity.py` compare at `rtol=0, atol=0` against `fused_experts`: Qwen3-A3B-30B shapes from the checked-in model config at 4096 and 4097 tokens on EP ranks 0 and 1; a 257-token, 16-expert case at top-k 2 and 4 whose routing leaves two local experts empty; repeated invocation with different inputs; and the FP8 path as a structural check. Three earlier attempts and their causes are in `test_report_2026-09-22_w6_fused_expert_arithmetic.md` section 8. | | Artifact identity | **CLOSED as document-only** by user decision. The finding stands: `resolve_grouped_gemm_backend` labels both vLLM paths `vllm_fused`, and `profiling_patch_tag` holds three historical free-text values in `a800/qwen3-a3b-30b-moe/moe.csv` while nothing in the source writes it. No column was added. `docs/profiling/README.md` records the operator's scope, the size of the pre-repair gap, and that a row cannot be checked for completeness from its own metadata. | | Suite re-run after the documentation and record commits | `pytest tests/unit -q --continue-on-collection-errors` under `frontier-py310` at `cad3afd`: 84 failed, 3778 passed, 49 skipped, 11 errors. Identical to the W4 baseline and to the earlier W6 measurement. | + +## Step 7 — Zero-payload collective through the collective-sim backend + +Full record: `test_report_2026-09-22_w7_collective_sim_zero_payload.md`. + +| Item | Result | +| --- | --- | +| Reachability | Confirmed by source. `moe_operator_times.py:512` computes `data_size_bytes = embedding_dim * 2 * routed_tokens` and hands it to `predict_all_to_all`, so an EP lane routing no token in a step asks for an empty transfer; `predict_reduce_scatter` floor-divides by the device count and reaches zero for any payload below it. `base_cc_backend._validate_data_size` rejects only negative sizes, so zero reaches the runner. | +| Defects, confirmed by execution against published `main` (`b8518af`) | (1) an explicit zero and a deleted field produce the identical `exit=2, Error: missing required fields: ['tensor_bytes']`; (2) `-1` passes validation and reaches flow generation; (3) `--tensor-bytes 0` against a spec of 32768 yields 32768. | +| Companion fix | `fwyc0573/frontier-htsim` branch `fix/zero-payload-input-handling`, commit `eb7bc4f`, draft PR 1. `tensor_bytes` merges through `set_if_none_or_empty`; the required-field check became a table carrying per field whether zero is legal; a negative payload is rejected in the runner and in `Scenario.validate()`. No change to flow generation or latency arithmetic. | +| Companion tests | `tests/test_zero_payload_input.py`, 9 tests: **9 passed** against `eb7bc4f`. Negative control with `htsim_runner.py` and `schema.py` restored to `HEAD`: **6 failed, 3 passed**, the failures reporting the missing-field error. | +| Frontier tests | `tests/unit/test_collective_sim_zero_payload.py`, 4 tests on the canonical `TP=4 x DP=2, EP=8` pod with `intra_server_model=nvlink_analytic`: **4 passed**. An empty all-to-all and an empty reduce-scatter each price at `7 x 0.5 us = 0.0035 ms`; a 1 MiB all-to-all prices at `0.0060486222 ms`; a negative payload raises from Frontier's own guard. Negative control at gitlink `b8518af`: **3 failed, 1 passed**. | +| Clean-checkout validation | Fresh clone of the branch: the module skips with the submodule absent (**1 skipped**); `git submodule update --init` checks out `eb7bc4f` from `https://github.com/fwyc0573/frontier-htsim.git`; `make -j` returns `build_exit=0`; the four tests then pass. The gitlink therefore resolves from the published remote, not from anything local to this host. | +| Governance scan exposed by the bump | The first post-bump suite gave 85 failures. Diffing the `FAILED` lists named one new failure, `test_model_architecture_registry.py::test_raw_model_profile_resolution_callsites_are_allowlisted`: it `ast.parse`s every file under `frontier/`, and the initialized submodule adds 38 vendored files, one of which raises `IndentationError`. Two other scans walked the same tree and tolerated it while silently measuring vendored files. Repair: `tests/frontier_sources.iter_frontier_sources()` yields the 413 Frontier-owned files and skips the vendored subtree; all three scans use it. The three modules then give **88 passed**. | +| Regression, default environment | `pytest tests/unit -q --continue-on-collection-errors` under `frontier-py310` with the submodule initialized and built: **84 failed, 3782 passed, 49 skipped, 11 errors**. Diffing the `FAILED` lists against the 84-failure baseline gives an empty set in both directions; the four extra passes are the new module. | +| Fidelity matrix | Not run. No Frontier source file changed, and the backend fix is unreachable without `--cc_backend_config_type collective_sim`, which no matrix case selects. The unchanged failure set and the pre-fix/post-fix negative controls are the evidence. | +| Limits | The `7 x 0.5 us` figure is the analytic NVLink model's own arithmetic, not a hardware measurement. Under the default `legacy_fabric` model with `collective_exclude_intra_server=True`, a single-server pod prices both an empty and a 1 MiB all-to-all at `0.0`, because all traffic is intra-server and excluded; that is the configuration's semantics, unrelated to the payload. The companion PR is draft, so the gitlink points at a branch commit and must be re-pointed at `main` once it merges. | From d881357a7f6d184bcbb5ee6486b3f71e101ca8ab Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 15:05:22 +0800 Subject: [PATCH 29/80] docs: complete the cluster-scheduler list and record the opt-in DP policy The implementation list named three of the six registered cluster schedulers. This adds the two sticky variants that were already registered and the new vllm_load_balancing policy, with the scope its constructor actually enforces: one co-location replica, vllm_v1, one pipeline stage, and either a MoE model or attn_dp=1. It also states what round-robin distributes over, since this PR changed that rotation to persist across scheduling calls. The eight pre-existing documentation-contract failures are unchanged by this edit; their FAILED list is identical before and after. --- AGENTS.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 435ea351..f9eaaa52 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,6 +11,7 @@ | 2026-09-05 | Clarified Replica-local collective backend materialization. | | 2026-09-05 | Added Python module size and plain ML-system naming guidance for refactors. | | 2026-09-06 | Added cleanup-first and split-analysis requirements for critical modules above 2,000 lines. | +| 2026-09-22 | Completed the cluster-scheduler implementation list and recorded the opt-in vLLM DP placement policy and its supported scope. | - Current public branch supports `co-location`, sequential PDD / `pd-disaggregation`, and sequential PD-AF / `pd-af-disaggregation`. - The public co-location, PDD, and PD-AF examples explicitly select `--cc_backend_config_type analytical` for one-click smoke runs using the built-in analytical model. @@ -611,9 +612,12 @@ The scheduling logic is split across four distinct layers to mirror real-world s 2. **Cluster Scheduler** (`ClusterSchedulerRegistry`): - **Role**: Manages workload distribution within a specific `ClusterType` (e.g., selecting which Replica gets a request). - **Implementations**: - - `RoundRobinClusterScheduler`: Distributes requests cyclically. + - `RoundRobinClusterScheduler`: Distributes requests cyclically over replicas and, inside each replica, over attention-DP lanes. The ordinal persists across scheduling calls, so an identical ordered request stream lands identically however it is divided between calls. - `LORClusterScheduler`: Least Outstanding Requests (load balancing). - `RandomClusterScheduler`: Random assignment. + - `StickyRoundRobinClusterScheduler`: Round-robin over targets, pinned per session so a session's later requests return to the same target. + - `StickyLORClusterScheduler`: Least Outstanding Requests with the same per-session pinning. + - `VllmLoadBalancingClusterScheduler`: Models vLLM V1's internal DP selection, choosing the lane with the lowest `waiting * 4 + running` score from a load snapshot the frontend observes with a delay. Opt-in and deliberately narrow: one `co-location` replica, the `vllm_v1` replica scheduler, one pipeline stage, and either a MoE model or `attn_dp=1`. The constructor rejects everything else. No placement or timing equivalence with a real vLLM deployment is claimed. 3. **Replica Scheduler** (`ReplicaSchedulerRegistry`): - **Role**: Operates at the level of a single `Replica` (GPU node/instance). From 8730509e2a1fdcc35a4bd36713b606ab6906e816 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 15:18:28 +0800 Subject: [PATCH 30/80] docs(records): record the Step 8 combined regression and the deferred findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 8 §14.1 ran the selected suites together on the integrated branch instead of package by package, and exercised the first-load predictor-cache path deliberately rather than inheriting a warm cache. - Unit: 84 failed, 3782 passed, 49 skipped, 11 errors. The FAILED set is identical to the recorded origin/main baseline in both directions. - Integration: 15 passed, 22 skipped, 5 errors. The added skip is the W6 GPU parity module; the 5 errors are the absent pinned PD-AF Reference checkout and are identical on the base. - All 16 release-supported architecture examples pass across co-location, sequential PDD, and sequential PD-AF in offline and online modes. - Four PP=2 runs pass, putting the changed cluster-scheduling, stage-dispatch, and metrics code on the multi-stage path that the PP1-only DP placement policy cannot reach by itself. - The trained-predictor path run cold and then warm against an empty scratch cache produces byte-identical request metrics, so the persisted-cache path reproduces the freshly trained one exactly. The repository cache was neither moved nor deleted. Step 8 §14.2 records the final diff review, its method, and the cleanup done in that pass. future.md records two deferred items: re-pointing the collective-sim gitlink at main once the companion PR merges, and the pre-existing tests/debug/ pointer defect that AGENTS.md, a scheduler docstring, and 10 baseline unit failures share. That defect predates this branch and its repair is a decision about the published test surface, so it is reported rather than fixed here. --- .../future.md | 64 ++++++ .../progress.md | 49 ++++- .../review.md | 43 ++++ ...eport_2026-09-22_w8_combined_regression.md | 199 ++++++++++++++++++ .../validation.md | 19 ++ 5 files changed, 371 insertions(+), 3 deletions(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/future.md create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w8_combined_regression.md diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/future.md b/task_memory/task_2026-09-21_issue26_correctness_pr/future.md new file mode 100644 index 00000000..edb12afb --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/future.md @@ -0,0 +1,64 @@ +# Issue 26 Correctness PR — Deferred work + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-22 | Created. Records the `tests/debug/` pointer defect found during Step 8 and the companion-PR gitlink follow-up. | + +## 1. `tests/debug/` is referenced but absent from the published repository + +**Found:** Step 8 §14.1, while running the two PP2 entry points `AGENTS.md` +names under "Tests". + +**Evidence** + +``` +bash: tests/debug/e2e-level/monolith_mode/scripts/test_dense_tp2_pp2_dummy.sh: No such file or directory +bash: tests/debug/e2e-level/monolith_mode/scripts/test_moe_tp2_ep2_pp2_dummy.sh: No such file or directory +``` + +`tests/debug/` exists neither on `fix/issue26-correctness-pr` nor on +`origin/main` (`1f694f7`). The published `tests/` tree is `analysis/`, +`comparison/`, `e2e/`, `fixtures/`, `integration/`, `performance/`, `unit/`. +The same section also names `comm_backend_tests/`, which does not exist either; +the CC-backend tests live in `tests/unit/test_cc_backend_*.py`. + +Three places still point into the removed tree: + +| Location | Reference | +| --- | --- | +| `AGENTS.md` §Tests | `comm_backend_tests/` and `debug/` directory entries, plus two `bash tests/debug/e2e-level/monolith_mode/scripts/*.sh` commands under "Start with:" | +| `tests/unit/test_colocation_release_review_contracts.py:13,14,89,90,139,140,182,193,203` | Resolves `tests/debug/e2e-level/monolith_mode/scripts/` and two scripts under it | +| `frontier/scheduler/replica_scheduler/vllm_v1_engine_replica_scheduler.py:16` | Docstring pointer to `tests/debug/flow-level/admission_control_dev_guide_en.md` | + +**Impact:** 10 of the 84 baseline unit failures are +`test_colocation_release_review_contracts.py` failing on the missing paths, and +the documented "start here" commands do not run. + +**Why it is deferred:** this is one pre-existing defect class inherited from the +release scrub that removed `tests/debug/`, and it is unrelated to Issue 26. +A correct repair is a decision about the published test surface — restore the +tree, or retarget the contract test and the documentation at entry points that +ship — not a documentation tweak. Making the doc read correctly while the +contract test still fails on the same paths would hide the real gap. + +**Suggested resolution, for whoever owns the release scrub** + +1. Decide whether the co-location review contracts should assert on shipped + scripts (`examples/architecture/co-location/**`) or on a restored + `tests/debug/` tree. +2. Retarget `test_colocation_release_review_contracts.py` accordingly. +3. Update `AGENTS.md` §Tests and the scheduler docstring to match. + +Equivalent PP2 coverage for this PR was obtained through the example scripts; +see `test_report_2026-09-22_w8_combined_regression.md` §5. + +## 2. Re-point the collective-sim gitlink at `main` + +`frontier/cc_backend/backends/collective-sim` currently points at `eb7bc4f` on +the companion branch `fix/zero-payload-input-handling` of +`fwyc0573/frontier-htsim`, because companion PR 1 is still draft. Once that PR +merges, bump the gitlink to the resulting commit on `main` and re-run +`tests/unit/test_collective_sim_zero_payload.py` plus the clean-checkout +validation described in `test_report_2026-09-22_w7_collective_sim_zero_payload.md`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 214a098d..909634e5 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -15,6 +15,7 @@ | 2026-09-22 | W7 investigated while the GPU job queued: candidate gitlink unpublished, three payload defects confirmed by execution, companion-repository decision pending. | | 2026-09-22 | W6 native parity PASS on H800: `exp-0922-145047-660565`, 8 of 8 at `rtol=0, atol=0`. Step 6 closed. | | 2026-09-22 | W7 authorized and delivered: companion fix published as `eb7bc4f` with draft PR 1, Frontier gitlink moved, Frontier-side test added, and the governance scans narrowed to Frontier-owned sources. Step 7 closed. | +| 2026-09-22 | Step 8 §14.1 run: unit and integration suites at the baseline failure set, 16 architecture examples, four PP=2 cases, and a cold-then-warm predictor-cache pair. `tests/debug/` pointer defect found, deferred to `future.md`. | ## Status @@ -23,9 +24,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | -| Current step | Steps 6 and 7 closed. Step 8 (combined regression and final PR review) is next. | +| Current step | Step 8 in progress. §14.1 combined regression complete and recorded; §14.2 diff review complete; §14.3 PR body update is the remaining work. | | Publication | PUSHED_VERIFIED. Draft PR 35 body carries the W6 and W7 sections; still draft. | -| Next action | Step 8: combined regression across the whole branch and a final read of PR 35. One follow-up carried forward: Frontier's gitlink points at a commit on the companion branch, so it must be re-pointed at `main` once `fwyc0573/frontier-htsim` PR 1 merges. | +| Next action | Step 8 §14.3: add the Step 8 results and the record links to PR 35's body, keep it draft, and report technical acceptance separately from GitHub's draft status. Two follow-ups carried forward, both in `future.md`: re-point the collective-sim gitlink at `main` once companion PR 1 merges, and repair the `tests/debug/` pointers that 10 baseline unit failures share. | ## Step status @@ -39,7 +40,7 @@ | 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | PUSHED_VERIFIED (records + PR 35 section) | REVIEWED (user chose to keep the single global field) | | 6 | Legacy fused-MoE profiling | PASS | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778). Native PASS: 8 of 8 in `tests/integration/test_moe_fused_expert_numerical_parity.py` at `rtol=0, atol=0` on H800, job `exp-0922-145047-660565` under `codesign` | PUSHED_VERIFIED (source, tests, docs, records, PR 35 section) | — | | 7 | Optional zero-payload backend | PASS. Companion fix published as `fwyc0573/frontier-htsim` `eb7bc4f` with draft PR 1; Frontier gitlink moved from `b8518af`; no Frontier source change | Companion 9 passed, negative control 6 of 9 fail on pristine sources. Frontier 4 passed, negative control 3 of 4 fail at the old gitlink. Clean checkout resolves `eb7bc4f` from the published remote, builds, and passes. Suite back to the 84-failure baseline with 3782 passing after narrowing three governance scans to Frontier-owned sources | PUSHED_VERIFIED | — | -| 8 | Combined regression, PR hand-off | NOT_STARTED | — | — | — | +| 8 | Combined regression, PR hand-off | IN_PROGRESS. §14.1 and §14.2 complete; §14.3 remaining | unit 84 failed / 3782 passed with a `FAILED` set identical to the `origin/main` baseline; integration 15 passed / 22 skipped / 5 errors, the errors environmental and identical on the base; 16 of 16 architecture examples pass; 4 of 4 `PP=2` cases pass; cold and warm predictor-cache runs byte-identical | — | — | ## Chronological updates @@ -115,3 +116,45 @@ One refinement over the audit's framing, from reading the code: `ForwardSyncStat Surface: about 3,500 lines across `replica_stage_schedule_event.py`, `sync_entry.py`, `prefill_collective.py`, `decode_collective.py`, `ep_wave_schedule.py`, `ep_wave_inputs.py` and `base_cluster_scheduler.py`, with 11 call sites of the sync-kind and sync-path selection. Blocked hunk carried from the audit: the candidate's decode final-metrics change calls `_create_corrected_execution_time_for_metrics`, which main deleted, so it needs rewriting against main's current execution-time ownership rather than porting. Resolved by exclusion: that hunk is I8, kept out of scope under Checkpoint D's "current-main metrics ownership" and recorded in the `design.md` scope table. + +### 2026-09-22 — Step 8 §14.1, combined regression + +Ran the selected suites together on `d881357` rather than package by package. +Full record: `test_report_2026-09-22_w8_combined_regression.md`; summary rows in +`validation.md`. + +- Re-fetched `origin/main` before the run. Still `1f694f7` and an ancestor of + `HEAD`, so no integration merge was needed and none was made. +- Unit: **84 failed, 3782 passed, 49 skipped, 11 errors**. The `FAILED` set is + identical to the recorded baseline in both directions. +- Integration: **15 passed, 22 skipped, 5 errors**. The added skip is the W6 GPU + parity module; the 5 errors are the absent pinned PD-AF Reference checkout and + are identical on the base. +- All 16 release-supported architecture examples pass, covering co-location, + sequential PDD, and sequential PD-AF in both offline and online modes. +- Four `PP=2` runs pass: co-location dense offline and online, co-location MoE + with `Attn_TP=4, MoE_TP=2, MoE_EP=2`, and sequential PDD dense with both roles + at `PP=2`. These put the changed cluster-scheduling, stage-dispatch, and + metrics code on the multi-stage path, which the PP1-only DP placement policy + cannot reach by itself. +- Ran the trained-predictor path cold and then warm against an empty scratch + cache directory via `--metrics_config_cache_dir`, leaving the repository + `cache/` untouched. Cold took 26.9 s and wrote 63 artifacts; warm took 2.1 s + and wrote none; their `request_metrics.csv` outputs are byte-identical. This + is the first-load path measured deliberately rather than inherited. + +**Finding, deferred.** `AGENTS.md` §Tests names `comm_backend_tests/`, `debug/`, +and two `bash tests/debug/e2e-level/monolith_mode/scripts/*.sh` commands. +`tests/debug/` exists neither here nor on `origin/main`. The same missing tree +causes 10 of the 84 baseline unit failures, in +`test_colocation_release_review_contracts.py`, and a docstring at +`vllm_v1_engine_replica_scheduler.py:16` still points into it. That is one +pre-existing defect class from the release scrub, unrelated to Issue 26, and +repairing it means deciding what the published test surface should assert +against. Recorded in `future.md` and left untouched; the PP2 coverage was taken +through the example scripts instead. + +**Hygiene.** `git status --porcelain` is empty afterwards. Example runs were +pointed at scratch metrics directories, and one leftover `outputs/examples/` +tree from an earlier iteration was removed after confirming it held 0 tracked +files; the 110 tracked files under `outputs/` are all still present. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index 48c4e92a..e0d182c1 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -15,6 +15,7 @@ | 2026-09-22 | W6 native parity PASS: 8 of 8 at `rtol=0, atol=0` on H800. W7 authorized and delivered; its blocker row closed and the delivery recorded. | | 2026-09-22 | W7 facts re-verified: the candidate gitlink is unpublished, the three payload defects are confirmed by execution against the published backend, and the fix needs companion-repository authorization. | | 2026-09-22 | W5 closed without source changes. Corrected the Mechanism A premise: the routing distribution has no per-role override on main, so both W5 mechanisms are unreachable from any released configuration. The user chose to keep the single global field; the drafted implementation was reverted and archived as a patch. | +| 2026-09-22 | Step 8 §14.2 recorded: final self-review of the whole branch diff at `d881357`, with the method stated and one deferred pre-existing defect. | ## Pinned source snapshot @@ -360,3 +361,45 @@ made incremental arrival reach the placement that batch arrival already had. That is the strongest available statement that W2 is a bug fix rather than a behavior change, and it is why the matrix is expected to move only the cases that enter the scheduler many times. + +## Step 8 §14.2 — Final review of the complete diff against the PR base + +| Field | Value | +| --- | --- | +| Reviewed revision | `d881357` | +| Base compared against | `refactor/oversized-module-split` @ `6ef0a3c` (32 commits) | +| Method | **Self-review.** One reviewer, the same agent that wrote the change, reading `git diff 6ef0a3c..d881357` restricted to `frontier/ tests/ docs/ examples/` — 1925 lines — line by line with surrounding context opened where a hunk did not stand alone. This is not independent review; no second party inspected the diff. | +| Task records excluded from the line-by-line pass | `task_memory/` and `AGENTS.md`, reviewed separately as documentation. | + +### Answers to the §14.2 questions + +| Question | Finding | +| --- | --- | +| Is every change linked to a demonstrated defect, an essential regression test, or a directly related simplification? | Yes. Each of W2, W3, W4, W6, W7 carries a negative control that fails against the unrepaired source, recorded in `validation.md`. W5 produced no source change. The only additions without a paired defect are `tests/frontier_sources.py`, which exists because the gitlink bump exposed a real scan defect, and the `AGENTS.md` scheduler list, which documents `VllmLoadBalancingClusterScheduler` added by W4. | +| Are state ownership and initialization explicit? Are repeated fallback checks or parallel state representations still present without a reason? | W3's waiting rooms are owned by one structure keyed by step id, initialized at creation and deleted at release; the tests assert no dangling room survives a cohort. No fallback chain was added. | +| Do source batches keep their own shape and progress? Are terminal events and ownership transitions unique? | Yes; this is what the W3 mixed-source tests and the four deliberate-defect controls check, including a control that deliberately merges two sources' progress and one that emits a duplicate terminal event. | +| Are config, layer, routing-runtime, precision, and measurement identities preserved through both fresh and cached paths? | Yes. The §14.1 cold/warm pair is direct evidence: the freshly trained predictors and the persisted ones produce byte-identical `request_metrics.csv`. | +| Is any operation omitted, counted twice, or relabeled without compatible metadata? | The one relabelling risk is W6's `resolve_grouped_gemm_backend`, which labels both vLLM paths `vllm_fused`. The user decided not to change the metadata; the limitation is written into `docs/profiling/README.md` instead. | +| Did the patch preserve current-main model/backend support and demand-driven reporting? | Yes. The unit `FAILED` set is identical to the baseline, all 16 architecture examples pass, and the stage-reporting path is unchanged. | +| Are tests checking production behavior rather than copying the implementation or replacing the behavior under test with a stub? | The W6 CPU tests replace the native calls with plain-Torch references and say so in the file; their authority is the native GPU parity run, which uses the real kernels. Every other new test drives the production object. | +| Are any workstation paths, credentials, datasets, weights, generated traces, caches, or unreachable submodule references staged? | No. The branch diff over `frontier/ tests/ docs/ examples/` was scanned for `/data/ycfeng`, `/home/brainpp`, `BRAINPP_`, `ACCESS_KEY`, `SECRET`, `password`, `token=` with no hit. The gitlink resolves from the published remote, proved by the clean-checkout run in the W7 report. | +| Can a reader understand the names and functions without the historical calibration conversation? | The names follow the surrounding ML-system vocabulary. `VllmLoadBalancingClusterScheduler`'s constructor rejects every configuration outside its narrow support, and `AGENTS.md` now states that no placement or timing equivalence with a real vLLM deployment is claimed. | + +### Cleanup performed in this pass + +| Item | Action | +| --- | --- | +| `set_if_unset` in the companion runner | Removed. It was a near-copy of `set_if_none_or_empty`, and the `--tensor-bytes` argparse default of `None` makes the two behave identically. | +| `pipeline_time` in `prefill_collective.py` | Verified the removal is safe: the value now comes from `final_timing.pipeline_time` at line 232. | +| `outputs/examples/` generated trees | Removed after confirming 0 tracked files there; the 110 tracked files under `outputs/` are untouched. | + +No speculative abstraction was added in the cleanup pass. The affected tests were +re-run afterwards, with results in `validation.md`. + +### Pre-existing defect found, not repaired + +`AGENTS.md` §Tests points at `tests/debug/` and `comm_backend_tests/`, neither of +which exists here or on `origin/main`; the same missing tree causes 10 of the 84 +baseline unit failures. Reproduced on the base, reported as a baseline failure, +and recorded in `future.md` rather than repaired, because the correct fix is a +decision about the published test surface and is unrelated to Issue 26. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w8_combined_regression.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w8_combined_regression.md new file mode 100644 index 00000000..22f903a0 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w8_combined_regression.md @@ -0,0 +1,199 @@ +# Step 8 — Combined regression on the integrated branch + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-22 | Initial record of the Step 8 combined regression (plan.md §14.1). | + +## 1. Scope + +This report covers `plan.md` §14.1: running the selected suites together on the +integrated branch rather than package by package, exercising the cold +(first-load) predictor-cache path deliberately instead of inheriting a warm +cache, and confirming that the supported co-location, sequential PDD, and +sequential PD-AF paths still run. + +- Branch: `fix/issue26-correctness-pr` +- Revision under test: `d881357` +- PR base: `refactor/oversized-module-split` @ `6ef0a3c` (32 commits ahead) +- Python: `/data/ycfeng/envs/frontier-py310/bin/python` (3.10, no `torch`) +- Environment: `PYTHONPATH=`, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1` + +`origin/main` was re-fetched before the run and was still `1f694f7`, which is an +ancestor of `HEAD`. No integration merge was needed and none was performed. + +## 2. Unit suite + +```bash +python -m pytest tests/unit -q --continue-on-collection-errors +``` + +`--continue-on-collection-errors` matches the method used for the recorded +baseline: 11 modules import `torch` or `matplotlib`, which the simulator +environment does not carry, so collection of those modules fails on both sides. + +Result on this branch: + +``` +84 failed, 3782 passed, 49 skipped, 11 errors in 88.59s +``` + +The gate is the FAILED **set**, not the counts. This branch's list was compared +against the recorded `origin/main` @ `1f694f7` baseline list (`base_failed.txt`, +84 entries): + +```bash +diff <(sort base_failed.txt) <(sort now_failed.txt) # empty +``` + +Empty in both directions: no test that passes on the base fails here, and no +baseline failure was silently repaired. The baseline's passed/skipped counts +were not separately recorded, so this report does not compare them; the 3782 +passed here includes this branch's own new tests. + +## 3. Integration suite + +```bash +python -m pytest tests/integration -q --continue-on-collection-errors +``` + +| Metric | Baseline | This branch | +| --- | --- | --- | +| passed | 15 | 15 | +| skipped | 21 | 22 | +| errors | 5 | 5 | + +The one added skip is `tests/integration/test_moe_fused_expert_numerical_parity.py`, +this branch's W6 module, which skips without a GPU and a matching vLLM build. Its +passing result on real hardware is recorded in +`test_report_2026-09-22_w6_fused_expert_arithmetic.md`. + +The 5 errors are all `tests/integration/test_pdaf_reference_lifecycle_observer.py` +and all report the same cause: + +``` +FileNotFoundError: [Errno 2] No such file or directory: +PosixPath('/data/ycfeng/stepfun-performance-optimization/Frontier/worktrees/ref-afd-readonly') +``` + +That pinned read-only PD-AF Reference checkout is not present on this host. The +errors are identical on the base and are environmental, not code defects. + +## 4. Architecture examples + +All 16 release-supported architecture example scripts were run end to end with +task-owned metrics directories: + +| Family | Scripts | Result | +| --- | --- | --- | +| co-location offline | `dense_model_basic`, `moe_model_basic`, `thinking_mode_basic`, `moe_spec_dec`, `moe_prefix_caching` | 5 passed | +| co-location online | `dense_model_basic_online`, `moe_model_basic_online` | 2 passed | +| PDD offline | `dense_model_basic`, `moe_model_basic` | 2 passed | +| PDD online | `dense_model_basic_online`, `moe_model_basic_online` | 2 passed | +| PD-AF offline | `dense_model_basic`, `moe_model_basic`, `moe_model_ep`, `moe_cuda_graph` | 4 passed | +| PD-AF online | `moe_cuda_graph_online` | 1 passed | + +**16 passed, 0 failed.** Every run wrote `request_metrics.csv` and +`system_metrics.json` under its run id. + +## 5. Pipeline-parallel and dense/MoE coverage + +§14.1 asks for pipeline and heterogeneous/dense-layer cases where the changed +shared code applies, even though the new DP placement strategy itself stays +PP1-only. The changed cluster-scheduling, stage-dispatch, and metrics code runs +on every pipeline stage, so `PP=2` exercises it on the multi-stage path. + +Driver: `w8_pp_and_cache.sh` (scratchpad), logs under `/data/ycfeng/tmp/w8_pp_cache/`. + +| Case | Configuration | Result | +| --- | --- | --- | +| `pp2_dense_tp2` | co-location offline dense, `TP=2, PP=2` | PASS (2.6 s) | +| `pp2_moe_tp2ep2` | co-location offline MoE, `Attn_TP=4, MoE_TP=2, MoE_EP=2, PP=2` | PASS (7.1 s) | +| `pp2_pdd_dense` | sequential PDD offline dense, `PREFILL_PP=2, DECODE_PP=2` | PASS (2.9 s) | +| `pp2_dense_online` | co-location online dense, `TP=2, PP=2` | PASS (3.5 s) | + +Each script echoes its resolved topology, and the logs confirm the intended +values (`Parallelism: TP=2, PP=2`; +`Parallelism: Attn_TP=4, MoE_TP=2, MoE_EP=2, PP=2`; +`Prefill parallelism: ... PP=2` / `Decode parallelism: ... PP=2`). + +Dense-layer coverage comes from two directions: the dense examples above, and +the MoE examples, whose shared-expert work uses the separate +`dense_mlp_hidden_dim` width through the ordinary linear-op path while routed +work uses `routed_mlp_hidden_dim`. Frontier has no "first `k` dense layers then +MoE" model field, so there is no third heterogeneous shape to exercise. + +## 6. Trained-predictor cache: cold then warm + +The two checked-in CSV smokes (`examples/profiling/smoke_simulator_*_csv.sh`) +disable dummy mode, so they run the trained-predictor path. Against the +repository `cache/`, which is already populated, they only ever measure the +cache-hit path. To exercise first-load behavior without moving or deleting the +repository cache, the same simulation was run twice against an empty scratch +cache directory via `--metrics_config_cache_dir`. + +| Run | Cache entries before | Wall time | Cache entries after | +| --- | --- | --- | --- | +| `csv_dense_cold` | 0 | 26.9 s | 63 | +| `csv_dense_warm` | 63 | 2.1 s | 63 | + +The cold run trained and persisted 63 predictor artifacts; the warm run loaded +them back, wrote nothing new, and finished in 8% of the time. Their +`request_metrics.csv` outputs are byte-identical: + +```bash +diff csv_dense_cold/request_metrics.csv csv_dense_warm/request_metrics.csv # empty +``` + +so the persisted-cache path reproduces the freshly trained path exactly +(`request_e2e_time = 16.77818517187422 ms`, `ttft = 8.430025150867172 ms` in +both). + +`csv_moe_repo_cache` then ran the MoE smoke against the repository cache and +passed (2.5 s). The repository `cache/` was neither moved nor deleted at any +point; it holds 0 tracked files and is a generated artifact. + +## 7. Working-tree hygiene + +After the run, `git status --porcelain` is empty. The example runs were pointed +at scratch metrics directories, and the one leftover `outputs/examples/` tree +from an earlier iteration was removed after confirming it contained 0 tracked +files; the 110 tracked files under `outputs/` are all still present on disk. + +## 8. Pre-existing defect found and deliberately not repaired + +`AGENTS.md` §Tests lists two entry points that do not exist: + +``` +bash tests/debug/e2e-level/monolith_mode/scripts/test_dense_tp2_pp2_dummy.sh +bash tests/debug/e2e-level/monolith_mode/scripts/test_moe_tp2_ep2_pp2_dummy.sh +``` + +`tests/debug/` is absent from this branch **and from `origin/main`**, as are the +`comm_backend_tests/` directory the same section names. The published `tests/` +tree contains `analysis/`, `comparison/`, `e2e/`, `fixtures/`, `integration/`, +`performance/`, and `unit/`. + +The same missing tree is the cause of 10 of the 84 baseline unit failures, in +`tests/unit/test_colocation_release_review_contracts.py`, which resolves paths +under `tests/debug/e2e-level/monolith_mode/scripts/`. A further reference +survives in a docstring at +`frontier/scheduler/replica_scheduler/vllm_v1_engine_replica_scheduler.py:16`. + +This is one pre-existing defect class inherited from the release scrub that +removed `tests/debug/`, and it is unrelated to Issue 26. Repairing it properly +means either restoring the tree or retargeting the contract test, which is a +decision about the published test surface, not a documentation tweak. It is +recorded in `future.md` and left untouched here. The PP2 coverage §14.1 asked +for was obtained through the existing example scripts instead, as §5 shows. + +## 9. Limits of this validation + +- CPU only. No native profiling suite and no vLLM serving or TTFT comparison was + run; §14.1 explicitly excludes both. +- The PD-AF Reference-checkout integration tests could not run on this host. +- The W6 GPU parity module is skipped here; its hardware result lives in its own + report. +- The example runs use dummy execution time except for the CSV smokes, so they + validate structure, lifecycle, and conservation rather than latency accuracy. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index c42a9da2..363638df 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -13,6 +13,7 @@ | 2026-09-22 | Step 6 native parity test recorded and submitted as `exp-0922-140423-075005`; artifact identity closed as document-only. | | 2026-09-22 | Step 6 native GPU parity recorded: 8 of 8 at `rtol=0, atol=0` on H800 under `codesign`. | | 2026-09-22 | Step 7 recorded: the companion backend fix, both negative controls, clean-checkout validation, and the governance-scan repair the gitlink bump exposed. | +| 2026-09-22 | Step 8 §14.1 recorded: combined unit/integration suites, 16 architecture examples, four PP=2 cases, and the cold-then-warm predictor-cache pair. | ## Environment @@ -481,3 +482,21 @@ Full record: `test_report_2026-09-22_w7_collective_sim_zero_payload.md`. | Regression, default environment | `pytest tests/unit -q --continue-on-collection-errors` under `frontier-py310` with the submodule initialized and built: **84 failed, 3782 passed, 49 skipped, 11 errors**. Diffing the `FAILED` lists against the 84-failure baseline gives an empty set in both directions; the four extra passes are the new module. | | Fidelity matrix | Not run. No Frontier source file changed, and the backend fix is unreachable without `--cc_backend_config_type collective_sim`, which no matrix case selects. The unchanged failure set and the pre-fix/post-fix negative controls are the evidence. | | Limits | The `7 x 0.5 us` figure is the analytic NVLink model's own arithmetic, not a hardware measurement. Under the default `legacy_fabric` model with `collective_exclude_intra_server=True`, a single-server pod prices both an empty and a 1 MiB all-to-all at `0.0`, because all traffic is intra-server and excluded; that is the configuration's semantics, unrelated to the payload. The companion PR is draft, so the gitlink points at a branch commit and must be re-pointed at `main` once it merges. | + +## Step 8 — Combined regression on the integrated branch (§14.1) + +Full record: `test_report_2026-09-22_w8_combined_regression.md`. + +| Item | Result | +| --- | --- | +| Revision under test | `d881357`, 32 commits ahead of the PR base `refactor/oversized-module-split` @ `6ef0a3c`. | +| Base re-fetch before handoff | `git fetch origin main` gives `1f694f7`, unchanged since Step 0 and an ancestor of `HEAD`. No integration merge was needed and none was made. | +| Unit suite | `pytest tests/unit -q --continue-on-collection-errors` under `frontier-py310`: **84 failed, 3782 passed, 49 skipped, 11 errors**. Diffing the `FAILED` list against the recorded `origin/main` baseline list gives an empty set in both directions. | +| Integration suite | `pytest tests/integration -q --continue-on-collection-errors`: **15 passed, 22 skipped, 5 errors** against a baseline of 15 / 21 / 5. The one added skip is this branch's W6 parity module, which needs a GPU. All 5 errors are `test_pdaf_reference_lifecycle_observer.py` reporting the absent pinned Reference checkout at `/data/ycfeng/stepfun-performance-optimization/Frontier/worktrees/ref-afd-readonly`; identical on the base and environmental. | +| Architecture examples | All 16 release-supported scripts run end to end: 5 co-location offline, 2 co-location online, 2 PDD offline, 2 PDD online, 4 PD-AF offline, 1 PD-AF online. **16 passed, 0 failed**, each writing `request_metrics.csv` and `system_metrics.json`. | +| Pipeline cases | Four `PP=2` runs, all PASS: co-location offline dense `TP=2`; co-location offline MoE `Attn_TP=4, MoE_TP=2, MoE_EP=2`; sequential PDD offline dense with `PREFILL_PP=2, DECODE_PP=2`; co-location online dense `TP=2`. Each script's echoed topology confirms the intended values in its log. The changed cluster-scheduling, stage-dispatch, and metrics code runs on every stage, so these exercise it on the multi-stage path even though the new DP placement itself stays PP1-only. | +| Dense-layer coverage | From the dense examples, and from the MoE examples whose shared-expert work uses the separate `dense_mlp_hidden_dim` width through the ordinary linear-op path. Frontier has no "first `k` dense layers then MoE" model field, so there is no third heterogeneous shape to exercise. | +| Cold predictor cache | The same trained-predictor simulation run twice against an empty scratch cache via `--metrics_config_cache_dir`, so the repository `cache/` was neither moved nor deleted. Cold: 0 entries before, **26.9 s**, 63 artifacts written. Warm: 63 before, **2.1 s**, nothing new written. Their `request_metrics.csv` outputs are byte-identical (`request_e2e_time = 16.77818517187422 ms`, `ttft = 8.430025150867172 ms`), so the persisted-cache path reproduces the freshly trained path exactly. | +| Working tree | `git status --porcelain` empty afterwards. The one leftover `outputs/examples/` tree was removed after confirming 0 tracked files there; the 110 tracked files under `outputs/` are all still present. | +| Pre-existing defect, deferred | `AGENTS.md` §Tests names `comm_backend_tests/`, `debug/`, and two `bash tests/debug/e2e-level/monolith_mode/scripts/*.sh` commands. `tests/debug/` exists neither here nor on `origin/main`. The same missing tree causes 10 of the 84 baseline unit failures in `test_colocation_release_review_contracts.py`, and a docstring at `vllm_v1_engine_replica_scheduler.py:16` still points into it. One pre-existing defect class from the release scrub, unrelated to Issue 26; recorded in `future.md` and not repaired here. The PP2 coverage was obtained through the example scripts instead. | +| Limits | CPU only. No native profiling suite and no vLLM serving or TTFT comparison, both excluded by §14.1. The PD-AF Reference-checkout tests could not run on this host. The example runs use dummy execution time except for the CSV smokes, so they validate structure, lifecycle, and conservation rather than latency accuracy. | From 75f355e3b2f87946d6168440e1984b2ec5f0e51e Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 15:22:12 +0800 Subject: [PATCH 31/80] docs(records): close Step 8 and write the completion archive MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the §14.2 diff review method and the §14.3 PR hand-off, and writes summary.md: the stacked-PR layout, the per-work-package outcomes, the delivered source/test/record paths, the observed validation results, the two deferred items, and what this validation does not establish. The §14.2 review is stated as a self-review by the same agent that wrote the change, not an independent review. --- .../progress.md | 28 ++++- .../summary.md | 111 +++++++++++++++++- 2 files changed, 135 insertions(+), 4 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 909634e5..cb59c205 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -16,6 +16,7 @@ | 2026-09-22 | W6 native parity PASS on H800: `exp-0922-145047-660565`, 8 of 8 at `rtol=0, atol=0`. Step 6 closed. | | 2026-09-22 | W7 authorized and delivered: companion fix published as `eb7bc4f` with draft PR 1, Frontier gitlink moved, Frontier-side test added, and the governance scans narrowed to Frontier-owned sources. Step 7 closed. | | 2026-09-22 | Step 8 §14.1 run: unit and integration suites at the baseline failure set, 16 architecture examples, four PP=2 cases, and a cold-then-warm predictor-cache pair. `tests/debug/` pointer defect found, deferred to `future.md`. | +| 2026-09-22 | Step 8 closed: §14.2 review recorded, PR 35 body updated with the Step 8 results and record links, `summary.md` written. Task technically complete; PR stays draft for user review. | ## Status @@ -24,9 +25,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | -| Current step | Step 8 in progress. §14.1 combined regression complete and recorded; §14.2 diff review complete; §14.3 PR body update is the remaining work. | +| Current step | Step 8 closed. All eight steps complete; `summary.md` is the completion archive. | | Publication | PUSHED_VERIFIED. Draft PR 35 body carries the W6 and W7 sections; still draft. | -| Next action | Step 8 §14.3: add the Step 8 results and the record links to PR 35's body, keep it draft, and report technical acceptance separately from GitHub's draft status. Two follow-ups carried forward, both in `future.md`: re-point the collective-sim gitlink at `main` once companion PR 1 merges, and repair the `tests/debug/` pointers that 10 baseline unit failures share. | +| Next action | **User review.** Technical acceptance is complete; PR 35 remains draft and nothing was merged. Two follow-ups carried forward, both in `future.md`: re-point the collective-sim gitlink at `main` once companion PR 1 merges, and repair the `tests/debug/` pointers that 10 baseline unit failures share. Retargeting PR 35's base to `main` waits on PR 34. | ## Step status @@ -40,7 +41,7 @@ | 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | PUSHED_VERIFIED (records + PR 35 section) | REVIEWED (user chose to keep the single global field) | | 6 | Legacy fused-MoE profiling | PASS | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778). Native PASS: 8 of 8 in `tests/integration/test_moe_fused_expert_numerical_parity.py` at `rtol=0, atol=0` on H800, job `exp-0922-145047-660565` under `codesign` | PUSHED_VERIFIED (source, tests, docs, records, PR 35 section) | — | | 7 | Optional zero-payload backend | PASS. Companion fix published as `fwyc0573/frontier-htsim` `eb7bc4f` with draft PR 1; Frontier gitlink moved from `b8518af`; no Frontier source change | Companion 9 passed, negative control 6 of 9 fail on pristine sources. Frontier 4 passed, negative control 3 of 4 fail at the old gitlink. Clean checkout resolves `eb7bc4f` from the published remote, builds, and passes. Suite back to the 84-failure baseline with 3782 passing after narrowing three governance scans to Frontier-owned sources | PUSHED_VERIFIED | — | -| 8 | Combined regression, PR hand-off | IN_PROGRESS. §14.1 and §14.2 complete; §14.3 remaining | unit 84 failed / 3782 passed with a `FAILED` set identical to the `origin/main` baseline; integration 15 passed / 22 skipped / 5 errors, the errors environmental and identical on the base; 16 of 16 architecture examples pass; 4 of 4 `PP=2` cases pass; cold and warm predictor-cache runs byte-identical | — | — | +| 8 | Combined regression, PR hand-off | PASS | unit 84 failed / 3782 passed with a `FAILED` set identical to the `origin/main` baseline; integration 15 passed / 22 skipped / 5 errors, the errors environmental and identical on the base; 16 of 16 architecture examples pass; 4 of 4 `PP=2` cases pass; cold and warm predictor-cache runs byte-identical | PUSHED_VERIFIED (records + PR 35 body carrying the Step 8 results, the record links and the implementation commits) | NOT_REVIEWED | ## Chronological updates @@ -158,3 +159,24 @@ through the example scripts instead. pointed at scratch metrics directories, and one leftover `outputs/examples/` tree from an earlier iteration was removed after confirming it held 0 tracked files; the 110 tracked files under `outputs/` are all still present. + +### 2026-09-22 — Step 8 §14.2 and §14.3, hand-off + +- §14.2: read `git diff 6ef0a3c..d881357` over `frontier/ tests/ docs/ examples/`, + 1925 lines, line by line. Answers to all nine review questions, the cleanup + done in the pass, and an explicit statement that this was a **self-review and + not independent** are in `review.md`. +- §14.3: updated PR 35's body in place rather than opening a duplicate, using + `gh api --method PATCH` because `gh pr edit` fails on this repository with a + Projects-classic GraphQL deprecation error. The body now carries the W6 and W7 + outcomes in its progress table, a combined-regression section with the numbers + and what they do not prove, the deferred `tests/debug/` defect, absolute links + to `plan.md` / `requirements.md` / `progress.md` / `review.md` / + `validation.md` / `future.md`, a table of the implementation commits, and a + status paragraph separating technical acceptance from GitHub's draft flag. + Relative links were replaced with blob URLs because a PR body does not resolve + repository-relative paths; the links were checked and return 200. +- Verified after the update: `draft: true`, `state: open`, base still + `refactor/oversized-module-split`, head `8730509`, and the body read back byte + for byte as sent. Nothing was merged, force-pushed, marked ready, or closed. +- `summary.md` written as the completion archive. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md b/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md index e85c75c8..3b1a3c69 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md @@ -5,5 +5,114 @@ | Date | Change | | --- | --- | | 2026-09-21 | Placeholder created at Step 0. | +| 2026-09-22 | Completion archive written at Step 8. | -Completion archive; to be written at Step 8. Until then `progress.md` is authoritative. +## Overview + +Seven correctness candidates were extracted from `bug/ttft-check` by source +audit, re-derived against current `main`, and each one was either implemented +with a negative control that fails on the unrepaired source, or closed with the +reason recorded. Nothing was merged wholesale from the candidate branch. + +Delivered as a stacked pair of draft PRs: + +| PR | Branch | Base | Role | +| --- | --- | --- | --- | +| [#34](https://github.com/NetX-lab/Frontier/pull/34) | `refactor/oversized-module-split` | `main` @ `1f694f7` | Brings the four modules this work edits under the 2,000-line gate, behavior unchanged | +| [#35](https://github.com/NetX-lab/Frontier/pull/35) | `fix/issue26-correctness-pr` | `refactor/oversized-module-split` @ `6ef0a3c` | The behavior changes; 33 commits, final head `8730509` | +| [fwyc0573/frontier-htsim#1](https://github.com/fwyc0573/frontier-htsim/pull/1) | `fix/zero-payload-input-handling` | `main` @ `b8518af` | Companion backend fix, commit `eb7bc4f` | + +Latency calibration and any Frontier-versus-vLLM end-to-end comparison were out +of scope throughout. Issue 26 stays open. All three PRs are draft. + +## Work packages + +| # | Fix | Outcome | +| --- | --- | --- | +| W2 | Round-robin DP placement keeps rotating across scheduling calls | Landed. `6ab521d`, tests strengthened in `ceac2b4`. | +| W3 | A monolithic Replica completes one shared forward across mixed prefill and decode source lanes | Landed. `65ed8a7`. | +| W4 | Opt-in vLLM-style DP request placement, off by default and bounded in its constructor | Landed. `10dd474`. | +| W5 | Routing implementation identity separated from expert-load distribution | **Closed, not ported**, by user decision after the premise check showed the collision unreachable from any released configuration. Drafted implementation reverted before commit and archived as `w5_reverted_moe_routing_runtime_path.patch`. | +| W6 | Legacy fused-MoE profiling performs the real gated expert computation | Landed. `7269bac`, native parity test `697f219`, identity limits documented in `79f599a`. | +| W7 | The collective-sim backend accepts an empty collective | Landed companion-side. Frontier gitlink moved in `1b95187`; governance scans narrowed in `beded3c`. | + +## Deliverables + +### Source + +| Path | Change | +| --- | --- | +| `frontier/scheduler/cluster_scheduler/round_robin_cluster_scheduler.py` | W2: DP lane index derives from the persistent counter, so rotation survives across calls | +| `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py` | W4: shared lane-resolution seam | +| `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py` | W4: the new opt-in policy (new file) | +| `frontier/scheduler/utils/vllm_dp_load_balancer.py` | W4: the delayed-snapshot load model (new file) | +| `frontier/scheduler/cluster_scheduler/cluster_scheduler_registry.py`, `frontier/types/cluster_scheduler_type.py`, `frontier/config/cluster_scheduler_config.py`, `frontier/config/config.py` | W4: registration through the existing registry | +| `frontier/scheduler/utils/forward_collective.py` | W3: the shared monolithic forward (new file) | +| `frontier/scheduler/utils/sync_entry.py`, `sync_state.py`, `forward_sync_state.py`, `prefill_collective.py`, `decode_collective.py`, `ep_wave_schedule.py`, `ep_wave_inputs.py` | W3: one lifecycle for mixed-source cohorts | +| `frontier/events/cluster_schedule_event.py`, `frontier/events/global_batch_end_event.py`, `frontier/scheduler/request_load.py`, `frontier/scheduler/replica_scheduler/*` | W3/W4: event and load-snapshot wiring | +| `frontier/profiling/moe/moe_vllm_kernel.py` | W6: the legacy path performs the gated expert computation | +| `frontier/cc_backend/backends/collective-sim` | W7: gitlink moved from `b8518af` to `eb7bc4f` | +| `docs/profiling/README.md` | W6: the operator's scope and its artifact-identity limits | + +### Tests + +| Path | Purpose | +| --- | --- | +| `tests/unit/test_cluster_scheduler_dp_lanes.py` | W2 placement, extended to state where each request lands | +| `tests/unit/test_monolithic_mixed_forward_sync.py`, `tests/integration/test_monolithic_mixed_forward_runtime.py` | W3, unit and real event loop | +| `tests/unit/test_vllm_dp_load_balancer.py`, `tests/integration/test_vllm_dp_placement_runtime.py` | W4, unit and real event loop | +| `tests/unit/test_moe_fused_expert_arithmetic.py` | W6 on CPU, against plain-Torch references | +| `tests/integration/test_moe_fused_expert_numerical_parity.py` | W6 against vLLM's own `fused_experts` on a GPU | +| `tests/unit/test_collective_sim_zero_payload.py` | W7 through the Frontier backend boundary | +| `tests/frontier_sources.py` | Enumerates Frontier-owned sources so governance scans skip the vendored submodule | + +Companion repository: `tests/test_zero_payload_input.py`, 9 tests, published with +a narrowed `.gitignore` so it ships while private working material does not. + +### Records + +All under `task_memory/task_2026-09-21_issue26_correctness_pr/`: `plan.md`, +`requirements.md`, `progress.md`, `review.md`, `validation.md`, `design.md`, +`future.md`, `summary.md`, three pinned-source audit reports, five test reports +(W3, W4, W6, W7, and this Step 8 combined regression; W2's measurement is +recorded in `validation.md`), and `w5_reverted_moe_routing_runtime_path.patch`. + +## Observed validation results + +| Check | Result | +| --- | --- | +| Unit suite at `d881357` | 84 failed, 3782 passed, 49 skipped, 11 errors. The `FAILED` set is identical to the recorded `origin/main` baseline in both directions. | +| Integration suite | 15 passed, 22 skipped, 5 errors. The added skip is the W6 GPU module; the 5 errors are the absent pinned PD-AF Reference checkout and are identical on the base. | +| Architecture examples | 16 of 16 pass, covering co-location, sequential PDD, and sequential PD-AF in offline and online modes. | +| Pipeline cases | 4 of 4 `PP=2` runs pass. | +| Predictor cache, cold then warm | Cold 26.9 s writing 63 artifacts; warm 2.1 s writing none; `request_metrics.csv` byte-identical. | +| W3 fidelity matrix | 71 of 71 cases identical, against an expectation recorded before the run. | +| W4 fidelity matrix | 71 of 71 cases identical, against an expectation recorded before the run. | +| W4 placement | Measured to place differently from round-robin under the same load, so the policy is not a renamed default. | +| W6 native parity | 8 of 8 at `rtol=0, atol=0` on an H800 (`exp-0922-145047-660565`, charged group `codesign`), vLLM 0.10.2, `VLLM_API_VERSION=0.10.x`. | +| W7 companion | 9 passed on the fix; 6 of 9 fail against pristine sources. | +| W7 Frontier | 4 passed; 3 of 4 fail at the old gitlink. A fresh clone resolves `eb7bc4f` from the published remote, builds, and passes. | +| Negative controls | W2 12 of 23, W3 four trees, W4 five trees, W6 one discriminating test, W7 both sides — each fails for its own stated reason on the unrepaired source. | + +Detailed commands, expectations, and limits are in `validation.md` and the +per-work-package reports. + +## Open and deferred work + +| Item | Where | +| --- | --- | +| Re-point the collective-sim gitlink at `main` once companion PR 1 merges. `.gitmodules` already records `branch = main`; `git submodule update --remote` would currently drop the fix. | `future.md` §2 | +| The pre-existing `tests/debug/` pointer defect: `AGENTS.md` §Tests, a docstring at `vllm_v1_engine_replica_scheduler.py:16`, and 10 of the 84 baseline unit failures all reference a tree that exists neither here nor on `main`. Reported, not repaired; its fix is a decision about the published test surface. | `future.md` §1 | +| Retarget PR 35's base to `main` once PR 34 merges. | PR 35 description | +| Issue 26 itself stays open; this PR is a subset of it. | PR 35 description | + +## Limits of what was validated + +CPU only, apart from the one W6 GPU parity job. No native profiling suite was +run as a gate, and no vLLM serving or TTFT comparison was performed — both are +outside this task and neither is needed to accept the PR. The PD-AF +Reference-checkout integration tests could not run on this host. Apart from the +two CSV smokes, the example runs use dummy execution time, so they validate +structure, lifecycle, and conservation rather than latency accuracy. The final +diff review was a self-review by the same agent that wrote the change, not an +independent one; `review.md` states this. From 0137269c7b8322ab848c544f450ee90bcc6de0ce Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 17:38:41 +0800 Subject: [PATCH 32/80] docs(records): plan Step 9, PP>1 support for the opt-in vLLM DP placement Record the user's request to remove the PP1 restriction of VllmLoadBalancingClusterScheduler inside this PR, the codesign-only GPU instruction, and the decisions D-a..D-g taken on 2026-09-22. plan.md gains amendment A12 and Step 9 (section 17): the pinned vLLM 0.10.2 count-publication semantics under the batch-queue stepping path, the equivalence argument that leaves the steady state unchanged, the schedule-only-iteration gap, the on_replica_batch_scheduled hook design, the report-key options, CPU packages P1-P6 and ground-truth packages G1-G5, the frontier-calibration v2 case binding, and the codebase-design framing. design.md gains the W9 analysis with the discriminating scenario derived from the score algebra. requirements.md keeps the verbatim requests and answers; progress.md records that no source or GPU action has started. --- .../design.md | 115 +++++++++++++++++ .../plan.md | 121 ++++++++++++++++++ .../progress.md | 7 + .../requirements.md | 29 +++++ 4 files changed, 272 insertions(+) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md index df106635..d1aeacfe 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md @@ -9,6 +9,7 @@ scope decisions and the pre-measurement expectation for that package. | --- | --- | | 2026-09-22 | Created. Source-backed design for Checkpoint D's W3 half, with the scope decisions and their evidence. | | 2026-09-22 | Restructured into per-work-package sections and added the W4 design, with the report-key identity measured at the emission boundary. | +| 2026-09-22 | Added the W9 design: vLLM 0.10.2 count publication under pipeline parallelism, the equivalence argument, the schedule-only gap, the report-key options, and the discriminating scenario. Analysis only; implementation pending. | # W3 — one shared monolithic forward @@ -264,3 +265,117 @@ delegates to the same helper they already called. a defect in the seams, not an approved behavior change — unlike W2 and W3, W4 has no reachable fidelity fix, so a single mismatch falsifies the change rather than confirming it. + +# W9 — schedule-time reports under pipeline parallelism + +Analysis recorded before implementation (plan §17). Everything here is read from +the pinned checkout `.real-engine/vLLM-BS` at `ea95f571e` and from the current +branch; nothing has been measured on a GPU yet. + +## Reference facts, verified in the pinned checkout + +| Fact | Source | +| --- | --- | +| `max_concurrent_batches = pipeline_parallel_size`; a value above one builds `batch_queue` and selects `step_with_batch_queue`. | `vllm/v1/executor/multiproc_executor.py:325-329`, `vllm/v1/engine/core.py:147-157` | +| One iteration schedules first (`waiting → running` for admitted requests; in-flight requests are skipped because `num_new_tokens == 0`), appends, and returns without completing anything when the queue still has room and the oldest batch is not done. Otherwise it pops the oldest, waits, and applies `update_from_output`. | `core.py:318-370`, `vllm/v1/core/sched/scheduler.py:436-441` | +| `_maybe_publish_request_counts()` runs after every iteration and publishes `(running, waiting)` when it changed, with `step_counter` and `current_wave`. `step_counter` increments afterwards, so the publication of iteration `k` carries `k-1`. | `core.py:1075-1087, 1089-1137` | +| Coordinator: a strictly greater `(wave, step)` latches the previous counts when `stats_changed`; an equal key applies without latching; a smaller key warns. Publications every `min_stats_update_interval_ms=100` while changed, 5000 ms otherwise, 50 ms first-collection wait. | `vllm/v1/engine/coordinator.py:116, 196-227, 280-312` | +| Frontend: `score = waiting * 4 + running`, first minimum from `eng_start_index`, `+client_count` waiting reservation until the next publication replaces `lb_engines`. Only `vllm serve` builds `DPLBAsyncMPClient`. | `vllm/v1/engine/core_client.py:85-103, 1131-1156` | +| `get_request_counts()` is `(len(running), len(waiting))`; preempted requests are back in `waiting`. | `scheduler.py:1502-1504` | +| No DP+PP prohibition; per-engine `world_size = PP*TP`. | `vllm/config/parallel.py:314`, `vllm/engine/arg_utils.py:1221-1269` | + +## Frontier facts on the current branch + +| Fact | Source | +| --- | --- | +| `on_schedule` admits while `_num_running_batches < _num_stages`; the counter falls in `on_batch_end`, called just before the cluster scheduler's completion report. | `base_replica_scheduler.py:1052-1057`, `global_batch_end_event.py:180-185` | +| `_running_requests` grows at admission; `get_request_load()` returns `(queue + preempted, len(_running_requests))`. | `vllm_v1_engine_replica_scheduler.py:948`, `vllm_v1_iteration_policy.py:528-545` | +| At admission a batch carries the provisional per-lane creation counter; the Replica-scoped key is assigned when the sync room opens during execution. | `base_replica_scheduler.py:460-467`, `forward_sync_state.py:152-158` | +| The only report boundary today is `on_replica_batch_end`, keyed by `ForwardSyncState.get_step_id(batch)`. | `vllm_load_balancing_cluster_scheduler.py`; W4 above | + +## Why the steady state already matches + +With a full batch queue of depth `PP`, vLLM iteration `k` schedules `B_k`, +then completes `B_(k-1)`, then publishes. Frontier admits `B_k` when +`B_(k-2)` ends (the counter drops below `PP`) and reports when `B_(k-1)` ends, +with `B_k` already admitted. Both sides therefore publish the state "after +`B_k` admission and `B_(k-1)` completion" at the completion of `B_(k-1)`. The +same holds for the populations: admission moves requests into `running` on both +sides, completion removes finished ones. **No change is needed for the steady +state**, and the current PP=1 equivalence is the degenerate case `PP=1`. + +## The gap: schedule-only iterations + +When the queue is empty (after idle, or after an empty-batch iteration) the +first admission returns without completing anything and vLLM publishes it +immediately: `waiting -n, running +n`, score `-3n`, with a key strictly greater +than the last completion's. Frontier stays silent until that batch ends, and +when a single `on_schedule` call admits two batches it never exposes the state +after the first. At PP=1 `step()` is atomic, so this cannot occur; the PP=1 +path is unaffected by construction, which is what makes a byte-identical PP=1 +fidelity check the right acceptance test. + +## The report key at the admission boundary + +The W4 key is valid because both lanes of a forward share the Replica-scoped +id and distinct forwards are strictly increasing. That id does not exist yet at +admission — only the provisional per-lane counter does, and the dense +multi-lane row in W4 shows what per-lane counters do to the latch. Options: + +| Option | Rule | Reference fidelity | Cost | +| --- | --- | --- | --- | +| K1 | Schedule-time report reuses `last_report_step` (equal key). | Applies counts; never latches. Misses the reference latch of the pre-admission state when the previous completion is still unpublished (resume from idle within 100 ms). | None. | +| K3 (recommended) | The cluster scheduler relabels keys in emission order: a completion mints a label on first sight of its cohort (both lanes share it); a schedule-only admission mints a fresh label. | Strictly increasing in emission order, equal only for peer lanes of one cohort — the reference's "(wave, step) strictly increasing; peer engines share a step". | A dict and a counter inside the module; no interface change. Only comparisons are used, so PP=1 results are unchanged. | +| Stride keys (`2*cohort±1`) | Arithmetic room between completion keys. | Arbitrary factor; no room for two consecutive schedule-only admissions at PP≥3. | Rejected. | + +The probe planned as P1 records `(lane, boundary, provisional id, resolved id, +load)` at PP=2 for four shapes and fixes the rule from evidence, the same way +W4's table did. + +## The discriminating scenario, derived + +Let engine `e` receive `k_e` requests in a burst and admit `a_e` of them in +its first batch. With the frontend reservation only, its score is `4k_e`; +after vLLM's schedule-only publication it is `4k_e - 3a_e`. The two views +order the engines differently when `k_0 > k_1` but +`4k_0 - 3a_0 < 4k_1 - 3a_1`, i.e. `3(a_0 - a_1) > 4(k_0 - k_1) ≥ 4`, so +`a_0 - a_1 ≥ 2`. The smallest instance: a burst of five whose reservation +alternation gives `e0: r1, r3, r5` and `e1: r2, r4`; `r2` has a long prompt that +fills `e1`'s chunk budget so `r4` waits (`a_1 = 1`); `r1, r3, r5` are short +(`a_0 = 3`). Reference publication after the schedule-only iteration: +`e0 = 3`, `e1 = 5`, so a probe request `r6` arriving after the first +coordinator publication (≥50 ms) and before `e1`'s first chunk completes goes to +`e0`. Current Frontier never reports before the first completion, the +reservations persist (`e0 = 12`, `e1 = 8`) and `r6` goes to `e1`. The fixed +module reports the admission and sends `r6` to `e0`. The scenario is robust to +timing as long as `e1`'s first chunk exceeds the window on both sides (vLLM: +chunk budget and prompt length; Frontier: `dummy_execution_time_ms`, decision +D-e). Three warmups and an idle gap of at least 5 s precede the burst so both +systems start from the quiet state the argument assumes. + +## Planned edits + +| File | Edit | +| --- | --- | +| `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py` | `on_replica_batch_scheduled(time, replica_id, replica_local_id, batch, pipeline_room_remaining)`, inert default. | +| `frontier/scheduler/replica_scheduler/base_replica_scheduler.py` | Call the hook in the MONOLITHIC/PREFILL admission loop after `self._num_running_batches += 1`, with `pipeline_room_remaining = self._num_running_batches < self._num_stages`. | +| `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py` | Drop the PP1 clause of the guard and its error text; report `lane.get_request_load()` at the hook when room remains; K1 or K3 key rule per P1. | +| `tests/unit/test_vllm_dp_load_balancer.py`, `tests/integration/test_vllm_dp_placement_runtime.py` | Guard case inverted; schedule-time report semantics; PP=2 dense and MoE cases; the discriminating scenario. | +| `AGENTS.md:620`, this file, `plan.md`, `progress.md`, `validation.md`, `review.md` | Wording and records. | + +## Fidelity expectation, stated before measuring + +- Every PP=1 `vllm_load_balancing` scenario: `request_metrics.csv` value-identical and `system_metrics.json` identical after removing timestamps and run ids. +- Every scenario of every other cluster scheduler: identical (the hook's default is inert; the only added work is one method call per admission). +- PP=2 with `vllm_load_balancing`: runs complete; placements differ from round-robin where the published load says they should; the discriminating request moves from `e1` to `e0`. +- Ground truth: T1 publication sequences match by boundary index; T2 placement of the discriminating request matches the fixed module and not the current one. + +## What this adds, and what it is not + +It adds vLLM's schedule-time publication to the existing report path and +removes a capability boundary that the candidate had inherited. It does not add +an event type, a flag, or a constant; it does not model multiple frontends, +hybrid or external load balancing, wave resets, or elastic EP; and it does not +claim latency equivalence of placements — the comparison is by boundary index, +with timing controlled only where the discriminating scenario needs it. + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md index 3e9182a5..6f50caf9 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md @@ -5,6 +5,7 @@ | Date | Change | | --- | --- | | 2026-09-21 | Landed the execution specification verbatim (Section "Execution Specification" below) and recorded the amendments agreed with the user before Step 0. | +| 2026-09-22 | Added amendment A12 and the Step 9 draft (§17) for PP>1 support of the opt-in vLLM DP placement; awaiting user approval. | ## Amendments (authoritative where they differ from the specification below) @@ -23,6 +24,7 @@ Facts were verified against `origin/main` at `1f694f7c549aa3aeeb7c5bbae04e119c09 | A9 | §3.3 publication, §3.5 approvals | User authorized (Q5) pushing both branches to `origin` (`NetX-lab/Frontier`) and creating/updating the two draft PRs for the whole task. Not authorized: merge, force-push, history rewrite, closing Issue 26. | Checkpoint pushes need no further approval. Q6=a: this session stops after the Step 0 push for user review. | | A10 | §2.1 candidate snapshot | Verified: merge-base is `d71ad80b…`; the final candidate commit `a7b3320` touches only `task_memory/` (1006 files, no source); the candidate gitlink is `e564935d…`; donor design documents exist at the archive top level; `tests/unit/test_moe_routing_runtime.py` already exists on main and is modified by the candidate. | No change to the specification; facts recorded for Step 1. | | A11 | §7.1 candidate scope | The candidate adds ~80 `tests/e2e|integration|performance/issue26_*` experiment scripts. | Default disposition `DROP` unless Step 1 finds a specific reusable helper (the specification already names `tests/integration/issue26_dp_coordinator_reference.py` for review). | +| A12 | §10 Step 4 "Initial support remains … PP1"; §14 Step 8 "remains PP1-only" | User request 2026-09-22: complete `vllm_load_balancing_cluster_scheduler.py` so it is not limited to PP=1, after research of both codebases and a real vLLM 0.10.2 PP>1 DP-scheduling comparison run under the calibration skill. Same day: GPU workers use `charged_group="codesign"` only; `steptron_ci` is suspended until the user allows it. | Step 9 (§17) supersedes the PP1 boundary once approved. Until approval, §17 is a plan only: no source, GPU, or publication action. | ## Execution Specification (verbatim copy of `.local-draft/Frontier_Issue26_Correctness_PR_Execution_Spec_2026-09-21.md`) @@ -841,3 +843,122 @@ Under [R1]: `frontier/profiling/moe/moe_vllm_kernel.py`, `frontier/entities/stag --- **First action for a fresh Claude Code session:** perform Step 0, publish the new worktree branch and tracked plan, then complete the source audit in Step 1. Do not begin by cherry-picking the calibration branch or launching a GPU job. + +## 17. Step 9 — PP>1 support for the opt-in vLLM DP placement + +**Status 2026-09-22:** the user answered every §17.7 decision the same day (verbatim in `requirements.md`); the plan below is final. No source edit and no GPU submission has been made; execution starts at `P1`/`G1` once the user confirms the start. Research followed the `codebase-design` skill (§17.10); the ground-truth comparison follows `frontier-calibration` v2 as written (§17.9). + +### 17.1 Goal and acceptance criteria + +`VllmLoadBalancingClusterScheduler` accepts `num_pipeline_stages > 1` and reproduces vLLM 0.10.2's per-iteration DP request-count publication under the batch-queue stepping path that PP>1 selects, verified against a real `vllm serve --data-parallel-size 2 --pipeline-parallel-size 2` deployment. + +| # | Criterion | Evidence | +| --- | --- | --- | +| C1 | Guard becomes `MONOLITHIC` + one Replica + `vllm_v1` + (MoE or `attn_dp == 1`); PP=2 and PP=3 runs with `vllm_load_balancing` complete every request for dense `attn_dp=1` and MoE `attn_dp=2`. | Integration cases in P4. | +| C2 | PP=1 behavior is unchanged: every existing `vllm_load_balancing` scenario has value-identical `request_metrics.csv` and identical `system_metrics.json` (timestamps/run ids removed, the Q11 rule). Other cluster schedulers are untouched (the hook is inert for them). | P5 byte comparison; Step 8 regression set rerun. | +| C3 | Semantic alignment (T1): at PP=2 the Frontier per-lane report sequence matches the vLLM per-engine `SchedulerStats` publication sequence for the same workload, compared by boundary index rather than wall-clock: same publications per engine during ramp-up, same `(running, waiting)` after each admission/completion boundary, same key-ordering pattern, no out-of-order warning in either system. | §17.6 extraction + `workflow-gap-analysis`. | +| C4 | Placement (T2): on the discriminating scenario of §17.6 the current code places the probe request differently from vLLM and the fixed code places it identically; all non-discriminating requests agree in both. | §17.6 comparison table. | +| C5 | Records: `design.md` section, `AGENTS.md:620` wording, test report, `validation.md` rows, the calibration skill's semantic-alignment rows and workflow-gap analysis. | Files listed in P6. | + +### 17.2 Reference semantics (pinned vLLM 0.10.2, `.real-engine/vLLM-BS` at `ea95f571e`) + +| Fact | Source | +| --- | --- | +| PP>1 makes `max_concurrent_batches = pipeline_parallel_size`, which builds the `batch_queue` and selects `step_with_batch_queue`. | `vllm/v1/executor/multiproc_executor.py:325-329`, `vllm/v1/engine/core.py:147-157` | +| One iteration: `scheduler.schedule()` (waiting→running for new requests; in-flight requests are skipped by the `num_new_tokens == 0` rule), append; if the queue still has room and the oldest batch is not done, return without completing anything (a **schedule-only iteration**); otherwise pop the oldest, wait, `update_from_output` (finished requests leave `running`). | `core.py:318-370`, `vllm/v1/core/sched/scheduler.py:436-441` | +| `_maybe_publish_request_counts()` runs after **every** iteration and publishes `(running, waiting)` whenever it changed, carrying `step_counter` and `current_wave`. `step_counter` is incremented afterwards in `_has_global_unfinished_reqs`, so the publication of iteration `k` carries `k-1`. | `core.py:1075-1087, 1089-1137` | +| Coordinator: a report whose `(wave, step)` is strictly greater than the last latches the previous counts when `stats_changed`; equal keys apply without latching; smaller keys warn. Publication every `stats_update_interval_ms` (100 ms) while changed, 5000 ms otherwise, with a 50 ms first-collection wait. | `vllm/v1/engine/coordinator.py:196-227, 280-312` | +| Frontend: `score = waiting * 4 + running`, first minimum from `eng_start_index`, local `+client_count` waiting reservation until the next publication. Only the online `vllm serve` path builds `DPLBAsyncMPClient`; offline `LLM` DP is SPMD without a balancer. | `vllm/v1/engine/core_client.py:85-103, 1131-1156` | +| No DP+PP prohibition: per-engine `world_size = PP*TP`; the `arg_utils` assertions concern hybrid/external LB and the `mp` backend only. | `vllm/config/parallel.py:314`, `vllm/engine/arg_utils.py:1221-1269` | +| Schedule log rows carry request ids and queue sizes but **no engine identity**; all DP engine processes inherit one `VLLM_FRONTIER_SCHED_LOG_PATH`. Coordinator publications are not logged. | `scheduler.py:91-92, 985-998` | + +### 17.3 Frontier model today and the gap + +| Fact | Source | +| --- | --- | +| `on_schedule` admits while `_num_running_batches < _num_stages`; `on_batch_end` decrements at the batch's last stage, immediately before the cluster-scheduler report. | `base_replica_scheduler.py:1052-1057`, `global_batch_end_event.py:180-185` | +| `_running_requests` grows at admission; `get_request_load()` returns `(queue + preempted, len(_running_requests))`, the same population as vLLM's `get_request_counts()`. | `vllm_v1_engine_replica_scheduler.py:948`, `vllm_v1_iteration_policy.py:528-545` | +| The only report boundary is `on_replica_batch_end` with key `ForwardSyncState.get_step_id(batch)`. | `vllm_load_balancing_cluster_scheduler.py`, `design.md` "The report key" | +| At admission a batch carries only the provisional per-lane creation counter; the Replica-scoped key is assigned when the shared sync room opens during execution. | `base_replica_scheduler.py:460-467`, `forward_sync_state.py:152-158` | + +Equivalence argument. With a full batch queue, vLLM iteration `k` = schedule `B_k` + complete `B_(k-1)` + publish. Frontier admits `B_k` at `B_(k-2)`'s end and reports at `B_(k-1)`'s end, when `B_k` is already admitted; both observe the same state. **Steady state needs no change.** The gap is the schedule-only iteration: after an idle period (or an empty-batch iteration) vLLM publishes the admission immediately (`waiting -n, running +n`, score `-3n`) with a key strictly greater than the last completion's, while Frontier stays silent until the batch ends and, when one `on_schedule` call admits two batches, never exposes the state after the first. At PP=1 `step()` is atomic, so schedule-only iterations cannot occur and the PP=1 path is unaffected by construction. The existing key cannot be reused at the admission boundary because it is not yet resolved there (the dense multi-lane INVALID row in `design.md` shows what per-lane counters do to the latch). + +### 17.4 Design + +| Id | Decision | Content | +| --- | --- | --- | +| D9-1 | Schedule-time hook | `BaseClusterScheduler.on_replica_batch_scheduled(time, replica_id, replica_local_id, batch, pipeline_room_remaining)` with an inert default, called from the MONOLITHIC/PREFILL branch of `BaseReplicaScheduler.on_schedule` right after `self._num_running_batches += 1`, with `pipeline_room_remaining = self._num_running_batches < self._num_stages`. `VllmLoadBalancingClusterScheduler` reports `lane.get_request_load()` only when room remains — at PP=1 that is never, so PP=1 behavior is unchanged. Rejected: reconstructing the report in `ReplicaScheduleEvent` after `on_schedule`, which cannot observe the intermediate state when one call admits two batches (vLLM publishes each admission in its own iteration). Name candidates under the AGENTS.md naming gate: `on_replica_batch_scheduled` (mirrors `on_replica_batch_end`; recommended) or `on_replica_admission`. | +| D9-2 | Report key | **K1** equal key (`last_report_step`): simplest, but misses the reference latch of the pre-admission state when the previous completion is still unpublished (resume from idle within 100 ms). **K3** order-preserving relabeling owned by the cluster scheduler: a completion mints a label on first sight of its cohort and both lanes of that cohort share it; a schedule-only admission mints a fresh label; labels strictly increase in emission order and are equal only for the same cohort — the reference's "(wave, step) strictly increasing, peer engines share a step". Only comparisons are used, so PP=1 behavior stays identical (C2 verifies). Recommended: K3, fixed after the P1 probe. Rejected: stride keys such as `2*cohort±1` (an arbitrary factor, and no room for two consecutive schedule-only admissions at PP≥3). | +| D9-3 | Guard | Drop `num_pipeline_stages == 1`; keep the other four clauses and the dense multi-lane rejection (its evidence is PP-independent); update the error text. | +| D9-4 | Determinism and flags | No new `EventType`, no new config flag, no balancer constant change (`design.md` event-type determinism; plan §10 "no tuning flags"). | +| D9-5 | Docs | `AGENTS.md:620` ("one pipeline stage" removed), `design.md` guard row plus a section "Schedule-time reports under pipeline parallelism" with the P1 probe table, `plan.md`/`progress.md`/`validation.md`/`review.md`. | + +### 17.5 Work packages (sequence `{P1, G1, G2} -> P2 -> {P3, P4, G3} -> G4 -> G5 -> P5 -> P6`) + +CPU packages `P*` change Frontier; ground-truth packages `G*` never change Frontier and run in parallel where the graph allows. GPU queue time on `codesign` is the expected critical path, so `G1`/`G2` start with `P1`. + +| Package | Content | Acceptance | +| --- | --- | --- | +| P1 Probe | Scratch script patches `on_replica_batch_end` and a temporary admission hook to record `(lane, boundary, provisional id, resolved id, load)` at PP=2 for four shapes (MoE `attn_dp=2` burst and staggered, MoE `attn_dp=1`, dense `attn_dp=1`). No source change. | Table added to `design.md`; K1/K3 decided from it. | +| P2 Implement | D9-1..D9-3 in `base_cluster_scheduler.py`, `base_replica_scheduler.py`, `vllm_load_balancing_cluster_scheduler.py` (about 80 lines). | Existing unit tests pass except the intentionally inverted guard case. | +| P3 Unit | `tests/unit/test_vllm_dp_load_balancer.py`: guard param at `:538` becomes positive; PP=1 never reports at schedule time; PP=2 reports iff room remained; key ordering/equality semantics; helper `num_pipeline_stages` parameter. | New tests fail before P2 and pass after. | +| P4 Integration | `tests/integration/test_vllm_dp_placement_runtime.py`: PP=2 dense `attn_dp=1` and MoE `attn_dp=2` (`moe_ep=2`) cases — completion, `routing_times == cluster_schedule_times`, schedule-time reports only when room remained, event-type set equal to the round-robin baseline, and the §17.6 discriminating scenario with `placements_fixed != placements_round_robin`. | Pass. | +| P5 Fidelity | Byte comparison of all PP=1 `vllm_load_balancing` scenarios before/after; Step 8 regression set (unit, integration, 16 examples) rerun. | C2. | +| P6 Records | Docs of D9-5, test report `test_report_2026-09-22_w9_pp_dp_placement.md`, commits per package, push (Q5), PR body update. | Pushed and verified. | +| G1 Ground-truth checkout | `.real-engine/vLLM-BS` is detached at `ea95f571e` with only the remote ref `origin/feature/frontier-comparison-instrumentation` (same commit). Create the local branch at that commit, commit the D-b instrumentation on it (`dp_rank` in the schedule row; `VLLM_FRONTIER_DP_STATS_LOG_PATH` JSONL in `DPEngineCoreProc._maybe_publish_request_counts` with `dp_rank, wave, step_counter, running, waiting, monotonic ts`), keep the tree clean. Record the tuple with `groundtruth_remote_tip=ea95f571e`, the diff artifact `remote_tip..HEAD` and its SHA-256. No push (D-b covers the commit only). | Clean tree on the exact branch; tuple recorded in the manifest. | +| G2 Harness and case | `tests/comparison/dp_placement_pp/`: `make_trace.py` (three warmups, ≥5 s idle gap, discriminating burst, steady-state segment → Frontier trace CSV and request-id namespace), `replay_client.py` (online token-id replay, `x-request-id`, dispatch offsets recorded), `extract_vllm_placement.py` (schedule log + DP-stats log → per-engine publication sequence and placement per request id), `compare_placement.py` (T1 boundary table, T2 placement table, `workflow_gap_table.csv` rows). Case directory `task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/` (`manifest.yaml`, `case_init`, `analysis/`, receipts); `runs/*` under `/data/ycfeng/tmp/issue26-correctness-pr/calibration/dp_pp_case_001/`, mirrored to `/mnt/codesign-exp/ycfeng/frontier/dp_pp_calibration/`. `semantic-alignment-check` table written before any run. | Semantic table `PASS`; Frontier-only pre-check shows the discriminating request differs between round-robin/current and the expected reference placement. | +| G3 S0 smoke | 2×H800, DP=2, PP=1: `groundtruth-run-check` (clean mode) → run → verify `vllm.__file__` resolves to the overlay, instrumentation rows exist with `dp_rank`, request-id mapping and extraction work end to end. | Run-check `PASS`; extraction produces one placement per formal request. | +| G4 S1 ground truth | 4×H800, DP=2, PP=2, TP=1, EP=2: `groundtruth-run-check` → `groundtruth-run` (clean mode, ≥3 warmups, E2E runtime metrics on, no CUDA-op or CPU probes; the schedule log and DP-stats log are scheduler-level workflow evidence declared in the manifest). Measure `e1`'s first-chunk duration; if it is below the window, retune the chunk budget/prompt length and rerun (S2). | Fresh artifacts, exit 0, tuple verified, first-chunk duration recorded. | +| G5 Simulator runs and analysis | `simulator-run` on the pre-change revision and on the post-P2 revision with the same trace; `workflow-gap-analysis` compares publication sequences (T1) and placements (T2). `e2e-metrics-gap` is not run: Frontier timing is dummy (D-e) and the entry's pinned normalizer is absent, which the skill treats as `FAIL`; recorded as not applicable to this step's acceptance. | C3 and C4 tables with `MATCH`/`MISMATCH` rows and source anchors. | + +### 17.6 Ground-truth comparison under `frontier-calibration` + +Workflow: `case_init` manifest → `parity-run` → `semantic-alignment-check` → `groundtruth-run-check` → `groundtruth-run` → `simulator-run` → `workflow-gap-analysis`; `$grill-me` questions for any setting the skill cannot resolve; human review before any code change that the comparison motivates. + +| Item | Setting | +| --- | --- | +| Topology | vLLM `--data-parallel-size 2 --pipeline-parallel-size 2 --tensor-parallel-size 1 --enable-expert-parallel` (EP = TP×DP = 2). Frontier `attn_dp=2, attn_tp=1, moe_tp=1, moe_ep=2, num_pipeline_stages=2, num_replicas=1`, `cluster_scheduler=vllm_load_balancing`, `replica_scheduler=vllm_v1`. 4×H800. | +| Model | `Qwen3MoeForCausalLM` from `data/config/models/Qwen3-30B-A3B-tiny.json` (8 layers, 16 experts, top-8; `SupportsPP` and `FusedMoE` EP in 0.10.2, `qwen3_moe.py:146,582,767`), `--load-format dummy`, `--skip-tokenizer-init`, served from a local config directory; Frontier loads the same JSON through `create_from_name`. | +| Semantic-alignment rows | DP/PP/TP/EP sizes, `max_num_batched_tokens`, `max_num_seqs`, block size, KV block count (Frontier `num_blocks` taken from vLLM's startup log), chunked prefill on, prefix caching off, FCFS policy, `stats_update_interval_ms=100`, dummy weights, tokenizer skipped, `ignore_eos`, request-id mapping, arrival-time origin. | +| Workload | One Frontier trace CSV (`arrived_at,num_prefill_tokens,num_decode_tokens`) is the single source. A replay client posts `/v1/completions` with `prompt=[token ids]`, `max_tokens=num_decode_tokens`, `ignore_eos=true`, header `x-request-id=` (propagated to the engine request id, `serving_engine.py:971-978`, so schedule-log `scheduled_new_req_ids` map back) at `arrived_at` offsets from one origin. | +| Discriminating scenario (T2) | Burst of 5 at `t0`: the frontend reservation alternates them (`e0: r1,r3,r5`, `e1: r2,r4`). `r2` has a long prompt that fills `e1`'s chunk budget so `r4` waits; `r1,r3,r5` are short. Probe `r6` arrives at `t0+~100 ms`, after the first coordinator publication (≥50 ms) and before `e1`'s first chunk completes. Reference publication S1: `e0 = 3 running → 3`, `e1 = 1 running + 1 waiting → 5`, so `r6 → e0`. Current Frontier never reports before the first completion, the reservations persist (`e0 12`, `e1 8`), so `r6 → e1`. Fixed Frontier reports S1 at admission, so `r6 → e0`. Robustness requires `e1`'s first chunk to exceed ~150 ms in both systems: vLLM through the chunk budget (8k-16k tokens, measured in S0), Frontier through decision D-e. A steady-state segment (staggered arrivals, long decodes) supplies T1. | +| Extraction | vLLM: per-engine publication log and schedule log (needs D-b). Frontier: balancer report trace (the hook `test_vllm_dp_placement_runtime.py` already uses), `metrics_ground_truth.jsonl`, placement ledger. Comparison script emits the T1 boundary table and the T2 placement table. | +| Harness location | `tests/comparison/dp_placement_pp/` (replay client, extraction, comparison) — pending D-a. | +| GPU job | StepMind `RJobBackend`, `charged_group="codesign"` only, `positive_tags=["H800"]`, `gpu=4, cpu=16, mem_gb=128`, image `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`, `code_mount_point` per D-f, libcuda path fix and internal PyPI mirror from handbook §10, vLLM-BS as a Python overlay (copy the checkout to worker-local disk, copy the image's compiled `vllm/*.so` and `vllm_flash_attn` in, `PYTHONPATH` first; the vLLM-BS delta touches no `csrc/`, `cmake/`, `setup.py`, or `requirements/` — verified). Durable logs under `/mnt/codesign-exp/ycfeng/frontier/dp_pp_calibration/`. Sequence: S0 2-GPU smoke (DP=2, PP=1: stack, overlay, client, extraction), S1 4-GPU DP=2×PP=2 (T1+T2), S2 rerun only if the workload needs retuning. Budget ≤ 3 jobs × ≤ 1 h; launcher kept alive locally; no resubmission while queued; verify creator, mount, `torch.cuda` device, outputs, terminal status. | + +### 17.7 Decisions (answered by the user on 2026-09-22; verbatim text in `requirements.md`) + +| Id | Decision | Effect on this plan | +| --- | --- | --- | +| D-a | "依据该skills" — follow `/home/brainpp/.claude/skills/frontier-calibration` as written. | Route `parity-run` (`semantic-alignment-check` → `groundtruth-run-check` → `groundtruth-run` → `simulator-run`), then `workflow-gap-analysis`. None of these entries binds a pinned helper file; the absent archive `/data/ycfeng/frontier-calibration-old-20260831/` affects only `e2e-metrics-gap`, `op-supplement`, and `dispatch-align-trace`, which are not on this case's path. If one of them becomes necessary it is `FAIL` per `tool-boundaries.md` and the case stops there; no substitute helper is used. The G2 scripts are the case's declared ground-truth client and analysis producers, not stand-ins for a listed helper. `$grill-me` is not installed; the decisions in this table were obtained by direct Q&A and are stored in the manifest as decision records with the user as `requesting_user` and `reviewer_identity`. | +| D-b | "授权" — the minimal vLLM-BS instrumentation commit is authorized. | G1. The commit stays local on the exact branch; pushing to `fwyc0573/vLLM-BS` was not requested and is not needed because the tuple records the remote tip plus the diff artifact. | +| D-c | tiny Qwen3-MoE + dummy weights + tokenizer skipped (no download). | §17.6 model row unchanged. | +| D-d | Hook name `on_replica_batch_scheduled`. | D9-1 final. The key rule (K1/K3) was not chosen by the user; K3 remains the recommendation and is fixed after the P1 probe. | +| D-e | Dummy mode first; if an unresolvable blocker appears, switch to H800 profiling mode. | Frontier runs use `dummy_execution_time_ms` sized so `e1`'s first chunk exceeds the window. The fallback is new H800 profiling CSVs and a trained predictor, taken only on a recorded blocker. | +| D-f | `code_mount_point=/data/ycfeng/Frontier`. | Parent mount covering the worktree and `.real-engine/vLLM-BS`; the worker copies vLLM-BS to local disk for the overlay. | +| D-g | The intended skill is `/home/brainpp/.claude/plugins/cache/claude-plugins-official/mattpocock-skills/1.2.3/skills/engineering/codebase-design`. | Read in full; its vocabulary and principles are applied in §17.10 and in `design.md` W9. | + +### 17.8 Limits + +Unchanged and not claimed: multiple Replicas, multiple frontends (`client_count > 1`), `data_parallel_hybrid_lb`/`external_lb`, wave-reset semantics, elastic EP, IPC timing, latency equivalence of placement. The hook is inert for PDD/PD-AF roles and the DECODE (M2N) branch of `on_schedule` is untouched. T1's boundary-index comparison is the primary evidence; T2 is confirmatory. + +### 17.9 Calibration case binding (`frontier-calibration` v2) + +| Contract item | Value for `dp_pp_case_001` | +| --- | --- | +| Entries used | `parity-run` → `semantic-alignment-check`, `groundtruth-run-check`, `groundtruth-run`, `simulator-run`; then `workflow-gap-analysis`. Not used: `e2e-metrics-gap` (dummy Frontier timing; pinned normalizer absent → `FAIL`), `dispatch-align-trace` (pinned helpers absent; dispatch offsets are still recorded by the client as audit evidence), operator/CPU/residual lanes. | +| Manifest | `case_id=dp_pp_case_001`, `run_generation`, `formal_request_ids`, three `warmup_request_ids`, requesting user and `reviewer_identity` (the user), `case_init.auto_recycle=false`, `groundtruth_weight_mode=dummy`, `real_weight_download=false`, model/dtype/trace/architecture/scheduler/parallel domains/backend/eager mode/KV budget, each mode's source path, producer profile, request-id namespace and encoding, artifact path and fresh-file assertion, decision records D-a..D-g. | +| Ground-truth checkout tuple | path `/data/ycfeng/Frontier/.real-engine/vLLM-BS`, branch `feature/frontier-comparison-instrumentation`, ref `refs/heads/feature/frontier-comparison-instrumentation`, commit = G1 instrumentation commit, `groundtruth_tree_dirty=false`, remote `https://github.com/fwyc0573/vLLM-BS.git`, `groundtruth_remote_tip=ea95f571e`, diff artifact `remote_tip..HEAD` with SHA-256, `groundtruth_overlay_patch_applied=false`. | +| Mode predicates | Clean ground truth: ≥3 warmups, E2E runtime metrics on, CUDA-op and CPU probes off. Frontier: clean-style E2E metrics plus `metrics_ground_truth.jsonl`. Disjoint run directories per mode and per Frontier revision. | +| Semantic rows | DP/PP/TP/EP sizes and meaning (attention vs MoE domains kept separate), `max_num_batched_tokens`, `max_num_seqs`, block size, KV block count, chunked prefill, prefix caching off, FCFS, `min_stats_update_interval_ms=100` (`coordinator.py:116`), 50 ms first-collection wait, dummy weights, tokenizer skipped, `ignore_eos`, request-id mapping, arrival-time origin, warmup/idle-gap layout, MoE routing audit rows (router path, `top_k=8`, renormalization, 16 experts, EP scope 2; routing distortion is diagnostic here because no numeric E2E gate is claimed). | +| Receipts | Caller-written command receipts (command, cwd, environment declarations, UTC start/end, exit code, artifact paths) for every command the case issues; `exec capture: UNKNOWN` as the contract states. | +| Code-change gate | The P2 change is the user-approved feature of this step, not a calibration repair. Any further Frontier change motivated by the G5 analysis needs `analysis_state=COMPLETE`, `status=PASS`, and the user's review `PASS` before it is applied (`repair-approval.md`). | + +### 17.10 Design vocabulary (`codebase-design`) + +- **Module.** `VllmLoadBalancingClusterScheduler`, with `VllmDPLoadBalancer` as an internal module. Its **interface** is `schedule_at`, `on_replica_batch_end`, and (new) `on_replica_batch_scheduled`, plus the facts a caller must know: reports are per lane, the schedule-time report is emitted once per admitted batch and only while pipeline room remains, at equal simulated time the completion report precedes the admissions it triggers, and the module never raises on an inert path. +- **Seam.** `BaseClusterScheduler.on_replica_batch_*` already has two **adapters** — the inert default used by every other cluster scheduler and this module — so the new hook extends a real seam rather than creating a hypothetical one. The call site in `on_schedule` is the only place that knows whether pipeline room remains, which is why the seam sits there and not in `ReplicaScheduleEvent`. +- **Depth.** The reservation, publish deadlines, latch, schedule-only detection, and the K3 relabeling all stay behind the same three methods; callers learn nothing new to gain PP>1 support. **Deletion test:** removing the hook would force every event that admits a batch to reconstruct vLLM's per-iteration publication — the complexity reappears across callers, so the module earns its keep. +- **Test surface.** Tests drive the module through its interface and assert observable placements and published counts; the relabeled keys are implementation and are not asserted directly. +- **Design it twice.** Three interface shapes were compared: (1) in-loop push hook with `pipeline_room_remaining` — chosen: smallest interface, the room predicate lives where it is known, one adapter per cluster scheduler; (2) event-level reconstruction after `on_schedule` — rejected: cannot observe the state after the first of two admissions in one call, so it is shallow and wrong; (3) pull-style `iter_admission_loads()` on the replica scheduler — rejected: widens the replica scheduler's interface for one caller and inverts the push direction the completion report already uses. + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index cb59c205..211d9d8f 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -180,3 +180,10 @@ files; the 110 tracked files under `outputs/` are all still present. `refactor/oversized-module-split`, head `8730509`, and the body read back byte for byte as sent. Nothing was merged, force-pushed, marked ready, or closed. - `summary.md` written as the completion archive. + +## Step 9 — PP>1 support for `vllm_load_balancing` (pending) + +| Date | State | Note | +| --- | --- | --- | +| 2026-09-22 | pending | Plan drafted in `plan.md` §17 with amendment A12 and `requirements.md` rows; awaiting user approval and the D-a..D-g answers. No source, GPU, or publication action taken. | +| 2026-09-22 | in-progress (planning closed) | User answered D-a..D-g; `plan.md` §17 finalized (decisions, G1–G5 ground-truth packages, calibration case binding, `codebase-design` vocabulary), `design.md` W9 written, `requirements.md` updated. Records committed and pushed (SHA in the commit log). No source edit, no GPU submission; next action: P1 probe + G1/G2 once the user confirms the start. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md index c1b3b9c7..c6f1192e 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md @@ -6,6 +6,7 @@ | --- | --- | | 2026-09-21 | Recorded the original request, the specification hand-off, and the decisions from the planning interview. | | 2026-09-22 | Recorded the W6 artifact-identity decision and the native GPU validation instruction. | +| 2026-09-22 | Recorded the PP>1 `vllm_load_balancing` request, the codesign-only GPU instruction, and the open Step 9 decisions. | ## [Original Request] 2026-09-21 @@ -39,3 +40,31 @@ The draft specification is landed verbatim in `plan.md` together with an Amendme | 2026-09-22 | W6 native validation: how should the GPU parity check be run? | **"请从 dockerhub 中找到 v0.10.2 的官方镜像（如果没有，fallback 到 >=0.10, <0.11），然后参考 hand-book 中对 docker 的使用在 gpu worker 上使用该镜像。如果你需要使用原来的 benchmark ... 中的测试 suits 和插桩，你需要 mount 该 repo 到 gpu worker。如果需要进行对比验证，则 follow skill：/home/brainpp/.claude/skills/frontier-calibration"** — `vllm/vllm-openai:v0.10.2` exists on Docker Hub, so no fallback was needed. It is pulled through the company docker.io proxy as `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`. The instrumented benchmark repository is not mounted: the parity check compares tensors from vLLM's own `fused_experts` inside one process and needs no serving instrumentation. The `frontier-calibration` skill is not invoked for the same reason — its workflow is E2E simulator-versus-served-vLLM calibration, not a kernel-level tensor comparison. | | 2026-09-22 | W7: authorize a companion-repository branch in `fwyc0573/frontier-htsim`, or record W7 as `EXCLUDED`. Asked with the option spelled out as "授权我在那个仓库建分支、提交测试和源码、推送并开 companion draft PR，然后再动 Frontier 的 gitlink。按计划这是 backend 先发布、Frontier 后跟进的顺序。" | **"1.授权"** — companion-repository work authorized. Delivered in that order: branch `fix/zero-payload-input-handling` pushed to `fwyc0573/frontier-htsim` at `eb7bc4f`, companion **draft** PR 1 opened, then Frontier's gitlink moved. PR 35 stays draft; nothing was merged. | | 2026-09-22 | `[Original Request]` "对于当前 task 的 gpu worker 集群应该使用 codesign 而不是 step_main" | All GPU submissions for this task use `charged_group="codesign"`. Applied from `exp-0922-142415-796404` onward; the first attempt `exp-0922-140423-075005` had used `steptron_ci`, found no capacity after ~20 minutes Pending, and was stopped. The successful native parity run `exp-0922-145047-660565` is on `codesign`. | + +## [Original Request] 2026-09-22 — PP>1 support for `vllm_load_balancing` + +"添加需求：我需要在当前pr中补全 vllm_load_balancing_cluster_scheduler.py 的模拟支持，使得其不被限制在pp=1；你需要基于frontier 和vllm的codebase进行充分调研（codebase design skill)和设计，并且运行vllm v0.10.2进行实际调度结果的对比（pp>1情况下的dp 调度策略；调用calibration来确保参数设定一致）。请你先设计落地该子任务的plan，在我批准之前暂不执行" + +Reading: (a) remove the PP1 restriction of `VllmLoadBalancingClusterScheduler` inside PR 35; (b) research both codebases and design first; (c) compare against a real vLLM v0.10.2 PP>1 DP-scheduling run, using the `frontier-calibration` skill to keep settings consistent; (d) deliver the plan first and do not execute before approval. Plan: `plan.md` §17 (Step 9) and amendment A12. + +## [Original Request] 2026-09-22 — GPU charged group + +"ps：后续的gpu worker集群只允许使用 codesign（暂停对steptron_ci的使用，直至得到我允许）" + +Rule: every GPU submission uses `charged_group="codesign"`; `steptron_ci` is suspended until the user allows it again. Also saved as a durable memory note. + +## [Original Request] 2026-09-22 — Step 9 decisions + +"d-a: /home/brainpp/.claude/skills/frontier-calibration 依据该skills D-b：授权 D-c：tiny Qwen3-MoE + dummy 权重 + 跳过 tokenizer（无需下载）； D-d：采用hook 名 on_replica_batch_scheduled D-e：暂时使用 dummy模式验证，如果过程中存在无法解决block转为h800 profiling模式； D-f：选择worker 挂载：code_mount_point=/data/ycfeng/Frontier（父目录，同时覆盖 .real-engine/vLLM-BS； D-g：/home/brainpp/.claude/plugins/cache/claude-plugins-official/mattpocock-skills/1.2.3/skills/engineering/codebase-design 完成上述问题确认，统一更新docs（确保上述执行plan和思路和已有观察被清晰记录）并提交push" + +| Id | Question | Decision | Recorded in | +| --- | --- | --- | --- | +| D-a | Calibration helper archive absent. | Follow `frontier-calibration` v2 as written. The case path (`parity-run` → `workflow-gap-analysis`) binds no pinned helper; entries that do (`e2e-metrics-gap`, `op-supplement`, `dispatch-align-trace`) are off-path and would be `FAIL` if needed. | `plan.md` §17.7, §17.9 | +| D-b | vLLM-BS instrumentation. | Authorized: local commit on `feature/frontier-comparison-instrumentation` (engine identity in schedule rows; per-engine publication log). Push not requested. | `plan.md` G1 | +| D-c | Model and tokenizer. | Tiny Qwen3-MoE config, dummy weights, tokenizer skipped. | `plan.md` §17.6 | +| D-d | Hook name and key rule. | `on_replica_batch_scheduled`. Key rule left to the P1 probe (K3 recommended). | `plan.md` D9-1, D9-2 | +| D-e | Frontier timing for the placement check. | Dummy mode first; switch to H800 profiling mode only on an unresolvable blocker. | `plan.md` §17.6, §17.7 | +| D-f | Worker mount. | `code_mount_point=/data/ycfeng/Frontier`. | `plan.md` §17.6 | +| D-g | "codebase design skill". | The `codebase-design` skill at the path above; applied in `plan.md` §17.10 and `design.md` W9. | `plan.md` §17.10 | + +Also requested: update the records so the execution plan, reasoning, and observations are clearly recorded, then commit and push. Not yet given: an explicit start signal for P1/G1. From f7c31e4169a696b220010ae0887d37e2d4b123b6 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 20:08:49 +0800 Subject: [PATCH 33/80] fix(scheduler): credit decoding requests at a dense layer inside a mixed batch A dense layer in a MoE model completes per source, outside the shared forward completion that credits a routed layer to every decoding request in the cohort. The decode handler credits its own batch, but the prefill handler credits nothing, so a request that had finished its prefill and was carried in a prefill-mode batch (chunked prefill mixes them) missed every dense layer: `MoE -> dense -> MoE` counted 1, 1, 2 instead of 1, 2, 3. `advance_decode_layer` now holds the validate-then-credit rule once, the shared forward and decode handlers use it, and `complete_dense_layer` credits the decoding members of a prefill-mode source before delegating. A request still prefilling has no decode layer to credit, and a PREFILL-role batch carries none, so the disaggregated roles are unchanged. Tests: a dense layer executed for a mixed source (+1 for the decoder, 0 for the prefiller, pure-prefill control credits nothing), the routed -> dense -> routed sequence, and a real-loop hybrid-layer variant whose credits are read before the token rollout resets them. On the pre-fix source that variant peaks four decode tokens at 3 of 4 layers. External review finding C35-01; evidence in task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_review_corrections.md. --- frontier/scheduler/utils/collective_timing.py | 15 +++ frontier/scheduler/utils/decode_collective.py | 9 +- frontier/scheduler/utils/dense_metrics.py | 20 +++- .../scheduler/utils/forward_collective.py | 9 +- .../test_monolithic_mixed_forward_runtime.py | 105 +++++++++++++++++- .../test_monolithic_mixed_forward_sync.py | 77 +++++++++++++ 6 files changed, 218 insertions(+), 17 deletions(-) diff --git a/frontier/scheduler/utils/collective_timing.py b/frontier/scheduler/utils/collective_timing.py index 4e93cdc2..d72bdc73 100644 --- a/frontier/scheduler/utils/collective_timing.py +++ b/frontier/scheduler/utils/collective_timing.py @@ -57,6 +57,21 @@ def validate_decode_layer_advance(requests: Iterable[Any], total_layers: int) -> ) +def advance_decode_layer(requests: Iterable[Any], total_layers: int) -> None: + """Credit one completed layer to each request, validating the whole set first. + + Every layer-completion path owes each decoding request exactly one credit + per executed layer, so the rule lives here rather than at each caller. + Validation runs over the full selection before any counter moves, so a + request that cannot advance leaves its peers untouched. + """ + + requests = list(requests) + validate_decode_layer_advance(requests, total_layers) + for request in requests: + request.mb_on_step_layer_count_increment(num_layers_completed=1) + + def prepare_prefill_final_timing( execution_time: Any, component_times_ms: Iterable[float], diff --git a/frontier/scheduler/utils/decode_collective.py b/frontier/scheduler/utils/decode_collective.py index e1a026b9..bcf944ef 100644 --- a/frontier/scheduler/utils/decode_collective.py +++ b/frontier/scheduler/utils/decode_collective.py @@ -10,7 +10,7 @@ attention_delay_seconds, prepare_decode_final_timing, select_active_batch, - validate_decode_layer_advance, + advance_decode_layer, ) from frontier.scheduler.utils.request_selection import collect_active_requests @@ -88,12 +88,9 @@ def handle_decode_sync_collective( predictor = stage_scheduler._execution_time_predictor active_requests = collect_active_requests(dp_batches.values()) if not layer_advance_done: - validate_decode_layer_advance( - active_requests, - scheduler._config.replica_config.model_config.num_layers, + advance_decode_layer( + active_requests, scheduler._config.replica_config.model_config.num_layers ) - for request in active_requests: - request.mb_on_step_layer_count_increment(num_layers_completed=1) num_layers = predictor._num_layers_per_pipeline_stage bounds_getter = getattr(scheduler, "get_pipeline_stage_layer_bounds", None) diff --git a/frontier/scheduler/utils/dense_metrics.py b/frontier/scheduler/utils/dense_metrics.py index da4715e7..1150dfab 100644 --- a/frontier/scheduler/utils/dense_metrics.py +++ b/frontier/scheduler/utils/dense_metrics.py @@ -8,10 +8,12 @@ from frontier.entities.execution_time import ExecutionTime from frontier.entities.stage_execution_time import StageExecutionTime +from frontier.scheduler.utils.collective_timing import advance_decode_layer from frontier.scheduler.utils.execution_time_metrics import ( build_metrics_execution_time, build_single_layer_metrics_execution_time, ) +from frontier.scheduler.utils.request_selection import collect_active_requests def complete_dense_layer( @@ -25,8 +27,24 @@ def complete_dense_layer( phase: str, metrics_store: Any, ) -> list: - """Advance a dense layer through the scheduler's existing phase handler.""" + """Advance a dense layer through the scheduler's existing phase handler. + + A dense layer completes per source, outside the shared forward completion + that credits a routed layer to every decoding request in the cohort. The + decode handler credits its own batch, but the prefill handler credits + nothing, so a decoding request carried in a prefill-mode batch (chunked + prefill mixes them) would miss this layer. Credit it here, once per + executed layer; a request still prefilling has no decode layer to credit. + """ if phase == "prefill": + advance_decode_layer( + ( + request + for request in collect_active_requests([batch]) + if request.is_prefill_complete + ), + scheduler._config.replica_config.model_config.num_layers, + ) return scheduler.on_prefill_sync_collective( time, replica_id, diff --git a/frontier/scheduler/utils/forward_collective.py b/frontier/scheduler/utils/forward_collective.py index a157a260..aff2d7de 100644 --- a/frontier/scheduler/utils/forward_collective.py +++ b/frontier/scheduler/utils/forward_collective.py @@ -4,7 +4,7 @@ from typing import Any -from frontier.scheduler.utils.collective_timing import validate_decode_layer_advance +from frontier.scheduler.utils.collective_timing import advance_decode_layer from frontier.scheduler.utils.forward_sync_state import source_forward_mode from frontier.scheduler.utils.request_selection import collect_active_requests @@ -84,10 +84,9 @@ def handle_forward_sync_collective( for request in collect_active_requests(source_batches.values()) if request.is_prefill_complete ] - num_layers = scheduler._config.replica_config.model_config.num_layers - validate_decode_layer_advance(decoding_requests, num_layers) - for request in decoding_requests: - request.mb_on_step_layer_count_increment(num_layers_completed=1) + advance_decode_layer( + decoding_requests, scheduler._config.replica_config.model_config.num_layers + ) stage_layer_end = _stage_layer_end(scheduler, stage_id) next_layer_id = layer_id + 1 diff --git a/tests/integration/test_monolithic_mixed_forward_runtime.py b/tests/integration/test_monolithic_mixed_forward_runtime.py index 8f7e90cb..56be25c9 100644 --- a/tests/integration/test_monolithic_mixed_forward_runtime.py +++ b/tests/integration/test_monolithic_mixed_forward_runtime.py @@ -22,7 +22,11 @@ import pytest -def test_shared_monolithic_forward_completes_every_request(tmp_path): +@pytest.fixture(scope="module") +def evidence(tmp_path_factory): + """Run every variant once in a child interpreter and return its evidence.""" + + tmp_path = tmp_path_factory.mktemp("shared_forward") # The child must import the same checkout this test file came from, not # whichever tree an editable install happens to point at. repo_root = Path(__file__).resolve().parents[2] @@ -43,18 +47,38 @@ def test_shared_monolithic_forward_completes_every_request(tmp_path): ) (tmp_path / "run.log").write_text(result.stdout) assert result.returncode == 0, result.stdout[-15000:] - evidence = json.loads((tmp_path / "shared_forward_evidence.json").read_text()) + return json.loads((tmp_path / "shared_forward_evidence.json").read_text()) + + +def test_shared_monolithic_forward_completes_every_request(evidence): # The run has to reach the shape under test, or it proves nothing. assert evidence["mixed_phase_cohorts"] > 0, evidence +def test_a_mixed_batch_crossing_a_dense_layer_is_credited_every_layer(evidence): + """`MoE -> dense -> MoE` in the real loop, with the credits read before reset. + + A dense layer completes per source, outside the shared completion that + credits routed layers, so a decoding request carried in a prefill-mode + batch used to miss it. Every decode token of every request must reach the + full layer count, and the run must actually have driven a mixed batch + through the dense layer for that to mean anything. + """ + + hybrid = evidence["hybrid_layers"] + assert hybrid["dense_layers"] == [1], hybrid + assert hybrid["mixed_dense_completions"] > 0, hybrid + assert hybrid["decode_tokens_credited"] > 0, hybrid + assert hybrid["layer_credit_peaks"] == {str(hybrid["num_layers"]): hybrid["decode_tokens_credited"]}, hybrid + + # Requests chosen so that chunked prefill leaves one lane prefilling while the # other has already started decoding: unequal prefill lengths, unequal decode # budgets, all arriving at once. REQUEST_SHAPES = ((32, 4), (16, 4), (24, 3), (16, 3)) -def _build_config(root, patch): +def _build_config(root, patch, *, moe_layers_enum=None): import pandas as pd from frontier.config import ( @@ -72,6 +96,14 @@ def _build_config(root, patch): from tests.integration.test_pr33_nondummy_acceptance import _model, _profiles model = _model("moe") + if moe_layers_enum is not None: + # Interleave dense layers into the MoE model: the layers left out of + # the map run an ordinary FFN and complete per source. + model.moe_layers_enum = moe_layers_enum + model._moe_layer_ids_cache = None + # A mixed model names its dense FFN width separately; the profile + # rows above are keyed by the same width, so the lookup still hits. + model.dense_mlp_hidden_dim = model.mlp_hidden_dim original = BaseModelConfig.create_from_name patch.setattr( BaseModelConfig, @@ -178,7 +210,7 @@ def _drained(room) -> bool: return True -def run_case(root: Path, *, reporting: bool): +def run_case(root: Path, *, reporting: bool, moe_layers_enum: str | None = None): from frontier.entities import Request from frontier.request_generator.synthetic_request_generator import ( SyntheticRequestGenerator, @@ -188,7 +220,9 @@ def run_case(root: Path, *, reporting: bool): from frontier.types import ClusterType with pytest.MonkeyPatch.context() as patch: - model, predictor_config, replica, classes = _build_config(root, patch) + model, predictor_config, replica, classes = _build_config( + root, patch, moe_layers_enum=moe_layers_enum + ) ( VllmV1SchedulerConfig, ClusterConfig, @@ -280,6 +314,34 @@ def observe_wave(scheduler, *, mode, batch, layer_id, cohort_batches=None, **kw) patch.setattr(bcs, "schedule_layer_wave", observe_wave) + # Layer credits, read as they are given: the counter is reset when a + # token rolls out, so the terminal state cannot show whether every + # layer was credited. Each credit is keyed by the decode token it + # belongs to, and the peak per token is what must reach num_layers. + credits: list[tuple[int, int, int]] = [] + real_increment = Request.mb_on_step_layer_count_increment + + def observe_increment(self, num_layers_completed=1): + real_increment(self, num_layers_completed=num_layers_completed) + credits.append( + (self.id, self.current_decode_token_index, self.completed_layer_count) + ) + + patch.setattr(Request, "mb_on_step_layer_count_increment", observe_increment) + + # Dense completions whose prefill-mode source also carries a request + # that has finished its own prefill: the shape the credit used to miss. + mixed_dense_completions = [0] + real_dense = bcs.complete_dense_layer + + def observe_dense(scheduler, **kwargs): + live = [r for r in kwargs["batch"].requests if not r.completed] + if kwargs["phase"] == "prefill" and any(r.is_prefill_complete for r in live): + mixed_dense_completions[0] += 1 + return real_dense(scheduler, **kwargs) + + patch.setattr(bcs, "complete_dense_layer", observe_dense) + simulator = Simulator(config) # The one injection point: wrap the predictor to record which batch each @@ -307,14 +369,37 @@ def observe_predict(batch, stage_id, cluster_type=None, **kwargs): ClusterType.MONOLITHIC ) mixed = [c for c in cohorts if len(set(c["members"].values())) > 1] + peaks: dict[tuple[int, int], int] = {} + for request_id, token_index, count in credits: + key = (request_id, token_index) + peaks[key] = max(peaks.get(key, 0), count) + peak_histogram: dict[str, int] = {} + for peak in peaks.values(): + peak_histogram[str(peak)] = peak_histogram.get(str(peak), 0) + 1 evidence = { "reporting": reporting, + "num_layers": model.num_layers, + "dense_layers": [ + layer for layer in range(model.num_layers) + if not model.is_moe_layer(layer) + ], "total_cohorts": len(cohorts), "mixed_phase_cohorts": len(mixed), + "mixed_dense_completions": mixed_dense_completions[0], + "decode_tokens_credited": len(peaks), + # peak layer count -> how many (request, decode token) pairs hit it + "layer_credit_peaks": peak_histogram, "completed_requests": sum(request.completed for request in requests), "makespan": simulator._time, } + # Every decode token of every request was credited exactly num_layers + # times, whichever handler completed each layer. A missed dense layer + # shows up here as a peak one short. + assert credits, evidence + assert all(count <= model.num_layers for _, _, count in credits), evidence + assert set(peaks.values()) == {model.num_layers}, evidence + # Every request finishes, exactly once, with every token accounted for. assert all(request.completed for request in requests), evidence for request, (prefill, decode) in zip(requests, REQUEST_SHAPES): @@ -371,13 +456,23 @@ def main(root: Path) -> None: assert ( evidence["on"]["mixed_phase_cohorts"] == evidence["off"]["mixed_phase_cohorts"] ) + # The same loop with a dense layer between routed layers: the credit path + # differs per layer kind, and a mixed batch must cross both kinds. + hybrid_root = root / "hybrid_layers" + hybrid_root.mkdir(parents=True, exist_ok=True) + hybrid = run_case(hybrid_root, reporting=False, moe_layers_enum="0,2,3") + assert hybrid["mixed_dense_completions"] > 0, hybrid + merged = dict(evidence["off"]) merged["reporting_variants"] = evidence + merged["hybrid_layers"] = hybrid (root / "shared_forward_evidence.json").write_text( json.dumps(merged, indent=2) + "\n" ) print("mixed_phase_cohorts:", merged["mixed_phase_cohorts"]) print("completed_requests:", merged["completed_requests"]) + print("hybrid mixed_dense_completions:", hybrid["mixed_dense_completions"]) + print("hybrid layer_credit_peaks:", hybrid["layer_credit_peaks"]) if __name__ == "__main__": diff --git a/tests/unit/test_monolithic_mixed_forward_sync.py b/tests/unit/test_monolithic_mixed_forward_sync.py index 1747cc0c..7de4563e 100644 --- a/tests/unit/test_monolithic_mixed_forward_sync.py +++ b/tests/unit/test_monolithic_mixed_forward_sync.py @@ -565,6 +565,83 @@ def test_a_dense_layer_labels_each_source_by_its_own_phase() -> None: } +@pytest.mark.parametrize("lane_zero_phase", ["mixed", "prefill"]) +def test_a_dense_layer_credits_only_the_requests_that_are_decoding( + lane_zero_phase: str, +) -> None: + """A dense layer completes per source, and each source credits its decoders. + + The decoding member of a mixed prefill-mode source advances by one layer, + the request still prefilling does not, and the pure-decode peer lane + advances by one through its own handler. A pure-prefill source is the + control: it has nothing to credit, and nothing is credited. + """ + + scheduler, _predictor, _context, stages = _build_scheduler( + dense_layers=frozenset({1}) + ) + batches = {0: PHASE_BUILDERS[lane_zero_phase](5), 1: _decode_batch(2)} + for lane, batch in batches.items(): + _admit(stages, batch, lane) + events: list = [] + for lane, batch in batches.items(): + events.extend(_enter(scheduler, batch, lane, 1, None)) + assert [type(event) for event in events] == [DenseLayerCompleteEvent] * 2 + # Completion runs through the real event handler, per source. + for event in events: + event.handle_event(_global(scheduler), None) + + decoding = [r for r in batches[0].requests if r.is_prefill_complete] + prefilling = [r for r in batches[0].requests if not r.is_prefill_complete] + assert len(decoding) == (1 if lane_zero_phase == "mixed" else 0) + assert [r.completed_layer_count for r in decoding] == [1] * len(decoding) + assert [r.completed_layer_count for r in prefilling] == [0] + assert [r.completed_layer_count for r in batches[1].requests] == [1, 1] + + +def test_a_mixed_source_is_credited_once_per_layer_across_routed_and_dense() -> None: + """`MoE -> dense -> MoE`: the decoding member counts 1, 2, 3 -- not 1, 1, 2. + + A routed layer completes once for the whole cohort and a dense layer once + per source, so the two paths credit through different handlers. What the + request sees must not depend on which one ran. + """ + + scheduler, _predictor, _context, stages = _build_scheduler( + dense_layers=frozenset({1}) + ) + batches = {0: _mixed_batch(4, 1), 1: _decode_batch(2)} + prefilling, decoding = batches[0].requests + peers = list(batches[1].requests) + for lane, batch in batches.items(): + _admit(stages, batch, lane) + + credits = [] + for layer_id in range(3): + events: list = [] + for lane, batch in batches.items(): + events.extend(_enter(scheduler, batch, lane, layer_id, None)) + completions = [ + event + for event in events + if isinstance( + event, + ( + PrefillSyncCollectiveEvent, + DecodeSyncCollectiveEvent, + DenseLayerCompleteEvent, + ), + ) + ] + assert len(completions) == (2 if layer_id == 1 else 1), events + for event in completions: + event.handle_event(_global(scheduler), None) + credits.append(decoding.completed_layer_count) + assert prefilling.completed_layer_count == 0 + assert [peer.completed_layer_count for peer in peers] == [layer_id + 1] * 2 + assert credits == [1, 2, 3] + + def test_a_disabled_metrics_store_costs_the_run_nothing() -> None: """Reporting is demand-driven at this boundary. From ca1b9b6e2a55395f9e5ce8c96d000eb9684fbdb1 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 20:08:49 +0800 Subject: [PATCH 34/80] test(profiling): pass the FP8 block shape and skip the CPU boundary tests without torch The FP8 native case passed `block_dims` but not `block_shape`, so the expert kernels read the block-quantized scales through their per-tensor path rather than the production block-quantized invocation that `profile_fused_moe_kernel` uses. Pass it, and pin on CPU that both GEMM invocations receive the block shape (and `None` when omitted), so a finite output cannot hide the wiring again. The CPU boundary module imported torch unconditionally and added a collection error to the minimal simulator environment; it now skips through `pytest.importorskip`. The W6 report restates the native result as seven reference comparisons at zero tolerance plus one FP8 structural check (FP8 numerics not established), replaces the legacy-equals-functional scope claim with a scope table (the functional entry aligns inside `fused_experts`, the low-level path aligns before the timed step), and the profiling guide says the two backends time different envelopes. The corrected native check is not re-run. External review findings C35-02, C35-03, C35-04. --- docs/profiling/README.md | 11 +++- ...t_2026-09-22_w6_fused_expert_arithmetic.md | 62 +++++++++++++++---- .../test_moe_fused_expert_numerical_parity.py | 4 ++ .../unit/test_moe_fused_expert_arithmetic.py | 48 ++++++++++++-- 4 files changed, 108 insertions(+), 17 deletions(-) diff --git a/docs/profiling/README.md b/docs/profiling/README.md index a947a277..4f7aee4c 100644 --- a/docs/profiling/README.md +++ b/docs/profiling/README.md @@ -216,7 +216,16 @@ expert GEMM with the routing weights applied, and the local reduction over one token's top-k expert outputs. That reduction is a per-token sum, not a collective, so it carries no communication cost. `MOE_FAMILY` has no separate operator for it, and the vLLM functional backend has always included it, so -counting it here counts it exactly once on both backends. +counting it here counts the reduction once on each backend. + +The two backends still time different envelopes. The low-level path runs block +alignment (`moe_align_block_size`) once before the timed step on caller-prepared +buffers; vLLM's `fused_experts` aligns inside the call, per chunk, with its own +workspace. A `moe_grouped_gemm` row from the functional backend therefore +includes preparation work that a low-level row does not, while Frontier adds +`moe_shuffling` as a separate term for both. Treat rows from the two backends as +different measurements under one operator name, and do not mix them in one +dataset. Before 2026-09-22 the vLLM 0.10.x low-level profiling path omitted the gated activation and the reduction. Rows produced by that path under-measure diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md index 4b77b716..cc13acd1 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md @@ -10,6 +10,7 @@ Date: 2026-09-22. Branch `fix/issue26-correctness-pr`, worktree | 2026-09-22 | Created: reachability check, magnitude estimate, source repair, CPU validation. Native GPU validation NOT_RUN. | | 2026-09-22 | Artifact identity decided as document-only. Native parity test added and submitted to an H800 worker as `exp-0922-140423-075005`; result pending. | | 2026-09-22 | Native parity PASS on H800 under `codesign`: `exp-0922-145047-660565`, 8 of 8 at `rtol=0, atol=0`. Three earlier attempts and their causes recorded in section 8. | +| 2026-09-22 | External review C35-02/C35-03: section 5 replaces the scope-equivalence claim with a scope table; section 8 restates the native result as seven reference comparisons plus one FP8 structural check, and records that the FP8 test omitted `block_shape` (test corrected; native rerun NOT_RUN). | ## 1. Is the defect reachable @@ -109,16 +110,38 @@ and the local top-k reduction. Both profile methods time the whole step — `_collect_cuda_event_stats` brackets `step_fn`, and `_collect_record_function_stats` wraps it in one `vidur_moe_grouped_gemm` scope. -This makes the legacy path measure the same scope the functional path already -measured: vLLM's `fused_experts` returns reduced hidden states, so `moe_sum` was -always inside the functional measurement. The repair removes a disagreement -between the two backends rather than creating one. +The repair makes the two backends agree on the **expert arithmetic**: vLLM's +`fused_experts` returns reduced hidden states, so the gated activation and +`moe_sum` were always inside the functional measurement, and the legacy path +now computes them too. It does **not** make the two timed scopes equal, and no +record may claim that it does (corrected 2026-09-22, external review C35-02; +the earlier text here said "the same scope"): + +| Work | Corrected legacy timed iteration (`_step` at `moe_vllm_kernel.py:920`) | Functional timed entry (`_step` at `:837`, calling `fused_experts`) | +| --- | --- | --- | +| GEMM1, gated activation, optional activation quantization, GEMM2, local top-k reduction | Included | Included, inside the complete expert operation | +| Block alignment / sorting (`moe_align_block_size`) | Outside: computed once before `_step` (`:897`) on caller-prepared buffers | Inside: vLLM 0.10.2 `fused_experts_impl` aligns per chunk (`fused_moe.py:1718`); a newer functional backend must be audited, not assumed | +| Internal allocations and chunk management | Caller-prepared buffers, one call | Entry-specific (`VLLM_FUSED_MOE_CHUNK_SIZE` loop, internal workspace); not assumed identical | +| DP/TP/EP communication | None added by the local `moe_sum` | None inferred from the local sum; inspect the selected entry | + +This matters because Frontier predicts `moe_shuffling` and `moe_grouped_gemm` +as separately additive terms in both accounting paths +(`time_components.py:505,508,531,534`; `moe_operator_times.py:129-143`). A +functional-backend row that already owns alignment inside `moe_grouped_gemm` +cannot simply be added to an independent shuffling prediction. Whether the +shuffling predictor is populated for functional-backend datasets was not +verified in W6: the boundary mismatch is confirmed, a live numerical double +count is not claimed. Numerical tensor parity (section 8) and timing-scope +parity are separate statements, reported separately. Frontier's `MOE_FAMILY` (`frontier/operators/families.py:77`) has exactly four operators — `moe_gating_linear`, `moe_gating_routing_topk`, `moe_shuffling`, `moe_grouped_gemm` — and none of them represents the reduction. Counting it -inside `moe_grouped_gemm` counts it exactly once without adding a fifth -operator, a new profiling column, or a new trained model. +inside `moe_grouped_gemm` counts the reduction once on each backend without +adding a fifth operator, a new profiling column, or a new trained model. Old +legacy rows remain incomplete, new rows cannot be told apart from the CSV +metadata alone (the user's decision: no metadata change), and re-profiling is +the remedy. Noted for the record, because it differs: the Frontier-instrumented reference vLLM puts `moe_sum` *outside* its `record_function("moe_grouped_gemm")` scope @@ -252,11 +275,28 @@ and both report `functional_fused_experts`. The pinned profiling range is | `test_repeated_invocations_do_not_reuse_a_stale_result` | PASSED | | `test_fp8_path_runs_on_the_gated_activation` | PASSED | -All eight compare at `rtol=0, atol=0`, so the repaired profiling path reproduces -vLLM 0.10.2's `fused_experts` output bit for bit on the production Qwen-A3B-30B -shapes at 4096 and 4097 tokens for two EP ranks, on the uneven-occupancy -boundary case at two top-k values, across repeated invocations, and on the FP8 -path. +Eight native tests passed: **seven reference-output comparison cases at zero +tolerance** (`rtol=0, atol=0` against vLLM 0.10.2's `fused_experts`: the +production Qwen-A3B-30B shapes at 4096 and 4097 tokens for two EP ranks, the +uneven-occupancy boundary case at two top-k values, and repeated invocations) +**and one FP8 structural/finite-output check**. The FP8 test asserts that the +quantizer receives the gated activation of shape `(M * top_k, width)` and that +the output is finite; it compares against no reference. **FP8 numerical +equivalence is not established.** (Wording corrected 2026-09-22, external review +C35-03; the earlier sentence here, "all eight compare at `rtol=0, atol=0`", +overstated the run.) + +As submitted, the FP8 test also omitted `block_shape=block_shape` when calling +`_run_fused_moe_iteration`, while passing `block_dims`. `block_dims` selects the +activation-quantization group; `block_shape` is what reaches the two expert +kernel invocations, and `profile_fused_moe_kernel` forwards it. The kernel +therefore read the block-quantized scales through its per-tensor path, so the +run above did not exercise the production block-quantized invocation. Corrected +2026-09-22: the test now passes `block_shape`, and +`tests/unit/test_moe_fused_expert_arithmetic.py` pins on CPU that both GEMM +invocations receive the block shape (and `None` when it is omitted). The +corrected native check has **not** been re-run: `NOT_RUN`, one H800 under +`codesign`, awaiting the user's go. ### Attempts diff --git a/tests/integration/test_moe_fused_expert_numerical_parity.py b/tests/integration/test_moe_fused_expert_numerical_parity.py index 9ed53711..8e68c728 100644 --- a/tests/integration/test_moe_fused_expert_numerical_parity.py +++ b/tests/integration/test_moe_fused_expert_numerical_parity.py @@ -377,6 +377,10 @@ def observing_quantize(tensor, *, group_size): w1_scale=w1_scale, w2_scale=w2_scale, use_fp8=True, + # The production block-quantized path hands the kernel its + # block shape; without it the kernel reads the scales as + # per-tensor and this check would exercise a different path. + block_shape=block_shape, ) finally: kernel.quantize_activations_to_fp8 = original_quantize diff --git a/tests/unit/test_moe_fused_expert_arithmetic.py b/tests/unit/test_moe_fused_expert_arithmetic.py index 125dc9df..136c826f 100644 --- a/tests/unit/test_moe_fused_expert_arithmetic.py +++ b/tests/unit/test_moe_fused_expert_arithmetic.py @@ -13,9 +13,10 @@ from types import SimpleNamespace import pytest -import torch -from frontier.profiling.moe import moe_vllm_kernel as kernel +torch = pytest.importorskip("torch", reason="the fused-MoE profiler is built on PyTorch") + +from frontier.profiling.moe import moe_vllm_kernel as kernel # noqa: E402 NUM_TOKENS = 5 @@ -83,13 +84,16 @@ def native_stubs(monkeypatch, expert_problem): calls = [] topk_ids = expert_problem.topk_ids - def fake_invoke_kernel(*, A, B, C, topk_weights, mul_routed_weight, top_k, **_): + def fake_invoke_kernel( + *, A, B, C, topk_weights, mul_routed_weight, top_k, block_shape=None, **_ + ): calls.append( SimpleNamespace( name="invoke_kernel", A=A, mul_routed_weight=mul_routed_weight, top_k=top_k, + block_shape=block_shape, ) ) rows = C.view(NUM_TOKENS, TOP_K, C.shape[-1]) @@ -126,12 +130,12 @@ def fake_moe_sum(inp, out): return calls -def _run(problem): +def _run(problem, **overrides): cache1 = torch.empty(NUM_TOKENS, TOP_K, 2 * EXPERT_HIDDEN_DIM) cache2 = torch.empty(NUM_TOKENS * TOP_K, EXPERT_HIDDEN_DIM) cache3 = torch.empty(NUM_TOKENS, TOP_K, HIDDEN_DIM) out = torch.empty(NUM_TOKENS, HIDDEN_DIM) - kernel._run_fused_moe_iteration( + arguments = dict( A=problem.A, w1=problem.w1, w2=problem.w2, @@ -147,6 +151,8 @@ def _run(problem): config={"BLOCK_SIZE_M": 16}, block_dims=None, ) + arguments.update(overrides) + kernel._run_fused_moe_iteration(**arguments) return SimpleNamespace(cache1=cache1, cache2=cache2, cache3=cache3, out=out) @@ -253,6 +259,38 @@ def fake_quantize(tensor, *, group_size): assert seen["group_size"] == 64 +def test_the_block_shape_reaches_both_expert_gemms( + monkeypatch, expert_problem, native_stubs +): + """Under block-quantized FP8 both GEMMs must see the same block shape. + + The kernel reads its scales per block only when told the block shape; a + call that drops it silently runs the per-tensor path instead. The native + FP8 check depends on this boundary, so it is pinned here on CPU. + """ + + monkeypatch.setattr( + kernel, + "quantize_activations_to_fp8", + lambda tensor, *, group_size: (tensor, torch.ones(1)), + ) + + _run(expert_problem, block_dims=(128, 64), use_fp8=True, block_shape=[128, 64]) + + gemms = [call for call in native_stubs if call.name == "invoke_kernel"] + assert len(gemms) == 2 + assert [call.block_shape for call in gemms] == [[128, 64], [128, 64]] + + +def test_a_missing_block_shape_reaches_the_gemms_as_none(expert_problem, native_stubs): + """The default is observable, so the FP8 test cannot omit it unnoticed.""" + + _run(expert_problem) + + gemms = [call for call in native_stubs if call.name == "invoke_kernel"] + assert [call.block_shape for call in gemms] == [None, None] + + def test_repeated_iterations_do_not_leak_a_previous_result( expert_problem, native_stubs ): From 57ffa5b623d7d47d9597e88e483ed7fa4d4dc118 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 20:08:49 +0800 Subject: [PATCH 35/80] docs(records): apply the 2026-09-22 external review to the records and the Step 9 plan Records: D2's scope-identifier clause marked superseded by the dated documentation-only decision; the W6 result stated as seven comparisons plus one FP8 structural check; the progress status table brought current; the review request and its scope recorded; a finding-by-finding disposition table (C34-01, C35-01..05, P9-01..06) with evidence; the verification report for packages B-E. Step 9 plan: section renumbered from 17 to 18 (the source index already held 17). The room-only hook rule and the "steady state needs no change" claim are replaced by the reference's engine-iteration state table and its preconditions; K1, K3 as written and stride keys are rejected as an acceptance basis, with the counterexample reproduced on this branch's balancer and six invariants for the rule to be chosen at a design checkpoint; the instrumentation covers the whole emission -> coordinator -> frontend -> routing chain; T1 uses a CPU reference-loop oracle and a causal join instead of boundary indices; the negative control is an explicit test-only variant because the unmodified constructor rejects PP2; PP3 gets a fixture with a valid layer count; the work graph and C1-C5 are revised. No Step 9 source change; execution not started. --- .../design.md | 133 +++++++++++---- .../plan.md | 155 ++++++++++++++---- .../progress.md | 29 +++- .../requirements.md | 26 ++- .../review.md | 37 ++++- .../summary.md | 5 +- ...st_report_2026-09-22_review_corrections.md | 98 +++++++++++ 7 files changed, 398 insertions(+), 85 deletions(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_review_corrections.md diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md index d1aeacfe..908363c5 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md @@ -10,6 +10,7 @@ scope decisions and the pre-measurement expectation for that package. | 2026-09-22 | Created. Source-backed design for Checkpoint D's W3 half, with the scope decisions and their evidence. | | 2026-09-22 | Restructured into per-work-package sections and added the W4 design, with the report-key identity measured at the emission boundary. | | 2026-09-22 | Added the W9 design: vLLM 0.10.2 count publication under pipeline parallelism, the equivalence argument, the schedule-only gap, the report-key options, and the discriminating scenario. Analysis only; implementation pending. | +| 2026-09-22 | W9 corrected per the external review (P9-01..P9-06): the steady-state claim withdrawn in favor of stated preconditions and the engine-iteration state table; K1, K3-as-written and stride keys rejected as acceptance basis, with the reproduced counterexample and six invariants; the discriminating scenario qualified as a conditional witness; planned edits extended (observation record, instrumentation chain, PP3 fixture, CPU reference loop, controls). Still analysis only. | # W3 — one shared monolithic forward @@ -268,7 +269,7 @@ than confirming it. # W9 — schedule-time reports under pipeline parallelism -Analysis recorded before implementation (plan §17). Everything here is read from +Analysis recorded before implementation (plan §18; corrected 2026-09-22 per the external review, see plan §18.11). Everything here is read from the pinned checkout `.real-engine/vLLM-BS` at `ea95f571e` and from the current branch; nothing has been measured on a GPU yet. @@ -293,27 +294,54 @@ branch; nothing has been measured on a GPU yet. | At admission a batch carries the provisional per-lane creation counter; the Replica-scoped key is assigned when the sync room opens during execution. | `base_replica_scheduler.py:460-467`, `forward_sync_state.py:152-158` | | The only report boundary today is `on_replica_batch_end`, keyed by `ForwardSyncState.get_step_id(batch)`. | `vllm_load_balancing_cluster_scheduler.py`; W4 above | -## Why the steady state already matches +## Where the steady state matches, and on what conditions -With a full batch queue of depth `PP`, vLLM iteration `k` schedules `B_k`, -then completes `B_(k-1)`, then publishes. Frontier admits `B_k` when -`B_(k-2)` ends (the counter drops below `PP`) and reports when `B_(k-1)` ends, -with `B_k` already admitted. Both sides therefore publish the state "after -`B_k` admission and `B_(k-1)` completion" at the completion of `B_(k-1)`. The -same holds for the populations: admission moves requests into `running` on both -sides, completion removes finished ones. **No change is needed for the steady -state**, and the current PP=1 equivalence is the degenerate case `PP=1`. +(Corrected 2026-09-22, P9-01. The earlier text here said "no change is needed +for the steady state"; that sentence is withdrawn.) -## The gap: schedule-only iterations +The reference decides each engine iteration by a conjunction, not by queue room: -When the queue is empty (after idle, or after an empty-batch iteration) the -first admission returns without completing anything and vLLM publishes it -immediately: `waiting -n, running +n`, score `-3n`, with a key strictly greater -than the last completion's. Frontier stays silent until that batch ends, and -when a single `on_schedule` call admits two batches it never exposes the state -after the first. At PP=1 `step()` is atomic, so this cannot occur; the PP=1 -path is unaffected by construction, which is what makes a byte-identical PP=1 -fidelity check the right acceptance test. +```python +model_executed = scheduler_output.total_num_scheduled_tokens > 0 +if (model_executed + and len(batch_queue) < batch_queue_size + and not batch_queue[-1][0].done()): + return None, True # admission-only iteration +# otherwise: pop the oldest output, wait, update_from_output, then publish +``` + +With queue depth `P`, appending `B_k` to a queue holding `P-1` earlier outputs +completes `B_(k-P+1)`; the `B_(k-1)` algebra of the first draft is the `P=2` +case. Frontier's completion report at `B_(k-P+1)`'s end shows the same combined +state only when `B_k` was admitted at `B_(k-P)`'s end, no empty iteration +intervened and no completion became visible between the two boundaries. Equal +queue occupancy does not prove equal `waiting`/`running` populations; request +membership, empty schedules, completions and their visibility times must +correspond. Those preconditions are a test table (plan §18.11), not an +assumption, and the PP=1 equivalence remains the degenerate case in which the +conjunction is never true because `step()` is atomic. + +## The gap: iterations the completion report cannot represent + +| State after a scheduling attempt | Reference | W9 must represent | +| --- | --- | --- | +| Nonzero tokens, room remains, oldest not ready | Return, publish changed counts | Admission-only observation | +| Nonzero tokens, room remains, oldest already ready | Apply oldest, publish | One combined observation, not an extra admission-only report | +| Nonzero tokens, queue full | Wait/apply oldest, publish | Completion-path observation | +| Zero-token schedule, work queued | No early return | Explicit mapping of the empty iteration | +| No new work, output queued | Drain, publish | Completion-only observation | + +The admission-only row is the visible one: after idle (or an empty-batch +iteration) vLLM publishes `waiting -n, running +n`, score `-3n`, with a key +strictly greater than the last completion's, while Frontier stays silent until +that batch ends and, when a single `on_schedule` call admits two batches, never +exposes the state after the first. But `pipeline_room_remaining` tests only the +second conjunct, so a hook keyed on it has no representation for the other +rows; the hook (name fixed by D-d) must pass an observation classified by this +table, captured where the state changes, with the publish decision kept in the +existing load-report owner. At PP=1 none of these iterations occur, which is +what makes a byte-identical PP=1 fidelity check the right acceptance test for +the unchanged path. ## The report key at the admission boundary @@ -322,15 +350,38 @@ id and distinct forwards are strictly increasing. That id does not exist yet at admission — only the provisional per-lane counter does, and the dense multi-lane row in W4 shows what per-lane counters do to the latch. Options: -| Option | Rule | Reference fidelity | Cost | -| --- | --- | --- | --- | -| K1 | Schedule-time report reuses `last_report_step` (equal key). | Applies counts; never latches. Misses the reference latch of the pre-admission state when the previous completion is still unpublished (resume from idle within 100 ms). | None. | -| K3 (recommended) | The cluster scheduler relabels keys in emission order: a completion mints a label on first sight of its cohort (both lanes share it); a schedule-only admission mints a fresh label. | Strictly increasing in emission order, equal only for peer lanes of one cohort — the reference's "(wave, step) strictly increasing; peer engines share a step". | A dict and a counter inside the module; no interface change. Only comparisons are used, so PP=1 results are unchanged. | -| Stride keys (`2*cohort±1`) | Arithmetic room between completion keys. | Arbitrary factor; no room for two consecutive schedule-only admissions at PP≥3. | Rejected. | +| Option | Rule | Verdict (2026-09-22, P9-02) | +| --- | --- | --- | +| K1 | Schedule-time report reuses `last_report_step` (equal key). | Rejected as acceptance basis: applies counts but never latches, so it misses the reference latch of the pre-admission state when the previous completion is still unpublished. | +| K3 as written | A completion mints a label on first sight of its cohort; every schedule-only admission mints a fresh label. | Rejected as written: a fresh label per admission *callback* equates callback order with iteration order and gives two peer lanes of one logical iteration different keys. Counterexample below. If "fresh label" was meant per shared logical iteration, the rule must first say how that iteration is identified. | +| Stride keys (`2*cohort±1`) | Arithmetic room between completion keys. | Rejected: arbitrary factor; no room for consecutive admission-only iterations at PP≥3. | -The probe planned as P1 records `(lane, boundary, provisional id, resolved id, -load)` at PP=2 for four shapes and fixes the rule from evidence, the same way -W4's table did. +**Counterexample, reproduced on this branch's `VllmDPLoadBalancer`** (zero +initial counts; lane 0 reports `waiting=0, running=3` at 10 ms, lane 1 reports +the same at 20 ms; a request is placed at 80 ms): + +| Key assignment | Frontend snapshot before the 80 ms placement | Last publication | Selected lane | +| --- | --- | --- | --- | +| One key for both reports | `[(0,3), (0,3)]` | 70 ms | 0 (first minimum) | +| Fresh key for the second report | `[(0,3), (0,0)]` — the partial snapshot latched at 20 ms | 20 ms | 1 | + +Identical inputs and loads, different published state. A second interleaving +must also be handled: one lane completes cohort `C`, proceeds to an +admission-only observation, and the peer's completion of `C` arrives later; +reusing `C`'s old label after minting the next one breaks strict emission +order, and the native coordinator's out-of-order warning is evidence to +analyze, not something to suppress by inventing newer identities. + +The rule is chosen at the design checkpoint (plan §18.5) after the P1 probes +establish logical-iteration membership: first identify the reference-equivalent +engine iteration, then reuse an existing scheduler iteration/forward identity if +it represents it, else a derived identity or a small report-state field. It +must satisfy: (1) peer observations of one logical iteration compare equal in +any callback order; (2) a new iteration orders after the previous one, with the +key captured at the observation boundary; (3) an iteration that schedules and +completes owns one report decision; (4) suppressed unchanged reports create no +fictitious messages; (5) PP3 allows consecutive admission-only iterations +without spacing constants; (6) bookkeeping is released with the in-flight work. ## The discriminating scenario, derived @@ -353,22 +404,38 @@ chunk budget and prompt length; Frontier: `dummy_execution_time_ms`, decision D-e). Three warmups and an idle gap of at least 5 s precede the burst so both systems start from the quiet state the argument assumes. +**Conditional witness (2026-09-22, P9-04).** The algebra holds for a state with +`k_e` assigned, `a_e` admitted, no intervening completion and no further +published state; `k=(3,2), a=(3,1)` is the target, not a guaranteed live +outcome. The trace must show the actual routing order of the burst, the +admissions, the in-flight requests, later scheduling attempts and the snapshot +applied at the frontend before `r6` was routed; otherwise the slice is +`SCENARIO_NOT_REACHED`, not a scheduler mismatch. HTTP concurrency does not fix +engine-receipt order, so request ids, dispatch order and receipt order are +recorded and qualified. Chunk sizes and decode lengths are frozen explicit +values; the balancer's constants are never tuned to make the witness occur. The +pre-change control for `r6` is the explicit test-only completion-reporting +baseline (guard lifted only), because the unmodified constructor rejects PP2 +and can only report that rejection. + ## Planned edits | File | Edit | | --- | --- | -| `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py` | `on_replica_batch_scheduled(time, replica_id, replica_local_id, batch, pipeline_room_remaining)`, inert default. | -| `frontier/scheduler/replica_scheduler/base_replica_scheduler.py` | Call the hook in the MONOLITHIC/PREFILL admission loop after `self._num_running_batches += 1`, with `pipeline_room_remaining = self._num_running_batches < self._num_stages`. | -| `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py` | Drop the PP1 clause of the guard and its error text; report `lane.get_request_load()` at the hook when room remains; K1 or K3 key rule per P1. | -| `tests/unit/test_vllm_dp_load_balancer.py`, `tests/integration/test_vllm_dp_placement_runtime.py` | Guard case inverted; schedule-time report semantics; PP=2 dense and MoE cases; the discriminating scenario. | +| `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py` | `on_replica_batch_scheduled(...)`, inert default; its payload is one observation of the engine iteration classified by the state table (fields fixed at the design checkpoint), not `pipeline_room_remaining` alone. | +| `frontier/scheduler/replica_scheduler/base_replica_scheduler.py` | Call the hook in the MONOLITHIC/PREFILL admission loop where the state changes; readiness of the oldest output is read from existing DES completion state. | +| `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py` | Drop the PP1 clause of the guard and its error text; keep the publish decision here; key rule per the design checkpoint, satisfying invariants 1–6. | +| `tests/comparison/dp_placement_pp/reference_loop.py` | Independent CPU reference-loop oracle (scripted admissions, empty schedules, completions, controllable readiness) — not a copy of the hook. | +| `tests/unit/test_vllm_dp_load_balancer.py`, `tests/integration/test_vllm_dp_placement_runtime.py` | Guard case inverted; the plan §18.11 behavioral matrix (PP2 both callback orders, oldest-ready, PP3 consecutive admission-only iterations on a 6- or 12-layer fixture, full queue, empty schedule, drain, idle peers, bounded bookkeeping); the discriminating scenario against the test-only control; the C35-01 hybrid-layer credit case at PP2. | +| `.real-engine/vLLM-BS` (local branch only, D-b) | Case-gated event chain: iteration result, emitted report, coordinator receive/publish with snapshot id, frontend application, frontend routing; named `waiting`/`running`; correlation ids; buffered per-process JSONL. | | `AGENTS.md:620`, this file, `plan.md`, `progress.md`, `validation.md`, `review.md` | Wording and records. | ## Fidelity expectation, stated before measuring - Every PP=1 `vllm_load_balancing` scenario: `request_metrics.csv` value-identical and `system_metrics.json` identical after removing timestamps and run ids. - Every scenario of every other cluster scheduler: identical (the hook's default is inert; the only added work is one method call per admission). -- PP=2 with `vllm_load_balancing`: runs complete; placements differ from round-robin where the published load says they should; the discriminating request moves from `e1` to `e0`. -- Ground truth: T1 publication sequences match by boundary index; T2 placement of the discriminating request matches the fixed module and not the current one. +- PP=2 and PP=3 with `vllm_load_balancing`: runs complete with request/token/owner conservation; on the qualified discriminating scenario the corrected module sends `r6` to `e0` while the test-only completion-reporting control sends it to `e1` (round-robin inequality alone proves nothing, since PP1 `vllm_load_balancing` already differs from round-robin). +- Ground truth: on a controlled or causally matched history, emitted loads, key equality/order, coordinator snapshots and frontend-visible counts agree with the reference; natural divergence is labeled by first cause; T2 matches the corrected module in a trace-qualified slice or is recorded as `SCENARIO_NOT_REACHED`. ## What this adds, and what it is not diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md index 6f50caf9..023884a7 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md @@ -5,7 +5,8 @@ | Date | Change | | --- | --- | | 2026-09-21 | Landed the execution specification verbatim (Section "Execution Specification" below) and recorded the amendments agreed with the user before Step 0. | -| 2026-09-22 | Added amendment A12 and the Step 9 draft (§17) for PP>1 support of the opt-in vLLM DP placement; awaiting user approval. | +| 2026-09-22 | Added amendment A12 and the Step 9 draft (§18; numbered §17 until 2026-09-22, when the duplicate number was fixed) for PP>1 support of the opt-in vLLM DP placement; awaiting user approval. | +| 2026-09-22 | Step 9 section renumbered §17 → §18 (the source index already held §17). §18 corrected per the 2026-09-22 external review, P9-01..P9-06: engine-iteration state table instead of a room-only hook rule; K1, K3-as-written and stride keys rejected as acceptance basis, six key invariants; instrumentation chain and T2 qualification; CPU reference-loop oracle and valid negative controls; PP3 fixture with a valid layer count; revised work graph and C1–C5. Corrections collected in §18.11. Execution still not started. | ## Amendments (authoritative where they differ from the specification below) @@ -844,23 +845,23 @@ Under [R1]: `frontier/profiling/moe/moe_vllm_kernel.py`, `frontier/entities/stag **First action for a fresh Claude Code session:** perform Step 0, publish the new worktree branch and tracked plan, then complete the source audit in Step 1. Do not begin by cherry-picking the calibration branch or launching a GPU job. -## 17. Step 9 — PP>1 support for the opt-in vLLM DP placement +## 18. Step 9 — PP>1 support for the opt-in vLLM DP placement -**Status 2026-09-22:** the user answered every §17.7 decision the same day (verbatim in `requirements.md`); the plan below is final. No source edit and no GPU submission has been made; execution starts at `P1`/`G1` once the user confirms the start. Research followed the `codebase-design` skill (§17.10); the ground-truth comparison follows `frontier-calibration` v2 as written (§17.9). +**Status 2026-09-22:** the user answered every §18.7 decision the same day (verbatim in `requirements.md`). Later the same day an external review of PR34/PR35 (`.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_Review_2026-09-22.md`, findings P9-01..P9-06) corrected this plan; the corrections are applied in place below and collected with their evidence in §18.11. No source edit and no GPU submission has been made; execution starts at the first node of the §18.5 graph once the user confirms the start. Research followed the `codebase-design` skill (§18.10); the ground-truth comparison follows `frontier-calibration` v2 as written (§18.9). -### 17.1 Goal and acceptance criteria +### 18.1 Goal and acceptance criteria -`VllmLoadBalancingClusterScheduler` accepts `num_pipeline_stages > 1` and reproduces vLLM 0.10.2's per-iteration DP request-count publication under the batch-queue stepping path that PP>1 selects, verified against a real `vllm serve --data-parallel-size 2 --pipeline-parallel-size 2` deployment. +`VllmLoadBalancingClusterScheduler` accepts valid `num_pipeline_stages > 1` configurations and reproduces vLLM 0.10.2's per-iteration DP request-count publication under the batch-queue stepping path that PP>1 selects — one observable engine scheduling iteration and its frontend-visible load, not a counter made monotonic after the fact — verified on a controlled or demonstrably matched iteration history against a real `vllm serve --data-parallel-size 2 --pipeline-parallel-size 2` deployment. | # | Criterion | Evidence | | --- | --- | --- | -| C1 | Guard becomes `MONOLITHIC` + one Replica + `vllm_v1` + (MoE or `attn_dp == 1`); PP=2 and PP=3 runs with `vllm_load_balancing` complete every request for dense `attn_dp=1` and MoE `attn_dp=2`. | Integration cases in P4. | -| C2 | PP=1 behavior is unchanged: every existing `vllm_load_balancing` scenario has value-identical `request_metrics.csv` and identical `system_metrics.json` (timestamps/run ids removed, the Q11 rule). Other cluster schedulers are untouched (the hook is inert for them). | P5 byte comparison; Step 8 regression set rerun. | -| C3 | Semantic alignment (T1): at PP=2 the Frontier per-lane report sequence matches the vLLM per-engine `SchedulerStats` publication sequence for the same workload, compared by boundary index rather than wall-clock: same publications per engine during ramp-up, same `(running, waiting)` after each admission/completion boundary, same key-ordering pattern, no out-of-order warning in either system. | §17.6 extraction + `workflow-gap-analysis`. | -| C4 | Placement (T2): on the discriminating scenario of §17.6 the current code places the probe request differently from vLLM and the fixed code places it identically; all non-discriminating requests agree in both. | §17.6 comparison table. | -| C5 | Records: `design.md` section, `AGENTS.md:620` wording, test report, `validation.md` rows, the calibration skill's semantic-alignment rows and workflow-gap analysis. | Files listed in P6. | +| C1 | Valid PP2 and PP3 configurations (layer count divisible by PP; `MONOLITHIC`, one Replica, `vllm_v1`, MoE or `attn_dp == 1` — the PP1 clause is the only guard removed) complete every request with request/token/owner conservation, for dense `attn_dp=1` and MoE `attn_dp=2`. PP3 uses a separate CPU fixture with a valid layer count (6 or 12); the native PP2 model stays the approved 8-layer tiny Qwen3-MoE. | P4 real-loop PP2 and PP3 cases; §18.11 behavioral matrix. | +| C2 | Previously supported behavior is unchanged under the stated comparison contract: every existing PP1 `vllm_load_balancing` scenario has value-identical `request_metrics.csv` and identical `system_metrics.json` (timestamps/run ids removed, the Q11 rule), with no additional admission-only report; every other cluster scheduler, including the supported disaggregated paths, has identical event outcomes (the hook is inert for them). | P5 byte comparison; Step 8 regression set rerun. | +| C3 | For a controlled or demonstrably matched iteration history, the emitted loads, the equality/order relation of logical-iteration keys, the coordinator snapshots and the frontend-visible counts agree with the reference. Natural-history divergence is classified by first cause (arrival/delivery order, batch composition, output readiness, count calculation, key grouping, snapshot publication, frontend selection), not hidden by re-indexing. Boundary-index comparison alone is not an alignment method. | CPU reference-loop oracle (P1) + causal join of the G4 trace (§18.11 instrumentation chain) + `workflow-gap-analysis`. | +| C4 | In a trace-qualified native discriminating slice (§18.6, qualified per §18.11: the intended snapshot was applied at the frontend before the probe was routed), the corrected placement matches the reference and the explicit test-only completion-reporting control fails for the expected reason. The actual unmodified PP2 baseline is reported as rejected by its constructor, not as a placement. Otherwise the slice is `SCENARIO_NOT_REACHED` with the failed precondition named. | §18.6 comparison table with the control column. | +| C5 | Source tuples, commands, every effective setting, observation completeness, negative controls and limitations are committed and reviewable: `design.md` section, `AGENTS.md:620` wording, test report, `validation.md` rows, the calibration case records. A clean worktree and an author-written receipt are provenance evidence, not a semantic PASS. A native out-of-order warning is evidence to analyze, not proof of a simulator bug. | Files listed in P6. | -### 17.2 Reference semantics (pinned vLLM 0.10.2, `.real-engine/vLLM-BS` at `ea95f571e`) +### 18.2 Reference semantics (pinned vLLM 0.10.2, `.real-engine/vLLM-BS` at `ea95f571e`) | Fact | Source | | --- | --- | @@ -872,7 +873,7 @@ Under [R1]: `frontier/profiling/moe/moe_vllm_kernel.py`, `frontier/entities/stag | No DP+PP prohibition: per-engine `world_size = PP*TP`; the `arg_utils` assertions concern hybrid/external LB and the `mp` backend only. | `vllm/config/parallel.py:314`, `vllm/engine/arg_utils.py:1221-1269` | | Schedule log rows carry request ids and queue sizes but **no engine identity**; all DP engine processes inherit one `VLLM_FRONTIER_SCHED_LOG_PATH`. Coordinator publications are not logged. | `scheduler.py:91-92, 985-998` | -### 17.3 Frontier model today and the gap +### 18.3 Frontier model today and the gap | Fact | Source | | --- | --- | @@ -881,37 +882,54 @@ Under [R1]: `frontier/profiling/moe/moe_vllm_kernel.py`, `frontier/entities/stag | The only report boundary is `on_replica_batch_end` with key `ForwardSyncState.get_step_id(batch)`. | `vllm_load_balancing_cluster_scheduler.py`, `design.md` "The report key" | | At admission a batch carries only the provisional per-lane creation counter; the Replica-scoped key is assigned when the shared sync room opens during execution. | `base_replica_scheduler.py:460-467`, `forward_sync_state.py:152-158` | -Equivalence argument. With a full batch queue, vLLM iteration `k` = schedule `B_k` + complete `B_(k-1)` + publish. Frontier admits `B_k` at `B_(k-2)`'s end and reports at `B_(k-1)`'s end, when `B_k` is already admitted; both observe the same state. **Steady state needs no change.** The gap is the schedule-only iteration: after an idle period (or an empty-batch iteration) vLLM publishes the admission immediately (`waiting -n, running +n`, score `-3n`) with a key strictly greater than the last completion's, while Frontier stays silent until the batch ends and, when one `on_schedule` call admits two batches, never exposes the state after the first. At PP=1 `step()` is atomic, so schedule-only iterations cannot occur and the PP=1 path is unaffected by construction. The existing key cannot be reused at the admission boundary because it is not yet resolved there (the dense multi-lane INVALID row in `design.md` shows what per-lane counters do to the latch). +Equivalence argument, as corrected by P9-01. The reference decides each engine iteration by the conjunction `model_executed and len(batch_queue) < batch_queue_size and not batch_queue[-1][0].done()`; only that branch returns without applying an output, and publication follows every iteration. With queue depth `P`, appending `B_k` to a queue holding `P-1` earlier outputs completes `B_(k-P+1)` — `B_(k-1)` is the `P=2` case only — and equal queue occupancy does not by itself prove equal `waiting`/`running` populations: request membership, empty schedules, completions and the time at which each change becomes visible must correspond. The narrow hypothesis retained: in a controlled execution history Frontier already exposes the correct combined state at completion boundaries where `B_k` was admitted at `B_(k-P)`'s end and reported at `B_(k-P+1)`'s end; §18.11 states these preconditions as a test table instead of the earlier sentence "steady state needs no change", which is withdrawn. The gap is every iteration the completion report cannot represent: the admission-only iteration (vLLM publishes `waiting -n, running +n`, score `-3n`, under a key strictly greater than the last completion's, while Frontier stays silent until the batch ends and, when one `on_schedule` call admits two batches, never exposes the state after the first), the iteration whose oldest output is already ready when room remains (one combined observation, not an extra admission-only report), the zero-token iteration with queued work, and the drain iteration. At PP=1 `step()` is atomic, so none of these occur and the PP=1 path is unaffected by construction. The existing key cannot be reused at the admission boundary because it is not yet resolved there (the dense multi-lane INVALID row in `design.md` shows what per-lane counters do to the latch), and a per-callback fresh key is not a valid substitute (P9-02, §18.11). -### 17.4 Design +### 18.4 Design | Id | Decision | Content | | --- | --- | --- | -| D9-1 | Schedule-time hook | `BaseClusterScheduler.on_replica_batch_scheduled(time, replica_id, replica_local_id, batch, pipeline_room_remaining)` with an inert default, called from the MONOLITHIC/PREFILL branch of `BaseReplicaScheduler.on_schedule` right after `self._num_running_batches += 1`, with `pipeline_room_remaining = self._num_running_batches < self._num_stages`. `VllmLoadBalancingClusterScheduler` reports `lane.get_request_load()` only when room remains — at PP=1 that is never, so PP=1 behavior is unchanged. Rejected: reconstructing the report in `ReplicaScheduleEvent` after `on_schedule`, which cannot observe the intermediate state when one call admits two batches (vLLM publishes each admission in its own iteration). Name candidates under the AGENTS.md naming gate: `on_replica_batch_scheduled` (mirrors `on_replica_batch_end`; recommended) or `on_replica_admission`. | -| D9-2 | Report key | **K1** equal key (`last_report_step`): simplest, but misses the reference latch of the pre-admission state when the previous completion is still unpublished (resume from idle within 100 ms). **K3** order-preserving relabeling owned by the cluster scheduler: a completion mints a label on first sight of its cohort and both lanes of that cohort share it; a schedule-only admission mints a fresh label; labels strictly increase in emission order and are equal only for the same cohort — the reference's "(wave, step) strictly increasing, peer engines share a step". Only comparisons are used, so PP=1 behavior stays identical (C2 verifies). Recommended: K3, fixed after the P1 probe. Rejected: stride keys such as `2*cohort±1` (an arbitrary factor, and no room for two consecutive schedule-only admissions at PP≥3). | -| D9-3 | Guard | Drop `num_pipeline_stages == 1`; keep the other four clauses and the dense multi-lane rejection (its evidence is PP-independent); update the error text. | +| D9-1 | Schedule-time hook (corrected per P9-01) | Name fixed by the user (D-d): `BaseClusterScheduler.on_replica_batch_scheduled(...)`, inert default, called from the MONOLITHIC/PREFILL admission branch of `BaseReplicaScheduler.on_schedule`. What it carries is **not** decided by queue room alone: `pipeline_room_remaining` tests one of the reference's three conditions and has no representation for the iteration whose oldest output is already ready, for the zero-token iteration, or for the drain. The hook passes one observation of the engine iteration as classified by the §18.11 state table — captured where the state changes, from existing scheduling/completion ownership (DES completion state stands in for `oldest.done()`; no futures or threads) — and the decision whether to publish stays in one place, the existing load-report owner in `VllmLoadBalancingClusterScheduler`. If a small explicit observation record is needed, prefer it to callers manufacturing a vLLM-specific heuristic from queue length; do not spread partly redundant booleans through the scheduler. Rejected, unchanged: reconstructing the report in `ReplicaScheduleEvent` after `on_schedule` (cannot see the state after the first of two admissions in one call). The exact fields are fixed at the design checkpoint of §18.5, after the P1 probes. | +| D9-2 | Report key (corrected per P9-02) | **Rejected as acceptance basis:** K1 (equal key; never latches, misses the reference latch of the pre-admission state), **K3 as written** (a fresh label per admission callback equates callback order with iteration order and gives two peer lanes of one logical iteration different keys — the executable counterexample in §18.11 shows the coordinator then latches a partial snapshot and routes differently), and stride keys (`2*cohort±1`; arbitrary factor, no room for consecutive admission-only iterations at PP≥3). **Rule to be chosen at the design checkpoint:** first identify the reference-equivalent logical engine iteration, then reuse an existing scheduler iteration/forward identity if it actually represents it; otherwise a derived identity or a small additional report-state field. Not a per-batch counter because it is available; not a global identity registry because the old getter is unavailable at admission. The chosen rule must satisfy invariants I1–I6 of §18.11, and the P1 probe must establish logical-iteration membership, not print provisional/resolved Batch ids and pick whichever looks monotonic. Only comparisons are used, so PP=1 behavior stays identical (C2 verifies). | +| D9-3 | Guard | Drop `num_pipeline_stages == 1`; keep the other four clauses and the dense multi-lane rejection (its evidence is PP-independent); update the error text. The layer-partition guard (`num_layers % num_pipeline_stages == 0`, `replica_config.py`) is untouched: removing the PP1 clause is not permission to bypass the other independent guards or to add uneven partitioning. | | D9-4 | Determinism and flags | No new `EventType`, no new config flag, no balancer constant change (`design.md` event-type determinism; plan §10 "no tuning flags"). | | D9-5 | Docs | `AGENTS.md:620` ("one pipeline stage" removed), `design.md` guard row plus a section "Schedule-time reports under pipeline parallelism" with the P1 probe table, `plan.md`/`progress.md`/`validation.md`/`review.md`. | -### 17.5 Work packages (sequence `{P1, G1, G2} -> P2 -> {P3, P4, G3} -> G4 -> G5 -> P5 -> P6`) +### 18.5 Work packages (sequence revised per P9-06) -CPU packages `P*` change Frontier; ground-truth packages `G*` never change Frontier and run in parallel where the graph allows. GPU queue time on `codesign` is the expected critical path, so `G1`/`G2` start with `P1`. +```text +existing PR corrections and scoped regressions (done 2026-09-22, review packages A–E) + -> publish the amended W9 plan and observation schema (this section) + -> explicit W9 start approval from the user + -> {expanded P1 CPU reference-loop and Frontier probes, + G1 local instrumentation, + G2 case and extraction preparation} + -> design checkpoint: source-backed state table + key-grouping rule (D9-1 fields, D9-2 rule) + -> {P2 implementation + P3 focused tests, + G3 native PP1 infrastructure smoke once G1/G2 are ready} + -> P4 real-loop PP2 and valid PP3 tests + -> G4 qualified native PP2 run + -> G5 causal comparison and gap classification + -> P5 unchanged-path regressions + -> P6 final records and review handoff +``` + +CPU packages `P*` change Frontier; ground-truth packages `G*` never change Frontier and run in parallel where the graph allows. GPU queue time on `codesign` is the expected critical path, so `G1`/`G2` start with `P1`. P2 implements the selected state model, not a test that repeats `pipeline_room_remaining`; the earlier "about 80 lines" estimate and "K1/K3 decided from P1" no longer constrain the decision. If the reference evidence shows the required state cannot be represented by a small change, record the missing responsibility and request a scoped decision before broadening the design; do not build a second trace framework or migrate unrelated scheduler code. | Package | Content | Acceptance | | --- | --- | --- | -| P1 Probe | Scratch script patches `on_replica_batch_end` and a temporary admission hook to record `(lane, boundary, provisional id, resolved id, load)` at PP=2 for four shapes (MoE `attn_dp=2` burst and staggered, MoE `attn_dp=1`, dense `attn_dp=1`). No source change. | Table added to `design.md`; K1/K3 decided from it. | -| P2 Implement | D9-1..D9-3 in `base_cluster_scheduler.py`, `base_replica_scheduler.py`, `vllm_load_balancing_cluster_scheduler.py` (about 80 lines). | Existing unit tests pass except the intentionally inverted guard case. | -| P3 Unit | `tests/unit/test_vllm_dp_load_balancer.py`: guard param at `:538` becomes positive; PP=1 never reports at schedule time; PP=2 reports iff room remained; key ordering/equality semantics; helper `num_pipeline_stages` parameter. | New tests fail before P2 and pass after. | -| P4 Integration | `tests/integration/test_vllm_dp_placement_runtime.py`: PP=2 dense `attn_dp=1` and MoE `attn_dp=2` (`moe_ep=2`) cases — completion, `routing_times == cluster_schedule_times`, schedule-time reports only when room remained, event-type set equal to the round-robin baseline, and the §17.6 discriminating scenario with `placements_fixed != placements_round_robin`. | Pass. | +| P1 Probes | (a) A small independent CPU reference-loop harness (`tests/comparison/dp_placement_pp/reference_loop.py`, not a copy of the proposed hook) that replays scripted admissions, empty schedules and completions with controllable output readiness through the reference's `step_with_batch_queue` conjunction, `_maybe_publish_request_counts`, the coordinator latch and the frontend scoring, and emits loads, key relations and visible snapshots. (b) Scratch Frontier probes at PP=2 and PP=3 for four shapes (MoE `attn_dp=2` burst and staggered, MoE `attn_dp=1`, dense `attn_dp=1`) recording `(lane, boundary, logical-iteration membership, load)` — membership, not merely provisional/resolved Batch ids. No source change. | State table (§18.11) confirmed or amended from evidence; the key-grouping rule proposed with its I1–I6 argument; both recorded in `design.md` for the design checkpoint. | +| P2 Implement | D9-1..D9-3 in `base_cluster_scheduler.py`, `base_replica_scheduler.py`, `vllm_load_balancing_cluster_scheduler.py`, implementing the state model fixed at the design checkpoint. | Existing unit tests pass except the intentionally inverted guard case; the §18.11 matrix rows that P3 owns pass. | +| P3 Unit | `tests/unit/test_vllm_dp_load_balancer.py`: guard param at `:538` becomes positive; PP=1 never reports at schedule time; PP=2 cold fill with peer lanes in both callback orders groups peer reports under one logical iteration (no partial peer snapshot); room remaining with the oldest output ready yields one combined report; PP=3 two admission-only iterations before the first completion are distinct iterations with peer equality inside each; full queue yields one report decision; idle/changed-to-zero peers; bounded bookkeeping after many iterations; helper `num_pipeline_stages` parameter. Expected counts, keys and placements are written independently, not taken from a run. | New tests fail before P2 and pass after. | +| P4 Integration | `tests/integration/test_vllm_dp_placement_runtime.py`: PP=2 dense `attn_dp=1` and MoE `attn_dp=2` (`moe_ep=2`) cases and a PP=3 case on a separate fixture with 6 or 12 layers (`_model()` today has 4; the tiny Qwen has 8; the divisibility guard stays) — completion with request/token/owner conservation, `routing_times == cluster_schedule_times`, reports only at classified iterations, event-type set equal to the round-robin baseline; the §18.6 discriminating scenario against the §18.11 controls (test-only guard-lifted completion-reporting baseline vs corrected implementation; `placements_fixed != placements_round_robin` is not sufficient because PP1 `vllm_load_balancing` already differs from round-robin); the C35-01 hybrid-layer mixed-batch credit case carried into the PP2 fixture. | Pass; each control fails for its stated reason. | | P5 Fidelity | Byte comparison of all PP=1 `vllm_load_balancing` scenarios before/after; Step 8 regression set (unit, integration, 16 examples) rerun. | C2. | | P6 Records | Docs of D9-5, test report `test_report_2026-09-22_w9_pp_dp_placement.md`, commits per package, push (Q5), PR body update. | Pushed and verified. | -| G1 Ground-truth checkout | `.real-engine/vLLM-BS` is detached at `ea95f571e` with only the remote ref `origin/feature/frontier-comparison-instrumentation` (same commit). Create the local branch at that commit, commit the D-b instrumentation on it (`dp_rank` in the schedule row; `VLLM_FRONTIER_DP_STATS_LOG_PATH` JSONL in `DPEngineCoreProc._maybe_publish_request_counts` with `dp_rank, wave, step_counter, running, waiting, monotonic ts`), keep the tree clean. Record the tuple with `groundtruth_remote_tip=ea95f571e`, the diff artifact `remote_tip..HEAD` and its SHA-256. No push (D-b covers the commit only). | Clean tree on the exact branch; tuple recorded in the manifest. | +| G1 Ground-truth checkout (instrumentation scope revised per P9-03) | `.real-engine/vLLM-BS` is detached at `ea95f571e` with only the remote ref `origin/feature/frontier-comparison-instrumentation` (same commit). Create the local branch at that commit and commit the D-b instrumentation on it: the case-gated event chain of §18.11 (engine iteration result, changed-count report emitted, coordinator receive/publish with a snapshot id, frontend snapshot application, frontend routing decision), named `waiting`/`running` fields, correlation ids rather than timestamps as the join, buffered per-process JSONL flushed at case completion, no CUDA synchronization or per-operator profiling. Changed files (to be confirmed at start approval): `vllm/v1/core/sched/scheduler.py` (schedule row gains `dp_rank`), `vllm/v1/engine/core.py` (iteration classification and emitted report), `vllm/v1/engine/coordinator.py` (receive/latch/publish with snapshot id), `vllm/v1/engine/core_client.py` (snapshot application and routing decision), plus one env-var switch. Keep the tree clean; record the tuple with `groundtruth_remote_tip=ea95f571e`, the diff artifact `remote_tip..HEAD` and its SHA-256. No push (D-b covers the commit only). | Clean tree on the exact branch; tuple and changed-file list recorded in the manifest. | | G2 Harness and case | `tests/comparison/dp_placement_pp/`: `make_trace.py` (three warmups, ≥5 s idle gap, discriminating burst, steady-state segment → Frontier trace CSV and request-id namespace), `replay_client.py` (online token-id replay, `x-request-id`, dispatch offsets recorded), `extract_vllm_placement.py` (schedule log + DP-stats log → per-engine publication sequence and placement per request id), `compare_placement.py` (T1 boundary table, T2 placement table, `workflow_gap_table.csv` rows). Case directory `task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/` (`manifest.yaml`, `case_init`, `analysis/`, receipts); `runs/*` under `/data/ycfeng/tmp/issue26-correctness-pr/calibration/dp_pp_case_001/`, mirrored to `/mnt/codesign-exp/ycfeng/frontier/dp_pp_calibration/`. `semantic-alignment-check` table written before any run. | Semantic table `PASS`; Frontier-only pre-check shows the discriminating request differs between round-robin/current and the expected reference placement. | | G3 S0 smoke | 2×H800, DP=2, PP=1: `groundtruth-run-check` (clean mode) → run → verify `vllm.__file__` resolves to the overlay, instrumentation rows exist with `dp_rank`, request-id mapping and extraction work end to end. | Run-check `PASS`; extraction produces one placement per formal request. | | G4 S1 ground truth | 4×H800, DP=2, PP=2, TP=1, EP=2: `groundtruth-run-check` → `groundtruth-run` (clean mode, ≥3 warmups, E2E runtime metrics on, no CUDA-op or CPU probes; the schedule log and DP-stats log are scheduler-level workflow evidence declared in the manifest). Measure `e1`'s first-chunk duration; if it is below the window, retune the chunk budget/prompt length and rerun (S2). | Fresh artifacts, exit 0, tuple verified, first-chunk duration recorded. | -| G5 Simulator runs and analysis | `simulator-run` on the pre-change revision and on the post-P2 revision with the same trace; `workflow-gap-analysis` compares publication sequences (T1) and placements (T2). `e2e-metrics-gap` is not run: Frontier timing is dummy (D-e) and the entry's pinned normalizer is absent, which the skill treats as `FAIL`; recorded as not applicable to this step's acceptance. | C3 and C4 tables with `MATCH`/`MISMATCH` rows and source anchors. | +| G5 Simulator runs and analysis (revised per P9-04) | `simulator-run` on the post-P2 revision and on the explicit test-only completion-reporting control (guard lifted, old reporting logic; the exact test-only change published) with the same trace; the unmodified pre-change revision is run once to record its constructor rejection under PP2. `workflow-gap-analysis` joins native rows to Frontier rows by causal inputs (same admissions/applied outputs), or replays a declared controlled history through the CPU reference loop and the Frontier observation path; a natural-history divergence is traced to its first cause and labeled (arrival/delivery order, batch composition, output readiness, count calculation, key grouping, snapshot publication, frontend selection). T2 is accepted only when the trace shows the intended snapshot applied at the frontend before the probe was routed; otherwise `SCENARIO_NOT_REACHED`. `e2e-metrics-gap` is not run: Frontier timing is dummy (D-e) and the entry's pinned normalizer is absent, which the skill treats as `FAIL`; recorded as not applicable to this step's acceptance. | C3 and C4 tables with `MATCH`/`MISMATCH`/`NOT_REACHED` rows, first-cause labels and source anchors. | -### 17.6 Ground-truth comparison under `frontier-calibration` +### 18.6 Ground-truth comparison under `frontier-calibration` Workflow: `case_init` manifest → `parity-run` → `semantic-alignment-check` → `groundtruth-run-check` → `groundtruth-run` → `simulator-run` → `workflow-gap-analysis`; `$grill-me` questions for any setting the skill cannot resolve; human review before any code change that the comparison motivates. @@ -921,28 +939,28 @@ Workflow: `case_init` manifest → `parity-run` → `semantic-alignment-check` | Model | `Qwen3MoeForCausalLM` from `data/config/models/Qwen3-30B-A3B-tiny.json` (8 layers, 16 experts, top-8; `SupportsPP` and `FusedMoE` EP in 0.10.2, `qwen3_moe.py:146,582,767`), `--load-format dummy`, `--skip-tokenizer-init`, served from a local config directory; Frontier loads the same JSON through `create_from_name`. | | Semantic-alignment rows | DP/PP/TP/EP sizes, `max_num_batched_tokens`, `max_num_seqs`, block size, KV block count (Frontier `num_blocks` taken from vLLM's startup log), chunked prefill on, prefix caching off, FCFS policy, `stats_update_interval_ms=100`, dummy weights, tokenizer skipped, `ignore_eos`, request-id mapping, arrival-time origin. | | Workload | One Frontier trace CSV (`arrived_at,num_prefill_tokens,num_decode_tokens`) is the single source. A replay client posts `/v1/completions` with `prompt=[token ids]`, `max_tokens=num_decode_tokens`, `ignore_eos=true`, header `x-request-id=` (propagated to the engine request id, `serving_engine.py:971-978`, so schedule-log `scheduled_new_req_ids` map back) at `arrived_at` offsets from one origin. | -| Discriminating scenario (T2) | Burst of 5 at `t0`: the frontend reservation alternates them (`e0: r1,r3,r5`, `e1: r2,r4`). `r2` has a long prompt that fills `e1`'s chunk budget so `r4` waits; `r1,r3,r5` are short. Probe `r6` arrives at `t0+~100 ms`, after the first coordinator publication (≥50 ms) and before `e1`'s first chunk completes. Reference publication S1: `e0 = 3 running → 3`, `e1 = 1 running + 1 waiting → 5`, so `r6 → e0`. Current Frontier never reports before the first completion, the reservations persist (`e0 12`, `e1 8`), so `r6 → e1`. Fixed Frontier reports S1 at admission, so `r6 → e0`. Robustness requires `e1`'s first chunk to exceed ~150 ms in both systems: vLLM through the chunk budget (8k-16k tokens, measured in S0), Frontier through decision D-e. A steady-state segment (staggered arrivals, long decodes) supplies T1. | -| Extraction | vLLM: per-engine publication log and schedule log (needs D-b). Frontier: balancer report trace (the hook `test_vllm_dp_placement_runtime.py` already uses), `metrics_ground_truth.jsonl`, placement ledger. Comparison script emits the T1 boundary table and the T2 placement table. | +| Discriminating scenario (T2) | Burst of 5 at `t0`: the frontend reservation alternates them (`e0: r1,r3,r5`, `e1: r2,r4`). `r2` has a long prompt that fills `e1`'s chunk budget so `r4` waits; `r1,r3,r5` are short. Probe `r6` arrives at `t0+~100 ms`, after the first coordinator publication (≥50 ms) and before `e1`'s first chunk completes. Reference publication S1: `e0 = 3 running → 3`, `e1 = 1 running + 1 waiting → 5`, so `r6 → e0`. Current Frontier never reports before the first completion, the reservations persist (`e0 12`, `e1 8`), so `r6 → e1`. Fixed Frontier reports S1 at admission, so `r6 → e0`. Robustness requires `e1`'s first chunk to exceed ~150 ms in both systems: vLLM through the chunk budget (8k-16k tokens, measured in S0), Frontier through decision D-e. A steady-state segment (staggered arrivals, long decodes) supplies T1. **Conditional witness (P9-04 §15.3):** the algebra `4k_e - 3a_e` holds for `k_e` assigned, `a_e` admitted and no intervening completion or published state; `k=(3,2), a=(3,1)` is the target, not a guaranteed live outcome. Before asserting `r6`'s lane, verify from the trace the actual routing order of the burst, the admissions, the in-flight requests, later scheduling attempts and the applied snapshot. HTTP-client concurrency does not fix engine-receipt order: preserve request ids, record dispatch and receipt order, and qualify it in the case. Long decodes and chunk sizes are explicit frozen values; the balancer's constants are never changed to make the witness occur; if the trace does not show the premise, the slice is `SCENARIO_NOT_REACHED`. | +| Extraction | vLLM: the §18.11 event chain (engine iteration, emitted report, coordinator receive/publish, frontend application, frontend routing), joined by correlation ids (needs D-b, scope per G1). Frontier: balancer report trace (the hook `test_vllm_dp_placement_runtime.py` already uses), `metrics_ground_truth.jsonl`, placement ledger. Comparison script emits the T1 causal-join table (rows matched on iteration inputs, divergences labeled by first cause) and the T2 placement table with the control column and qualification status. | | Harness location | `tests/comparison/dp_placement_pp/` (replay client, extraction, comparison) — pending D-a. | | GPU job | StepMind `RJobBackend`, `charged_group="codesign"` only, `positive_tags=["H800"]`, `gpu=4, cpu=16, mem_gb=128`, image `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`, `code_mount_point` per D-f, libcuda path fix and internal PyPI mirror from handbook §10, vLLM-BS as a Python overlay (copy the checkout to worker-local disk, copy the image's compiled `vllm/*.so` and `vllm_flash_attn` in, `PYTHONPATH` first; the vLLM-BS delta touches no `csrc/`, `cmake/`, `setup.py`, or `requirements/` — verified). Durable logs under `/mnt/codesign-exp/ycfeng/frontier/dp_pp_calibration/`. Sequence: S0 2-GPU smoke (DP=2, PP=1: stack, overlay, client, extraction), S1 4-GPU DP=2×PP=2 (T1+T2), S2 rerun only if the workload needs retuning. Budget ≤ 3 jobs × ≤ 1 h; launcher kept alive locally; no resubmission while queued; verify creator, mount, `torch.cuda` device, outputs, terminal status. | -### 17.7 Decisions (answered by the user on 2026-09-22; verbatim text in `requirements.md`) +### 18.7 Decisions (answered by the user on 2026-09-22; verbatim text in `requirements.md`) | Id | Decision | Effect on this plan | | --- | --- | --- | | D-a | "依据该skills" — follow `/home/brainpp/.claude/skills/frontier-calibration` as written. | Route `parity-run` (`semantic-alignment-check` → `groundtruth-run-check` → `groundtruth-run` → `simulator-run`), then `workflow-gap-analysis`. None of these entries binds a pinned helper file; the absent archive `/data/ycfeng/frontier-calibration-old-20260831/` affects only `e2e-metrics-gap`, `op-supplement`, and `dispatch-align-trace`, which are not on this case's path. If one of them becomes necessary it is `FAIL` per `tool-boundaries.md` and the case stops there; no substitute helper is used. The G2 scripts are the case's declared ground-truth client and analysis producers, not stand-ins for a listed helper. `$grill-me` is not installed; the decisions in this table were obtained by direct Q&A and are stored in the manifest as decision records with the user as `requesting_user` and `reviewer_identity`. | | D-b | "授权" — the minimal vLLM-BS instrumentation commit is authorized. | G1. The commit stays local on the exact branch; pushing to `fwyc0573/vLLM-BS` was not requested and is not needed because the tuple records the remote tip plus the diff artifact. | -| D-c | tiny Qwen3-MoE + dummy weights + tokenizer skipped (no download). | §17.6 model row unchanged. | +| D-c | tiny Qwen3-MoE + dummy weights + tokenizer skipped (no download). | §18.6 model row unchanged. | | D-d | Hook name `on_replica_batch_scheduled`. | D9-1 final. The key rule (K1/K3) was not chosen by the user; K3 remains the recommendation and is fixed after the P1 probe. | | D-e | Dummy mode first; if an unresolvable blocker appears, switch to H800 profiling mode. | Frontier runs use `dummy_execution_time_ms` sized so `e1`'s first chunk exceeds the window. The fallback is new H800 profiling CSVs and a trained predictor, taken only on a recorded blocker. | | D-f | `code_mount_point=/data/ycfeng/Frontier`. | Parent mount covering the worktree and `.real-engine/vLLM-BS`; the worker copies vLLM-BS to local disk for the overlay. | -| D-g | The intended skill is `/home/brainpp/.claude/plugins/cache/claude-plugins-official/mattpocock-skills/1.2.3/skills/engineering/codebase-design`. | Read in full; its vocabulary and principles are applied in §17.10 and in `design.md` W9. | +| D-g | The intended skill is `/home/brainpp/.claude/plugins/cache/claude-plugins-official/mattpocock-skills/1.2.3/skills/engineering/codebase-design`. | Read in full; its vocabulary and principles are applied in §18.10 and in `design.md` W9. | -### 17.8 Limits +### 18.8 Limits -Unchanged and not claimed: multiple Replicas, multiple frontends (`client_count > 1`), `data_parallel_hybrid_lb`/`external_lb`, wave-reset semantics, elastic EP, IPC timing, latency equivalence of placement. The hook is inert for PDD/PD-AF roles and the DECODE (M2N) branch of `on_schedule` is untouched. T1's boundary-index comparison is the primary evidence; T2 is confirmatory. +Unchanged and not claimed: multiple Replicas, multiple frontends (`client_count > 1`), `data_parallel_hybrid_lb`/`external_lb`, wave-reset semantics, elastic EP, IPC timing, latency equivalence of placement. The hook is inert for PDD/PD-AF roles and the DECODE (M2N) branch of `on_schedule` is untouched. T1 on a controlled or causally matched history is the primary evidence; T2 is confirmatory and conditional on trace qualification. Comparing by boundary index alone is not an alignment method (P9-04). -### 17.9 Calibration case binding (`frontier-calibration` v2) +### 18.9 Calibration case binding (`frontier-calibration` v2) | Contract item | Value for `dp_pp_case_001` | | --- | --- | @@ -954,7 +972,7 @@ Unchanged and not claimed: multiple Replicas, multiple frontends (`client_count | Receipts | Caller-written command receipts (command, cwd, environment declarations, UTC start/end, exit code, artifact paths) for every command the case issues; `exec capture: UNKNOWN` as the contract states. | | Code-change gate | The P2 change is the user-approved feature of this step, not a calibration repair. Any further Frontier change motivated by the G5 analysis needs `analysis_state=COMPLETE`, `status=PASS`, and the user's review `PASS` before it is applied (`repair-approval.md`). | -### 17.10 Design vocabulary (`codebase-design`) +### 18.10 Design vocabulary (`codebase-design`) - **Module.** `VllmLoadBalancingClusterScheduler`, with `VllmDPLoadBalancer` as an internal module. Its **interface** is `schedule_at`, `on_replica_batch_end`, and (new) `on_replica_batch_scheduled`, plus the facts a caller must know: reports are per lane, the schedule-time report is emitted once per admitted batch and only while pipeline room remains, at equal simulated time the completion report precedes the admissions it triggers, and the module never raises on an inert path. - **Seam.** `BaseClusterScheduler.on_replica_batch_*` already has two **adapters** — the inert default used by every other cluster scheduler and this module — so the new hook extends a real seam rather than creating a hypothetical one. The call site in `on_schedule` is the only place that knows whether pipeline room remains, which is why the seam sits there and not in `ReplicaScheduleEvent`. @@ -962,3 +980,70 @@ Unchanged and not claimed: multiple Replicas, multiple frontends (`client_count - **Test surface.** Tests drive the module through its interface and assert observable placements and published counts; the relabeled keys are implementation and are not asserted directly. - **Design it twice.** Three interface shapes were compared: (1) in-loop push hook with `pipeline_room_remaining` — chosen: smallest interface, the room predicate lives where it is known, one adapter per cluster scheduler; (2) event-level reconstruction after `on_schedule` — rejected: cannot observe the state after the first of two admissions in one call, so it is shallow and wrong; (3) pull-style `iter_admission_loads()` on the replica scheduler — rejected: widens the replica scheduler's interface for one caller and inverts the push direction the completion report already uses. +### 18.11 Corrections from the 2026-09-22 external review (P9-01..P9-06) + +Recorded the day the review arrived; every item below is a plan/record change, not a source change. The review document is `.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_Review_2026-09-22.md` (local, not committed). + +**P9-01 — the engine iteration, not the queue slot.** Reference branch (`core.py`, `step_with_batch_queue`): `model_executed = total_num_scheduled_tokens > 0; if model_executed and len(batch_queue) < batch_queue_size and not batch_queue[-1][0].done(): return None, True`; otherwise the oldest queued output is processed before the iteration returns, and an empty scheduled output can be enqueued before that completion path. Publication follows the iteration. The state table the hook must represent: + +| State after a scheduling attempt | Reference behavior | What W9 must represent | +| --- | --- | --- | +| Nonzero tokens, room remains, oldest result not ready | Return without applying an output, then publish changed counts | Admission-only observation | +| Nonzero tokens, room remains, oldest result already ready | Apply the oldest output, then publish changed counts | One combined admission/completion observation, not an extra admission-only report | +| Nonzero tokens, queue reaches capacity | Wait for / apply the oldest output, then publish changed counts | Completion-path observation | +| Zero-token scheduled output, prior work queued | Does not take the early-return branch | Explicit mapping of the empty iteration; the current hook design has no representation for it | +| No new request work, prior output queued | Drain an output, publish changed counts | Completion-only observation | + +Steady-state preconditions (replacing "needs no change"): with depth `P`, `B_k`'s append completes `B_(k-P+1)`; Frontier's completion report at `B_(k-P+1)`'s end shows the combined state only if `B_k` was admitted at `B_(k-P)`'s end, no empty iteration intervened, and no completion became visible between the two boundaries. These rows are tested, not assumed (P3/P4). + +**P9-02 — K3 as written is not order-preserving.** The coordinator distinguishes equal keys (apply without latch) from strictly greater keys (latch the previous counts). A global `next_label += 1` per report callback preserves neither the equality class of two peer lanes reporting one logical iteration nor the source ordering. Executable counterexample, reproduced on this branch's `VllmDPLoadBalancer` on 2026-09-22 (zero initial counts; lane 0 reports `waiting=0, running=3` at 10 ms, lane 1 the same at 20 ms; a request is placed at 80 ms): with one iteration key for both reports the frontend sees `[(0,3),(0,3)]`, last publication 70 ms, and selects lane 0 by first minimum; with a fresh key for the second report the coordinator latches the partial snapshot `[(0,3),(0,0)]` at 20 ms and the request goes to lane 1. Same inputs, different published state. Second interleaving to cover: one lane completes cohort `C`, moves on to an admission-only observation, and the peer's completion of `C` arrives later; reusing `C`'s label after minting the next one contradicts strict emission order, and suppressing the native out-of-order warning by inventing newer identities is not a fix. Invariants for the chosen rule: + +1. Peer observations of one logical iteration compare equal whatever the callback order. +2. A new logical iteration orders after the previous one; the key is captured at the observation boundary, not read later from a mutable Batch. +3. An iteration that both schedules new work and completes older work owns one report decision. +4. Suppressed unchanged-count reports create no fictitious coordinator messages. +5. PP3 allows more than one admission-only iteration before a completion without spacing constants. +6. Bookkeeping kept for in-flight work is released when no pending observation can refer to it; it does not grow with the lifetime number of batches. + +**P9-03 — observe the whole path that determines a placement.** An emission log in `_maybe_publish_request_counts` shows neither when the coordinator received the report, which previous-step snapshot it latched, when it published, nor when the frontend applied it; the schedule log shows where a request was admitted, not the frontend's decision or the load estimate it used. Minimal case-gated chain: + +| Observation | Minimum fields | +| --- | --- | +| Engine iteration result | engine/lane id, `(wave, step)`, scheduled request ids/tokens, whether an older output was applied, queue occupancy, readiness classification of the chosen branch | +| Changed-count report emitted | engine id, `(wave, step)`, named `waiting` and `running`, local timestamp | +| Coordinator receives/publishes | id of the received report; snapshot id and the published per-engine counts; link from a previous-step snapshot to its inputs | +| Frontend applies a snapshot | snapshot id and resulting counts | +| Frontend routes a request | request id, chosen engine, snapshot id or counts used, local reservation update | + +Named fields because the reference count accessor and Frontier's `RequestLoad` do not share positional order everywhere. Correlation ids are the join; a timestamp is never a causal id. Buffered per-process files, flushed at case completion; no CUDA synchronization, per-operator profiling or per-record `fsync`. The local-commit / no-vLLM-push boundary stands (D-b). + +**P9-04 — comparable history and valid controls.** Native GPU timing and dummy timing can batch the same arrivals differently, and under PP the `oldest.done()` branch depends on that timing, so the fifth report on each side need not describe the same admissions. T1 therefore (a) tests the state transformation first on the CPU reference loop with scripted inputs, (b) joins native and Frontier rows only where causal inputs match or replays a declared controlled history through both, and (c) labels any natural divergence by first cause. Controls: + +| Control | Establishes | +| --- | --- | +| Unmodified `0137269` (or the pre-P2 tip) under PP2 | Constructor rejection — a capability result, not a placement | +| Test-only completion-reporting baseline, guard lifted only, change published | Diagnostic behavior of the old reporting logic under PP2 | +| Corrected PP2 implementation | Proposed behavior under the same qualified inputs | +| Round-robin (optional) | A different policy; not the causal baseline for the reporting fix | + +**P9-05 — PP3 and lifecycle coverage.** The tiny Qwen3-MoE has 8 layers and Frontier requires `num_layers % PP == 0`, so PP3 needs its own CPU fixture (6 or 12 layers); the eight-layer native PP2 configuration stays. Behavioral matrix, owned by P3/P4/P5: + +| Case | Assertions | +| --- | --- | +| Existing PP1 dense DP1 and MoE DP>1 controls | Placements, results and visible snapshots unchanged; no added admission-only report | +| PP2 cold fill, peer lanes in both callback orders | One logical iteration groups the peers; no partial snapshot manufactured by relabeling | +| PP2, room remains, oldest output ready | One combined report, no spurious intermediate report | +| PP3, two admission-only iterations before the first completion | Distinct iterations, peer equality within each, no stride assumptions | +| Full queue: new batch scheduled, older completed | One report decision with the correct post-iteration populations | +| Empty schedule with queued output | Drain behavior and changed counts represented | +| Last request finishes; quiet interval; new work | Counts return to zero; new ordering coherent | +| Idle peer and changed-to-zero peer | Unchanged zeros may be suppressed; a real change never disappears | +| Mixed phases and dense layers inside a MoE model | C35-01 layer credits, request/token conservation, released ownership (carried from the 2026-09-22 fix) | +| Invalid dense DP>1 and invalid layer partition | Existing explicit errors remain | +| Other cluster schedulers, including supported disaggregated paths | Hook inert; event outcomes unchanged | +| Many completed iterations | Report-key bookkeeping bounded by live work | + +Exact baseline comparison only for behavior meant to stay unchanged; new PP2/PP3 behavior needs independently written expected counts, placements and transitions. + +**P9-06 — work graph and acceptance language.** Applied in §18.5 and §18.1. A native out-of-order warning is evidence to analyze (the reference applies the counts after warning), not proof of a simulator bug nor a reason to rewrite native ordering. + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 211d9d8f..54f4b3ef 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -17,6 +17,7 @@ | 2026-09-22 | W7 authorized and delivered: companion fix published as `eb7bc4f` with draft PR 1, Frontier gitlink moved, Frontier-side test added, and the governance scans narrowed to Frontier-owned sources. Step 7 closed. | | 2026-09-22 | Step 8 §14.1 run: unit and integration suites at the baseline failure set, 16 architecture examples, four PP=2 cases, and a cold-then-warm predictor-cache pair. `tests/debug/` pointer defect found, deferred to `future.md`. | | 2026-09-22 | Step 8 closed: §14.2 review recorded, PR 35 body updated with the Step 8 results and record links, `summary.md` written. Task technically complete; PR stays draft for user review. | +| 2026-09-22 | External review of PR34/PR35 applied (packages A–E; F excluded by the user): C34-01 merged in from the PR34 branch; C35-01 fixed with unit and real-loop hybrid-layer coverage and a negative control; C35-03/04 test wiring and optional-torch skip; C35-02/05 wording and record consistency; P9-01..06 folded into `plan.md` §18 and `design.md` W9. Step 9 remains planned, not started. See `test_report_2026-09-22_review_corrections.md` and the disposition table in `review.md`. | ## Status @@ -25,9 +26,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | -| Current step | Step 8 closed. All eight steps complete; `summary.md` is the completion archive. | -| Publication | PUSHED_VERIFIED. Draft PR 35 body carries the W6 and W7 sections; still draft. | -| Next action | **User review.** Technical acceptance is complete; PR 35 remains draft and nothing was merged. Two follow-ups carried forward, both in `future.md`: re-point the collective-sim gitlink at `main` once companion PR 1 merges, and repair the `tests/debug/` pointers that 10 baseline unit failures share. Retargeting PR 35's base to `main` waits on PR 34. | +| Current step | Step 8 closed; external review corrections A–E applied 2026-09-22 (`test_report_2026-09-22_review_corrections.md`). Step 9 is planned (`plan.md` §18, corrected per the review) and **not started**. | +| Publication | PUSHED_VERIFIED (SHAs in the commit log; PR34 correction `2310417` merged in as `0d025f8`). Draft PR 35 body synchronized with the corrections; still draft. | +| Next action | **User decisions.** (1) Go/no-go for re-running the corrected FP8 native check on one H800 under `codesign` (`NOT_RUN`). (2) Start signal for Step 9, at the first node of the `plan.md` §18.5 graph. PR 35 remains draft and nothing was merged. Carried forward in `future.md`: re-point the collective-sim gitlink at `main` once companion PR 1 merges, and repair the `tests/debug/` pointers that 10 baseline unit failures share. Retargeting PR 35's base to `main` waits on PR 34. | ## Step status @@ -41,7 +42,9 @@ | 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | PUSHED_VERIFIED (records + PR 35 section) | REVIEWED (user chose to keep the single global field) | | 6 | Legacy fused-MoE profiling | PASS | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778). Native PASS: 8 of 8 in `tests/integration/test_moe_fused_expert_numerical_parity.py` at `rtol=0, atol=0` on H800, job `exp-0922-145047-660565` under `codesign` | PUSHED_VERIFIED (source, tests, docs, records, PR 35 section) | — | | 7 | Optional zero-payload backend | PASS. Companion fix published as `fwyc0573/frontier-htsim` `eb7bc4f` with draft PR 1; Frontier gitlink moved from `b8518af`; no Frontier source change | Companion 9 passed, negative control 6 of 9 fail on pristine sources. Frontier 4 passed, negative control 3 of 4 fail at the old gitlink. Clean checkout resolves `eb7bc4f` from the published remote, builds, and passes. Suite back to the 84-failure baseline with 3782 passing after narrowing three governance scans to Frontier-owned sources | PUSHED_VERIFIED | — | -| 8 | Combined regression, PR hand-off | PASS | unit 84 failed / 3782 passed with a `FAILED` set identical to the `origin/main` baseline; integration 15 passed / 22 skipped / 5 errors, the errors environmental and identical on the base; 16 of 16 architecture examples pass; 4 of 4 `PP=2` cases pass; cold and warm predictor-cache runs byte-identical | PUSHED_VERIFIED (records + PR 35 body carrying the Step 8 results, the record links and the implementation commits) | NOT_REVIEWED | +| 8 | Combined regression, PR hand-off | PASS | unit 84 failed / 3782 passed with a `FAILED` set identical to the `origin/main` baseline; integration 15 passed / 22 skipped / 5 errors, the errors environmental and identical on the base; 16 of 16 architecture examples pass; 4 of 4 `PP=2` cases pass; cold and warm predictor-cache runs byte-identical | PUSHED_VERIFIED (records + PR 35 body carrying the Step 8 results, the record links and the implementation commits) | REVIEWED (external review 2026-09-22; corrections below) | +| 8+ | External review corrections A–E | PASS | unit 84 failed / 3789 passed / 50 skipped / 10 errors with the `FAILED` set identical to the baseline (+7 passes are the new tests, +1 skip and −1 error are the optional-torch module); mixed-forward unit 26 passed; real-loop hybrid-layer case 2 passed with the negative control failing on the pre-fix source; arithmetic 9 passed under torch | PUSHED_VERIFIED | NOT_REVIEWED | +| 9 | PP>1 support for `vllm_load_balancing` | PLANNED — plan corrected per the external review (`plan.md` §18.11); **not started** | n/a | PUSHED_VERIFIED (records only) | awaiting the user's start signal | ## Chronological updates @@ -181,9 +184,23 @@ files; the 110 tracked files under `outputs/` are all still present. for byte as sent. Nothing was merged, force-pushed, marked ready, or closed. - `summary.md` written as the completion archive. +## External review corrections (2026-09-22) + +Review document: `.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_Review_2026-09-22.md` (local). Finding-by-finding disposition: `review.md`, "External review 2026-09-22 — findings disposition". Evidence: `test_report_2026-09-22_review_corrections.md`. + +| Package | Finding | Where | State | +| --- | --- | --- | --- | +| A | C34-01 predictor-cache eligibility by executed case list | PR34 branch `2310417`, merged here as `0d025f8` | completed | +| B | C35-01 decode credit at a dense layer for a mixed source | `collective_timing.advance_decode_layer`, `dense_metrics.complete_dense_layer`; unit + real-loop tests | completed | +| C | C35-03 FP8 `block_shape` wiring + CPU boundary test; C35-04 optional-torch skip | `tests/integration/test_moe_fused_expert_numerical_parity.py`, `tests/unit/test_moe_fused_expert_arithmetic.py` | completed; native rerun `NOT_RUN` | +| D | C35-02 scope table; C35-03 seven-plus-one wording; C35-05 records consistency and PR bodies | W6 report §5/§8, `docs/profiling/README.md`, `summary.md`, `review.md` D2, this file, PR34/PR35 bodies | completed | +| E | P9-01..P9-06 plan corrections | `plan.md` §18 (renumbered from §17) and §18.11, `design.md` W9 | completed (records only) | +| F | W9 implementation | — | **not started** (user: 暂不开启) | + ## Step 9 — PP>1 support for `vllm_load_balancing` (pending) | Date | State | Note | | --- | --- | --- | -| 2026-09-22 | pending | Plan drafted in `plan.md` §17 with amendment A12 and `requirements.md` rows; awaiting user approval and the D-a..D-g answers. No source, GPU, or publication action taken. | -| 2026-09-22 | in-progress (planning closed) | User answered D-a..D-g; `plan.md` §17 finalized (decisions, G1–G5 ground-truth packages, calibration case binding, `codebase-design` vocabulary), `design.md` W9 written, `requirements.md` updated. Records committed and pushed (SHA in the commit log). No source edit, no GPU submission; next action: P1 probe + G1/G2 once the user confirms the start. | +| 2026-09-22 | pending | Plan drafted in `plan.md` §18 with amendment A12 and `requirements.md` rows; awaiting user approval and the D-a..D-g answers. No source, GPU, or publication action taken. | +| 2026-09-22 | in-progress (planning closed) | User answered D-a..D-g; `plan.md` §18 finalized (decisions, G1–G5 ground-truth packages, calibration case binding, `codebase-design` vocabulary), `design.md` W9 written, `requirements.md` updated. Records committed and pushed (SHA in the commit log). No source edit, no GPU submission; next action: P1 probe + G1/G2 once the user confirms the start. | +| 2026-09-22 | in-progress (planning corrected) | External review P9-01..P9-06 applied: `plan.md` §17 → §18 (duplicate number), state table and preconditions replace the room-only hook rule and the steady-state claim, K1/K3/stride rejected as acceptance basis with invariants I1–I6, instrumentation chain and T2 qualification, CPU reference-loop oracle and valid controls, PP3 fixture with a valid layer count, revised graph and C1–C5 (`plan.md` §18.11, `design.md` W9). Records only; execution still awaits the user's start signal. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md index c6f1192e..7fa6f734 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md @@ -45,7 +45,7 @@ The draft specification is landed verbatim in `plan.md` together with an Amendme "添加需求：我需要在当前pr中补全 vllm_load_balancing_cluster_scheduler.py 的模拟支持，使得其不被限制在pp=1；你需要基于frontier 和vllm的codebase进行充分调研（codebase design skill)和设计，并且运行vllm v0.10.2进行实际调度结果的对比（pp>1情况下的dp 调度策略；调用calibration来确保参数设定一致）。请你先设计落地该子任务的plan，在我批准之前暂不执行" -Reading: (a) remove the PP1 restriction of `VllmLoadBalancingClusterScheduler` inside PR 35; (b) research both codebases and design first; (c) compare against a real vLLM v0.10.2 PP>1 DP-scheduling run, using the `frontier-calibration` skill to keep settings consistent; (d) deliver the plan first and do not execute before approval. Plan: `plan.md` §17 (Step 9) and amendment A12. +Reading: (a) remove the PP1 restriction of `VllmLoadBalancingClusterScheduler` inside PR 35; (b) research both codebases and design first; (c) compare against a real vLLM v0.10.2 PP>1 DP-scheduling run, using the `frontier-calibration` skill to keep settings consistent; (d) deliver the plan first and do not execute before approval. Plan: `plan.md` §18 (Step 9) and amendment A12. ## [Original Request] 2026-09-22 — GPU charged group @@ -59,12 +59,26 @@ Rule: every GPU submission uses `charged_group="codesign"`; `steptron_ci` is sus | Id | Question | Decision | Recorded in | | --- | --- | --- | --- | -| D-a | Calibration helper archive absent. | Follow `frontier-calibration` v2 as written. The case path (`parity-run` → `workflow-gap-analysis`) binds no pinned helper; entries that do (`e2e-metrics-gap`, `op-supplement`, `dispatch-align-trace`) are off-path and would be `FAIL` if needed. | `plan.md` §17.7, §17.9 | +| D-a | Calibration helper archive absent. | Follow `frontier-calibration` v2 as written. The case path (`parity-run` → `workflow-gap-analysis`) binds no pinned helper; entries that do (`e2e-metrics-gap`, `op-supplement`, `dispatch-align-trace`) are off-path and would be `FAIL` if needed. | `plan.md` §18.7, §18.9 | | D-b | vLLM-BS instrumentation. | Authorized: local commit on `feature/frontier-comparison-instrumentation` (engine identity in schedule rows; per-engine publication log). Push not requested. | `plan.md` G1 | -| D-c | Model and tokenizer. | Tiny Qwen3-MoE config, dummy weights, tokenizer skipped. | `plan.md` §17.6 | +| D-c | Model and tokenizer. | Tiny Qwen3-MoE config, dummy weights, tokenizer skipped. | `plan.md` §18.6 | | D-d | Hook name and key rule. | `on_replica_batch_scheduled`. Key rule left to the P1 probe (K3 recommended). | `plan.md` D9-1, D9-2 | -| D-e | Frontier timing for the placement check. | Dummy mode first; switch to H800 profiling mode only on an unresolvable blocker. | `plan.md` §17.6, §17.7 | -| D-f | Worker mount. | `code_mount_point=/data/ycfeng/Frontier`. | `plan.md` §17.6 | -| D-g | "codebase design skill". | The `codebase-design` skill at the path above; applied in `plan.md` §17.10 and `design.md` W9. | `plan.md` §17.10 | +| D-e | Frontier timing for the placement check. | Dummy mode first; switch to H800 profiling mode only on an unresolvable blocker. | `plan.md` §18.6, §18.7 | +| D-f | Worker mount. | `code_mount_point=/data/ycfeng/Frontier`. | `plan.md` §18.6 | +| D-g | "codebase design skill". | The `codebase-design` skill at the path above; applied in `plan.md` §18.10 and `design.md` W9. | `plan.md` §18.10 | Also requested: update the records so the execution plan, reasoning, and observations are clearly recorded, then commit and push. Not yet given: an explicit start signal for P1/G1. + +## [Original Request] 2026-09-22 — External review corrections + +"充分阅读理解/data/ycfeng/Frontier/.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_Review_2026-09-22.md，逐条校对，采纳正确和高价值建议，修正补充代码和docs。暂不开启new subtask的执行。" + +Reading: read the external review in full, verify each finding against the source, adopt the correct and high-value ones, and fix or supplement code and docs. Do **not** start the new sub-task, i.e. the Step 9 / W9 implementation (the review's package F). + +| Item | Decision / disposition | +| --- | --- | +| Scope | Review packages A (PR34 cache eligibility), B (mixed-batch dense-layer credit), C (FP8 test wiring, optional-torch skip), D (evidence wording, records consistency, PR bodies) and E (Step 9 plan corrections, records only) are adopted. Package F (W9 implementation) is excluded by the user's instruction; no W9 source change and no GPU submission. | +| Findings verified before editing | C34-01, C35-01..05 confirmed from source; P9-01..P9-06 accepted (P9-02 reproduced on the real balancer; P9-05 divisibility guard and layer counts confirmed). The review's "PR35 mergeable=false" was stale: both PRs report `MERGEABLE`; PR34 is not a draft on GitHub. | +| GPU | The corrected FP8 native check (package C) was **not** re-run on a worker; it needs one H800 under `codesign` and a fresh go from the user. Recorded as `NOT_RUN`. | +| Publication | PR34 correction committed and pushed first, then merged (not rebased) into PR35; PR35 commits pushed; both PR bodies updated through the API. Draft state of PR35 untouched. | + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index e0d182c1..e5263421 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -16,6 +16,7 @@ | 2026-09-22 | W7 facts re-verified: the candidate gitlink is unpublished, the three payload defects are confirmed by execution against the published backend, and the fix needs companion-repository authorization. | | 2026-09-22 | W5 closed without source changes. Corrected the Mechanism A premise: the routing distribution has no per-role override on main, so both W5 mechanisms are unreachable from any released configuration. The user chose to keep the single global field; the drafted implementation was reverted and archived as a patch. | | 2026-09-22 | Step 8 §14.2 recorded: final self-review of the whole branch diff at `d881357`, with the method stated and one deferred pre-existing defect. | +| 2026-09-22 | External review of PR34/PR35 (`.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_Review_2026-09-22.md`) verified finding by finding; D2's scope-identifier clause marked SUPERSEDED; disposition table added at the end. | ## Pinned source snapshot @@ -236,10 +237,18 @@ unconditionally (`:308-310`). W4 must not add a hard runtime assertion on report order that the reference does not have. Key equality per shared forward is a test invariant, not a runtime abort condition. -### D2 — RESOLVED: include the local reduction, behind a narrow versioned scope identifier +### D2 — RESOLVED: include the local reduction (the scope-identifier clause is SUPERSEDED) -Approved as written, including the compatibility and cache policy in §3.5 and the -validation set in §3.6. +**Superseded in part, 2026-09-22.** The "narrow versioned scope identifier" and +its §3.5 compatibility/cache policy were overtaken by the user's later decision, +recorded in the W6 table above under "Decided: artifact identity": 不改 +metadata，只记录限制 — document the limitation, change no metadata. The +inclusion of the local reduction, the §3.3 verification table and the +"Consequence adopted" paragraph below remain in force. Only one rule is active: +the documentation-only decision (external review C35-05). + +Approved as written at the time, including the compatibility and cache policy in +§3.5 and the validation set in §3.6. The maintainer's §3.3 correction is confirmed in this tree: @@ -403,3 +412,25 @@ which exists here or on `origin/main`; the same missing tree causes 10 of the 84 baseline unit failures. Reproduced on the base, reported as a baseline failure, and recorded in `future.md` rather than repaired, because the correct fix is a decision about the published test surface and is unrelated to Issue 26. + +## External review 2026-09-22 — findings disposition + +Reviewed revisions: PR34 `6ef0a3c`, PR35 `0137269`. Each finding was checked against source before any edit. Statuses: `FIXED` (code or record changed and verified), `ACCEPTED_LIMITATION` (true, recorded, not changed), `SUPERSEDED` (overtaken by a dated decision), `OPEN` (still to do). Commit SHAs are in the branch log; the evidence file is `test_report_2026-09-22_review_corrections.md` unless stated. + +| Finding | Verified as | Status | Where / evidence | +| --- | --- | --- | --- | +| C34-01 cache comparison eligibility ignores `cases_executed_in_last_run` | Confirmed: `compare_labels` read only `cache_clean_before_run` and `case_filter`; `--start`/`--limit` leave no filter | FIXED | PR34 `2310417` (runner + 4 gate tests), merged as `0d025f8`; Checkpoint B verdict re-derived, unchanged (`task_2026-09-21_oversized_module_split/test_report_2026-09-22_cache_eligibility_correction.md`) | +| C35-01 decoding request in a prefill-mode mixed batch uncredited at a dense layer | Confirmed by call path: `complete_dense_layer(phase="prefill")` → `handle_prefill_sync_collective`, which credits nothing; only the shared forward and decode helpers credit | FIXED | `advance_decode_layer` helper (validate then increment) used by all three completion paths; dense prefill-mode source credits its decoding members. Unit: mixed source at a dense layer +1 for the decoder, 0 for the prefiller, pure-prefill control credits nothing; `MoE -> dense -> MoE` credits 1, 2, 3. Real loop: hybrid `moe_layers_enum="0,2,3"`, 4 mixed dense completions, 10 decode tokens all peaking at 4 layers; the pre-fix source peaks 4 of them at 3 | +| C35-02 W6 report claims legacy scope equals functional scope | Confirmed: the functional entry aligns inside `fused_experts` (vLLM 0.10.2 `fused_moe.py:1718`), the legacy path aligns before `_step`; shuffling and grouped GEMM are additive in both accounting paths | FIXED (records) | W6 report §5 scope table; `docs/profiling/README.md`; `summary.md`; PR35 body. Live double count for functional datasets: not verified, not claimed | +| C35-03 FP8 test omits `block_shape`; "8 of 8 at `rtol=0, atol=0`" overstates | Confirmed: `block_dims` was passed, `block_shape` was not; the FP8 test asserts shape and finiteness only | FIXED (test + wording); native rerun OPEN | `block_shape=block_shape` added; CPU test pins both GEMM invocations receive it (`[128, 64]`) and `None` when omitted; report §8, `summary.md`, PR35 body restated as seven comparisons plus one structural check. Native rerun on one H800 under `codesign`: `NOT_RUN`, awaiting the user's go | +| C35-04 unconditional `import torch` adds a collection error | Confirmed: 11 collection errors in the minimal environment versus 10 on the base | FIXED | `pytest.importorskip("torch")` before importing the profiler module; minimal env: `1 skipped`; torch env: 9 passed; unit suite errors back to 10 | +| C35-05 records inconsistent (D2 metadata rule vs documentation-only; W2-checkpoint diff claim; blanket vLLM-comparison exclusion; PR34 "Draft") | Confirmed on all four points | FIXED (records) | D2 heading marked SUPERSEDED in part with a link to the dated decision; PR35 body scopes the `ceac2b4` diff claim to the W2 checkpoint and amends the exclusion for the authorized scheduler-level comparison; `progress.md` status table current; PR34 body says "open for review" | +| Review's "PR35 mergeable=false" | Stale: GitHub reports `MERGEABLE` for both PRs; PR34 `isDraft=false` | ACCEPTED_LIMITATION (of the review) | `gh pr view` 2026-09-22 | +| P9-01 room-only hook rule; "steady state needs no change" | Accepted: the reference branch is a three-way conjunction; depth-`P` completes `B_(k-P+1)` | FIXED (plan) | `plan.md` §18.3, D9-1, §18.11 state table; `design.md` W9 | +| P9-02 K3 as written breaks peer-key equality | Reproduced on this branch's balancer: equal keys → lane 0; fresh key → partial snapshot `[(0,3),(0,0)]`, lane 1 | FIXED (plan) | K1, K3-as-written, stride rejected as acceptance basis; invariants I1–I6; rule deferred to the design checkpoint after P1 establishes iteration membership | +| P9-03 emission log alone cannot show the placement path | Accepted | FIXED (plan) | Instrumentation chain (iteration, emission, coordinator receive/publish with snapshot id, frontend application, routing), named fields, correlation ids, changed-file list under G1, T2 `SCENARIO_NOT_REACHED` rule | +| P9-04 boundary index is not alignment; unmodified PP2 baseline cannot run | Accepted; constructor rejects PP>1 at `0137269` | FIXED (plan) | CPU reference-loop oracle, causal join, first-cause labels, controls table (rejected production / test-only guard-lifted / corrected / round-robin optional) | +| P9-05 PP3 needs a valid layer count; behavioral matrix | Accepted: tiny Qwen has 8 layers, `_model()` fixture 4, `num_layers % PP == 0` enforced | FIXED (plan) | PP3 CPU fixture with 6 or 12 layers; matrix in §18.11 | +| P9-06 work graph and acceptance language | Accepted | FIXED (plan) | §18.5 graph, §18.1 C1–C5 | +| Package F (W9 implementation) | — | OPEN by instruction | Not started (user: 暂不开启 new subtask) | + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md b/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md index 3b1a3c69..0802f63d 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md @@ -6,6 +6,7 @@ | --- | --- | | 2026-09-21 | Placeholder created at Step 0. | | 2026-09-22 | Completion archive written at Step 8. | +| 2026-09-22 | External review corrections: W3 dense-layer credit for decoding requests in a mixed batch (C35-01); W6 native result restated as seven comparisons plus one FP8 structural check, FP8 `block_shape` wiring corrected (C35-02/03); optional-torch skip (C35-04); records aligned (C35-05). Step 9 remains planned, not started. | ## Overview @@ -30,7 +31,7 @@ of scope throughout. Issue 26 stays open. All three PRs are draft. | # | Fix | Outcome | | --- | --- | --- | | W2 | Round-robin DP placement keeps rotating across scheduling calls | Landed. `6ab521d`, tests strengthened in `ceac2b4`. | -| W3 | A monolithic Replica completes one shared forward across mixed prefill and decode source lanes | Landed. `65ed8a7`. | +| W3 | A monolithic Replica completes one shared forward across mixed prefill and decode source lanes | Landed. `65ed8a7`. A decoding request inside a prefill-mode mixed batch was still missing its credit at a dense layer; repaired 2026-09-22 (C35-01, `test_report_2026-09-22_review_corrections.md`). | | W4 | Opt-in vLLM-style DP request placement, off by default and bounded in its constructor | Landed. `10dd474`. | | W5 | Routing implementation identity separated from expert-load distribution | **Closed, not ported**, by user decision after the premise check showed the collision unreachable from any released configuration. Drafted implementation reverted before commit and archived as `w5_reverted_moe_routing_runtime_path.patch`. | | W6 | Legacy fused-MoE profiling performs the real gated expert computation | Landed. `7269bac`, native parity test `697f219`, identity limits documented in `79f599a`. | @@ -89,7 +90,7 @@ recorded in `validation.md`), and `w5_reverted_moe_routing_runtime_path.patch`. | W3 fidelity matrix | 71 of 71 cases identical, against an expectation recorded before the run. | | W4 fidelity matrix | 71 of 71 cases identical, against an expectation recorded before the run. | | W4 placement | Measured to place differently from round-robin under the same load, so the policy is not a renamed default. | -| W6 native parity | 8 of 8 at `rtol=0, atol=0` on an H800 (`exp-0922-145047-660565`, charged group `codesign`), vLLM 0.10.2, `VLLM_API_VERSION=0.10.x`. | +| W6 native parity | Eight native tests passed on an H800 (`exp-0922-145047-660565`, charged group `codesign`, vLLM 0.10.2, `VLLM_API_VERSION=0.10.x`): seven reference-output comparisons at `rtol=0, atol=0` and one FP8 structural/finite-output check. FP8 numerical equivalence is not established. The FP8 check as run omitted the production `block_shape`; corrected 2026-09-22 (C35-03), native rerun NOT_RUN. | | W7 companion | 9 passed on the fix; 6 of 9 fail against pristine sources. | | W7 Frontier | 4 passed; 3 of 4 fail at the old gitlink. A fresh clone resolves `eb7bc4f` from the published remote, builds, and passes. | | Negative controls | W2 12 of 23, W3 four trees, W4 five trees, W6 one discriminating test, W7 both sides — each fails for its own stated reason on the unrepaired source. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_review_corrections.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_review_corrections.md new file mode 100644 index 00000000..c7957be2 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_review_corrections.md @@ -0,0 +1,98 @@ +# Test Report 2026-09-22 — External review corrections (packages B, C, D, E) + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-22 | Created: verification of the review corrections applied on `fix/issue26-correctness-pr`. Package A is reported in the PR34 task directory (`test_report_2026-09-22_cache_eligibility_correction.md`). | + +## Scope + +Review document: `.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_Review_2026-09-22.md` +(local, not committed). Packages adopted: A (PR34), B, C, D, E. Package F (the +W9 implementation) was excluded by the user and is not started. Finding-level +dispositions are in `review.md`, "External review 2026-09-22 — findings +disposition". + +Environment: simulator interpreter `/data/ycfeng/envs/frontier-py310/bin/python` +(Python 3.10, no torch); torch interpreter +`/data/ycfeng/envs/openmopd-py312/bin/python` (Python 3.12, torch 2.8.0, vLLM +0.11.0). `PYTHONPATH` at the worktree root, `OMP_NUM_THREADS=1`, +`OPENBLAS_NUM_THREADS=1`, run from +`/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr` after merging the PR34 +correction (`0d025f8`). + +## Package B — C35-01: decode credit at a dense layer for a mixed source + +### Change + +| File | Change | +| --- | --- | +| `frontier/scheduler/utils/collective_timing.py` | `advance_decode_layer(requests, total_layers)`: validate the whole selection, then credit one layer to each request. | +| `frontier/scheduler/utils/forward_collective.py`, `decode_collective.py` | Use the helper in place of the duplicated validate-then-increment loop (behavior-preserving). | +| `frontier/scheduler/utils/dense_metrics.py` | `complete_dense_layer(phase="prefill")` credits the source batch's decoding members (`collect_active_requests` filtered by `is_prefill_complete`) before delegating to the prefill handler. A dense layer completes per source, outside the shared forward completion that credits routed layers, and the prefill handler credits nothing. | + +No double credit: the shared forward path credits at `handle_forward_sync_collective` and enters the per-phase helpers with the credit already done; the dense path credits once per source (prefill-mode source here, decode-mode source in `handle_decode_sync_collective`). A PREFILL-role batch holds no request with `is_prefill_complete` set during its layers (`Request` sets it at prefill completion), so the new call is a no-op for the disaggregated roles. + +### Verification + +| # | Check | Command | Expected | Actual | Result | +| --- | --- | --- | --- | --- | --- | +| B1 | Mixed-forward unit tests, including the three new ones | `python -m pytest tests/unit/test_monolithic_mixed_forward_sync.py -q -p no:cacheprovider` | 23 existing + 3 new pass | 26 passed in 1.63 s | PASS | +| B2 | Dense layer executed for a mixed source (`test_a_dense_layer_credits_only_the_requests_that_are_decoding[mixed]`) | in B1 | decoding member `completed_layer_count == 1`, prefilling member `0`, decode-lane peers `1` | as expected | PASS | +| B3 | Pure-phase control (`[prefill]` parametrization) | in B1 | pure-prefill source credits nothing; decode lane `1` each | as expected | PASS | +| B4 | `MoE -> dense -> MoE` (`test_a_mixed_source_is_credited_once_per_layer_across_routed_and_dense`) | in B1 | decoding member counts `[1, 2, 3]`; prefiller `0`; peers `layer + 1` | `[1, 2, 3]` | PASS | +| B5 | Real loop, hybrid layers (`moe_layers_enum="0,2,3"`, `dense_mlp_hidden_dim=64`, chunked prefill, `attn_dp=2`, `moe_ep=2`, trained predictor on constant synthetic rows) | `python -m pytest tests/integration/test_monolithic_mixed_forward_runtime.py -q -p no:cacheprovider` | both tests pass; a mixed batch actually crosses the dense layer; every decode token credited exactly `num_layers` | 2 passed in 7.17 s; child evidence: `mixed_dense_completions: 4`, `decode_tokens_credited: 10`, `layer_credit_peaks: {"4": 10}`, `mixed_phase_cohorts: 4`, 4 of 4 requests complete, waiting rooms drained | PASS | +| B6 | Negative control: same hybrid case on the pre-fix `dense_metrics.py` (file restored from `HEAD`, then the fix copied back) | scratch driver calling `run_case(..., moe_layers_enum="0,2,3")` | the credit assertion fails | `AssertionError`; evidence `layer_credit_peaks: {"3": 4, "4": 6}` — the four decode tokens carried through the dense layer inside a mixed batch peaked one layer short | PASS (detects the defect) | +| B7 | Neighbors of the changed helpers | `python -m pytest tests/unit/test_monolithic_mixed_forward_sync.py tests/unit/test_collective_timing.py tests/unit/test_execution_time_metrics_ownership.py tests/unit/test_moe_routing_conservation.py tests/unit/test_stage_reporting_contract.py -q` | all pass | 72 passed in 2.00 s (before the three new tests were added; 26 of the 23 above included) | PASS | +| B8 | Whole unit suite versus the Step 8 baseline | `python -m pytest tests/unit -q -p no:cacheprovider -rfE --continue-on-collection-errors` | `FAILED` set identical to `base_failed.txt`; count deltas explained | 84 failed / 3789 passed / 50 skipped / 10 errors in 109.9 s (Step 8: 84 / 3782 / 49 / 11). `diff <(sort base) <(sort now)` empty. +7 passes = 3 new mixed-forward tests + 4 gate tests merged from PR34; +1 skip and −1 error = the optional-torch module (package C) | PASS | + +Observation versus inference: B5 and B6 are observed runs; the claim that the +disaggregated roles are untouched is by construction (B8 shows the unchanged +failure set, but the suite has no PREFILL-role hybrid-layer case). + +## Package C — C35-03 / C35-04: FP8 test wiring and optional-torch collection + +### Change + +| File | Change | +| --- | --- | +| `tests/integration/test_moe_fused_expert_numerical_parity.py` | FP8 case passes `block_shape=block_shape` to `_run_fused_moe_iteration`, matching `profile_fused_moe_kernel`. | +| `tests/unit/test_moe_fused_expert_arithmetic.py` | `torch = pytest.importorskip("torch", ...)` before importing the profiler module; the `_invoke_kernel` stub records `block_shape`; two new tests pin that both GEMM invocations receive `[128, 64]` under FP8 and `None` when omitted; `_run` accepts overrides. | + +### Verification + +| # | Check | Command | Expected | Actual | Result | +| --- | --- | --- | --- | --- | --- | +| C1 | Arithmetic boundary tests under torch | `openmopd-py312 python -m pytest tests/unit/test_moe_fused_expert_arithmetic.py -q -p no:cacheprovider` | 7 existing + 2 new pass | 9 passed in 6.31 s | PASS | +| C2 | Minimal environment collection | `frontier-py310 python -m pytest tests/unit/test_moe_fused_expert_arithmetic.py -q -p no:cacheprovider` | module skips instead of erroring | 1 skipped in 0.06 s | PASS | +| C3 | Unit-suite collection errors | B8 | 10 (the base's count), not 11 | 10 errors | PASS | +| C4 | Corrected FP8 native check on the approved worker | not run | — | `NOT_RUN`: needs one H800 under `codesign` and the user's go | NOT_RUN | + +The native run recorded in the W6 report (`exp-0922-145047-660565`) stands as +evidence for the arithmetic repair: seven reference comparisons at +`rtol=0, atol=0` plus one FP8 structural check that, as run, exercised the +per-tensor scale path. FP8 numerical equivalence is not established. + +## Packages D and E — records only + +No production code changed. Files: W6 report §5 (scope table) and §8 (seven +plus one, `block_shape` omission), `docs/profiling/README.md` (backend +envelopes differ), `summary.md`, `review.md` (D2 clause superseded; disposition +table), `progress.md` (status table, step rows, package table), +`requirements.md` (this request), `plan.md` (§17 → §18 renumber; §18.1, §18.3, +§18.4, §18.5, §18.6, §18.8 corrected; §18.11 added), `design.md` W9 (steady +state, gap table, key verdicts with the reproduced counterexample, conditional +witness, planned edits, expectations). PR34 and PR35 bodies updated through the +API; PR35 stays draft. + +## Limits + +- The corrected FP8 native check has not been re-run on a GPU. +- The real-loop hybrid case uses constant synthetic profiling rows and a + four-layer synthetic MoE model, as the W3 runtime test does; it proves the + accounting, not latency fidelity. +- The disaggregated-role control is by construction plus the unchanged unit + failure set; no PREFILL-role hybrid-layer run was added. +- Package E changes plans and designs only; none of its statements is a test + result for W9. From c2313225837292d2c5549e4cda4fe0bb3726646a Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 20:13:39 +0800 Subject: [PATCH 36/80] docs(records): record the review-correction publication SHAs and the PR 35 description update --- task_memory/task_2026-09-21_issue26_correctness_pr/progress.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 54f4b3ef..157e769a 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -27,7 +27,7 @@ | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | | Current step | Step 8 closed; external review corrections A–E applied 2026-09-22 (`test_report_2026-09-22_review_corrections.md`). Step 9 is planned (`plan.md` §18, corrected per the review) and **not started**. | -| Publication | PUSHED_VERIFIED (SHAs in the commit log; PR34 correction `2310417` merged in as `0d025f8`). Draft PR 35 body synchronized with the corrections; still draft. | +| Publication | PUSHED_VERIFIED: `f7c31e4` (C35-01 source + tests), `ca1b9b6` (FP8 `block_shape`, optional-torch skip, W6 report, profiling guide), `57ffa5b` (records, Step 9 plan corrections); remote head `57ffa5b` confirmed; PR34 correction `2310417` merged in as `0d025f8`. Draft PR 35 body PATCHed 2026-09-22T12:12Z through `gh api` and read back; still draft, MERGEABLE, base `refactor/oversized-module-split`. | | Next action | **User decisions.** (1) Go/no-go for re-running the corrected FP8 native check on one H800 under `codesign` (`NOT_RUN`). (2) Start signal for Step 9, at the first node of the `plan.md` §18.5 graph. PR 35 remains draft and nothing was merged. Carried forward in `future.md`: re-point the collective-sim gitlink at `main` once companion PR 1 merges, and repair the `tests/debug/` pointers that 10 baseline unit failures share. Retargeting PR 35's base to `main` waits on PR 34. | ## Step status From 9df18d19c31d6075ee4b5bb749d0395e1ae3ee91 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 20:37:03 +0800 Subject: [PATCH 37/80] docs(records): FP8 native rerun passes; second Step 9 plan review against the core-module quality gates The corrected FP8 native case (block_shape now forwarded) was re-run on one H800 under codesign at the user's authorization: 8 passed, exit 0. It remains a structural check; FP8 numerics are still not compared against a reference. W6 report, corrections report, review, validation and progress record the run. Step 9's plan was reviewed a second time at the user's direction, for grounding in the codebase, readability, value, and the bans on hard-coding, temporary patches, over-defense, redundancy and vague names. Eight findings (plan section 18.12): the DES has no "ready but unapplied" state, so the hook carries the completion hook's signature and no readiness classifier; the completion key names the scheduling iteration and is right only at PP=1, so the key rule applies to both observation kinds; the Replica's next forward id is the first existing identity to test, with its invariant-5 gap to be measured rather than assumed; the call site uses the constructor-required cluster scheduler without getattr/hasattr; the CPU oracle models the engine loop only and feeds the real balancer; names stay plain; validation code stays out of frontier/; DP engines are not iteration-lockstep in the reference. Records only; no Step 9 source change; execution still awaits the start signal. --- .../design.md | 63 +++++++++++++++++-- .../plan.md | 36 ++++++++--- .../progress.md | 10 ++- .../requirements.md | 8 +++ .../review.md | 19 +++++- .../summary.md | 1 + ...st_report_2026-09-22_review_corrections.md | 5 +- ...t_2026-09-22_w6_fused_expert_arithmetic.md | 19 +++++- .../validation.md | 1 + 9 files changed, 142 insertions(+), 20 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md index 908363c5..70114c26 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md @@ -7,6 +7,7 @@ scope decisions and the pre-measurement expectation for that package. | Date | Change | | --- | --- | +| 2026-09-22 | W9 second review (user-directed quality gates): section "What the code already provides" added; planned-edits rows for the hook payload, the call site and the CPU oracle amended; plan §18.12 R9-01..R9-08. | | 2026-09-22 | Created. Source-backed design for Checkpoint D's W3 half, with the scope decisions and their evidence. | | 2026-09-22 | Restructured into per-work-package sections and added the W4 design, with the report-key identity measured at the emission boundary. | | 2026-09-22 | Added the W9 design: vLLM 0.10.2 count publication under pipeline parallelism, the equivalence argument, the schedule-only gap, the report-key options, and the discriminating scenario. Analysis only; implementation pending. | @@ -383,6 +384,60 @@ completes owns one report decision; (4) suppressed unchanged reports create no fictitious messages; (5) PP3 allows consecutive admission-only iterations without spacing constants; (6) bookkeeping is released with the in-flight work. +## What the code already provides (second review, 2026-09-22) + +Read against `c231322` with the user's gates for core-module changes (plan +§18.12). Three facts narrow the design. + +**A completion is atomic in the DES.** `GlobalBatchEndEvent` runs +`replica_scheduler.on_batch_end` and then `cluster_scheduler.on_replica_batch_end` +at the batch's end time, and the lane's next `ReplicaScheduleEvent` follows at +the same time. There is no "oldest output ready but not yet applied" state, so +the hook has nothing to classify: it carries `(time, replica_id, +replica_local_id, batch)` exactly as the completion hook does, and the policy +scheduler reads the post-admission load through `get_request_load()`, which +already reflects the admission because `_running_requests` grows inside +`_get_next_batch`. The zero-token iteration publishes nothing in the reference +(`_maybe_publish_request_counts` emits only changed counts) and needs no +observation here. Of the five reference rows only the admission-only row is new. + +**The completion key names the wrong iteration at PP>1.** `on_replica_batch_end` +keys by `ForwardSyncState.get_step_id(batch)`, assigned when the batch's own +forward opened. The reference publishes a completion under the iteration that +applied the output, which at depth `P` is `P-1` iterations later. At PP=1 the +two are the same iteration; at PP>1 the batch key would order every completion +before the admissions emitted while it ran. The key rule therefore applies to +both observation kinds and is read at the observation boundary. + +**A Replica-scoped, monotonic, lane-equal value already exists.** +`ForwardSyncState._next_step_id_by_replica[replica_id]` is the id the next +forward on the Replica will take: equal for all lanes between room openings, +strictly greater than every open or completed step, advanced only by +`resolve_step`/`close_step`. As the key for both kinds it meets invariants 1–4 +and 6 with no new bookkeeping. Its gap is invariant 5: several admissions on +one lane while stage 0 is busy read one value, where the reference gives +strictly increasing steps and latches the intermediate state for one publish +interval. P1 measures whether that case occurs in the target scenario (one +admission per lane) and whether the reference actually keeps peer steps equal +there — `_has_global_unfinished_reqs` all-reduces every 32 steps, so it may +not. The decision stays at the design checkpoint; if a derived identity is +needed, the reason is this measurement, not a preference. + +**Call site and layering.** `self._cluster_scheduler` is constructor-required +(`TypeError` when `None`), so the call in `on_schedule` is unconditional; the +two existing `getattr(self, "_cluster_scheduler", None)` reach-ups in +`_create_batch` are the pattern not to repeat. The completion hook is invoked +by an event and the admission hook by the replica scheduler; the asymmetry is +deliberate, because the per-admission state after the first of two admissions +in one call is visible only inside the loop. + +**The CPU oracle models the engine loop only.** `VllmDPLoadBalancer` already +reproduces the coordinator latch/publish and the frontend score with cited +constants; `reference_loop.py` scripts the `step_with_batch_queue` conjunction, +the changed-count emission and the per-iteration step counter, and feeds the +real balancer. Comparing a Frontier-driven balancer against it isolates the +one mapping W9 changes. + ## The discriminating scenario, derived Let engine `e` receive `k_e` requests in a burst and admit `a_e` of them in @@ -422,10 +477,10 @@ and can only report that rejection. | File | Edit | | --- | --- | -| `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py` | `on_replica_batch_scheduled(...)`, inert default; its payload is one observation of the engine iteration classified by the state table (fields fixed at the design checkpoint), not `pipeline_room_remaining` alone. | -| `frontier/scheduler/replica_scheduler/base_replica_scheduler.py` | Call the hook in the MONOLITHIC/PREFILL admission loop where the state changes; readiness of the oldest output is read from existing DES completion state. | -| `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py` | Drop the PP1 clause of the guard and its error text; keep the publish decision here; key rule per the design checkpoint, satisfying invariants 1–6. | -| `tests/comparison/dp_placement_pp/reference_loop.py` | Independent CPU reference-loop oracle (scripted admissions, empty schedules, completions, controllable readiness) — not a copy of the hook. | +| `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py` | `on_replica_batch_scheduled(time, replica_id, replica_local_id, batch)`, the completion hook's signature, inert default. No readiness field or observation record (second review, R9-01). | +| `frontier/scheduler/replica_scheduler/base_replica_scheduler.py` | Call the hook once per admitted batch in the MONOLITHIC/PREFILL admission loop, after `_num_running_batches += 1`, through the constructor-required `self._cluster_scheduler` without `getattr`/`hasattr` (R9-04). | +| `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py` | Drop the PP1 clause of the guard and its error text; implement `on_replica_batch_scheduled`; key both observation kinds by the observing iteration per the design checkpoint (first candidate: the Replica's next forward id through a plain `ForwardSyncState` accessor), satisfying invariants 1–6. | +| `tests/comparison/dp_placement_pp/reference_loop.py` | CPU oracle of the engine iteration only (scripted admissions, empty schedules, completions, controllable readiness; conjunction, changed-count emission, step counter), feeding the real `VllmDPLoadBalancer`; not a second coordinator/frontend model (R9-05). | | `tests/unit/test_vllm_dp_load_balancer.py`, `tests/integration/test_vllm_dp_placement_runtime.py` | Guard case inverted; the plan §18.11 behavioral matrix (PP2 both callback orders, oldest-ready, PP3 consecutive admission-only iterations on a 6- or 12-layer fixture, full queue, empty schedule, drain, idle peers, bounded bookkeeping); the discriminating scenario against the test-only control; the C35-01 hybrid-layer credit case at PP2. | | `.real-engine/vLLM-BS` (local branch only, D-b) | Case-gated event chain: iteration result, emitted report, coordinator receive/publish with snapshot id, frontend application, frontend routing; named `waiting`/`running`; correlation ids; buffered per-process JSONL. | | `AGENTS.md:620`, this file, `plan.md`, `progress.md`, `validation.md`, `review.md` | Wording and records. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md index 023884a7..2403ab93 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-22 | §18 second review at the user's direction (codebase integration, readability, value, no hard-coding/patches/over-defense/redundancy, plain names): findings R9-01..R9-08 in §18.12; D9-1, D9-2, P1(a), §18.10 and the §18.11 representation column amended in place. Execution still not started. | | 2026-09-21 | Landed the execution specification verbatim (Section "Execution Specification" below) and recorded the amendments agreed with the user before Step 0. | | 2026-09-22 | Added amendment A12 and the Step 9 draft (§18; numbered §17 until 2026-09-22, when the duplicate number was fixed) for PP>1 support of the opt-in vLLM DP placement; awaiting user approval. | | 2026-09-22 | Step 9 section renumbered §17 → §18 (the source index already held §17). §18 corrected per the 2026-09-22 external review, P9-01..P9-06: engine-iteration state table instead of a room-only hook rule; K1, K3-as-written and stride keys rejected as acceptance basis, six key invariants; instrumentation chain and T2 qualification; CPU reference-loop oracle and valid negative controls; PP3 fixture with a valid layer count; revised work graph and C1–C5. Corrections collected in §18.11. Execution still not started. | @@ -847,7 +848,7 @@ Under [R1]: `frontier/profiling/moe/moe_vllm_kernel.py`, `frontier/entities/stag ## 18. Step 9 — PP>1 support for the opt-in vLLM DP placement -**Status 2026-09-22:** the user answered every §18.7 decision the same day (verbatim in `requirements.md`). Later the same day an external review of PR34/PR35 (`.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_Review_2026-09-22.md`, findings P9-01..P9-06) corrected this plan; the corrections are applied in place below and collected with their evidence in §18.11. No source edit and no GPU submission has been made; execution starts at the first node of the §18.5 graph once the user confirms the start. Research followed the `codebase-design` skill (§18.10); the ground-truth comparison follows `frontier-calibration` v2 as written (§18.9). +**Status 2026-09-22:** the user answered every §18.7 decision the same day (verbatim in `requirements.md`). Later the same day an external review of PR34/PR35 (`.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_Review_2026-09-22.md`, findings P9-01..P9-06) corrected this plan; the corrections are applied in place below and collected with their evidence in §18.11. A second review at the user's direction (2026-09-22, quality gates for core-module changes) is recorded in §18.12 and amended D9-1, D9-2, P1 and §18.10 in place. No Step 9 source edit has been made; execution starts at the first node of the §18.5 graph once the user confirms the start. Research followed the `codebase-design` skill (§18.10); the ground-truth comparison follows `frontier-calibration` v2 as written (§18.9). ### 18.1 Goal and acceptance criteria @@ -867,7 +868,7 @@ Under [R1]: `frontier/profiling/moe/moe_vllm_kernel.py`, `frontier/entities/stag | --- | --- | | PP>1 makes `max_concurrent_batches = pipeline_parallel_size`, which builds the `batch_queue` and selects `step_with_batch_queue`. | `vllm/v1/executor/multiproc_executor.py:325-329`, `vllm/v1/engine/core.py:147-157` | | One iteration: `scheduler.schedule()` (waiting→running for new requests; in-flight requests are skipped by the `num_new_tokens == 0` rule), append; if the queue still has room and the oldest batch is not done, return without completing anything (a **schedule-only iteration**); otherwise pop the oldest, wait, `update_from_output` (finished requests leave `running`). | `core.py:318-370`, `vllm/v1/core/sched/scheduler.py:436-441` | -| `_maybe_publish_request_counts()` runs after **every** iteration and publishes `(running, waiting)` whenever it changed, carrying `step_counter` and `current_wave`. `step_counter` is incremented afterwards in `_has_global_unfinished_reqs`, so the publication of iteration `k` carries `k-1`. | `core.py:1075-1087, 1089-1137` | +| `_maybe_publish_request_counts()` runs after **every** iteration and publishes `(running, waiting)` whenever it changed, carrying `step_counter` and `current_wave`. `step_counter` is incremented afterwards in `_has_global_unfinished_reqs`, so the publication of iteration `k` carries `k-1`. That method all-reduces only every 32 steps, so DP engines' step counters are not held in lockstep per iteration (R9-08). | `core.py:1075-1087, 1089-1137` | | Coordinator: a report whose `(wave, step)` is strictly greater than the last latches the previous counts when `stats_changed`; equal keys apply without latching; smaller keys warn. Publication every `stats_update_interval_ms` (100 ms) while changed, 5000 ms otherwise, with a 50 ms first-collection wait. | `vllm/v1/engine/coordinator.py:196-227, 280-312` | | Frontend: `score = waiting * 4 + running`, first minimum from `eng_start_index`, local `+client_count` waiting reservation until the next publication. Only the online `vllm serve` path builds `DPLBAsyncMPClient`; offline `LLM` DP is SPMD without a balancer. | `vllm/v1/engine/core_client.py:85-103, 1131-1156` | | No DP+PP prohibition: per-engine `world_size = PP*TP`; the `arg_utils` assertions concern hybrid/external LB and the `mp` backend only. | `vllm/config/parallel.py:314`, `vllm/engine/arg_utils.py:1221-1269` | @@ -888,8 +889,8 @@ Equivalence argument, as corrected by P9-01. The reference decides each engine i | Id | Decision | Content | | --- | --- | --- | -| D9-1 | Schedule-time hook (corrected per P9-01) | Name fixed by the user (D-d): `BaseClusterScheduler.on_replica_batch_scheduled(...)`, inert default, called from the MONOLITHIC/PREFILL admission branch of `BaseReplicaScheduler.on_schedule`. What it carries is **not** decided by queue room alone: `pipeline_room_remaining` tests one of the reference's three conditions and has no representation for the iteration whose oldest output is already ready, for the zero-token iteration, or for the drain. The hook passes one observation of the engine iteration as classified by the §18.11 state table — captured where the state changes, from existing scheduling/completion ownership (DES completion state stands in for `oldest.done()`; no futures or threads) — and the decision whether to publish stays in one place, the existing load-report owner in `VllmLoadBalancingClusterScheduler`. If a small explicit observation record is needed, prefer it to callers manufacturing a vLLM-specific heuristic from queue length; do not spread partly redundant booleans through the scheduler. Rejected, unchanged: reconstructing the report in `ReplicaScheduleEvent` after `on_schedule` (cannot see the state after the first of two admissions in one call). The exact fields are fixed at the design checkpoint of §18.5, after the P1 probes. | -| D9-2 | Report key (corrected per P9-02) | **Rejected as acceptance basis:** K1 (equal key; never latches, misses the reference latch of the pre-admission state), **K3 as written** (a fresh label per admission callback equates callback order with iteration order and gives two peer lanes of one logical iteration different keys — the executable counterexample in §18.11 shows the coordinator then latches a partial snapshot and routes differently), and stride keys (`2*cohort±1`; arbitrary factor, no room for consecutive admission-only iterations at PP≥3). **Rule to be chosen at the design checkpoint:** first identify the reference-equivalent logical engine iteration, then reuse an existing scheduler iteration/forward identity if it actually represents it; otherwise a derived identity or a small additional report-state field. Not a per-batch counter because it is available; not a global identity registry because the old getter is unavailable at admission. The chosen rule must satisfy invariants I1–I6 of §18.11, and the P1 probe must establish logical-iteration membership, not print provisional/resolved Batch ids and pick whichever looks monotonic. Only comparisons are used, so PP=1 behavior stays identical (C2 verifies). | +| D9-1 | Schedule-time hook (corrected per P9-01; payload fixed per §18.12 R9-01/R9-04) | Name fixed by the user (D-d): `BaseClusterScheduler.on_replica_batch_scheduled(time, replica_id, replica_local_id, batch)`, the same signature as `on_replica_batch_end`, inert default (`return None`). Called once per admitted batch from the MONOLITHIC/PREFILL admission loop of `BaseReplicaScheduler.on_schedule`, immediately after `self._num_running_batches += 1`, through the constructor-required `self._cluster_scheduler` (no `getattr`/`hasattr`). No readiness flag, no `pipeline_room_remaining`, no observation record: the DES has no "ready but unapplied" state (R9-01), so the hook carries the lane, the batch and the time, and the policy scheduler reads the post-admission load through the existing `get_request_load()`. The publish decision stays in `VllmLoadBalancingClusterScheduler`. Rejected, unchanged: reconstructing the report in `ReplicaScheduleEvent` after `on_schedule` (cannot see the state after the first of two admissions in one call). | +| D9-2 | Report key (corrected per P9-02; candidates per §18.12 R9-02/R9-03) | **Rejected as acceptance basis:** K1 (equal key; never latches, misses the reference latch of the pre-admission state), **K3 as written** (a fresh label per admission callback equates callback order with iteration order and gives two peer lanes of one logical iteration different keys — the executable counterexample in §18.11 shows the coordinator then latches a partial snapshot and routes differently), and stride keys (`2*cohort±1`; arbitrary factor, no room for consecutive admission-only iterations at PP≥3). **Rule to be chosen at the design checkpoint:** first identify the reference-equivalent logical engine iteration, then reuse an existing scheduler iteration/forward identity if it actually represents it; otherwise a derived identity or a small additional report-state field. Not a per-batch counter because it is available; not a global identity registry because the old getter is unavailable at admission. The chosen rule must satisfy invariants I1–I6 of §18.11, and the P1 probe must establish logical-iteration membership, not print provisional/resolved Batch ids and pick whichever looks monotonic. Only comparisons are used, so PP=1 behavior stays identical (C2 verifies). **Consequence stated by R9-02:** the rule keys *both* observation kinds by the observing iteration; the current completion key `ForwardSyncState.get_step_id(batch)` names the iteration that scheduled the batch, which coincides with the applying iteration only at PP=1. **First candidate to test in P1 (R9-03):** the Replica-scoped `ForwardSyncState._next_step_id_by_replica` value read at the observation boundary, through a plain accessor; it needs no new bookkeeping and meets I1–I4 and I6, and its known gap is I5 (consecutive admission-only iterations on one lane while stage 0 is busy collapse to one key). Whether the reference itself keeps peer keys equal in that case is a P1/G4 question (R9-08), not an assumption. | | D9-3 | Guard | Drop `num_pipeline_stages == 1`; keep the other four clauses and the dense multi-lane rejection (its evidence is PP-independent); update the error text. The layer-partition guard (`num_layers % num_pipeline_stages == 0`, `replica_config.py`) is untouched: removing the PP1 clause is not permission to bypass the other independent guards or to add uneven partitioning. | | D9-4 | Determinism and flags | No new `EventType`, no new config flag, no balancer constant change (`design.md` event-type determinism; plan §10 "no tuning flags"). | | D9-5 | Docs | `AGENTS.md:620` ("one pipeline stage" removed), `design.md` guard row plus a section "Schedule-time reports under pipeline parallelism" with the P1 probe table, `plan.md`/`progress.md`/`validation.md`/`review.md`. | @@ -917,7 +918,7 @@ CPU packages `P*` change Frontier; ground-truth packages `G*` never change Front | Package | Content | Acceptance | | --- | --- | --- | -| P1 Probes | (a) A small independent CPU reference-loop harness (`tests/comparison/dp_placement_pp/reference_loop.py`, not a copy of the proposed hook) that replays scripted admissions, empty schedules and completions with controllable output readiness through the reference's `step_with_batch_queue` conjunction, `_maybe_publish_request_counts`, the coordinator latch and the frontend scoring, and emits loads, key relations and visible snapshots. (b) Scratch Frontier probes at PP=2 and PP=3 for four shapes (MoE `attn_dp=2` burst and staggered, MoE `attn_dp=1`, dense `attn_dp=1`) recording `(lane, boundary, logical-iteration membership, load)` — membership, not merely provisional/resolved Batch ids. No source change. | State table (§18.11) confirmed or amended from evidence; the key-grouping rule proposed with its I1–I6 argument; both recorded in `design.md` for the design checkpoint. | +| P1 Probes | (a) A small CPU reference-loop harness (`tests/comparison/dp_placement_pp/reference_loop.py`) that models only the engine iteration — the `step_with_batch_queue` conjunction, `_maybe_publish_request_counts` changed-count emission and the per-iteration `step_counter` — over scripted admissions, empty schedules and completions with controllable output readiness, and feeds the emitted `(engine, step, counts)` reports into a `VllmDPLoadBalancer` instance (R9-05: the coordinator latch and frontend scoring are already modeled there and are not re-implemented). A Frontier-driven balancer instance is compared against it on the same scripted history, which isolates the iteration-to-report mapping W9 changes. (b) Scratch Frontier probes at PP=2 and PP=3 for four shapes (MoE `attn_dp=2` burst and staggered, MoE `attn_dp=1`, dense `attn_dp=1`) recording `(lane, boundary, logical-iteration membership, load)` — membership, not merely provisional/resolved Batch ids. No source change. | State table (§18.11) confirmed or amended from evidence; the key-grouping rule proposed with its I1–I6 argument; both recorded in `design.md` for the design checkpoint. | | P2 Implement | D9-1..D9-3 in `base_cluster_scheduler.py`, `base_replica_scheduler.py`, `vllm_load_balancing_cluster_scheduler.py`, implementing the state model fixed at the design checkpoint. | Existing unit tests pass except the intentionally inverted guard case; the §18.11 matrix rows that P3 owns pass. | | P3 Unit | `tests/unit/test_vllm_dp_load_balancer.py`: guard param at `:538` becomes positive; PP=1 never reports at schedule time; PP=2 cold fill with peer lanes in both callback orders groups peer reports under one logical iteration (no partial peer snapshot); room remaining with the oldest output ready yields one combined report; PP=3 two admission-only iterations before the first completion are distinct iterations with peer equality inside each; full queue yields one report decision; idle/changed-to-zero peers; bounded bookkeeping after many iterations; helper `num_pipeline_stages` parameter. Expected counts, keys and placements are written independently, not taken from a run. | New tests fail before P2 and pass after. | | P4 Integration | `tests/integration/test_vllm_dp_placement_runtime.py`: PP=2 dense `attn_dp=1` and MoE `attn_dp=2` (`moe_ep=2`) cases and a PP=3 case on a separate fixture with 6 or 12 layers (`_model()` today has 4; the tiny Qwen has 8; the divisibility guard stays) — completion with request/token/owner conservation, `routing_times == cluster_schedule_times`, reports only at classified iterations, event-type set equal to the round-robin baseline; the §18.6 discriminating scenario against the §18.11 controls (test-only guard-lifted completion-reporting baseline vs corrected implementation; `placements_fixed != placements_round_robin` is not sufficient because PP1 `vllm_load_balancing` already differs from round-robin); the C35-01 hybrid-layer mixed-batch credit case carried into the PP2 fixture. | Pass; each control fails for its stated reason. | @@ -974,11 +975,11 @@ Unchanged and not claimed: multiple Replicas, multiple frontends (`client_count ### 18.10 Design vocabulary (`codebase-design`) -- **Module.** `VllmLoadBalancingClusterScheduler`, with `VllmDPLoadBalancer` as an internal module. Its **interface** is `schedule_at`, `on_replica_batch_end`, and (new) `on_replica_batch_scheduled`, plus the facts a caller must know: reports are per lane, the schedule-time report is emitted once per admitted batch and only while pipeline room remains, at equal simulated time the completion report precedes the admissions it triggers, and the module never raises on an inert path. +- **Module.** `VllmLoadBalancingClusterScheduler`, with `VllmDPLoadBalancer` as an internal module. Its **interface** is `schedule_at`, `on_replica_batch_end`, and (new) `on_replica_batch_scheduled`, plus the facts a caller must know: reports are per lane, the schedule-time report is emitted once per admitted batch, at equal simulated time the completion report precedes the admissions it triggers, and the module never raises on an inert path. - **Seam.** `BaseClusterScheduler.on_replica_batch_*` already has two **adapters** — the inert default used by every other cluster scheduler and this module — so the new hook extends a real seam rather than creating a hypothetical one. The call site in `on_schedule` is the only place that knows whether pipeline room remains, which is why the seam sits there and not in `ReplicaScheduleEvent`. -- **Depth.** The reservation, publish deadlines, latch, schedule-only detection, and the K3 relabeling all stay behind the same three methods; callers learn nothing new to gain PP>1 support. **Deletion test:** removing the hook would force every event that admits a batch to reconstruct vLLM's per-iteration publication — the complexity reappears across callers, so the module earns its keep. +- **Depth.** The reservation, publish deadlines, latch, the admission observation and the key rule all stay behind the same three methods; callers learn nothing new to gain PP>1 support. **Deletion test:** removing the hook would force every event that admits a batch to reconstruct vLLM's per-iteration publication — the complexity reappears across callers, so the module earns its keep. - **Test surface.** Tests drive the module through its interface and assert observable placements and published counts; the relabeled keys are implementation and are not asserted directly. -- **Design it twice.** Three interface shapes were compared: (1) in-loop push hook with `pipeline_room_remaining` — chosen: smallest interface, the room predicate lives where it is known, one adapter per cluster scheduler; (2) event-level reconstruction after `on_schedule` — rejected: cannot observe the state after the first of two admissions in one call, so it is shallow and wrong; (3) pull-style `iter_admission_loads()` on the replica scheduler — rejected: widens the replica scheduler's interface for one caller and inverts the push direction the completion report already uses. +- **Design it twice.** Three interface shapes were compared: (1) in-loop push hook with the completion hook's signature — chosen: smallest interface, called where the per-admission state is visible, one adapter per cluster scheduler (R9-01 removed the `pipeline_room_remaining` argument the first draft gave it); (2) event-level reconstruction after `on_schedule` — rejected: cannot observe the state after the first of two admissions in one call, so it is shallow and wrong; (3) pull-style `iter_admission_loads()` on the replica scheduler — rejected: widens the replica scheduler's interface for one caller and inverts the push direction the completion report already uses. ### 18.11 Corrections from the 2026-09-22 external review (P9-01..P9-06) @@ -994,6 +995,8 @@ Recorded the day the review arrived; every item below is a plan/record change, n | Zero-token scheduled output, prior work queued | Does not take the early-return branch | Explicit mapping of the empty iteration; the current hook design has no representation for it | | No new request work, prior output queued | Drain an output, publish changed counts | Completion-only observation | +Representation column revised by §18.12 R9-01: in the DES a completion is atomic at its end event and itself triggers the lane's same-time schedule, so rows 2, 3 and 5 are the existing completion report (row 2's merge with the following admission is the key rule's invariant 3), row 4 changes no counts and needs no observation, and only row 1 is new. The reference column stands. + Steady-state preconditions (replacing "needs no change"): with depth `P`, `B_k`'s append completes `B_(k-P+1)`; Frontier's completion report at `B_(k-P+1)`'s end shows the combined state only if `B_k` was admitted at `B_(k-P)`'s end, no empty iteration intervened, and no completion became visible between the two boundaries. These rows are tested, not assumed (P3/P4). **P9-02 — K3 as written is not order-preserving.** The coordinator distinguishes equal keys (apply without latch) from strictly greater keys (latch the previous counts). A global `next_label += 1` per report callback preserves neither the equality class of two peer lanes reporting one logical iteration nor the source ordering. Executable counterexample, reproduced on this branch's `VllmDPLoadBalancer` on 2026-09-22 (zero initial counts; lane 0 reports `waiting=0, running=3` at 10 ms, lane 1 the same at 20 ms; a request is placed at 80 ms): with one iteration key for both reports the frontend sees `[(0,3),(0,3)]`, last publication 70 ms, and selects lane 0 by first minimum; with a fresh key for the second report the coordinator latches the partial snapshot `[(0,3),(0,0)]` at 20 ms and the request goes to lane 1. Same inputs, different published state. Second interleaving to cover: one lane completes cohort `C`, moves on to an admission-only observation, and the peer's completion of `C` arrives later; reusing `C`'s label after minting the next one contradicts strict emission order, and suppressing the native out-of-order warning by inventing newer identities is not a fix. Invariants for the chosen rule: @@ -1047,3 +1050,20 @@ Exact baseline comparison only for behavior meant to stay unchanged; new PP2/PP3 **P9-06 — work graph and acceptance language.** Applied in §18.5 and §18.1. A native out-of-order warning is evidence to analyze (the reference applies the counts after warning), not proof of a simulator bug nor a reason to rewrite native ordering. +### 18.12 Second review (2026-09-22, user-directed): codebase integration and quality gates + +Review question, as set by the user: is every planned change to Frontier's core modules grounded in the whole codebase, readable and maintainable, high-value (fidelity or simulation function, not replaceable), and free of hard-coding, temporary patches, over-defensive code, redundant design and vague names. Findings are against the code on `c231322`; each states the source it rests on and what it changes in this plan. None of them changes source now. + +| Id | Finding (source) | Effect on the plan | +| --- | --- | --- | +| R9-01 | **The state table over-specifies what Frontier must represent.** `GlobalBatchEndEvent` applies a batch's completion atomically (`replica_scheduler.on_batch_end`, then `cluster_scheduler.on_replica_batch_end`, `global_batch_end_event.py:180-185`) and the lane's next `ReplicaScheduleEvent` follows at the same simulated time. There is no "oldest output ready but unapplied" state to classify. A zero-token iteration changes no counts, and `_maybe_publish_request_counts` (`core.py:1075-1087`) publishes only changed counts, so it needs no Frontier observation; its `step_counter` increment shifts later key *values*, and only comparisons are used. Of the five §18.11 rows, only the admission-only row is new; rows 2, 3, 5 are the existing completion report and row 2's merge with the same-time admission is invariant I3 of the key rule. | D9-1 payload reduced to the completion hook's signature; no readiness field, no observation record; §18.11 representation column annotated. Avoids a redundant classifier and booleans threaded through `on_schedule`. | +| R9-02 | **The existing completion key changes meaning under PP.** `on_replica_batch_end` keys by `ForwardSyncState.get_step_id(batch)`, the forward identity resolved when the batch's own forward opened (`forward_sync_state.py`, `resolve_step`). vLLM keys a publication by the iteration whose counts changed; a completion is published under the iteration that *applied* the output, `P-1` iterations after the one that scheduled it. The two coincide only at PP=1. Keeping the batch key at PP>1 would order every completion report before the admission reports emitted while it was in flight — the P9-02 "second interleaving" is the normal case, not an edge. | D9-2 now requires both observation kinds to be keyed by the observing iteration. C2's PP=1 byte comparison is the check that the changed key keeps identical comparisons at PP=1. | +| R9-03 | **An existing identity is the first candidate; do not mint a new counter before testing it.** `ForwardSyncState._next_step_id_by_replica[replica_id]` is Replica-scoped, monotonic, equal for every lane between room openings and strictly greater than every open or completed step. Read at an observation boundary it satisfies I1, I2, I3 (a completion and the admission it triggers at the same time read the same value), I4 and I6 with no bookkeeping. Known gap: I5 — on one lane, admissions made while stage 0 is still busy (cold fill of several slots in one `on_schedule` call) read the same value, whereas the reference gives them strictly increasing steps and latches the intermediate state for one publish interval. | P1(b) probes this candidate first and measures the I5 case; the design checkpoint decides whether the I5 gap is material (the §18.6 scenario admits one batch per lane) or needs a derived identity. A plain accessor on `ForwardSyncState` (named for what it returns) is the only new surface if adopted. | +| R9-04 | **Call site and layering.** `BaseReplicaScheduler` requires `cluster_scheduler` at construction (`TypeError` otherwise) and stores it as `self._cluster_scheduler` (`base_replica_scheduler.py:43-56`). The two `getattr(self, "_cluster_scheduler", None)`/`hasattr(...)` reach-ups at `:423` and `:468` are the over-defensive pattern the gates forbid; the new call must not add a third. The completion hook is invoked by an event, the admission hook by the replica scheduler, because per-admission state is visible only inside the loop; record the asymmetry and its reason in `design.md` so nobody later "fixes" it by moving the call to `ReplicaScheduleEvent`. | D9-1 wording; `design.md` planned-edits row. | +| R9-05 | **The CPU oracle must not re-implement what `VllmDPLoadBalancer` already models.** The balancer reproduces the coordinator latch/publish and the frontend score with line-cited constants (`vllm_dp_load_balancer.py`). A second implementation of the same in `tests/comparison/` would duplicate W4 and could encode one misunderstanding twice. | P1(a) narrowed to the engine-iteration loop feeding the real balancer; comparison isolates the iteration-to-report mapping. | +| R9-06 | **Test surface and names.** `test_vllm_dp_placement_runtime.py` asserts only key ordering and lane distinctness per key (`keys == sorted(keys)`, `len(lanes) == len(set(lanes))`); these hold under any rule meeting I1–I2 and stay as the PP>1 assertions too. New names stay plain: the user's hook name; an accessor named for its value; no "observation", "classifier" or "snapshot manager" objects. `AGENTS.md:620` is the sentence to edit ("one pipeline stage"). | P3/P4 acceptance wording; D9-5 anchor verified. | +| R9-07 | **Value and size.** The Frontier change is a guard clause, one inert base method, one call in `on_schedule`, one method in the policy scheduler and possibly one accessor; it completes a user-requested simulation capability and touches no other cluster scheduler's behavior. The G-packages (vLLM-BS instrumentation, three GPU jobs) are validation and must not leak into `frontier/`: no env-var switches, no trace fields on Frontier events; Frontier-side evidence uses the runtime test's existing report trace and `metrics_ground_truth.jsonl`. | Boundary stated for P2/G1. | +| R9-08 | **Reference precision.** `_has_global_unfinished_reqs` increments `step_counter` every iteration but all-reduces only every 32 steps (`core.py:1131-1135`), so DP engines are not iteration-lockstep in the reference; peer-key equality is an idealization inherited from W4 and, under PP with independent admission-only iterations, peer steps can drift until the next forward aligns them. | §18.2 row amended; "key grouping" stays a first-cause label in G5, and the I5 question of R9-03 is answered from the G4 trace, not assumed either way. | + +Gate check of the plan after these amendments: no hard-coded constants beyond the cited reference values already in `vllm_dp_load_balancer.py`; no temporary patch (the guard is removed, not bypassed); no new defensive branches (the hook is unconditional on a constructor-required reference); no redundant state (no readiness classifier, no second coordinator model); names are the user's hook name and plain accessors. + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 157e769a..70a1412e 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-22 | FP8 native rerun PASS (`exp-0922-202645-561899`, 8 passed, exit 0). Second Step 9 plan review (plan §18.12, R9-01..R9-08) recorded; Step 9 still not started. | | 2026-09-21 | Step 0 started: records landed, environment created, baseline pending. | | 2026-09-22 | Maintainer review dispositions recorded; Checkpoint C closed: parent merged, W2 tests strengthened, W2 re-measured with one harness revision. | | 2026-09-22 | Checkpoint D first half: W3, the shared monolithic forward lifecycle, implemented, tested against four deliberate-defect controls, and committed as `65ed8a7`. | @@ -28,7 +29,7 @@ | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | | Current step | Step 8 closed; external review corrections A–E applied 2026-09-22 (`test_report_2026-09-22_review_corrections.md`). Step 9 is planned (`plan.md` §18, corrected per the review) and **not started**. | | Publication | PUSHED_VERIFIED: `f7c31e4` (C35-01 source + tests), `ca1b9b6` (FP8 `block_shape`, optional-torch skip, W6 report, profiling guide), `57ffa5b` (records, Step 9 plan corrections); remote head `57ffa5b` confirmed; PR34 correction `2310417` merged in as `0d025f8`. Draft PR 35 body PATCHed 2026-09-22T12:12Z through `gh api` and read back; still draft, MERGEABLE, base `refactor/oversized-module-split`. | -| Next action | **User decisions.** (1) Go/no-go for re-running the corrected FP8 native check on one H800 under `codesign` (`NOT_RUN`). (2) Start signal for Step 9, at the first node of the `plan.md` §18.5 graph. PR 35 remains draft and nothing was merged. Carried forward in `future.md`: re-point the collective-sim gitlink at `main` once companion PR 1 merges, and repair the `tests/debug/` pointers that 10 baseline unit failures share. Retargeting PR 35's base to `main` waits on PR 34. | +| Next action | **User decision.** Start signal for Step 9, at the first node of the `plan.md` §18.5 graph (plan amended by the §18.12 second review). The FP8 native rerun is done and PASS. PR 35 remains draft and nothing was merged. Carried forward in `future.md`: re-point the collective-sim gitlink at `main` once companion PR 1 merges, and repair the `tests/debug/` pointers that 10 baseline unit failures share. Retargeting PR 35's base to `main` waits on PR 34. | ## Step status @@ -40,7 +41,7 @@ | 3 | Shared monolithic forward | PASS | unit PASS (23 new, 3717 total, failure set identical to the parent); integration PASS (real event loop, 4 mixed-phase cohorts); four deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 4 | Opt-in vLLM DP placement | PASS | unit PASS (61 new, 3778 total, failure set identical to the parent); integration PASS (3 cases in the real event loop, including a placement that diverges from round-robin); five deliberate-defect controls each fail for their own reason; matrix PASS, 71 of 71 identical against the stated expectation | PUSHED_VERIFIED | NOT_REVIEWED | | 5 | Routing implementation identity | CLOSED, NOT PORTED (user decision 2026-09-22) | n/a: no source change; restored files re-run, failure set identical to the parent (torch-missing only) | PUSHED_VERIFIED (records + PR 35 section) | REVIEWED (user chose to keep the single global field) | -| 6 | Legacy fused-MoE profiling | PASS | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778). Native PASS: 8 of 8 in `tests/integration/test_moe_fused_expert_numerical_parity.py` at `rtol=0, atol=0` on H800, job `exp-0922-145047-660565` under `codesign` | PUSHED_VERIFIED (source, tests, docs, records, PR 35 section) | — | +| 6 | Legacy fused-MoE profiling | PASS | CPU PASS (7 new tests; HEAD comparison shows the same single environment-dependent failure; default-environment suite unchanged at 84/3778). Native PASS: 8 of 8 in `tests/integration/test_moe_fused_expert_numerical_parity.py` on H800 — seven comparisons at `rtol=0, atol=0` plus the FP8 structural check — job `exp-0922-145047-660565`, re-run with the corrected `block_shape` wiring as `exp-0922-202645-561899` (8 passed), both under `codesign` | PUSHED_VERIFIED (source, tests, docs, records, PR 35 section) | — | | 7 | Optional zero-payload backend | PASS. Companion fix published as `fwyc0573/frontier-htsim` `eb7bc4f` with draft PR 1; Frontier gitlink moved from `b8518af`; no Frontier source change | Companion 9 passed, negative control 6 of 9 fail on pristine sources. Frontier 4 passed, negative control 3 of 4 fail at the old gitlink. Clean checkout resolves `eb7bc4f` from the published remote, builds, and passes. Suite back to the 84-failure baseline with 3782 passing after narrowing three governance scans to Frontier-owned sources | PUSHED_VERIFIED | — | | 8 | Combined regression, PR hand-off | PASS | unit 84 failed / 3782 passed with a `FAILED` set identical to the `origin/main` baseline; integration 15 passed / 22 skipped / 5 errors, the errors environmental and identical on the base; 16 of 16 architecture examples pass; 4 of 4 `PP=2` cases pass; cold and warm predictor-cache runs byte-identical | PUSHED_VERIFIED (records + PR 35 body carrying the Step 8 results, the record links and the implementation commits) | REVIEWED (external review 2026-09-22; corrections below) | | 8+ | External review corrections A–E | PASS | unit 84 failed / 3789 passed / 50 skipped / 10 errors with the `FAILED` set identical to the baseline (+7 passes are the new tests, +1 skip and −1 error are the optional-torch module); mixed-forward unit 26 passed; real-loop hybrid-layer case 2 passed with the negative control failing on the pre-fix source; arithmetic 9 passed under torch | PUSHED_VERIFIED | NOT_REVIEWED | @@ -192,7 +193,7 @@ Review document: `.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_ | --- | --- | --- | --- | | A | C34-01 predictor-cache eligibility by executed case list | PR34 branch `2310417`, merged here as `0d025f8` | completed | | B | C35-01 decode credit at a dense layer for a mixed source | `collective_timing.advance_decode_layer`, `dense_metrics.complete_dense_layer`; unit + real-loop tests | completed | -| C | C35-03 FP8 `block_shape` wiring + CPU boundary test; C35-04 optional-torch skip | `tests/integration/test_moe_fused_expert_numerical_parity.py`, `tests/unit/test_moe_fused_expert_arithmetic.py` | completed; native rerun `NOT_RUN` | +| C | C35-03 FP8 `block_shape` wiring + CPU boundary test; C35-04 optional-torch skip | `tests/integration/test_moe_fused_expert_numerical_parity.py`, `tests/unit/test_moe_fused_expert_arithmetic.py` | completed; native rerun PASS `exp-0922-202645-561899` | | D | C35-02 scope table; C35-03 seven-plus-one wording; C35-05 records consistency and PR bodies | W6 report §5/§8, `docs/profiling/README.md`, `summary.md`, `review.md` D2, this file, PR34/PR35 bodies | completed | | E | P9-01..P9-06 plan corrections | `plan.md` §18 (renumbered from §17) and §18.11, `design.md` W9 | completed (records only) | | F | W9 implementation | — | **not started** (user: 暂不开启) | @@ -204,3 +205,6 @@ Review document: `.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_ | 2026-09-22 | pending | Plan drafted in `plan.md` §18 with amendment A12 and `requirements.md` rows; awaiting user approval and the D-a..D-g answers. No source, GPU, or publication action taken. | | 2026-09-22 | in-progress (planning closed) | User answered D-a..D-g; `plan.md` §18 finalized (decisions, G1–G5 ground-truth packages, calibration case binding, `codebase-design` vocabulary), `design.md` W9 written, `requirements.md` updated. Records committed and pushed (SHA in the commit log). No source edit, no GPU submission; next action: P1 probe + G1/G2 once the user confirms the start. | | 2026-09-22 | in-progress (planning corrected) | External review P9-01..P9-06 applied: `plan.md` §17 → §18 (duplicate number), state table and preconditions replace the room-only hook rule and the steady-state claim, K1/K3/stride rejected as acceptance basis with invariants I1–I6, instrumentation chain and T2 qualification, CPU reference-loop oracle and valid controls, PP3 fixture with a valid layer count, revised graph and C1–C5 (`plan.md` §18.11, `design.md` W9). Records only; execution still awaits the user's start signal. | + +- 2026-09-22: FP8 native rerun authorized by the user. Same launcher as `exp-0922-145047-660565`, worktree clean at `c231322`: `exp-0922-202645-561899`, `codesign` / H800 (`gpu-h800-0095`), `Succeeded`, `8 passed in 14.27s`, `W6:PARITY_EXIT=0`. Log via `logs_replica` only (`logs_rjob` empty); the first fetch two minutes after completion returned a truncated window, the second fetch five minutes later returned the full tail. Records: W6 report §8, corrections report C4, review.md C35-03, validation.md Step 6. +- 2026-09-22: Second Step 9 plan review at the user's direction (quality gates). Eight findings R9-01..R9-08 recorded in `plan.md` §18.12 with amendments to D9-1 (hook payload = completion hook signature), D9-2 (key both observation kinds by the observing iteration; first candidate `ForwardSyncState._next_step_id_by_replica`, I5 gap to be measured), P1(a) (oracle = engine loop feeding the real balancer), §18.10, §18.11 representation column, §18.2 (32-step all-reduce); `design.md` section "What the code already provides"; `review.md` disposition table. Records only; no Step 9 source change; execution awaits the start signal. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md index 7fa6f734..5ea099cf 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md @@ -82,3 +82,11 @@ Reading: read the external review in full, verify each finding against the sourc | GPU | The corrected FP8 native check (package C) was **not** re-run on a worker; it needs one H800 under `codesign` and a fresh go from the user. Recorded as `NOT_RUN`. | | Publication | PR34 correction committed and pushed first, then merged (not rebased) into PR35; PR35 commits pushed; both PR bodies updated through the API. Draft state of PR35 untouched. | +## [Original Request] 2026-09-22 — FP8 native rerun authorization and second Step 9 plan review + +> 授权：FP8 native 重跑（1×H800 codesign）；再次review plan §18，review的核心为： 确保当前计划的代码模块的实现/改动/重构是基于整体codebase的，记住，对frontier 核心模块的代码的修改和实现上，确保可读性和可维护，任何引入的修改和实现都应该是高价值的（要么对fidelity有收益，要么与模拟功能直接相关，不可替代），禁止hard-coding，禁止临时补丁，禁止过度防御，禁止冗余性设计和实现，禁止使用ai味命名函数和变量。 + +| Item | Decision / outcome | +| --- | --- | +| FP8 native rerun | Authorized and executed: `exp-0922-202645-561899`, `codesign` / 1×H800, 8 passed in 14.27 s, exit 0 (W6 report §8). | +| Plan §18 second review | Performed against `c231322` with the stated gates; findings R9-01..R9-08 in `plan.md` §18.12, amendments to D9-1, D9-2, P1, §18.10, §18.11 and `design.md`. Records only; no Step 9 source change; execution still awaits the user's start signal. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index e5263421..ce36fb54 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-22 | Second Step 9 plan review at the user's direction (quality gates for core-module changes): findings R9-01..R9-08 recorded with dispositions; plan §18.12, design.md. | | 2026-09-21 | Created with the pinned source snapshot. | | 2026-09-21 | Step 1 complete: candidate and vLLM audits landed, dispositions recorded, two decision checkpoints raised. | | 2026-09-21 | Corrected the W3 and W4 rows: the step-id namespace is not partitioned by sync kind, only the open-step binding table is. Verified against `forward_sync_state.py` at `c18eb2c`. | @@ -422,7 +423,7 @@ Reviewed revisions: PR34 `6ef0a3c`, PR35 `0137269`. Each finding was checked aga | C34-01 cache comparison eligibility ignores `cases_executed_in_last_run` | Confirmed: `compare_labels` read only `cache_clean_before_run` and `case_filter`; `--start`/`--limit` leave no filter | FIXED | PR34 `2310417` (runner + 4 gate tests), merged as `0d025f8`; Checkpoint B verdict re-derived, unchanged (`task_2026-09-21_oversized_module_split/test_report_2026-09-22_cache_eligibility_correction.md`) | | C35-01 decoding request in a prefill-mode mixed batch uncredited at a dense layer | Confirmed by call path: `complete_dense_layer(phase="prefill")` → `handle_prefill_sync_collective`, which credits nothing; only the shared forward and decode helpers credit | FIXED | `advance_decode_layer` helper (validate then increment) used by all three completion paths; dense prefill-mode source credits its decoding members. Unit: mixed source at a dense layer +1 for the decoder, 0 for the prefiller, pure-prefill control credits nothing; `MoE -> dense -> MoE` credits 1, 2, 3. Real loop: hybrid `moe_layers_enum="0,2,3"`, 4 mixed dense completions, 10 decode tokens all peaking at 4 layers; the pre-fix source peaks 4 of them at 3 | | C35-02 W6 report claims legacy scope equals functional scope | Confirmed: the functional entry aligns inside `fused_experts` (vLLM 0.10.2 `fused_moe.py:1718`), the legacy path aligns before `_step`; shuffling and grouped GEMM are additive in both accounting paths | FIXED (records) | W6 report §5 scope table; `docs/profiling/README.md`; `summary.md`; PR35 body. Live double count for functional datasets: not verified, not claimed | -| C35-03 FP8 test omits `block_shape`; "8 of 8 at `rtol=0, atol=0`" overstates | Confirmed: `block_dims` was passed, `block_shape` was not; the FP8 test asserts shape and finiteness only | FIXED (test + wording); native rerun OPEN | `block_shape=block_shape` added; CPU test pins both GEMM invocations receive it (`[128, 64]`) and `None` when omitted; report §8, `summary.md`, PR35 body restated as seven comparisons plus one structural check. Native rerun on one H800 under `codesign`: `NOT_RUN`, awaiting the user's go | +| C35-03 FP8 test omits `block_shape`; "8 of 8 at `rtol=0, atol=0`" overstates | Confirmed: `block_dims` was passed, `block_shape` was not; the FP8 test asserts shape and finiteness only | FIXED (test + wording + native rerun) | `block_shape=block_shape` added; CPU test pins both GEMM invocations receive it (`[128, 64]`) and `None` when omitted; report §8, `summary.md`, PR35 body restated as seven comparisons plus one structural check. Native rerun authorized and executed 2026-09-22: `exp-0922-202645-561899`, 8 passed in 14.27 s, exit 0 (W6 report §8) | | C35-04 unconditional `import torch` adds a collection error | Confirmed: 11 collection errors in the minimal environment versus 10 on the base | FIXED | `pytest.importorskip("torch")` before importing the profiler module; minimal env: `1 skipped`; torch env: 9 passed; unit suite errors back to 10 | | C35-05 records inconsistent (D2 metadata rule vs documentation-only; W2-checkpoint diff claim; blanket vLLM-comparison exclusion; PR34 "Draft") | Confirmed on all four points | FIXED (records) | D2 heading marked SUPERSEDED in part with a link to the dated decision; PR35 body scopes the `ceac2b4` diff claim to the W2 checkpoint and amends the exclusion for the authorized scheduler-level comparison; `progress.md` status table current; PR34 body says "open for review" | | Review's "PR35 mergeable=false" | Stale: GitHub reports `MERGEABLE` for both PRs; PR34 `isDraft=false` | ACCEPTED_LIMITATION (of the review) | `gh pr view` 2026-09-22 | @@ -434,3 +435,19 @@ Reviewed revisions: PR34 `6ef0a3c`, PR35 `0137269`. Each finding was checked aga | P9-06 work graph and acceptance language | Accepted | FIXED (plan) | §18.5 graph, §18.1 C1–C5 | | Package F (W9 implementation) | — | OPEN by instruction | Not started (user: 暂不开启 new subtask) | +## Second plan review 2026-09-22 — user-directed quality gates for Step 9 + +Reviewer: this session, against `c231322`, at the user's direction ("确保当前计划的代码模块的实现/改动/重构是基于整体codebase的 ... 禁止hard-coding，禁止临时补丁，禁止过度防御，禁止冗余性设计和实现，禁止使用ai味命名函数和变量"). Inspected: `vllm_load_balancing_cluster_scheduler.py`, `vllm_dp_load_balancer.py`, `base_cluster_scheduler.py:417-470`, `base_replica_scheduler.py:36-60, 440-480, 1048-1063`, `forward_sync_state.py`, `global_batch_end_event.py:150-215`, `replica_schedule_event.py:80-175`, `vllm_v1_iteration_policy.py:533-545`, both DP-placement test modules, `AGENTS.md:620`, reference `core.py:318-372, 1075-1137`, `coordinator.py:280-312`. Full text in `plan.md` §18.12. + +| Id | Gate | Finding | Disposition | +| --- | --- | --- | --- | +| R9-01 | redundancy / over-design | Readiness classification and `pipeline_room_remaining` have no DES counterpart; only the admission-only row is new | Plan amended: hook payload = completion hook signature | +| R9-02 | grounded in codebase | Completion key `get_step_id(batch)` names the scheduling iteration; correct only at PP=1 | D9-2 amended: key both kinds by the observing iteration; C2 verifies PP=1 | +| R9-03 | reuse before inventing | `ForwardSyncState._next_step_id_by_replica` meets I1–I4, I6; I5 gap measured, not assumed | P1 tests it first; decision at the checkpoint | +| R9-04 | over-defense / layering | Constructor-required `_cluster_scheduler`; do not repeat the `getattr`/`hasattr` reach-ups | D9-1 wording; asymmetry recorded in `design.md` | +| R9-05 | redundancy | Oracle must not re-implement the balancer's coordinator/frontend | P1(a) narrowed to the engine loop | +| R9-06 | naming / test surface | Plain names; existing key assertions remain valid under I1–I2 | P3/P4 wording | +| R9-07 | value / size | Frontier change is small and user-requested; validation must not leak into `frontier/` | Boundary stated | +| R9-08 | reference precision | DP engines not iteration-lockstep (all-reduce every 32 steps) | §18.2 row amended; G5 first-cause label | + +No source change results from this review; Step 9 execution remains unstarted pending the user's start signal. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md b/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md index 0802f63d..64bcf94e 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-22 | FP8 native rerun with the corrected `block_shape` wiring PASS (`exp-0922-202645-561899`, 8 passed). Step 9 plan reviewed a second time against the user's quality gates (plan §18.12); still not started. | | 2026-09-21 | Placeholder created at Step 0. | | 2026-09-22 | Completion archive written at Step 8. | | 2026-09-22 | External review corrections: W3 dense-layer credit for decoding requests in a mixed batch (C35-01); W6 native result restated as seven comparisons plus one FP8 structural check, FP8 `block_shape` wiring corrected (C35-02/03); optional-torch skip (C35-04); records aligned (C35-05). Step 9 remains planned, not started. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_review_corrections.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_review_corrections.md index c7957be2..722eea46 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_review_corrections.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_review_corrections.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-22 | C4 executed: corrected FP8 native case passed on H800 (`exp-0922-202645-561899`). Limits updated. | | 2026-09-22 | Created: verification of the review corrections applied on `fix/issue26-correctness-pr`. Package A is reported in the PR34 task directory (`test_report_2026-09-22_cache_eligibility_correction.md`). | ## Scope @@ -67,7 +68,7 @@ failure set, but the suite has no PREFILL-role hybrid-layer case). | C1 | Arithmetic boundary tests under torch | `openmopd-py312 python -m pytest tests/unit/test_moe_fused_expert_arithmetic.py -q -p no:cacheprovider` | 7 existing + 2 new pass | 9 passed in 6.31 s | PASS | | C2 | Minimal environment collection | `frontier-py310 python -m pytest tests/unit/test_moe_fused_expert_arithmetic.py -q -p no:cacheprovider` | module skips instead of erroring | 1 skipped in 0.06 s | PASS | | C3 | Unit-suite collection errors | B8 | 10 (the base's count), not 11 | 10 errors | PASS | -| C4 | Corrected FP8 native check on the approved worker | not run | — | `NOT_RUN`: needs one H800 under `codesign` and the user's go | NOT_RUN | +| C4 | Corrected FP8 native check on the approved worker | StepMind `RJobBackend`, `codesign` / H800, image `vllm/vllm-openai:v0.10.2`, `pytest -q -rA tests/integration/test_moe_fused_expert_numerical_parity.py` | 8 passed, exit 0; FP8 case runs with `block_shape=[128, 64]` | `exp-0922-202645-561899`, `gpu-h800-0095`, **8 passed in 14.27 s**, exit 0; details in the W6 report §8 | PASS | The native run recorded in the W6 report (`exp-0922-145047-660565`) stands as evidence for the arithmetic repair: seven reference comparisons at @@ -88,7 +89,7 @@ API; PR35 stays draft. ## Limits -- The corrected FP8 native check has not been re-run on a GPU. +- The corrected FP8 native check passed on a GPU; it remains a structural check (shape and finiteness), so FP8 numerical equivalence is still not established. - The real-loop hybrid case uses constant synthetic profiling rows and a four-layer synthetic MoE model, as the W3 runtime test does; it proves the accounting, not latency fidelity. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md index cc13acd1..5b401660 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w6_fused_expert_arithmetic.md @@ -7,6 +7,7 @@ Date: 2026-09-22. Branch `fix/issue26-correctness-pr`, worktree | Date | Change | | --- | --- | +| 2026-09-22 | Corrected FP8 case re-run natively: `exp-0922-202645-561899` (`codesign` / H800, `gpu-h800-0095`), 8 passed in 14.27 s, exit 0. Section 8 records the run. | | 2026-09-22 | Created: reachability check, magnitude estimate, source repair, CPU validation. Native GPU validation NOT_RUN. | | 2026-09-22 | Artifact identity decided as document-only. Native parity test added and submitted to an H800 worker as `exp-0922-140423-075005`; result pending. | | 2026-09-22 | Native parity PASS on H800 under `codesign`: `exp-0922-145047-660565`, 8 of 8 at `rtol=0, atol=0`. Three earlier attempts and their causes recorded in section 8. | @@ -295,8 +296,22 @@ run above did not exercise the production block-quantized invocation. Corrected 2026-09-22: the test now passes `block_shape`, and `tests/unit/test_moe_fused_expert_arithmetic.py` pins on CPU that both GEMM invocations receive the block shape (and `None` when it is omitted). The -corrected native check has **not** been re-run: `NOT_RUN`, one H800 under -`codesign`, awaiting the user's go. +corrected native check was re-run on 2026-09-22 with the user's authorization: + +| Field | Value | +| --- | --- | +| Job | `exp-0922-202645-561899`, creator `i-fengyicheng`, `codesign` / `H800`, 1 GPU / 8 CPU / 64000Mi, RJob `Succeeded` | +| Node | `gpu-h800-0095.host.platform.shaipower.com`, `NVIDIA H800`, torch 2.8.0+cu128, vLLM 0.10.2 (`VLLM_API_VERSION=0.10.x`, `FP8_AVAILABLE=True`), Python 3.12.11 | +| Image / mount | `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`; `100.96.128.195:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr` at the same path, worktree clean at `c231322` (test source `ca1b9b6`) | +| Command | `python3 -m pytest -q -rA -p no:cacheprovider --no-header tests/integration/test_moe_fused_expert_numerical_parity.py` (same launcher as the run above; unchanged since it) | +| Result | **8 passed in 14.27 s**, `W6:PARITY_EXIT=0`, worker exit 0. The same eight test ids as the table above, including `test_fp8_path_runs_on_the_gated_activation`, now executed with `block_shape=[128, 64]` reaching both expert GEMM invocations. | +| Log | `/data/ycfeng/tmp/issue26-correctness-pr/w6_native_fp8_rerun_exp-0922-202645-561899.log` (retrieved through `logs_replica`; `logs_rjob` returned nothing, as before) | + +What this establishes: the block-quantized FP8 invocation that `profile_fused_moe_kernel` +uses runs on the gated activation and returns finite output of the expected shape on +the real kernels. What it still does not establish: FP8 numerical equivalence, because +the FP8 case compares against no reference. The seven zero-tolerance comparisons +passed again unchanged. ### Attempts diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index 363638df..a0d37744 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -465,6 +465,7 @@ Full record: `test_report_2026-09-22_w6_fused_expert_arithmetic.md`. | Native GPU parity | **PASS. 8 passed in 13.70 s** on `NVIDIA H800` (`gpu-h800-0110`), job `exp-0922-145047-660565`, creator `i-fengyicheng`, charged group `codesign` per the user's 2026-09-22 instruction, 1 GPU, image `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`, NFS source `100.96.128.195:/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`. Worker: Python 3.12.11, torch 2.8.0+cu128, vLLM 0.10.2, `VLLM_API_VERSION=0.10.x`, `FP8_AVAILABLE=True`. All 8 cases of `tests/integration/test_moe_fused_expert_numerical_parity.py` compare at `rtol=0, atol=0` against `fused_experts`: Qwen3-A3B-30B shapes from the checked-in model config at 4096 and 4097 tokens on EP ranks 0 and 1; a 257-token, 16-expert case at top-k 2 and 4 whose routing leaves two local experts empty; repeated invocation with different inputs; and the FP8 path as a structural check. Three earlier attempts and their causes are in `test_report_2026-09-22_w6_fused_expert_arithmetic.md` section 8. | | Artifact identity | **CLOSED as document-only** by user decision. The finding stands: `resolve_grouped_gemm_backend` labels both vLLM paths `vllm_fused`, and `profiling_patch_tag` holds three historical free-text values in `a800/qwen3-a3b-30b-moe/moe.csv` while nothing in the source writes it. No column was added. `docs/profiling/README.md` records the operator's scope, the size of the pre-repair gap, and that a row cannot be checked for completeness from its own metadata. | | Suite re-run after the documentation and record commits | `pytest tests/unit -q --continue-on-collection-errors` under `frontier-py310` at `cad3afd`: 84 failed, 3778 passed, 49 skipped, 11 errors. Identical to the W4 baseline and to the earlier W6 measurement. | +| Native rerun with the corrected FP8 wiring (external review C35-03) | `exp-0922-202645-561899`, `codesign` / H800 (`gpu-h800-0095`), vLLM 0.10.2 image, worktree `c231322`: **8 passed in 14.27 s**, exit 0. Seven zero-tolerance comparisons plus the FP8 structural check, now with `block_shape` reaching both GEMMs. FP8 numerics still not compared against a reference. | ## Step 7 — Zero-payload collective through the collective-sim backend From e955406ae0098405b5a7f63d7a27223a2be3bee7 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 21:00:14 +0800 Subject: [PATCH 38/80] docs(plan): R9-01 keeps the pipeline-room test in the policy, computed from existing getters The room conjunct of the reference iteration is real and is what folds an admission that fills the pipeline into the following completion, which is also why PP=1 never publishes an admission on its own. The hook signature stays that of the completion hook; the policy reads num_running_batches and num_pipeline_stages itself. One mechanism for every PP. --- .../task_2026-09-21_issue26_correctness_pr/design.md | 10 ++++++++-- .../task_2026-09-21_issue26_correctness_pr/plan.md | 8 ++++---- .../task_2026-09-21_issue26_correctness_pr/review.md | 2 +- 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md index 70114c26..2ee65372 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md @@ -397,7 +397,13 @@ the hook has nothing to classify: it carries `(time, replica_id, replica_local_id, batch)` exactly as the completion hook does, and the policy scheduler reads the post-admission load through `get_request_load()`, which already reflects the admission because `_running_requests` grows inside -`_get_next_batch`. The zero-token iteration publishes nothing in the reference +`_get_next_batch`. The reference's other conjunct, room in the batch queue, is +real and is the lane's `num_running_batches < num_pipeline_stages` after the +admission; the policy reads both from existing getters and publishes the +admission on its own only while room remains, otherwise the admission is +folded into the completion the engine then blocks on. At PP=1 the single slot +is always filled, so this one rule reproduces today's completion-only +reporting without a PP special case. The zero-token iteration publishes nothing in the reference (`_maybe_publish_request_counts` emits only changed counts) and needs no observation here. Of the five reference rows only the admission-only row is new. @@ -479,7 +485,7 @@ and can only report that rejection. | --- | --- | | `frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py` | `on_replica_batch_scheduled(time, replica_id, replica_local_id, batch)`, the completion hook's signature, inert default. No readiness field or observation record (second review, R9-01). | | `frontier/scheduler/replica_scheduler/base_replica_scheduler.py` | Call the hook once per admitted batch in the MONOLITHIC/PREFILL admission loop, after `_num_running_batches += 1`, through the constructor-required `self._cluster_scheduler` without `getattr`/`hasattr` (R9-04). | -| `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py` | Drop the PP1 clause of the guard and its error text; implement `on_replica_batch_scheduled`; key both observation kinds by the observing iteration per the design checkpoint (first candidate: the Replica's next forward id through a plain `ForwardSyncState` accessor), satisfying invariants 1–6. | +| `frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py` | Drop the PP1 clause of the guard and its error text; implement `on_replica_batch_scheduled`, publishing an admission on its own only while `num_running_batches < num_pipeline_stages` (existing getters); key both observation kinds by the observing iteration per the design checkpoint (first candidate: the Replica's next forward id through a plain `ForwardSyncState` accessor), satisfying invariants 1–6. | | `tests/comparison/dp_placement_pp/reference_loop.py` | CPU oracle of the engine iteration only (scripted admissions, empty schedules, completions, controllable readiness; conjunction, changed-count emission, step counter), feeding the real `VllmDPLoadBalancer`; not a second coordinator/frontend model (R9-05). | | `tests/unit/test_vllm_dp_load_balancer.py`, `tests/integration/test_vllm_dp_placement_runtime.py` | Guard case inverted; the plan §18.11 behavioral matrix (PP2 both callback orders, oldest-ready, PP3 consecutive admission-only iterations on a 6- or 12-layer fixture, full queue, empty schedule, drain, idle peers, bounded bookkeeping); the discriminating scenario against the test-only control; the C35-01 hybrid-layer credit case at PP2. | | `.real-engine/vLLM-BS` (local branch only, D-b) | Case-gated event chain: iteration result, emitted report, coordinator receive/publish with snapshot id, frontend application, frontend routing; named `waiting`/`running`; correlation ids; buffered per-process JSONL. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md index 2403ab93..b4b5d1f2 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md @@ -889,7 +889,7 @@ Equivalence argument, as corrected by P9-01. The reference decides each engine i | Id | Decision | Content | | --- | --- | --- | -| D9-1 | Schedule-time hook (corrected per P9-01; payload fixed per §18.12 R9-01/R9-04) | Name fixed by the user (D-d): `BaseClusterScheduler.on_replica_batch_scheduled(time, replica_id, replica_local_id, batch)`, the same signature as `on_replica_batch_end`, inert default (`return None`). Called once per admitted batch from the MONOLITHIC/PREFILL admission loop of `BaseReplicaScheduler.on_schedule`, immediately after `self._num_running_batches += 1`, through the constructor-required `self._cluster_scheduler` (no `getattr`/`hasattr`). No readiness flag, no `pipeline_room_remaining`, no observation record: the DES has no "ready but unapplied" state (R9-01), so the hook carries the lane, the batch and the time, and the policy scheduler reads the post-admission load through the existing `get_request_load()`. The publish decision stays in `VllmLoadBalancingClusterScheduler`. Rejected, unchanged: reconstructing the report in `ReplicaScheduleEvent` after `on_schedule` (cannot see the state after the first of two admissions in one call). | +| D9-1 | Schedule-time hook (corrected per P9-01; payload fixed per §18.12 R9-01/R9-04) | Name fixed by the user (D-d): `BaseClusterScheduler.on_replica_batch_scheduled(time, replica_id, replica_local_id, batch)`, the same signature as `on_replica_batch_end`, inert default (`return None`). Called once per admitted batch from the MONOLITHIC/PREFILL admission loop of `BaseReplicaScheduler.on_schedule`, immediately after `self._num_running_batches += 1`, through the constructor-required `self._cluster_scheduler` (no `getattr`/`hasattr`). No readiness flag and no observation record: the DES has no "ready but unapplied" state (R9-01). Whether pipeline room remains after the admission is the reference's second conjunct and does decide whether the admission is published on its own or folded into the completion the engine then waits for; the policy scheduler computes it from the lane's existing `num_running_batches` property and `replica_config.num_pipeline_stages`, so it is not a hook argument. The policy reads the post-admission load through `get_request_load()`. One rule for every PP: at PP=1 the single slot is always filled, so no admission is ever published on its own. The publish decision stays in `VllmLoadBalancingClusterScheduler`. Rejected, unchanged: reconstructing the report in `ReplicaScheduleEvent` after `on_schedule` (cannot see the state after the first of two admissions in one call). | | D9-2 | Report key (corrected per P9-02; candidates per §18.12 R9-02/R9-03) | **Rejected as acceptance basis:** K1 (equal key; never latches, misses the reference latch of the pre-admission state), **K3 as written** (a fresh label per admission callback equates callback order with iteration order and gives two peer lanes of one logical iteration different keys — the executable counterexample in §18.11 shows the coordinator then latches a partial snapshot and routes differently), and stride keys (`2*cohort±1`; arbitrary factor, no room for consecutive admission-only iterations at PP≥3). **Rule to be chosen at the design checkpoint:** first identify the reference-equivalent logical engine iteration, then reuse an existing scheduler iteration/forward identity if it actually represents it; otherwise a derived identity or a small additional report-state field. Not a per-batch counter because it is available; not a global identity registry because the old getter is unavailable at admission. The chosen rule must satisfy invariants I1–I6 of §18.11, and the P1 probe must establish logical-iteration membership, not print provisional/resolved Batch ids and pick whichever looks monotonic. Only comparisons are used, so PP=1 behavior stays identical (C2 verifies). **Consequence stated by R9-02:** the rule keys *both* observation kinds by the observing iteration; the current completion key `ForwardSyncState.get_step_id(batch)` names the iteration that scheduled the batch, which coincides with the applying iteration only at PP=1. **First candidate to test in P1 (R9-03):** the Replica-scoped `ForwardSyncState._next_step_id_by_replica` value read at the observation boundary, through a plain accessor; it needs no new bookkeeping and meets I1–I4 and I6, and its known gap is I5 (consecutive admission-only iterations on one lane while stage 0 is busy collapse to one key). Whether the reference itself keeps peer keys equal in that case is a P1/G4 question (R9-08), not an assumption. | | D9-3 | Guard | Drop `num_pipeline_stages == 1`; keep the other four clauses and the dense multi-lane rejection (its evidence is PP-independent); update the error text. The layer-partition guard (`num_layers % num_pipeline_stages == 0`, `replica_config.py`) is untouched: removing the PP1 clause is not permission to bypass the other independent guards or to add uneven partitioning. | | D9-4 | Determinism and flags | No new `EventType`, no new config flag, no balancer constant change (`design.md` event-type determinism; plan §10 "no tuning flags"). | @@ -975,11 +975,11 @@ Unchanged and not claimed: multiple Replicas, multiple frontends (`client_count ### 18.10 Design vocabulary (`codebase-design`) -- **Module.** `VllmLoadBalancingClusterScheduler`, with `VllmDPLoadBalancer` as an internal module. Its **interface** is `schedule_at`, `on_replica_batch_end`, and (new) `on_replica_batch_scheduled`, plus the facts a caller must know: reports are per lane, the schedule-time report is emitted once per admitted batch, at equal simulated time the completion report precedes the admissions it triggers, and the module never raises on an inert path. +- **Module.** `VllmLoadBalancingClusterScheduler`, with `VllmDPLoadBalancer` as an internal module. Its **interface** is `schedule_at`, `on_replica_batch_end`, and (new) `on_replica_batch_scheduled`, plus the facts a caller must know: reports are per lane, an admission is published on its own only while pipeline room remains after it (otherwise it is folded into the completion the engine then waits for), at equal simulated time the completion report precedes the admissions it triggers, and the module never raises on an inert path. - **Seam.** `BaseClusterScheduler.on_replica_batch_*` already has two **adapters** — the inert default used by every other cluster scheduler and this module — so the new hook extends a real seam rather than creating a hypothetical one. The call site in `on_schedule` is the only place that knows whether pipeline room remains, which is why the seam sits there and not in `ReplicaScheduleEvent`. - **Depth.** The reservation, publish deadlines, latch, the admission observation and the key rule all stay behind the same three methods; callers learn nothing new to gain PP>1 support. **Deletion test:** removing the hook would force every event that admits a batch to reconstruct vLLM's per-iteration publication — the complexity reappears across callers, so the module earns its keep. - **Test surface.** Tests drive the module through its interface and assert observable placements and published counts; the relabeled keys are implementation and are not asserted directly. -- **Design it twice.** Three interface shapes were compared: (1) in-loop push hook with the completion hook's signature — chosen: smallest interface, called where the per-admission state is visible, one adapter per cluster scheduler (R9-01 removed the `pipeline_room_remaining` argument the first draft gave it); (2) event-level reconstruction after `on_schedule` — rejected: cannot observe the state after the first of two admissions in one call, so it is shallow and wrong; (3) pull-style `iter_admission_loads()` on the replica scheduler — rejected: widens the replica scheduler's interface for one caller and inverts the push direction the completion report already uses. +- **Design it twice.** Three interface shapes were compared: (1) in-loop push hook with the completion hook's signature — chosen: smallest interface, called where the per-admission state is visible, one adapter per cluster scheduler (R9-01 moved the room test from a hook argument into the policy, which reads the lane's existing `num_running_batches`); (2) event-level reconstruction after `on_schedule` — rejected: cannot observe the state after the first of two admissions in one call, so it is shallow and wrong; (3) pull-style `iter_admission_loads()` on the replica scheduler — rejected: widens the replica scheduler's interface for one caller and inverts the push direction the completion report already uses. ### 18.11 Corrections from the 2026-09-22 external review (P9-01..P9-06) @@ -1056,7 +1056,7 @@ Review question, as set by the user: is every planned change to Frontier's core | Id | Finding (source) | Effect on the plan | | --- | --- | --- | -| R9-01 | **The state table over-specifies what Frontier must represent.** `GlobalBatchEndEvent` applies a batch's completion atomically (`replica_scheduler.on_batch_end`, then `cluster_scheduler.on_replica_batch_end`, `global_batch_end_event.py:180-185`) and the lane's next `ReplicaScheduleEvent` follows at the same simulated time. There is no "oldest output ready but unapplied" state to classify. A zero-token iteration changes no counts, and `_maybe_publish_request_counts` (`core.py:1075-1087`) publishes only changed counts, so it needs no Frontier observation; its `step_counter` increment shifts later key *values*, and only comparisons are used. Of the five §18.11 rows, only the admission-only row is new; rows 2, 3, 5 are the existing completion report and row 2's merge with the same-time admission is invariant I3 of the key rule. | D9-1 payload reduced to the completion hook's signature; no readiness field, no observation record; §18.11 representation column annotated. Avoids a redundant classifier and booleans threaded through `on_schedule`. | +| R9-01 | **The state table over-specifies what Frontier must represent.** `GlobalBatchEndEvent` applies a batch's completion atomically (`replica_scheduler.on_batch_end`, then `cluster_scheduler.on_replica_batch_end`, `global_batch_end_event.py:180-185`) and the lane's next `ReplicaScheduleEvent` follows at the same simulated time. There is no "oldest output ready but unapplied" state to classify. A zero-token iteration changes no counts, and `_maybe_publish_request_counts` (`core.py:1075-1087`) publishes only changed counts, so it needs no Frontier observation; its `step_counter` increment shifts later key *values*, and only comparisons are used. The room conjunct (`len(batch_queue) < batch_queue_size`) is real: it is `num_running_batches < num_pipeline_stages` after the increment, and an admission that fills the pipeline is published together with the completion the engine then blocks on — which is exactly the PP=1 behavior, where the single slot is always filled. Of the five §18.11 rows, only the admission-only row is new; rows 2, 3, 5 are the existing completion report and row 2's merge with the same-time admission is invariant I3 of the key rule. | D9-1 payload reduced to the completion hook's signature; no readiness field, no observation record; the room test is computed by the policy from the lane's existing `num_running_batches` property, not passed in. One mechanism for all PP. §18.11 representation column annotated. | | R9-02 | **The existing completion key changes meaning under PP.** `on_replica_batch_end` keys by `ForwardSyncState.get_step_id(batch)`, the forward identity resolved when the batch's own forward opened (`forward_sync_state.py`, `resolve_step`). vLLM keys a publication by the iteration whose counts changed; a completion is published under the iteration that *applied* the output, `P-1` iterations after the one that scheduled it. The two coincide only at PP=1. Keeping the batch key at PP>1 would order every completion report before the admission reports emitted while it was in flight — the P9-02 "second interleaving" is the normal case, not an edge. | D9-2 now requires both observation kinds to be keyed by the observing iteration. C2's PP=1 byte comparison is the check that the changed key keeps identical comparisons at PP=1. | | R9-03 | **An existing identity is the first candidate; do not mint a new counter before testing it.** `ForwardSyncState._next_step_id_by_replica[replica_id]` is Replica-scoped, monotonic, equal for every lane between room openings and strictly greater than every open or completed step. Read at an observation boundary it satisfies I1, I2, I3 (a completion and the admission it triggers at the same time read the same value), I4 and I6 with no bookkeeping. Known gap: I5 — on one lane, admissions made while stage 0 is still busy (cold fill of several slots in one `on_schedule` call) read the same value, whereas the reference gives them strictly increasing steps and latches the intermediate state for one publish interval. | P1(b) probes this candidate first and measures the I5 case; the design checkpoint decides whether the I5 gap is material (the §18.6 scenario admits one batch per lane) or needs a derived identity. A plain accessor on `ForwardSyncState` (named for what it returns) is the only new surface if adopted. | | R9-04 | **Call site and layering.** `BaseReplicaScheduler` requires `cluster_scheduler` at construction (`TypeError` otherwise) and stores it as `self._cluster_scheduler` (`base_replica_scheduler.py:43-56`). The two `getattr(self, "_cluster_scheduler", None)`/`hasattr(...)` reach-ups at `:423` and `:468` are the over-defensive pattern the gates forbid; the new call must not add a third. The completion hook is invoked by an event, the admission hook by the replica scheduler, because per-admission state is visible only inside the loop; record the asymmetry and its reason in `design.md` so nobody later "fixes" it by moving the call to `ReplicaScheduleEvent`. | D9-1 wording; `design.md` planned-edits row. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index ce36fb54..bee5d3cd 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -441,7 +441,7 @@ Reviewer: this session, against `c231322`, at the user's direction ("确保当 | Id | Gate | Finding | Disposition | | --- | --- | --- | --- | -| R9-01 | redundancy / over-design | Readiness classification and `pipeline_room_remaining` have no DES counterpart; only the admission-only row is new | Plan amended: hook payload = completion hook signature | +| R9-01 | redundancy / over-design | Readiness classification has no DES counterpart; the room test is real but computed by the policy from the lane's existing `num_running_batches`; only the admission-only row is new; one rule for all PP | Plan amended: hook payload = completion hook signature | | R9-02 | grounded in codebase | Completion key `get_step_id(batch)` names the scheduling iteration; correct only at PP=1 | D9-2 amended: key both kinds by the observing iteration; C2 verifies PP=1 | | R9-03 | reuse before inventing | `ForwardSyncState._next_step_id_by_replica` meets I1–I4, I6; I5 gap measured, not assumed | P1 tests it first; decision at the checkpoint | | R9-04 | over-defense / layering | Constructor-required `_cluster_scheduler`; do not repeat the `getattr`/`hasattr` reach-ups | D9-1 wording; asymmetry recorded in `design.md` | From cf1cb74360e5a782257fe14ed57876def20f5025 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 21:47:24 +0800 Subject: [PATCH 39/80] test(dp-placement): add the vLLM engine-iteration reference loop for PP>1 The completion report Frontier publishes today is exact only when the engine has no batch queue. Above pipeline_parallel_size 1 an iteration can schedule new work, return without applying an older output, and still publish changed request counts, so a placement can depend on state Frontier never reports. reference_loop.py models that iteration and nothing else: the early-return conjunction from step_with_batch_queue, changed-count emission, and the per engine step counter. Emitted reports feed the existing VllmDPLoadBalancer, so the coordinator latch and the frontend score stay modeled in one place. The unit test pins the five reference states as executable expectations, including the two that no stride rule can express: a depth-three queue admits twice before the first completion, and both admissions publish. --- .../dp_placement_pp/reference_loop.py | 186 ++++++++++++++++++ .../unit/test_dp_placement_reference_loop.py | 113 +++++++++++ 2 files changed, 299 insertions(+) create mode 100644 tests/comparison/dp_placement_pp/reference_loop.py create mode 100644 tests/unit/test_dp_placement_reference_loop.py diff --git a/tests/comparison/dp_placement_pp/reference_loop.py b/tests/comparison/dp_placement_pp/reference_loop.py new file mode 100644 index 00000000..ba2c0fff --- /dev/null +++ b/tests/comparison/dp_placement_pp/reference_loop.py @@ -0,0 +1,186 @@ +"""vLLM V1's data-parallel engine iteration, replayed on the CPU. + +Reference: vLLM v0.10.2 as checked out in `.real-engine/vLLM-BS` at +`ea95f571e`. `EngineCore.step_with_batch_queue` (`vllm/v1/engine/core.py`) +decides what one iteration does, `DPEngineCoreProc._maybe_publish_request_counts` +decides whether that iteration publishes its populations, and `run_busy_loop` +advances the step counter that orders the publications. + +Why it exists: under pipeline parallelism an engine publishes counts at +iterations a completion report cannot express, and the mapping from iteration +to publication is what Step 9 changes in Frontier. Replaying a scripted history +here gives an expectation written from the reference instead of from a Frontier +run. + +What it deliberately leaves out: the coordinator's latch and the frontend's +score. `VllmDPLoadBalancer` already models both from the same source, so +`ReferenceDeployment` feeds the real balancer rather than carrying a second +copy of them. Wave resets are outside Step 9's scope, so the step counter here +only increases; the coordinator compares keys and never reads their magnitude. +The counter also advances on an iteration the reference would skip while every +engine is idle, which shifts later key values but not their order or equality. +""" + +from __future__ import annotations + +from collections import deque +from dataclasses import dataclass + +from frontier.scheduler.request_load import RequestLoad +from frontier.scheduler.utils.vllm_dp_load_balancer import VllmDPLoadBalancer + + +@dataclass(frozen=True) +class ScheduledBatch: + """What one `schedule()` result does to an engine's populations. + + `admitted` requests move from waiting to running when the batch is + scheduled. `finished` requests leave running when the batch's output is + applied, which under pipeline parallelism is a later iteration. + `num_scheduled_tokens` is zero for the empty scheduler output a busy engine + can still produce; the reference reads it as `model_executed`. + """ + + admitted: int = 0 + finished: int = 0 + num_scheduled_tokens: int = 1 + + +@dataclass(frozen=True) +class Iteration: + """The scripted inputs of one engine iteration. + + `batch` is what `schedule()` returned, or `None` when the engine had + nothing to schedule and only drains a queued output. `oldest_ready` is + whether the oldest queued batch's future had already completed, the + reference's third condition for returning without applying an output. + """ + + arrivals: int = 0 + batch: ScheduledBatch | None = None + oldest_ready: bool = False + + +@dataclass(frozen=True) +class IterationRecord: + """What one iteration did, and whether it published its populations.""" + + step: int + scheduled: bool + applied_output: bool + load: RequestLoad + published: bool + + +@dataclass(frozen=True) +class Publication: + """One engine's changed counts, as the coordinator receives them.""" + + time: float + engine: int + step: int + load: RequestLoad + + +class ReferenceEngine: + """One DP engine's busy loop over a scripted history. + + `queue_depth` is `batch_queue_size`, which the reference sets to the + pipeline-parallel size. At depth one the blocking path runs every + iteration, which is what the engine does when the batch queue is absent: + the same mechanism, not a special case. + """ + + def __init__(self, queue_depth: int) -> None: + if type(queue_depth) is not int or queue_depth < 1: + raise ValueError( + f"queue depth must be a positive int, got {queue_depth!r}" + ) + self._depth = queue_depth + self._queue: deque[ScheduledBatch] = deque() + self.waiting = 0 + self.running = 0 + self.step_counter = 0 + self._last_load = RequestLoad(0, 0) + + @property + def load(self) -> RequestLoad: + return RequestLoad(self.waiting, self.running) + + def step(self, iteration: Iteration) -> IterationRecord: + """Run one iteration and return what the engine would publish.""" + + self.waiting += iteration.arrivals + batch = iteration.batch + if batch is None and not self._queue: + raise ValueError( + "the reference steps an engine only while it holds requests or " + "a queued batch" + ) + if batch is not None: + if batch.admitted > self.waiting: + raise ValueError( + f"cannot admit {batch.admitted} of {self.waiting} waiting " + "requests" + ) + self.waiting -= batch.admitted + self.running += batch.admitted + self._queue.appendleft(batch) + if ( + batch.num_scheduled_tokens > 0 + and len(self._queue) < self._depth + and not iteration.oldest_ready + ): + return self._record(scheduled=True, applied_output=False) + + oldest = self._queue.pop() + if oldest.finished > self.running: + raise ValueError( + f"cannot finish {oldest.finished} of {self.running} running " + "requests" + ) + self.running -= oldest.finished + return self._record(scheduled=batch is not None, applied_output=True) + + def _record(self, *, scheduled: bool, applied_output: bool) -> IterationRecord: + """Publish changed counts under the pre-increment step counter.""" + + load = self.load + published = load != self._last_load + if published: + self._last_load = load + record = IterationRecord( + step=self.step_counter, + scheduled=scheduled, + applied_output=applied_output, + load=load, + published=published, + ) + self.step_counter += 1 + return record + + +class ReferenceDeployment: + """DP engines behind one coordinator and one frontend. + + The engines are scripted here; the coordinator's latch, its publish + deadlines and the frontend's score come from `VllmDPLoadBalancer`. Times + are simulated seconds, as they are for the balancer. + """ + + def __init__(self, *, num_engines: int, queue_depth: int) -> None: + self.engines = [ReferenceEngine(queue_depth) for _ in range(num_engines)] + self.balancer = VllmDPLoadBalancer(num_engines) + self.publications: list[Publication] = [] + + def step(self, time: float, engine: int, iteration: Iteration) -> IterationRecord: + record = self.engines[engine].step(iteration) + if record.published: + self.balancer.report(time, engine, record.step, record.load) + self.publications.append( + Publication(time, engine, record.step, record.load) + ) + return record + + def route(self, time: float) -> int: + return self.balancer.select(time) diff --git a/tests/unit/test_dp_placement_reference_loop.py b/tests/unit/test_dp_placement_reference_loop.py new file mode 100644 index 00000000..5120d171 --- /dev/null +++ b/tests/unit/test_dp_placement_reference_loop.py @@ -0,0 +1,113 @@ +"""Pin vLLM V1's iteration-to-publication mapping under pipeline parallelism. + +These are expectations read from the reference (`.real-engine/vLLM-BS` at +`ea95f571e`), not from a Frontier run. They are what a Frontier schedule-time +report has to reproduce, and they are the reason the completion report alone +cannot: at depth one every iteration both schedules and applies, while above it +an iteration can publish a changed population without completing anything. +""" + +from __future__ import annotations + +import pytest + +from frontier.scheduler.request_load import RequestLoad +from tests.comparison.dp_placement_pp.reference_loop import ( + Iteration, + ReferenceDeployment, + ReferenceEngine, + ScheduledBatch, +) + + +def run(depth: int, script: list[Iteration]) -> list[tuple]: + """Return one engine's (step, scheduled, applied, load, published) rows.""" + + engine = ReferenceEngine(depth) + return [ + (r.step, r.scheduled, r.applied_output, tuple(r.load), r.published) + for r in (engine.step(iteration) for iteration in script) + ] + + +def test_without_a_batch_queue_every_iteration_applies_what_it_scheduled(): + rows = run(1, [ + Iteration(arrivals=2, batch=ScheduledBatch(admitted=2)), + Iteration(batch=ScheduledBatch(finished=1)), + Iteration(batch=ScheduledBatch(finished=1)), + ]) + assert [row[2] for row in rows] == [True, True, True] + assert [row[3] for row in rows] == [(0, 2), (0, 1), (0, 0)] + + +def test_a_cold_fill_publishes_an_admission_before_anything_completes(): + rows = run(2, [ + Iteration(arrivals=3, batch=ScheduledBatch(admitted=3)), + Iteration(arrivals=1, batch=ScheduledBatch(admitted=1)), + ]) + assert rows[0][1:] == (True, False, (0, 3), True) + assert rows[1][1:] == (True, True, (0, 4), True) + + +def test_an_already_ready_output_is_applied_in_the_iteration_that_schedules(): + rows = run(3, [ + Iteration(arrivals=4, batch=ScheduledBatch(admitted=2)), + Iteration(batch=ScheduledBatch(admitted=2), oldest_ready=True), + ]) + assert rows[0][1:3] == (True, False) + assert rows[1][1:] == (True, True, (0, 4), True) + + +def test_an_empty_scheduler_output_does_not_take_the_early_return(): + rows = run(3, [ + Iteration(arrivals=2, batch=ScheduledBatch(admitted=2)), + Iteration(batch=ScheduledBatch(num_scheduled_tokens=0)), + ]) + assert rows[1][1:] == (True, True, (0, 2), False) + + +def test_a_drain_publishes_without_scheduling(): + rows = run(2, [ + Iteration(arrivals=2, batch=ScheduledBatch(admitted=2, finished=2)), + Iteration(batch=None), + ]) + assert rows[1][1:] == (False, True, (0, 0), True) + + +def test_depth_three_allows_two_admissions_before_the_first_completion(): + rows = run(3, [ + Iteration(arrivals=6, batch=ScheduledBatch(admitted=2)), + Iteration(batch=ScheduledBatch(admitted=2)), + Iteration(batch=ScheduledBatch(admitted=2)), + ]) + assert [row[2] for row in rows] == [False, False, True] + assert [row[0] for row in rows] == [0, 1, 2] + assert all(row[4] for row in rows) + + +def test_an_engine_is_stepped_only_while_it_holds_work(): + with pytest.raises(ValueError, match="holds requests or"): + ReferenceEngine(2).step(Iteration()) + + +def test_peers_at_the_same_iteration_index_publish_under_one_key(): + deployment = ReferenceDeployment(num_engines=2, queue_depth=2) + first = deployment.step(0.010, 0, Iteration(arrivals=3, batch=ScheduledBatch(admitted=3))) + second = deployment.step(0.020, 1, Iteration(arrivals=3, batch=ScheduledBatch(admitted=3))) + + assert first.step == second.step + # Equal keys apply without latching, so the frontend sees both lanes. + assert deployment.route(0.080) == 0 + assert deployment.balancer.frontend_counts[1] == RequestLoad(0, 3) + + +def test_an_engine_that_runs_an_extra_iteration_moves_its_key_ahead(): + """Step counters are per engine, so peer equality is not guaranteed.""" + + deployment = ReferenceDeployment(num_engines=2, queue_depth=2) + deployment.step(0.005, 0, Iteration(arrivals=2, batch=ScheduledBatch(admitted=2))) + deployment.step(0.006, 0, Iteration(batch=ScheduledBatch())) + peer = deployment.step(0.020, 1, Iteration(arrivals=3, batch=ScheduledBatch(admitted=3))) + + assert deployment.engines[0].step_counter == 2 + assert peer.step == 0 From 9e3afe84f85b65d6407ae33661635d4f9f95afed Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 21:47:24 +0800 Subject: [PATCH 40/80] docs(step9): record the P1 evidence, the design checkpoint, and W9-01 P1(a) confirms the plan state table as written and shows that at depth one every iteration both schedules and applies, which is why the current completion-only report is already exact at PP=1. P1(b) probed three shapes and found the fourth unrunnable. MoE attn_dp=2 with num_pipeline_stages=2 drains the event queue with requests unfinished: stage admission mints a ticket per arriving batch and admits only the strict FIFO head, so a lane that admits num_pipeline_stages batches in one round places its peer behind a ticket belonging to a lane that is already busy. The three files involved are byte-identical to main, and nothing exercised the combination because every attn_dp>1 test uses one pipeline stage. The design checkpoint closes with the hook payload settled and the report key open. The ForwardSyncState forward id satisfies every invariant that the runnable shapes can test but collapses consecutive cold-fill admissions into one key, and the alternative that fixes that breaks peer equality. Deciding between them requires the shape that deadlocks. --- .../design.md | 46 ++++++ .../issues.md | 133 ++++++++++++++++++ .../plan.md | 32 ++++- .../progress.md | 15 ++ 4 files changed, 225 insertions(+), 1 deletion(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/issues.md diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md index 2ee65372..4bfd2dc1 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md @@ -7,6 +7,7 @@ scope decisions and the pre-measurement expectation for that package. | Date | Change | | --- | --- | +| 2026-09-22 | Added the design checkpoint section: D9-1 payload settled from the P1 oracle; D9-2 key left open because the `ForwardSyncState` candidate fails invariant I5 at PP>1 and the I1/I5 trade-off is only observable on the shape blocked by W9-01. | | 2026-09-22 | W9 second review (user-directed quality gates): section "What the code already provides" added; planned-edits rows for the hook payload, the call site and the CPU oracle amended; plan §18.12 R9-01..R9-08. | | 2026-09-22 | Created. Source-backed design for Checkpoint D's W3 half, with the scope decisions and their evidence. | | 2026-09-22 | Restructured into per-work-package sections and added the W4 design, with the report-key identity measured at the emission boundary. | @@ -507,3 +508,48 @@ hybrid or external load balancing, wave resets, or elastic EP; and it does not claim latency equivalence of placements — the comparison is by boundary index, with timing controlled only where the discriminating scenario needs it. + +## Design checkpoint: what P1 settled and what it did not (2026-09-22) + +### D9-1 payload — settled + +The oracle confirms there is nothing to classify at the boundary. In the +reference an iteration's branch depends on whether the oldest queued output is +already done; in the DES a completion is atomic at its end event, so the +admission hook needs the same four values the completion hook already takes, +`(time, replica_id, replica_local_id, batch)`, and the policy reads the +post-admission population through `get_request_load()`. The pipeline-room test +stays, computed by the policy from `num_running_batches` and +`num_pipeline_stages` (R9-01); it is what makes PP=1 report nothing extra. + +### D9-2 key — not settled, and the reason is structural + +The first candidate was the Replica-scoped forward id that `ForwardSyncState` +hands out, read at the boundary. Measured on the three shapes that run: + +| Invariant | `attn_dp=2, PP=1` | `attn_dp=1, PP=2` | `attn_dp=1, PP=3` | +| --- | --- | --- | --- | +| I1 peers of one iteration compare equal | holds (both lanes read the same value at both boundaries) | not observable (one lane) | not observable | +| I2 new iteration orders after the previous | holds (0, 6, 12, 18, 24) | holds after the fill | holds after the fill | +| I3 one decision per iteration | holds (completion and the admission it triggers share the value) | holds | holds | +| I5 consecutive admission-only iterations stay distinct | vacuous (none occur) | **fails**: both cold-fill admissions read 0 | **fails**: all three read 0 | +| I6 bookkeeping released | holds (no new state) | holds | holds | + +The failure is not incidental. The forward id advances when a forward room +opens, which happens once per executed forward, while the reference key advances +once per engine iteration. At PP>1 a lane admits up to `num_pipeline_stages` +batches before the first forward completes, so every one of those admissions +reads the same id. That is precisely the cold fill the discriminating scenario +in plan §18.6 turns on, so the candidate cannot be accepted. + +The obvious repair, a per-lane observation counter, restores I5 by construction +and breaks I1 by construction: two lanes advance independently, so peer +observations of one logical iteration no longer compare equal, which is the +counterexample recorded above. Any rule that satisfies both has to derive a +shared identity that still advances per observation, and whether a given rule +does can only be decided by observing a shape with `attn_dp > 1` **and** +`PP > 1`. That shape deadlocks today (W9-01 in `issues.md`). + +The checkpoint therefore closes with D9-1 fixed and D9-2 open. Implementing a +key rule now would mean choosing between two invariants with no way to test the +choice, which is the kind of unfalsifiable design the gates exclude. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md new file mode 100644 index 00000000..ebc7187a --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md @@ -0,0 +1,133 @@ +# Issues + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-22 | Created; recorded W9-01 (stage-admission deadlock at PP>1 with attn_dp>1) and W9-02 (PP=3 x attn_dp=2 topology rejection) found during Step 9 P1(b). | + +## W9-01 Stage admission deadlocks when `num_pipeline_stages > 1` and `attn_dp > 1` + +Status: open, pre-existing on `main`, blocks Step 9 acceptance criteria C1 and C4. +Found: 2026-09-22, Step 9 package P1(b) boundary probe. + +### Symptom + +A MONOLITHIC MoE replica with `attn_dp=2`, `moe_expert_parallel_size=2` and +`num_pipeline_stages=2` drains the event queue with scheduler state still +non-empty. Requests neither complete nor raise; the simulation ends early and +reports fewer completed requests than were generated. + +Reproduced in fresh processes, deterministic in the request count: + +| Shape | Requests | Result | +| --- | --- | --- | +| `attn_dp=2, moe_ep=2, PP=2` | 2 | completes | +| `attn_dp=2, moe_ep=2, PP=2` | 3 | completes | +| `attn_dp=2, moe_ep=2, PP=2` | 4 | drained, 0 completed | +| `attn_dp=2, moe_ep=2, PP=2` | 6 | drained, 0 completed | +| `attn_dp=2, moe_ep=2, PP=2` | 8 | drained, 0 completed | +| `attn_dp=1, moe_ep=1, PP=2` | 6 | completes | +| `attn_dp=1, moe_ep=1, PP=3` | 6 | completes | +| `attn_dp=2, moe_ep=2, PP=1` | 6 | completes | + +The threshold is the point at which both lanes can hold more than one batch in +flight at once, which is what `num_pipeline_stages > 1` permits. + +### Mechanism + +`ReplicaStageScheduler.add_batch` mints a `StageAdmissionTicket` through +`StageExecutionContext.enqueue_full_stage` at batch *arrival*, and +`try_acquire` admits a ticket only when it is the strict head of the +per-`(replica, stage)` `_ready_fifo`: + +```python +if not self._ready_fifo or self._ready_fifo[0] != ticket: + return False +``` + +At `num_pipeline_stages = 1` each lane has at most one batch in flight, so +tickets interleave one per lane and the head is always acquirable. At +`num_pipeline_stages > 1` `BaseReplicaScheduler.on_schedule` admits up to +`num_pipeline_stages` batches for one lane in a single scheduling round, so +that lane enqueues `num_pipeline_stages` tickets before its peer enqueues its +first. The peer's ticket then sits behind a ticket whose own lane scheduler is +already `_is_busy`, so that head can never be acquired, and the shared forward +cohort never assembles. Neither lane can progress and no event is left. + +Observed event sequence (MoE `attn_dp=2, moe_ep=2, PP=2`, 4 requests): + +``` +ReplicaStageScheduleEvent(lane 1) -> PrefillSyncEvent +ReplicaStageScheduleEvent(lane 1) -> [] # scheduler busy +ReplicaStageScheduleEvent(lane 0) -> [] # head ticket belongs to busy lane 1 +ReplicaStageScheduleEvent(lane 0) -> [] +``` + +Stage `(0, 0)` state at drain: + +``` +capacity=2 active_seqs=[0] waiting_fifo_seqs=[1, 2, 3] +sealed=False ep_active=False +``` + +`full_stage_capacity` is `replica_dp_size` (2 here), so capacity is not the +constraint; strict FIFO head ordering is. + +### Scope + +Pre-existing, not introduced by this PR. The three files involved are +byte-identical to `main`: + +- `frontier/scheduler/replica_stage_scheduler/stage_execution_context.py` +- `frontier/scheduler/replica_stage_scheduler/replica_stage_schduler.py` +- `frontier/scheduler/utils/stage_contexts.py` + +It is unobserved today because nothing exercises the combination: every +Simulator-level test with `attn_dp > 1` uses `num_pipeline_stages = 1`, and no +shipped example sets `attn_dp > 1` at all. + +### Why it blocks Step 9 + +The vLLM DP placement policy is only meaningful when `attn_dp > 1`; with +`attn_dp = 1` there is a single engine and `select` is degenerate. Step 9's +acceptance criterion C1 requires MoE `attn_dp=2` to complete at PP2, and the +discriminating scenario in plan section 18.6 is +`--data-parallel-size 2 --pipeline-parallel-size 2`. Both require this shape to +run. The design checkpoint is blocked for the same reason: invariant I1 (peers +of one forward share the report key) can only be observed on a shape with both +`attn_dp > 1` and `PP > 1`. + +### Options + +1. Fix the ordering in the shared stage-admission path as part of Step 9. + Acquisition would have to consider the first ticket that is actually + acquirable for its lane rather than the global FIFO head, keeping the + existing anti-starvation intent. This touches infrastructure shared by + co-location, PDD and PD-AF, so it needs its own regression matrix. +2. Fix it as a separate correctness item with its own validation, and pause + Step 9's PP>1 packages until it lands. +3. Restrict Step 9 to `attn_dp = 1` at PP>1. This satisfies nothing: the policy + has no effect at `attn_dp = 1`, so it would ship a PP>1 claim with no + evidence for the only configuration the policy affects. + +Recommendation: option 2. The defect is independent of the placement policy, +predates both PRs, and changing shared admission ordering under a feature branch +would mix an infrastructure fidelity fix into a feature PR. + +## W9-02 `attn_dp=2, moe_ep=2, num_pipeline_stages=3` is rejected at config time + +Status: open, expected behavior, affects plan wording only. +Found: 2026-09-22, Step 9 package P1(b). + +Construction fails with: + +``` +collective-sim physical topology requires cluster_total_devices 6 to be +divisible by node size 4 +``` + +The rejection is correct: 2 lanes x 3 stages is 6 devices against a node size of +4. Plan acceptance criterion C1 lists MoE `attn_dp=2` at both PP2 and PP3; the +PP3 row must be `attn_dp=1`, or must choose a device count that divides the node +size. C1 was amended accordingly. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md index b4b5d1f2..b862a338 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-22 | Step 9 execution started. §18.13 added: P1(a) oracle complete with the state-table evidence, P1(b) blocked by the pre-existing stage-admission deadlock W9-01 (`issues.md`), design checkpoint D9-2 left open because the candidate key fails invariant I5 and I1/I5 can only be settled on the deadlocking shape. C1 amended per W9-02: the PP3 row uses `attn_dp=1`. | | 2026-09-22 | §18 second review at the user's direction (codebase integration, readability, value, no hard-coding/patches/over-defense/redundancy, plain names): findings R9-01..R9-08 in §18.12; D9-1, D9-2, P1(a), §18.10 and the §18.11 representation column amended in place. Execution still not started. | | 2026-09-21 | Landed the execution specification verbatim (Section "Execution Specification" below) and recorded the amendments agreed with the user before Step 0. | | 2026-09-22 | Added amendment A12 and the Step 9 draft (§18; numbered §17 until 2026-09-22, when the duplicate number was fixed) for PP>1 support of the opt-in vLLM DP placement; awaiting user approval. | @@ -856,7 +857,7 @@ Under [R1]: `frontier/profiling/moe/moe_vllm_kernel.py`, `frontier/entities/stag | # | Criterion | Evidence | | --- | --- | --- | -| C1 | Valid PP2 and PP3 configurations (layer count divisible by PP; `MONOLITHIC`, one Replica, `vllm_v1`, MoE or `attn_dp == 1` — the PP1 clause is the only guard removed) complete every request with request/token/owner conservation, for dense `attn_dp=1` and MoE `attn_dp=2`. PP3 uses a separate CPU fixture with a valid layer count (6 or 12); the native PP2 model stays the approved 8-layer tiny Qwen3-MoE. | P4 real-loop PP2 and PP3 cases; §18.11 behavioral matrix. | +| C1 | Valid PP2 and PP3 configurations (layer count divisible by PP; `MONOLITHIC`, one Replica, `vllm_v1`, MoE or `attn_dp == 1` — the PP1 clause is the only guard removed) complete every request with request/token/owner conservation, for dense `attn_dp=1` and MoE `attn_dp=2` at PP2. **Amended 2026-09-22 (W9-02):** the PP3 row uses `attn_dp=1`; `attn_dp=2, moe_ep=2, PP=3` is rejected at construction because 6 devices do not divide the node size of 4. PP3 uses a separate CPU fixture with a valid layer count (6 or 12); the native PP2 model stays the approved 8-layer tiny Qwen3-MoE. | P4 real-loop PP2 and PP3 cases; §18.11 behavioral matrix. | | C2 | Previously supported behavior is unchanged under the stated comparison contract: every existing PP1 `vllm_load_balancing` scenario has value-identical `request_metrics.csv` and identical `system_metrics.json` (timestamps/run ids removed, the Q11 rule), with no additional admission-only report; every other cluster scheduler, including the supported disaggregated paths, has identical event outcomes (the hook is inert for them). | P5 byte comparison; Step 8 regression set rerun. | | C3 | For a controlled or demonstrably matched iteration history, the emitted loads, the equality/order relation of logical-iteration keys, the coordinator snapshots and the frontend-visible counts agree with the reference. Natural-history divergence is classified by first cause (arrival/delivery order, batch composition, output readiness, count calculation, key grouping, snapshot publication, frontend selection), not hidden by re-indexing. Boundary-index comparison alone is not an alignment method. | CPU reference-loop oracle (P1) + causal join of the G4 trace (§18.11 instrumentation chain) + `workflow-gap-analysis`. | | C4 | In a trace-qualified native discriminating slice (§18.6, qualified per §18.11: the intended snapshot was applied at the frontend before the probe was routed), the corrected placement matches the reference and the explicit test-only completion-reporting control fails for the expected reason. The actual unmodified PP2 baseline is reported as rejected by its constructor, not as a placement. Otherwise the slice is `SCENARIO_NOT_REACHED` with the failed precondition named. | §18.6 comparison table with the control column. | @@ -1050,6 +1051,35 @@ Exact baseline comparison only for behavior meant to stay unchanged; new PP2/PP3 **P9-06 — work graph and acceptance language.** Applied in §18.5 and §18.1. A native out-of-order warning is evidence to analyze (the reference applies the counts after warning), not proof of a simulator bug nor a reason to rewrite native ordering. +### 18.13 Execution status and the P1(b) blocker (2026-09-22) + +Start approval was given ("开始执行step9"). P1(a) is complete and P1(b) stopped on a pre-existing runtime defect that blocks the design checkpoint. + +**P1(a) reference-loop oracle — done.** `tests/comparison/dp_placement_pp/reference_loop.py` models the engine iteration only (the `step_with_batch_queue` conjunction, changed-count emission, per-iteration `step_counter`) and feeds emitted reports into the real `VllmDPLoadBalancer`. `tests/unit/test_dp_placement_reference_loop.py` pins the §18.11 state table: 9 tests, all pass. Findings: + +| §18.11 row | Oracle result | +| --- | --- | +| Depth 1 (PP=1) | Every iteration both schedules and applies; the admission-only row cannot occur. Frontier's completion-only report is already exact at PP=1. | +| Room remains, oldest not ready | `scheduled=True, applied=False`, counts published. The admission-only observation is real. | +| Room remains, oldest ready | One combined publication, not two. Confirms invariant I3. | +| Zero-token schedule | Does not early-return; applies the oldest; publishes only if counts changed. | +| Drain | `scheduled=False, applied=True`, published. | +| Depth 3 | Two consecutive admission-only iterations, both published, steps 0 and 1. No stride constant can reproduce this (I5). | +| Peer keys | Equal only while two engines sit at the same iteration index; one extra iteration on a lane moves its counter ahead. Confirms R9-08: step counters are per engine. | + +**P1(b) Frontier boundary probe — blocked.** Three shapes ran; the fourth deadlocks. Recorded as W9-01 in `issues.md`. + +| Shape | Boundaries | Candidate key `ForwardSyncState._next_step_id_by_replica` | +| --- | --- | --- | +| `attn_dp=2, moe_ep=2, PP=1` | 24, 6/6 completed | Peers of one forward always read the same value; values advance 0, 6, 12, 18, 24 (one per layer); each completion and the admission it triggers share a value. I1, I2, I3, I4, I6 hold. | +| `attn_dp=1, moe_ep=1, PP=2` | 28, 6/6 completed | The two cold-fill admissions **both read 0**. Afterwards each completion/admission pair reads a common increasing value. | +| `attn_dp=1, moe_ep=1, PP=3` | 32, 6/6 completed | The three cold-fill admissions **all read 0**, then pairs read 12, 18, 24, ... | +| `attn_dp=2, moe_ep=2, PP=2` | — | Event queue drains with requests unfinished (W9-01). | + +**Design-checkpoint conclusion.** The candidate key satisfies I1–I4 and I6 but **fails I5**: it advances when a forward room opens, not once per engine iteration, so consecutive admissions on one lane while a stage is busy collapse into one key. That is exactly the cold fill the discriminating scenario in §18.6 depends on. A per-lane counter would fix I5 but breaks I1, and whether a candidate satisfies both can only be observed on a shape with `attn_dp > 1` **and** `PP > 1` — the shape W9-01 deadlocks. The key rule therefore cannot be fixed at this checkpoint, and D9-2 stays open. + +**Consequence.** Step 9 packages P2, P3, P4, G3, G4 and G5 all depend on the design checkpoint or on a running `attn_dp=2, PP=2` shape. They are paused pending the user's scope decision on W9-01. P1 and its records are complete. + ### 18.12 Second review (2026-09-22, user-directed): codebase integration and quality gates Review question, as set by the user: is every planned change to Frontier's core modules grounded in the whole codebase, readable and maintainable, high-value (fidelity or simulation function, not replaceable), and free of hard-coding, temporary patches, over-defensive code, redundant design and vague names. Findings are against the code on `c231322`; each states the source it rests on and what it changes in this plan. None of them changes source now. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 70a1412e..c69385cc 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -208,3 +208,18 @@ Review document: `.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_ - 2026-09-22: FP8 native rerun authorized by the user. Same launcher as `exp-0922-145047-660565`, worktree clean at `c231322`: `exp-0922-202645-561899`, `codesign` / H800 (`gpu-h800-0095`), `Succeeded`, `8 passed in 14.27s`, `W6:PARITY_EXIT=0`. Log via `logs_replica` only (`logs_rjob` empty); the first fetch two minutes after completion returned a truncated window, the second fetch five minutes later returned the full tail. Records: W6 report §8, corrections report C4, review.md C35-03, validation.md Step 6. - 2026-09-22: Second Step 9 plan review at the user's direction (quality gates). Eight findings R9-01..R9-08 recorded in `plan.md` §18.12 with amendments to D9-1 (hook payload = completion hook signature), D9-2 (key both observation kinds by the observing iteration; first candidate `ForwardSyncState._next_step_id_by_replica`, I5 gap to be measured), P1(a) (oracle = engine loop feeding the real balancer), §18.10, §18.11 representation column, §18.2 (32-step all-reduce); `design.md` section "What the code already provides"; `review.md` disposition table. Records only; no Step 9 source change; execution awaits the start signal. + +## Step 9 execution — P1 (2026-09-22) + +User start signal: "开始执行step9", with the quality gates repeated (readability, maintainability, high-value changes only, no hard-coding, no temporary patches, no over-defense, no redundancy, plain names). + +| Package | State | Evidence | +| --- | --- | --- | +| P1(a) reference-loop oracle | completed | `tests/comparison/dp_placement_pp/reference_loop.py`; `tests/unit/test_dp_placement_reference_loop.py` (9 passed, 1.21 s, `frontier-py310`). §18.11 state table confirmed as written; PP=1 shown to degenerate to "every iteration schedules and applies"; depth 3 shown to allow two consecutive admission-only publications, which rules out any stride constant. | +| P1(b) Frontier boundary probe | blocked | Three shapes probed (`attn_dp=2 PP=1`, `attn_dp=1 PP=2`, `attn_dp=1 PP=3`), tables in `plan.md` §18.13. The fourth shape, MoE `attn_dp=2, moe_ep=2, PP=2`, drains the event queue with requests unfinished — pre-existing defect W9-01 in `issues.md`. | +| Design checkpoint (D9-1, D9-2) | open | D9-1's payload is settled (the completion hook signature already carries lane, load and a key source). D9-2 is not: the candidate key `ForwardSyncState._next_step_id_by_replica` satisfies I1, I2, I3, I4 and I6 on the runnable shapes but fails I5, and no alternative can be checked against I1 without a running `attn_dp>1, PP>1` shape. | +| P2–P6, G3–G5 | paused | All depend on the design checkpoint or on that shape. | + +W9-01 is not caused by this PR: `stage_execution_context.py`, `replica_stage_schduler.py` and `stage_contexts.py` are byte-identical to `main`. It is unobserved because every Simulator-level test with `attn_dp > 1` uses `num_pipeline_stages = 1` and no shipped example sets `attn_dp > 1`. Scope decision requested from the user; recommendation is to fix it as a separate correctness item rather than inside this feature branch. + +W9-02: `attn_dp=2, moe_ep=2, PP=3` is rejected at construction (6 devices against node size 4). Plan C1's PP3 row amended to `attn_dp=1`. From 4a510d42d040a1a3c2eb1e438f5ea2c44165ee80 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Tue, 22 Sep 2026 21:57:50 +0800 Subject: [PATCH 41/80] docs(step9): record G1 instrumentation, the case binding, and Step 9 evidence The ground-truth checkout now carries the placement chain on its own branch, committed locally and not pushed. The case manifest records the checkout tuple, the diff artifact and its hash, the four record kinds with their join keys, and the two decisions that are still blocked. Package G1 needed one file fewer than planned: the scheduler needs no dp_rank column because the scheduler output already carries the scheduled request ids and the record is written by the engine that owns the rank. --- .../calibration/dp_pp_case_001/case_init.md | 21 + .../dp_pp_case_001/g1_instrumentation.diff | 412 ++++++++++++++++++ .../calibration/dp_pp_case_001/manifest.yaml | 134 ++++++ .../progress.md | 32 ++ ...st_report_2026-09-22_w9_pp_dp_placement.md | 155 +++++++ .../validation.md | 20 + 6 files changed, 774 insertions(+) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/case_init.md create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/g1_instrumentation.diff create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/manifest.yaml create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w9_pp_dp_placement.md diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/case_init.md b/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/case_init.md new file mode 100644 index 00000000..3667bd52 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/case_init.md @@ -0,0 +1,21 @@ +# case_init — dp_pp_case_001 + +Immutable record. Written 2026-09-22 when package G1 completed. + +| Field | Value | +| --- | --- | +| `case_id` | `dp_pp_case_001` | +| `run_generation` | 1 | +| `requesting_user` | `i-fengyicheng` | +| `reviewer_identity` | `i-fengyicheng` | +| `auto_recycle` | `false` | +| Ground-truth checkout | `/data/ycfeng/Frontier/.real-engine/vLLM-BS` | +| Ground-truth branch | `feature/frontier-comparison-instrumentation` | +| Ground-truth commit | `494b9f327036d4493034a9b37ebb343354884e01` | +| Ground-truth remote tip | `ea95f571e20937c7c908c6d59ddd1cd6bf9268f1` | +| Tree dirty | `false` | +| Diff artifact | `g1_instrumentation.diff`, SHA-256 `84fc24db0e2411268a93f8be7ca5f8e4e5072ea09d86063cac2cfb98381feb2c` | +| Weight mode | `dummy`, no real weight download | + +No run has been launched. `manifest.yaml` carries the two decisions that are +still `BLOCKED`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/g1_instrumentation.diff b/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/g1_instrumentation.diff new file mode 100644 index 00000000..5d788f9a --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/g1_instrumentation.diff @@ -0,0 +1,412 @@ +diff --git a/vllm/v1/engine/coordinator.py b/vllm/v1/engine/coordinator.py +index 596edfdbe..1fd9ad16f 100644 +--- a/vllm/v1/engine/coordinator.py ++++ b/vllm/v1/engine/coordinator.py +@@ -12,6 +12,7 @@ import zmq + from vllm.config import ParallelConfig + from vllm.logger import init_logger + from vllm.utils import get_mp_context, make_zmq_socket, set_process_title ++from vllm.v1 import frontier_trace + from vllm.v1.engine import EngineCoreOutputs, EngineCoreRequestType + from vllm.v1.serial_utils import MsgpackDecoder + from vllm.v1.utils import get_engine_client_zmq_addr, shutdown +@@ -157,6 +158,10 @@ class DPCoordinatorProc: + last_stats_wave = -1 + last_step_counts: Optional[list[list[int]]] = None + ++ # Identifies each set of counts sent to the front ends, so a placement ++ # can be traced back to the engine reports it was computed from. ++ snapshot_id = 0 ++ + with make_zmq_socket( + path=front_publish_address, # IPC + ctx=self.ctx, +@@ -208,12 +213,23 @@ class DPCoordinatorProc: + if last_step_counts is not None: + engine_req_counts_list = last_step_counts + last_step_counts = None ++ counts_source = "latched_previous_step" + else: + engine_req_counts_list = self._get_engine_counts() + stats_changed = False ++ counts_source = "current" ++ ++ snapshot_id += 1 ++ frontier_trace.log_dp_placement_record( ++ "coordinator_publish", ++ snapshot=snapshot_id, ++ counts=engine_req_counts_list, ++ counts_source=counts_source, ++ wave=current_wave, ++ engines_running=engines_running) + + to_publish = (engine_req_counts_list, current_wave, +- engines_running) ++ engines_running, snapshot_id) + publish_front.send(msgspec.msgpack.encode(to_publish)) + last_publish_time = int(time.time() * 1000) + continue +@@ -290,21 +306,35 @@ class DPCoordinatorProc: + stats = self.engines[eng_index].request_counts + stats_step = scheduler_stats.step_counter + stats_wave = scheduler_stats.current_wave ++ disposition = "applied_without_latch" + if (stats_wave > last_stats_wave + or stats_wave == last_stats_wave + and stats_step > last_stats_step): + if stats_changed: + last_step_counts = self._get_engine_counts( + do_copy=True) ++ disposition = "latched_previous_step" ++ else: ++ disposition = "advanced_without_latch" + last_stats_step = stats_step + last_stats_wave = stats_wave + elif stats_wave != last_stats_wave or ( + stats_step != last_stats_step): ++ disposition = "out_of_order" + logger.warning( + "Received stats for out-of-order " + "step (%d, %d) from engine %d (expected " + "> (%d, %d))", stats_wave, stats_step, + eng_index, last_stats_wave, last_stats_step) ++ frontier_trace.log_dp_placement_record( ++ "coordinator_receive", ++ engine=eng_index, ++ wave=stats_wave, ++ step=stats_step, ++ waiting=scheduler_stats.num_waiting_reqs, ++ running=scheduler_stats.num_running_reqs, ++ disposition=disposition, ++ latched_counts=last_step_counts) + stats[0] = scheduler_stats.num_waiting_reqs + stats[1] = scheduler_stats.num_running_reqs + stats_changed = True +@@ -335,7 +365,8 @@ class DPCoordinatorProc: + self._send_start_wave(publish_back, wave, eng_index) + + if wave_state_changed: +- message = (None, current_wave, engines_running) ++ message = (None, current_wave, engines_running, ++ snapshot_id) + publish_front.send(msgspec.msgpack.encode(message)) + + @staticmethod +diff --git a/vllm/v1/engine/core.py b/vllm/v1/engine/core.py +index bfc29fc78..c61236776 100644 +--- a/vllm/v1/engine/core.py ++++ b/vllm/v1/engine/core.py +@@ -45,6 +45,7 @@ from vllm.v1.engine.utils import (EngineHandshakeMetadata, EngineZmqAddresses, + from vllm.v1.executor.abstract import Executor + from vllm.v1.kv_cache_interface import KVCacheConfig + from vllm.v1.metrics.stats import SchedulerStats ++from vllm.v1 import frontier_trace + from vllm.v1.outputs import ModelRunnerOutput + from vllm.v1.request import Request, RequestStatus + from vllm.v1.serial_utils import MsgpackDecoder, MsgpackEncoder +@@ -156,6 +157,12 @@ class EngineCore: + self.batch_queue_size) + self.batch_queue = deque(maxlen=self.batch_queue_size) + ++ # Classification of the most recent iteration, written by the step ++ # methods and consumed by the data-parallel busy loop, which is where ++ # the wave, step counter and published counts are known. Stays None ++ # while Frontier placement logging is off. ++ self.frontier_iteration: Optional[dict[str, Any]] = None ++ + self.request_block_hasher: Optional[Callable[[Request], + list[BlockHash]]] = None + if (self.vllm_config.cache_config.enable_prefix_caching +@@ -305,9 +312,48 @@ class EngineCore: + engine_core_outputs = self.scheduler.update_from_output( + scheduler_output, model_output) # type: ignore + ++ self._record_frontier_iteration(scheduler_output, ++ applied_output=True, ++ queue_occupancy=0) ++ + return (engine_core_outputs, + scheduler_output.total_num_scheduled_tokens > 0) + ++ def _record_frontier_iteration(self, ++ scheduled_output: Optional[SchedulerOutput], ++ *, applied_output: bool, ++ queue_occupancy: int) -> None: ++ """Classify one engine iteration for Frontier placement analysis. ++ ++ Which branch an iteration takes is what decides whether it publishes ++ request counts on its own, so the branch is recorded rather than ++ inferred later from the counts. ++ """ ++ if not frontier_trace.is_dp_placement_logging_enabled(): ++ return ++ ++ if scheduled_output is None: ++ branch = "applied_without_scheduling" ++ elif applied_output: ++ branch = "applied_after_scheduling" ++ else: ++ branch = "scheduled_without_applying" ++ ++ self.frontier_iteration = { ++ "branch": ++ branch, ++ "applied_output": ++ applied_output, ++ "queue_occupancy": ++ queue_occupancy, ++ "scheduled_new_req_ids": ++ [data.req_id for data in scheduled_output.scheduled_new_reqs] ++ if scheduled_output is not None else [], ++ "num_scheduled_tokens": ++ scheduled_output.total_num_scheduled_tokens ++ if scheduled_output is not None else 0, ++ } ++ + def post_step(self, model_executed: bool) -> None: + if self.use_spec_decode and model_executed: + # Take the draft token ids. +@@ -339,17 +385,22 @@ class EngineCore: + assert len(batch_queue) < self.batch_queue_size + + model_executed = False ++ scheduled_output: Optional[SchedulerOutput] = None + if self.scheduler.has_requests(): +- scheduler_output = self.scheduler.schedule() +- future = self.model_executor.execute_model(scheduler_output) ++ scheduled_output = self.scheduler.schedule() ++ future = self.model_executor.execute_model(scheduled_output) + batch_queue.appendleft( +- (future, scheduler_output)) # type: ignore[arg-type] ++ (future, scheduled_output)) # type: ignore[arg-type] + +- model_executed = scheduler_output.total_num_scheduled_tokens > 0 ++ model_executed = scheduled_output.total_num_scheduled_tokens > 0 + if model_executed and len(batch_queue) < self.batch_queue_size \ + and not batch_queue[-1][0].done(): + # Don't block on next worker response unless the queue is full + # or there are no more requests to schedule. ++ self._record_frontier_iteration( ++ scheduled_output, ++ applied_output=False, ++ queue_occupancy=len(batch_queue)) + return None, True + + elif not batch_queue: +@@ -366,6 +417,10 @@ class EngineCore: + engine_core_outputs = self.scheduler.update_from_output( + scheduler_output, model_output) + ++ self._record_frontier_iteration(scheduled_output, ++ applied_output=True, ++ queue_occupancy=len(batch_queue)) ++ + return engine_core_outputs, model_executed + + def shutdown(self): +@@ -1072,9 +1127,10 @@ class DPEngineCoreProc(EngineCoreProc): + else: + super()._handle_client_request(request_type, request) + +- def _maybe_publish_request_counts(self): ++ def _maybe_publish_request_counts(self) -> bool: ++ """Returns whether this iteration published its request counts.""" + if not self.publish_dp_lb_stats: +- return ++ return False + + # Publish our request counts (if they've changed). + counts = self.scheduler.get_request_counts() +@@ -1085,6 +1141,31 @@ class DPEngineCoreProc(EngineCoreProc): + current_wave=self.current_wave) + self.output_queue.put_nowait( + (-1, EngineCoreOutputs(scheduler_stats=stats))) ++ return True ++ return False ++ ++ def _log_frontier_iteration(self, published: bool) -> None: ++ """Emit the iteration the step methods classified. ++ ++ `step_counter` is read before `_has_global_unfinished_reqs` advances ++ it, so it is the same value a published report carried, which makes ++ `(engine, wave, step)` the join for the whole placement chain. ++ """ ++ iteration = self.frontier_iteration ++ if iteration is None: ++ return ++ self.frontier_iteration = None ++ ++ num_running_reqs, num_waiting_reqs = self.scheduler.get_request_counts() ++ frontier_trace.log_dp_placement_record( ++ "engine_iteration", ++ engine=self.dp_rank, ++ wave=self.current_wave, ++ step=self.step_counter, ++ waiting=num_waiting_reqs, ++ running=num_running_reqs, ++ published=published, ++ **iteration) + + def run_busy_loop(self): + """Core busy loop of the EngineCore for data parallel case.""" +@@ -1096,7 +1177,7 @@ class DPEngineCoreProc(EngineCoreProc): + + # 2) Step the engine core. + executed = self._process_engine_step() +- self._maybe_publish_request_counts() ++ self._log_frontier_iteration(self._maybe_publish_request_counts()) + + local_unfinished_reqs = self.scheduler.has_unfinished_requests() + if not executed: +diff --git a/vllm/v1/engine/core_client.py b/vllm/v1/engine/core_client.py +index 605bedaf1..daf8a67ef 100644 +--- a/vllm/v1/engine/core_client.py ++++ b/vllm/v1/engine/core_client.py +@@ -29,6 +29,7 @@ from vllm.v1.engine import (EngineCoreOutputs, EngineCoreRequest, + EngineCoreRequestType, + ReconfigureDistributedRequest, ReconfigureRankType, + UtilityOutput) ++from vllm.v1 import frontier_trace + from vllm.v1.engine.coordinator import DPCoordinator + from vllm.v1.engine.core import EngineCore, EngineCoreProc + from vllm.v1.engine.exceptions import EngineDeadError +@@ -977,6 +978,10 @@ class DPAsyncMPClient(AsyncMPClient): + # List of [waiting, running] pair per engine. + # Used only by DPLBAsyncMPClient subclass. + self.lb_engines: list[list[int]] = [[0, 0] for _ in self.core_engines] ++ # Coordinator snapshot the counts above came from; 0 until the first ++ # one arrives, so a placement made from the initial zeros is visible ++ # as such. ++ self.lb_snapshot: int = 0 + + self.first_req_sock_addr = get_open_zmq_inproc_path() + self.first_req_send_socket = self.resources.first_req_send_socket = ( +@@ -1070,12 +1075,20 @@ class DPAsyncMPClient(AsyncMPClient): + continue + + # Update local load-balancing state. +- counts, wave, running = msgspec.msgpack.decode(buf) ++ counts, wave, running, snapshot = msgspec.msgpack.decode( ++ buf) + self.current_wave = wave + self.engines_running = running + if counts is not None: + sliced_counts = counts[count_slice] + self.lb_engines = sliced_counts ++ self.lb_snapshot = snapshot ++ frontier_trace.log_dp_placement_record( ++ "frontend_snapshot", ++ snapshot=snapshot, ++ counts=sliced_counts, ++ wave=wave, ++ engines_running=running) + logger.debug("Received counts: %s (%s)", sliced_counts, + count_slice) + +@@ -1147,6 +1160,15 @@ class DPLBAsyncMPClient(DPAsyncMPClient): + if score < min_score: + min_score = score + eng_index = idx ++ frontier_trace.log_dp_placement_record( ++ "frontend_route", ++ request_id=request.request_id, ++ engine=eng_index, ++ snapshot=self.lb_snapshot, ++ counts=[list(counts) for counts in current_counts], ++ score=min_score, ++ start_index=self.eng_start_index, ++ reservation=self.client_count) + # Increment local waiting count for better balancing between stats + # updates from the coordinator (which happen every 100ms). + current_counts[eng_index][0] += self.client_count +diff --git a/vllm/v1/frontier_trace.py b/vllm/v1/frontier_trace.py +index d9bddeea5..a727a86b9 100644 +--- a/vllm/v1/frontier_trace.py ++++ b/vllm/v1/frontier_trace.py +@@ -2,14 +2,18 @@ + # SPDX-FileCopyrightText: Copyright contributors to the vLLM project + """Runtime gate for Frontier trace logging.""" + ++import atexit + from contextlib import contextmanager + import json + import os ++import time + from typing import Any, Mapping + + _SKIP_WARMUP = os.environ.get("VLLM_FRONTIER_TRACE_SKIP_WARMUP", "0") == "1" + _TRACE_ACTIVE = not _SKIP_WARMUP + _PP_BOUNDARY_LOG_ENV_VAR = "VLLM_FRONTIER_PP_BOUNDARY_LOG_PATH" ++_DP_PLACEMENT_LOG_DIR_ENV_VAR = "VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR" ++_DP_PLACEMENT_FLUSH_EVERY = 1024 + _PP_BOUNDARY_REQUIRED_FIELDS = ( + "model_name", + "timestamp", +@@ -99,3 +103,67 @@ def disable_for_warmup(): + yield + finally: + _TRACE_ACTIVE = True ++ ++ ++# Data-parallel placement records. Every process that takes part in a placement ++# decision -- each engine core, the coordinator, each API server -- writes its ++# own file, so records never interleave and the reader can tell the roles ++# apart. They are buffered because an engine core writes one per iteration of ++# its busy loop; they reach disk every _DP_PLACEMENT_FLUSH_EVERY records and at ++# normal process exit. ++_dp_placement_records: list[dict[str, Any]] = [] ++_dp_placement_seq = 0 ++ ++ ++def get_dp_placement_log_dir() -> str: ++ return os.environ.get(_DP_PLACEMENT_LOG_DIR_ENV_VAR, "") ++ ++ ++def is_dp_placement_logging_enabled() -> bool: ++ return is_active() and bool(get_dp_placement_log_dir()) ++ ++ ++def log_dp_placement_record(kind: str, **fields: Any) -> None: ++ """Buffer one placement record of the given kind. ++ ++ `seq` orders the records one process wrote and is the tie-break when two ++ of them carry the same correlation id. Timestamps are for reading; the ++ join between processes is always a correlation id. ++ """ ++ if not is_dp_placement_logging_enabled(): ++ return ++ ++ global _dp_placement_seq ++ record: dict[str, Any] = { ++ "kind": kind, ++ "pid": os.getpid(), ++ "seq": _dp_placement_seq, ++ "monotonic": time.monotonic(), ++ } ++ record.update(fields) ++ _dp_placement_records.append(record) ++ _dp_placement_seq += 1 ++ ++ if len(_dp_placement_records) >= _DP_PLACEMENT_FLUSH_EVERY: ++ flush_dp_placement_records() ++ ++ ++def flush_dp_placement_records() -> None: ++ if not _dp_placement_records: ++ return ++ ++ log_dir = get_dp_placement_log_dir() ++ os.makedirs(log_dir, exist_ok=True) ++ log_path = os.path.join(log_dir, f"dp_placement_{os.getpid()}.jsonl") ++ try: ++ with open(log_path, "a", encoding="utf-8") as file: ++ for record in _dp_placement_records: ++ file.write(json.dumps(record) + "\n") ++ except OSError as exc: ++ raise RuntimeError( ++ f"Failed to write Frontier DP placement log file: {log_path}" ++ ) from exc ++ _dp_placement_records.clear() ++ ++ ++atexit.register(flush_dp_placement_records) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/manifest.yaml b/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/manifest.yaml new file mode 100644 index 00000000..4055d691 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/manifest.yaml @@ -0,0 +1,134 @@ +# Frontier calibration case: vLLM DP placement under pipeline parallelism. +# Fields marked BLOCKED are unresolved decisions. Per the v2 contract a runner +# never fills them from an unstated default; they are answered before the case +# leaves package G2. + +case_id: dp_pp_case_001 +run_generation: 1 +created_at_utc: "2026-09-22" +requesting_user: i-fengyicheng +reviewer_identity: i-fengyicheng +auto_recycle: false + +purpose: >- + Compare the vLLM V1 data-parallel placement chain against Frontier's + vllm_load_balancing cluster scheduler at pipeline_parallel_size 2, where an + engine iteration can publish changed request counts without completing any + batch. Task step 9, plan section 18.6. + +selected_checkout: + frontier_worktree: /data/ycfeng/Frontier/.worktrees/issue26-correctness-pr + frontier_branch: fix/issue26-correctness-pr + frontier_commit: BLOCKED # pinned at simulator-run, after package P2 + +groundtruth_checkout_path: /data/ycfeng/Frontier/.real-engine/vLLM-BS +groundtruth_branch: feature/frontier-comparison-instrumentation +groundtruth_ref: refs/heads/feature/frontier-comparison-instrumentation +groundtruth_commit: 494b9f327036d4493034a9b37ebb343354884e01 +groundtruth_tree_dirty: false +groundtruth_remote_url: https://github.com/fwyc0573/vLLM-BS.git +groundtruth_remote_tip: ea95f571e20937c7c908c6d59ddd1cd6bf9268f1 +groundtruth_diff_artifact: g1_instrumentation.diff +groundtruth_diff_sha256: 84fc24db0e2411268a93f8be7ca5f8e4e5072ea09d86063cac2cfb98381feb2c +groundtruth_dirty_patch_sha256: null +groundtruth_overlay_patch_applied: false +groundtruth_weight_mode: dummy +real_weight_download: false + +groundtruth_instrumentation: + gate_env_var: VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR + output: one JSONL file per process, named dp_placement_.jsonl + buffering: 1024 records, plus a flush at normal process exit + changed_files: + - vllm/v1/frontier_trace.py + - vllm/v1/engine/core.py + - vllm/v1/engine/coordinator.py + - vllm/v1/engine/core_client.py + record_kinds: + engine_iteration: + written_by: DPEngineCoreProc.run_busy_loop + join_key: [engine, wave, step] + fields: [branch, applied_output, queue_occupancy, scheduled_new_req_ids, + num_scheduled_tokens, waiting, running, published] + coordinator_receive: + written_by: DPCoordinatorProc.process_input_socket + join_key: [engine, wave, step] + fields: [waiting, running, disposition, latched_counts] + coordinator_publish: + written_by: DPCoordinatorProc.process_input_socket + join_key: [snapshot] + fields: [counts, counts_source, wave, engines_running] + frontend_snapshot: + written_by: DPLBAsyncMPClient stats update task + join_key: [snapshot] + fields: [counts, wave, engines_running] + frontend_route: + written_by: DPLBAsyncMPClient.get_core_engine_for_request + join_key: [request_id, snapshot] + fields: [engine, counts, score, start_index, reservation] + wire_change: >- + The coordinator's front-end publication carries a fourth element, the + snapshot id. Both the only sender and the only receiver are in this + checkout. Without it an applied snapshot cannot be tied to the engine + reports it came from, which is the join T1 needs. + scheduler_py_unchanged: >- + The plan listed vllm/v1/core/sched/scheduler.py as a likely change for a + dp_rank column. It is not needed: SchedulerOutput already carries + scheduled_new_reqs and total_num_scheduled_tokens, and the engine record is + written by the engine that owns the rank. + +topology: + vllm: "--data-parallel-size 2 --pipeline-parallel-size 2 --tensor-parallel-size 1 --enable-expert-parallel" + frontier: + attn_dp: 2 + attn_tp: 1 + moe_tp: 1 + moe_ep: 2 + num_pipeline_stages: 2 + num_replicas: 1 + cluster_scheduler: vllm_load_balancing + replica_scheduler: vllm_v1 + devices: 4x H800 + blocked_by: >- + W9-01. Frontier cannot run attn_dp=2 with num_pipeline_stages=2 today; the + stage admission FIFO deadlocks. The vLLM side of the case is unaffected. + +model: + name: Qwen3MoeForCausalLM + config: data/config/models/Qwen3-30B-A3B-tiny.json + layers: 8 + experts: 16 + top_k: 8 + load_format: dummy + skip_tokenizer_init: true + +workload: + source: single Frontier trace CSV (arrived_at, num_prefill_tokens, num_decode_tokens) + request_id_namespace: BLOCKED # set by G2 make_trace.py + request_id_encoding: "x-request-id header, propagated to the engine request id" + formal_request_ids: BLOCKED + warmup_request_ids: BLOCKED + warmups: 3 + +modes: + groundtruth_clean: + status: NOT_RUN + artifact_path: /mnt/codesign-exp/ycfeng/frontier/dp_pp_calibration/ + simulator: + status: NOT_RUN + simulator_control_completion_reporting: + status: NOT_RUN + +decisions: + - decision_id: D-gpu-authorization + question: >- + Does the S0 2-GPU smoke and the S1 4-GPU ground-truth run have the user's + GPU authorization on charged_group codesign? + status: BLOCKED + - decision_id: W9-01-scope + question: >- + Is the pre-existing stage admission deadlock fixed inside this feature + branch, fixed as a separate correctness item, or does step 9 pause? + status: BLOCKED + +status: INITIALIZED diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index c69385cc..49b351c5 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-22 | Step 9 execution started: P1(a) oracle completed, P1(b) blocked by W9-01, design checkpoint partially settled, G1 ground-truth instrumentation completed and case binding written. | | 2026-09-22 | FP8 native rerun PASS (`exp-0922-202645-561899`, 8 passed, exit 0). Second Step 9 plan review (plan §18.12, R9-01..R9-08) recorded; Step 9 still not started. | | 2026-09-21 | Step 0 started: records landed, environment created, baseline pending. | | 2026-09-22 | Maintainer review dispositions recorded; Checkpoint C closed: parent merged, W2 tests strengthened, W2 re-measured with one harness revision. | @@ -223,3 +224,34 @@ User start signal: "开始执行step9", with the quality gates repeated (readabi W9-01 is not caused by this PR: `stage_execution_context.py`, `replica_stage_schduler.py` and `stage_contexts.py` are byte-identical to `main`. It is unobserved because every Simulator-level test with `attn_dp > 1` uses `num_pipeline_stages = 1` and no shipped example sets `attn_dp > 1`. Scope decision requested from the user; recommendation is to fix it as a separate correctness item rather than inside this feature branch. W9-02: `attn_dp=2, moe_ep=2, PP=3` is rejected at construction (6 devices against node size 4). Plan C1's PP3 row amended to `attn_dp=1`. + +### G1 ground-truth instrumentation (2026-09-22, completed) + +Runs in parallel with P1 in the work graph and does not depend on W9-01. + +`/data/ycfeng/Frontier/.real-engine/vLLM-BS` now has the local branch +`feature/frontier-comparison-instrumentation` created at the remote tip +`ea95f571e`, with the instrumentation committed as `494b9f327`. Tree clean, +nothing pushed, per decision D-b. + +Five observations, four record kinds, one env gate +(`VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR`), buffered one file per process. The +engine iteration and its publication are one record because they happen in the +same turn of the busy loop under the same `(engine, wave, step)` key. Coordinator +publications carry a snapshot id that is sent on to the front ends, which is +what makes an applied snapshot and the placement made from it traceable back to +the engine reports behind them. + +`vllm/v1/core/sched/scheduler.py` turned out not to need a change: the plan +expected a `dp_rank` column, but `SchedulerOutput` already carries the +scheduled request ids and the record is written by the engine that owns the +rank. The changed-file list is four files, 212 insertions, 10 deletions. + +Case binding written to `calibration/dp_pp_case_001/` (`manifest.yaml`, +`case_init.md`, `g1_instrumentation.diff` with SHA-256 +`84fc24db0e2411268a93f8be7ca5f8e4e5072ea09d86063cac2cfb98381feb2c`). Two +manifest decisions are `BLOCKED`: GPU authorization for the S0/S1 runs, and the +W9-01 scope decision. + +Evidence: `test_report_2026-09-22_w9_pp_dp_placement.md` §3. The instrumented +engine paths are not executed yet; that is package G3 and needs a GPU host. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w9_pp_dp_placement.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w9_pp_dp_placement.md new file mode 100644 index 00000000..241da2fd --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w9_pp_dp_placement.md @@ -0,0 +1,155 @@ +# Test report — Step 9, PP>1 support for the vLLM DP placement policy + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-22 | Created. Covers packages P1(a), P1(b) and G1. | + +Environment for every CPU check below: + +| Item | Value | +| --- | --- | +| Host | `kun-workspace-vgen2` | +| Worktree | `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`, branch `fix/issue26-correctness-pr` | +| Interpreter | `/data/ycfeng/envs/frontier-py310/bin/python` (Python 3.10, no `torch`) | +| `PYTHONPATH` | the worktree root | + +## 1. P1(a) — reference-loop oracle + +Command: + +```bash +PYTHONPATH=$PWD /data/ycfeng/envs/frontier-py310/bin/python -m pytest \ + tests/unit/test_dp_placement_reference_loop.py -q -p no:cacheprovider +``` + +Expected: every row of the plan's §18.11 state table is reproduced by a model +of the vLLM engine iteration alone, and the model drives the real +`VllmDPLoadBalancer` rather than a second copy of it. + +Actual: `9 passed in 1.21s`. **PASS.** + +Together with `tests/unit/test_vllm_dp_load_balancer.py`: +`70 passed in 1.44s`. **PASS.** + +Row-by-row result: + +| Scripted iteration | Expected | Observed | Verdict | +| --- | --- | --- | --- | +| Depth 1, three iterations | Each schedules and applies; loads (0,2), (0,1), (0,0) | as expected | PASS | +| Depth 2, cold fill | First `scheduled=True, applied=False`, load (0,3), published; second applies | as expected | PASS | +| Depth 3, oldest already ready | One combined publication, load (0,4) | one publication | PASS | +| Depth 3, zero-token schedule | No early return; applies; counts unchanged so nothing published | `published=False` | PASS | +| Depth 2, drain | `scheduled=False, applied=True`, published, load (0,0) | as expected | PASS | +| Depth 3, three admissions | Steps 0, 1, 2; first two do not apply; all published | as expected | PASS | +| Engine stepped with no work | `ValueError` | raised | PASS | +| Two peers at the same index | Equal step, both counts visible to the frontend | equal, frontend sees `[(0,3), (0,3)]` | PASS | +| One peer one iteration ahead | Counters diverge | 2 against 0 | PASS | + +Conclusion carried into the design: at depth 1 the admission-only iteration +cannot occur, so Frontier's completion-only report is already exact at PP=1; +above depth 1 it is not, and no fixed stride between completion keys can hold +because depth 3 produces two consecutive admission-only publications. + +## 2. P1(b) — Frontier boundary probe + +Scratch driver: `$SCRATCH/w9/probe_frontier_boundaries.py` (one process per +shape; `IS_MOE` is a process global in this codebase). Each shape builds a +`MONOLITHIC` single-Replica `vllm_v1` configuration on a 6-layer model so that +PP 1, 2 and 3 all divide the layer count, runs the Simulator, and records at +every admission and completion the lane, the slot occupancy and the value of +`ForwardSyncState._next_step_id_by_replica`, the candidate report key. + +Expected: every shape completes all six requests, and the candidate key is +equal for peers of one forward and distinct for distinct engine iterations. + +| Shape | Requests completed | Boundaries | Candidate key behavior | Verdict | +| --- | --- | --- | --- | --- | +| `attn_dp=2, moe_ep=2, PP=1` | 6/6 | 24 | Peers always equal; values 0, 6, 12, 18, 24; each completion shares a value with the admission it triggers | PASS | +| `attn_dp=1, moe_ep=1, PP=2` | 6/6 | 28 | Both cold-fill admissions read 0 | FAIL (invariant I5) | +| `attn_dp=1, moe_ep=1, PP=3` | 6/6 | 32 | All three cold-fill admissions read 0 | FAIL (invariant I5) | +| `attn_dp=2, moe_ep=2, PP=2` | 0/4 | — | Event queue drained with requests unfinished | FAIL (W9-01) | +| `attn_dp=2, moe_ep=2, PP=3` | — | — | Rejected at construction | FAIL (W9-02) | + +Request-count sensitivity of the deadlock, fresh process each time: + +| Requests | 2 | 3 | 4 | 6 | 8 | +| --- | --- | --- | --- | --- | --- | +| `attn_dp=2, moe_ep=2, PP=2` | completes | completes | drained | drained | drained | + +Scheduler state at drain, stage `(0, 0)`: + +``` +capacity=2 active_seqs=[0] waiting_fifo_seqs=[1, 2, 3] +sealed=False ep_active=False +``` + +W9-02 rejection message: + +``` +collective-sim physical topology requires cluster_total_devices 6 to be +divisible by node size 4 +``` + +Both are recorded in `issues.md`. The three source files involved in W9-01 are +byte-identical to `origin/main`, verified with `git rev-parse HEAD:` +against `origin/main:`. + +## 3. G1 — ground-truth instrumentation + +Checkout `/data/ycfeng/Frontier/.real-engine/vLLM-BS`, local branch +`feature/frontier-comparison-instrumentation` created at the remote tip +`ea95f571e` and committed as `494b9f327`. Tree clean, nothing pushed (D-b). + +Changed files, 212 insertions and 10 deletions: + +| File | Change | +| --- | --- | +| `vllm/v1/frontier_trace.py` | Buffered per-process JSONL writer gated by `VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR` | +| `vllm/v1/engine/core.py` | Iteration classification at both step methods; publish helper returns whether it published; the DP busy loop emits the record | +| `vllm/v1/engine/coordinator.py` | Receive disposition and publication snapshot id, sent on to the front ends | +| `vllm/v1/engine/core_client.py` | Applied snapshot and routing decision | + +`vllm/v1/core/sched/scheduler.py` is not changed. The plan expected a `dp_rank` +column there; it is unnecessary because `SchedulerOutput` already carries the +scheduled request ids and the record is written by the engine that owns the +rank. + +Writer check, since the module is pure standard library and can be loaded +without `torch`: + +```bash +/data/ycfeng/envs/frontier-py310/bin/python $SCRATCH/w9/check_trace_writer.py +``` + +Expected: nothing is written while the gate is off; records buffer until the +bound or an explicit flush; one file per process id; `seq` is dense and +ordered; a second flush is a no-op; the warmup gate suppresses records. + +Actual: `frontier_trace DP placement writer: all checks passed`. **PASS.** +Sample record: + +```json +{"engine": 1, "kind": "engine_iteration", "monotonic": 549631.140559739, + "pid": 3180674, "published": true, "running": 0, "seq": 0, "step": 0, + "waiting": 0} +``` + +Syntax check of all four changed files with `python -m py_compile`: **PASS.** + +Verification limit: the instrumented engine paths have not been executed. They +need a GPU host with the compiled vLLM extensions, which is package G3. What is +established here is that the writer behaves as specified and that the four +files parse; that the records are emitted at the right points is asserted from +source reading, not from a run. + +## 4. Summary + +| Package | Verdict | +| --- | --- | +| P1(a) | PASS | +| P1(b) | Evidence collected on three shapes; BLOCKED on the fourth by W9-01 | +| Design checkpoint D9-1 | Settled | +| Design checkpoint D9-2 | Open; the candidate key fails invariant I5 and the alternatives cannot be tested until W9-01 is resolved | +| G1 | PASS for what is testable without a GPU | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md index a0d37744..096fc8c7 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/validation.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-22 | Step 9 partial validation added: reference-loop oracle, Frontier boundary probes on three shapes, the two probe failures (invariant I5, W9-01), and the ground-truth writer checks. | | 2026-09-21 | Created. Environment recorded; baseline results recorded in the refactor task's Step 0 report because both branches share the same base commit. | | 2026-09-21 | Step 1 recorded: audit spot checks and the vLLM reference identity check. | | 2026-09-21 | Step 2 recorded: unit sensitivity and the fidelity measurement against a stated expectation. | @@ -501,3 +502,22 @@ Full record: `test_report_2026-09-22_w8_combined_regression.md`. | Working tree | `git status --porcelain` empty afterwards. The one leftover `outputs/examples/` tree was removed after confirming 0 tracked files there; the 110 tracked files under `outputs/` are all still present. | | Pre-existing defect, deferred | `AGENTS.md` §Tests names `comm_backend_tests/`, `debug/`, and two `bash tests/debug/e2e-level/monolith_mode/scripts/*.sh` commands. `tests/debug/` exists neither here nor on `origin/main`. The same missing tree causes 10 of the 84 baseline unit failures in `test_colocation_release_review_contracts.py`, and a docstring at `vllm_v1_engine_replica_scheduler.py:16` still points into it. One pre-existing defect class from the release scrub, unrelated to Issue 26; recorded in `future.md` and not repaired here. The PP2 coverage was obtained through the example scripts instead. | | Limits | CPU only. No native profiling suite and no vLLM serving or TTFT comparison, both excluded by §14.1. The PD-AF Reference-checkout tests could not run on this host. The example runs use dummy execution time except for the CSV smokes, so they validate structure, lifecycle, and conservation rather than latency accuracy. | + +## Step 9 — PP>1 support for the vLLM DP placement policy (2026-09-22, partial) + +| Item | Command | Expected | Actual | Outcome | +| --- | --- | --- | --- | --- | +| Reference-loop oracle | `pytest tests/unit/test_dp_placement_reference_loop.py -q` | Every §18.11 state-table row reproduced from a model of the engine iteration alone | `9 passed in 1.21s` | PASS | +| Oracle with the existing balancer suite | `pytest tests/unit/test_dp_placement_reference_loop.py tests/unit/test_vllm_dp_load_balancer.py -q` | No interference with the shipped balancer tests | `70 passed in 1.44s` | PASS | +| Frontier boundary probe, `attn_dp=2 PP=1` | scratch `probe_frontier_boundaries.py` | 6/6 requests complete; peer keys equal | 6/6, 24 boundaries, peers equal at every boundary | PASS | +| Frontier boundary probe, `attn_dp=1 PP=2` | same | Consecutive admissions carry distinct keys | Both cold-fill admissions read key 0 | FAIL, invariant I5 | +| Frontier boundary probe, `attn_dp=1 PP=3` | same | Same | All three cold-fill admissions read key 0 | FAIL, invariant I5 | +| Frontier boundary probe, `attn_dp=2 PP=2` | same | 6/6 requests complete | Event queue drained with requests unfinished | FAIL, W9-01 | +| Ground-truth writer | scratch `check_trace_writer.py` | Gate off writes nothing; buffering, per-process file, dense `seq`, idempotent flush, warmup gate | All checks passed | PASS | +| Ground-truth syntax | `python -m py_compile` on the four changed vLLM files | Parse | All four parse | PASS | + +Limits. The instrumented vLLM paths have not been executed; that needs a GPU +host and is package G3. The two probe FAIL rows are findings, not regressions: +the I5 failures are the measurement the design checkpoint asked for, and W9-01 +is a pre-existing defect on `main`. Evidence: +`test_report_2026-09-22_w9_pp_dp_placement.md`. From 4c2d5731dc06358bd8f827e2984ba7c16ee68614 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 10:46:30 +0800 Subject: [PATCH 42/80] docs(step9): record the W9-01 fix on PR 36 and the resume order W9-01 was fixed as a separate correctness item on fix/stage-admission-ordering (draft PR 36, rule dac4e69) and validated against vLLM DP=2/PP=2. Record the resolution, answer the case manifest's scope decision, and copy the branch summary and test report. Step 9's PP>1 packages stay paused until PR 36 is merged forward and G3b passes as the composition check with W3. --- .../calibration/dp_pp_case_001/manifest.yaml | 8 +- .../issues.md | 35 ++- .../progress.md | 3 + .../w9_01_stage_admission_ordering/summary.md | 57 ++++ ...ort_2026-09-23_stage_admission_ordering.md | 272 ++++++++++++++++++ 5 files changed, 373 insertions(+), 2 deletions(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/manifest.yaml b/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/manifest.yaml index 4055d691..a09cb25c 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/manifest.yaml +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/calibration/dp_pp_case_001/manifest.yaml @@ -129,6 +129,12 @@ decisions: question: >- Is the pre-existing stage admission deadlock fixed inside this feature branch, fixed as a separate correctness item, or does step 9 pause? - status: BLOCKED + answer: "采纳你的推荐，继续" + outcome: >- + Fixed as a separate correctness item on fix/stage-admission-ordering, + draft PR 36 (rule dac4e69). The Frontier side of this case stays blocked + until PR 36 merges into main, is merged forward, and passes the G3b + composition check (issues.md W9-01, Resolution). + decided_at_utc: "2026-09-22" status: INITIALIZED diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md index ebc7187a..593dc15d 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md @@ -4,11 +4,14 @@ | Date | Change | | --- | --- | +| 2026-09-23 | W9-01: fixed on `fix/stage-admission-ordering` (draft PR 36) under option 2; resolution recorded, summary and test report copied to `w9_01_stage_admission_ordering/`. | | 2026-09-22 | Created; recorded W9-01 (stage-admission deadlock at PP>1 with attn_dp>1) and W9-02 (PP=3 x attn_dp=2 topology rejection) found during Step 9 P1(b). | ## W9-01 Stage admission deadlocks when `num_pipeline_stages > 1` and `attn_dp > 1` -Status: open, pre-existing on `main`, blocks Step 9 acceptance criteria C1 and C4. +Status: fixed on `fix/stage-admission-ordering` (draft PR 36), not yet on this +branch. Step 9's PP>1 packages stay paused until PR 36 merges into `main`, is +merged forward here, and passes the composition check in Resolution below. Found: 2026-09-22, Step 9 package P1(b) boundary probe. ### Symptom @@ -115,6 +118,36 @@ Recommendation: option 2. The defect is independent of the placement policy, predates both PRs, and changing shared admission ordering under a feature branch would mix an infrastructure fidelity fix into a feature PR. +### Resolution + +Option 2 was taken. The fix lives on its own branch and PR: + +| Item | Value | +| --- | --- | +| Branch / PR | `fix/stage-admission-ordering`, draft https://github.com/NetX-lab/Frontier/pull/36, head `4bcd616` | +| Rule commit | `dac4e69`: `StageExecutionContext.try_acquire` refuses a full-stage ticket only when an EP wave is queued ahead of it. EP waves keep the strict FIFO-head rule. | +| Acceptance rules | `aeeca93` (plan D-9) | +| Records | `fc34341`, `4bcd616`; copies in `w9_01_stage_admission_ordering/` (`summary.md`, `test_report_2026-09-23_stage_admission_ordering.md`) | + +Observed on that branch (details in the copied test report): + +- The 18 base admission deadlocks complete with requests and tokens + conserved. +- 50/50 unchanged cases are byte-identical. +- G2 shows no regressions. +- The Step 9 probe shape, MoE `attn_dp=2, moe_ep=2, PP=2`, completes 6/6. +- vLLM DP=2/PP=2 on 4×H800 gives 50 MATCH, 0 MISMATCH and 2 INFORMATIONAL + (dense V5, per D-9). + +Remaining here, in order: + +1. After PR 36 merges into `main`, merge `main` forward into this branch. +2. Rerun the PR 36 matrix group G3b (mixed prefill/decode, `attn_dp > 1`, + `PP > 1`) on the merged tree as the composition check with W3. +3. If G3b passes, resume Step 9 P1(b) and the design checkpoint D9-2. + +Step 9 C1's PP3 row stays on `attn_dp=1` (W9-02). + ## W9-02 `attn_dp=2, moe_ep=2, num_pipeline_stages=3` is rejected at config time Status: open, expected behavior, affects plan wording only. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 49b351c5..cfeb7cca 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | W9-01 fixed on `fix/stage-admission-ordering` (draft PR 36); Step 9 PP>1 packages stay paused until it merges forward and passes G3b. | | 2026-09-22 | Step 9 execution started: P1(a) oracle completed, P1(b) blocked by W9-01, design checkpoint partially settled, G1 ground-truth instrumentation completed and case binding written. | | 2026-09-22 | FP8 native rerun PASS (`exp-0922-202645-561899`, 8 passed, exit 0). Second Step 9 plan review (plan §18.12, R9-01..R9-08) recorded; Step 9 still not started. | | 2026-09-21 | Step 0 started: records landed, environment created, baseline pending. | @@ -223,6 +224,8 @@ User start signal: "开始执行step9", with the quality gates repeated (readabi W9-01 is not caused by this PR: `stage_execution_context.py`, `replica_stage_schduler.py` and `stage_contexts.py` are byte-identical to `main`. It is unobserved because every Simulator-level test with `attn_dp > 1` uses `num_pipeline_stages = 1` and no shipped example sets `attn_dp > 1`. Scope decision requested from the user; recommendation is to fix it as a separate correctness item rather than inside this feature branch. +Scope decision (2026-09-22): "采纳你的推荐，继续" — option 2, a separate correctness item. Fixed on `fix/stage-admission-ordering`, draft PR 36, rule commit `dac4e69`, validated against vLLM DP=2/PP=2 on 4×H800. Branch records are copied to `w9_01_stage_admission_ordering/`. Resume order: PR 36 merges, `main` is merged forward here, G3b reruns as the composition check with W3, then P1(b) and D9-2 (`issues.md` W9-01, Resolution). + W9-02: `attn_dp=2, moe_ep=2, PP=3` is rejected at construction (6 devices against node size 4). Plan C1's PP3 row amended to `attn_dp=1`. ### G1 ground-truth instrumentation (2026-09-22, completed) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md new file mode 100644 index 00000000..13979b22 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md @@ -0,0 +1,57 @@ +# Stage admission ordering under pipeline parallelism — Summary + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-23 | Created at P4: fix, tests, P0–P3 and the vLLM comparison complete under D-9. | + +## Overview + +W9-01: with `attn_dp > 1` and `num_pipeline_stages > 1`, a busy lane's queued +ticket at the head of a stage's ready FIFO refused another lane's runnable +batch. MoE runs drained with requests unfinished (admission deadlock). Dense +runs completed but started the lanes one forward apart. + +The fix (plan D-1, option B) changes one predicate in +`StageExecutionContext.try_acquire`. A full-stage ticket is refused only by an +EP wave queued ahead of it. EP waves keep the strict FIFO-head rule. The +admitted ticket leaves the FIFO by `remove(ticket)`. + +## Deliverables + +| Item | Path / commit | +| --- | --- | +| Rule and P2 tests | `dac4e69`: `frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`, `tests/unit/test_stage_execution_context.py`, `tests/unit/test_shared_forward_group_admission.py`, `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`, `tests/integration/test_stage_admission_pipeline_lanes.py` | +| Case matrix | `tests/e2e/stage_admission_matrix.py` (`a054d87`, `5ade853`, `aeeca93`) | +| vLLM comparison | `tests/comparison/stage_admission_pp/{vllm_burst_driver.py,run_vllm_worker.sh,compare_lanes.py}` (`799ccb4`, `a1b9819`, `aeeca93`) | +| Test report | `test_report_2026-09-23_stage_admission_ordering.md` | +| Calibration case | `calibration/stage_admission_case_001/` (manifest, inputs incl. `groundtruth_overlay.patch`, two vLLM runs, `analysis/`) | +| Evidence | `evidence/` (base negative controls, G2 comparisons, path-T explanation, Step 9 probe, co-execution decomposition script) | +| Branch / PR | `fix/stage-admission-ordering`, draft PR https://github.com/NetX-lab/Frontier/pull/36 | + +## Validation (observed) + +| Criterion | Result | +| --- | --- | +| C1 | 18 base admission deadlocks (G3a 10, G3b 6, G7 2) complete with requests and tokens conserved | +| C2 | 50/50 unchanged cases byte-identical (30 release recipes, every `PP=1` cell, G5) | +| C3 | 6 T cases identical. The other 8 change start times only: same batches, same component durations, no self-overlap, `peak_lanes ≤ attn_dp`. All 4 witnesses have a strictly larger co-execution fraction (D-9). | +| C4 | `tests/unit` and `tests/integration`: no regression, no new failure, skips and collection errors unchanged | +| C5 | One predicate plus docstrings; no flag, field, fallback, wake-up or special case | +| C6 | Step 9 probe shape MoE `attn_dp=2, moe_ep=2, PP=2` completes 6/6 (base drains) | +| C7 | vLLM DP=2/PP=2 on 4×H800, run `sa-pp-20260923b`: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5, D-9). MoE: 26/26 rows MATCH. Dense: V1–V4 MATCH in every round. The base fails its negative controls: MoE deadlock, dense pairing and co-start. | + +Decisions taken during execution: + +| Id | Decision | +| --- | --- | +| R-7 | The vLLM ground truth uses the four-argument `topk_softmax`, applied as a recorded overlay patch. The checkout is unchanged. | +| R-8 / D-9 | Witnesses are judged by co-execution fraction. V5 gates MoE only. | + +## Open and deferred work + +- PR 35 (`fix/issue26-correctness-pr`) merges this branch forward after it lands and reruns G3b as the composition check with W3. Only then does Step 9 resume (C6). +- `PP=3` with `attn_dp=2` stays rejected by the node-size rule on the default backends (W9-02), outside this fix. +- vLLM-BS: the fork's Python `topk_softmax` still passes five arguments. So does its test `tests/model_executor/test_enabled_custom_ops.py::test_topk_softmax_wrapper_forwards_renormalize`. The four-argument form was applied only as this case's overlay patch. +- Dense per-rank duration variance, which vLLM shows and the dummy predictor lacks, is an execution-time-model topic. It is not part of admission. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md new file mode 100644 index 00000000..239ff593 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md @@ -0,0 +1,272 @@ +# Test report — stage admission ordering (P0–P3, P5) + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-23 | D-9 adopted ("采纳你的推荐，继续"): C3 witnesses judged by co-execution fraction, V5 gated on MoE only. Both comparisons rerun (`aeeca93`); all criteria pass. | +| 2026-09-23 | Created. P0–P3 and P5 executed; two plan stop conditions reached (C3 witness metric at `attn_dp=4`, C7 V5 on the dense shape). P4 push held for the user's decision. | + +## 1. Result + +| Criterion | Result | Section | +| --- | --- | --- | +| C1 repaired liveness | PASS: all 10 G3a `admission_deadlock` cases complete with conservation; so do the 6 G3b and 2 G7 MoE deadlocks. | §4.1 | +| C2 unchanged controls | PASS: 50 of 50 U cases byte-identical. | §4.2 | +| C3 timing change | PASS (D-9): 6 T cases identical, 8 differ. All 8 differences are start times only (same batches, same component durations), with no self-overlap and `peak_lanes ≤ attn_dp`. All 4 contention witnesses have a strictly larger co-execution fraction. The first comparison stopped on the original absolute-overlap rule; see §4.3. | §4.3 | +| C4 existing tests | PASS: no base-passed node regresses, no new failure or error, skips and collection errors unchanged. | §4.4 | +| C5 rule shape | PASS by review: one predicate, docstrings state the contract, no flag, field, fallback, wake-up, PP branch, second queue or capacity-1 case. | §3 | +| C6 Step 9 probe | Informational: MoE `attn_dp=2, moe_ep=2, PP=2` completes 6/6 (base drains). `PP=3` stops on the known W9-02 node-size rejection. | §4.5 | +| C7 vLLM comparison | PASS (D-9): 50 rows MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5). MoE matches on all 26 rows, V5 included; dense matches V1–V4 in every round. The negative controls fail on the base as planned. The first comparison stopped on dense V5; see §5.3. | §5 | + +Observed facts are separated from inferences. Inferences are marked +"Inference". + +## 2. Environment and commits + +| Item | Value | +| --- | --- | +| Host | `kun-workspace-vgen2` (CPU) | +| Interpreter | `/data/ycfeng/envs/frontier-py310/bin/python`, Python 3.10.6; distribution digest `ecd50ea8…1902620` for both sets | +| Environment | `PYTHONPATH=`, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`, `TMPDIR=/data/ycfeng/tmp/stage_admission_ordering/pytest-tmp` | +| Base set `base` | run at `a054d87` (harness only; `frontier/` identical to `1f694f7`) | +| After set `after` | run at `dac4e69`, tree clean outside `task_memory/`; 98 cases in 90 s with `--jobs 8` | +| Rule commit | `dac4e69` fix(scheduler): order full-stage admission only behind queued EP waves | +| Harness commits | `a054d87`, `5ade853` (matrix), `799ccb4` (vLLM comparison), `a1b9819` (recorded overlay patch), `aeeca93` (D-9 witness and V5 rules) | +| Scratch root | `/data/ycfeng/tmp/stage_admission_ordering/{base,after,base-rerun,base-pytest,after-pytest,step9_probe}` | + +Commands: + +```bash +python -m tests.e2e.stage_admission_matrix run --set after --jobs 8 +python -m tests.e2e.stage_admission_matrix compare --before base --after after \ + --output /data/ycfeng/tmp/stage_admission_ordering/compare_base_after_d9.json # first run: compare_base_after.json +python -m pytest tests/ -q -p no:cacheprovider --continue-on-collection-errors \ + --junitxml=/.xml # suite in {unit, integration}, base and after +python task_memory/.../evidence/explain_t_path.py +python -m tests.comparison.stage_admission_pp.compare_lanes \ + --vllm-run calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b \ + --before base --after after --output calibration/stage_admission_case_001/analysis +``` + +## 3. P1 rule + +`StageExecutionContext.try_acquire` (`frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`): +an EP wave must be the FIFO head; a full-stage ticket is refused only by an EP +wave queued ahead of it; the admitted ticket leaves the FIFO by +`remove(ticket)`. `_validate_ticket` already rejects a ticket that is neither +queued nor active, so the scan always finds the ticket or an earlier wave. +One file, +21/−7 lines. P1 acceptance: `tests/unit/test_stage_execution_context.py` +and `tests/unit/test_shared_forward_group_admission.py` gave 34 passed with no +assertion change. + +## 4. P2 and P3 + +### 4.0 P2 tests and base negative controls + +The new tests were copied into a `git archive 799ccb4` export (rule as on +`1f694f7`) and run there; the log is `evidence/base_negative_controls.log`. + +| Test | Expected on base | Observed on base | After P1 | +| --- | --- | --- | --- | +| (a) `test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave` | fails at first assertion | fails at line 111, `try_acquire(full1)` is False | pass | +| (a) `test_queued_ep_wave_orders_full_stage_work_on_both_sides` | pass | pass | pass | +| (a) `test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket` | fails | fails at line 141 | pass | +| (a′) `test_decode_ffn_dense_groups_keep_counter_order_around_a_queued_ep_wave` | pass | pass | pass | +| (b) `test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0,1]` | fails at the other lane's first admission | both fail at line 71, `pop_batch_if_not_busy()` is None | pass | +| (c) `test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4, G3a-moe-dp4-pp2-n8]` | `admission_deadlock` | both `admission_deadlock` | pass: (4, 64, 4) and (8, 128, 8) | +| (c) `test_dense_lanes_start_in_the_same_first_forward` | fails only the same-start assertion | fails `0.05 == 0.0`; lane 1 runs `[0, 0.05]`, lane 0 starts at `0.05` | pass: both lanes start at 0.0 | + +### 4.1 C1 — path L (18 cases, all PASS) + +Observed `(requests, prefill tokens, decode tokens)` after P1 equals the +generated workload in every case. + +| Cases | Observed | +| --- | --- | +| G3a `dp2-pp{2,3}-n{4,8,12}`, `dp4-pp{2,3}-n{8,12}` (10) | n4: (4, 64, 4); n8: (8, 128, 8); n12: (12, 192, 12) | +| G3b `dp2-pp{2,3}-n{4,8}`, `dp4-pp{2,3}-n8` (6) | n4: (4, 64, 12); n8: (8, 128, 24); no mixed-phase failure | +| G7 MoE `dp2-pp2-n{8,16}` (2) | (8, 2048, 8); (16, 4096, 16) | + +R0 (informational): the three base deadlocks `moe-dp2-pp2-n4`, `moe-dp2-pp2-n6`, +`moe-dp4-pp2-n8` now succeed; `moe-dp2-pp3-n6` remains +`configuration_rejection` (node-size rule, D-6); the other 12 stay `success`. + +### 4.2 C2 — path U (50 cases, all PASS) + +Byte-identical `sha256sums.txt`: G1 30 recipes (10 PD-AF included), `PP=1` +cells of G3a (6), G3b (4) and G4 (4, `attn_dp=4, PP=1` included), G5 6. + +### 4.3 C3 — path T (14 cases) + +Identical hashes (6): G3a/G3b/G4 `dp4-pp{2,3}-n4` (one batch per lane). + +Differing (8). `evidence/explain_t_path.py` checks, per stage and lane, that +the ordered batch list, the forward duration and the `execution_time` +component ledger are equal before and after; output +`evidence/p3_t_path_explanation.json`. All 8: `same_batches_and_component_durations = true`, +no self-overlap, `peak_lanes ≤ attn_dp`. Differing files are the ledger, +`request_metrics.csv` and `system_metrics.json` only. Stage 0: + +| Case | W | multi-lane time before → after | co-execution fraction before → after | peak lanes | first stage-0 starts before → after | Verdict | +| --- | --- | --- | --- | --- | --- | --- | +| G4-dense-dp2-pp2-n4 | | 0.25 → 0.30 | 0.714 → 1.0 | 2 → 2 | {1: 0, 0: 0.05} → all 0 | EXPLAIN | +| G4-dense-dp2-pp2-n8 | W | 0.45 → 0.50 | 0.818 → 1.0 | 2 → 2 | {1: 0, 0: 0.05} → all 0 | EXPLAIN | +| G4-dense-dp2-pp3-n4 | | 0.108 → 0.216 | 0.333 → 1.0 | 2 → 2 | {1: 0, 0: 0.036} → all 0 | EXPLAIN | +| G4-dense-dp2-pp3-n8 | W | 0.216 → 0.396 | 0.375 → 1.0 | 2 → 2 | {1: 0, 0: 0.072} → all 0 | EXPLAIN | +| G4-dense-dp4-pp2-n8 | W | 0.55 → 0.30 | 0.846 → 1.0 | 2 → 4 | {1: 0, 2: 0.05, 3: 0.10, 0: 0.15} → all 0 | EXPLAIN (first run: STOP) | +| G4-dense-dp4-pp3-n8 | W | 0.396 → 0.216 | 0.846 → 1.0 | 2 → 4 | {1: 0, 2: 0.036, 3: 0.072, 0: 0.108} → all 0 | EXPLAIN (first run: STOP) | +| G7-dense-dp2-pp2-n8 | | 0.36 → 0.48 | 0.60 → 1.0 | 2 → 2 | {1: 0, 0: 0.12} → all 0 | EXPLAIN | +| G7-dense-dp2-pp2-n16 | | 0.84 → 0.96 | 0.778 → 1.0 | 2 → 2 | {1: 0, 0: 0.12} → all 0 | EXPLAIN | + +Why the two witnesses fail the stated rule (observed from the ledgers): at +`attn_dp=4` the base admits the lanes two at a time; after P1 all four lanes +start every forward together. The stage busy period shrinks from 0.65 to 0.30 +(PP=2) and from 0.468 to 0.216 (PP=3), so the time with two or more lanes busy +shrinks with it, although it is now the whole busy period. Request E2E for +`dp4-pp2-n8` drops from 500–700 ms to 300–350 ms with identical batches. +Inference: absolute `multi_lane_busy_time` measures overlap only while the +busy period stays the same length; it cannot express "more overlap" when the +fix compresses the timeline, which happens whenever the base serialized more +than two lanes. The first comparison stopped here with nothing adjusted. +Under D-9 the witness condition is the co-execution fraction, which strictly +increases in all four witnesses (0.818, 0.375, 0.846, 0.846 → 1.0); the rerun +(`compare_base_after_d9.json`) gives U 50 PASS, L 18 PASS, T 6 PASS and +8 EXPLAIN, and no STOP. + +### 4.4 C4 — G2 test identities + +| Suite | Base | After | Regressions | New failures | Skip / collection changes | Only after | +| --- | --- | --- | --- | --- | --- | --- | +| `tests/unit` | 84 failed, 3644 passed, 49 skipped, 10 errors | 84 failed, 3650 passed, 49 skipped, 10 errors | 0 | 0 | none; `ERROR` lines identical | the 6 new P2 unit tests, all passed | +| `tests/integration` | 11 passed, 21 skipped, 5 errors | 14 passed, 21 skipped, 5 errors | 0 | 0 | none; `ERROR` lines identical | the 3 new P2(c) tests, all passed | + +Evidence: `evidence/g2_unit_compare.json`, `evidence/g2_integration_compare.json`; +junit XML under the scratch root. + +### 4.5 C6 — Step 9 boundary probe + +The original `probe_main.py` wraps `BaseClusterScheduler.on_replica_batch_end`, +a seam that exists only on the PR 35 branch, so it raises `AttributeError` +on this branch. `evidence/step9_probe/probe_completion.py` reuses its +`build_config` unchanged (a100, 6 requests, 16/3 tokens, Poisson) and reports +completion, one process per shape. + +| Shape | Base | After | +| --- | --- | --- | +| MoE `attn_dp=2, moe_ep=2, PP=1` | — | 6/6 | +| MoE `attn_dp=2, moe_ep=2, PP=2` | drain, "Sequential simulation ended with non-empty scheduler state" | 6/6 | +| MoE `attn_dp=2, moe_ep=2, PP=3` | — | `ValueError`: collective-sim node-size rule (W9-02, unchanged) | +| dense `attn_dp=1, PP=2` | — | 6/6 | + +Composition with PR 35 W3 stays a parent-task check after merge-forward. + +## 5. C7 — vLLM comparison (P5) + +### 5.1 Ground-truth runs + +| Run | RJob | Result | +| --- | --- | --- | +| `sa-pp-20260923a` | `exp-0923-022226-151935`, codesign, 4×H800, creator `i-fengyicheng`, NFS `100.96.128.195:/data/ycfeng/Frontier` | dense complete; MoE failed in `profile_run`: `_moe_C::topk_softmax() expected at most 4 argument(s) but received 5`. Job `Failed`. | +| `sa-pp-20260923b` | `exp-0923-024146-345158`, same shape and mount | MoE and dense complete; job `Succeeded`; worker status 0 | + +Cause of the run-a failure (observed): fork commit `1109c4f16` changed +`vllm/_custom_ops.py::topk_softmax` and the `vllm_topk_softmax` call in +`fused_moe.py` to pass a fifth `renormalize` argument, but the fork's own +`csrc/moe/torch_bindings.cpp` (unchanged from `upstream-v0.10.2`) and the +v0.10.2 image both declare the four-argument op. The user decided on +2026-09-23: "topk_softmax 统一修复为4 个参数的版本". Run b applies +`inputs/groundtruth_overlay.patch` (SHA-256 `8d476789…3a9c81`) to the accepted +overlay: it restores the upstream four-argument wrapper and call. The worker +records `_custom_ops.py` as byte-identical to the image's after the patch. +Numerics are unchanged: `vllm_topk_softmax` renormalizes in Python after the +call in both versions. The checkout `494b9f327` is not modified. + +vLLM run b: 7 rounds per model (1 warmup + 2 bursts × 3), 152 `pp_boundary` records per +model, no preemption, placement records for every request with none +misplaced; `num_gpu_blocks` 600666 (MoE) and 304854 (dense). + +### 5.2 Workflow-gap table (run b) + +`analysis/workflow_gap_table.csv`, `analysis/lane_metrics.json`, +`analysis/workflow_gap_status.json`. + +| Check | MoE n8 | MoE n16 | Dense n8 | Dense n16 | +| --- | --- | --- | --- | --- | +| V1 completion | 3/3 MATCH; base `admission_deadlock` | 3/3 MATCH; base `admission_deadlock` | 3/3 MATCH | 3/3 MATCH | +| V2 lane sequences | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | +| V3 stage-0 pairing | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH; base pairs 0↔3, 2↔5, …, 6↔none | 3/3 MATCH; base shifted by one forward | +| V4 co-start (vLLM / after / base) | 0.009–0.063 / 0.0 / — | 0.005–0.024 / 0.0 / — | 0.008–0.248 / 0.0 / 1.0 | 0.046–0.171 / 0.0 / 1.0 | +| V5 co-execution (vLLM mean / after / base) | 0.976 / 1.0 / — MATCH | 0.948 / 1.0 / — MATCH | 0.706 / 1.0 / 0.600 INFORMATIONAL (first run: MISMATCH) | 0.865 / 1.0 / 0.778 INFORMATIONAL (first run: MISMATCH) | + +Run a (dense only, same scripts): V1–V4 all MATCH; V5 vLLM mean 0.714 (n8) +and 0.685 (n16): MISMATCH under the first rule, INFORMATIONAL under D-9. + +### 5.3 V5 on the dense shape + +`evidence/decompose_co_execution.py` splits the stage-0 non-overlap of each +M3 pair into `|Δstart| + |Δend|` +(`analysis/co_execution_decomposition_sa-pp-20260923{a,b}.json`). + +| Shape (run b) | vLLM M5 per round | Σ start offsets (ms) | Σ end offsets (ms) | stage-0 duration median (ms), CV | +| --- | --- | --- | --- | --- | +| MoE n8 | 0.977, 0.974, 0.977 | 0.32–0.54 | 0.19–0.20 | 5.3–8.5, 0.07–0.09 | +| MoE n16 | 0.978, 0.937, 0.928 | 0.70–3.83 | 0.26–0.69 | 5.3–7.5, 0.05–0.07 | +| Dense n8 | 0.657, 0.851, 0.609 | 1.49–2.63 | 0.26–4.40 | 3.0–3.8, 0.13–0.29 | +| Dense n16 | 0.926, 0.833, 0.837 | 1.02–3.60 | 0.60–2.81 | 2.6–2.7, 0.10–0.19 | + +Observed: + +- vLLM's dense M5 varies between rounds more than the V5 bound: 0.537–0.926 + across the 12 dense rounds of runs a and b. The n16 means of the two runs + differ by 0.18. +- In every vLLM round the pairing (V3) and the one-to-one lane sequences (V2) + match the after revision, and the first forwards co-start (V4). +- The non-overlap consists of per-pair start offsets of up to about 1.5 ms + (`forward_start_ts` is taken before the per-forward DP metadata exchange) + and end offsets from per-rank duration variation. +- MoE ends align within 0.2–0.7 ms in total. + +Inference: in MoE the EP collectives inside each forward hold the two ranks +together, so vLLM's co-execution is close to Frontier's 1.0. The dense ranks +meet once per forward and then run host-bound forwards of about 3 ms whose +durations vary per rank. The dummy predictor gives both lanes the same +duration, so Frontier's co-execution is exactly 1.0 whenever the lanes +co-start. The residual is a duration-variance property of the ground truth +that the dummy predictor does not model. It is not an admission difference: +admission is what V1–V4 measure, and they match. The dense base (0.600, +0.778) is numerically closer to vLLM only because base serialization removes +overlap; its pairing (V3) and co-start (V4) are wrong in every round. + +`compare_lanes.py` labels every `MISMATCH` with the admission owner +`stage_execution_context.py`; on the evidence above, these two rows belong to +the execution-time model instead. The first comparison stopped here with +nothing adjusted. Under D-9 dense V5 is reported, not gated; the rerun gives +`workflow_gap_status.json` status PASS with 0 mismatches. The P5a synthetic +check (`analysis/synthetic_check.py`) still flags its planted dummy-shifted +dense round through V3 and V4. + +## 6. Decisions + +Both stops were resolved by D-9 (`plan.md`), adopted by the user on +2026-09-23 ("采纳你的推荐，继续"): + +1. C3: a contention witness passes on a strictly larger co-execution fraction + `multi_lane_busy_time / busy_time`; the self-overlap and `peak_lanes` checks + are unchanged. +2. C7: V5 gates the MoE shape only; the dense value is reported with the + decomposition of §5.3. C7 rests on V1–V4 for both models, V5 for MoE, and the + base negative controls. + +## 7. Verification limits + +- The Frontier side runs the dummy predictor; no latency or duration + parity is claimed (D-8). Dense co-execution against vLLM is therefore not a + gate (D-9). +- vLLM instrumented mode synchronizes after each forward; stage-1 intervals + use a wall/monotonic offset and are informational. +- The vLLM ground truth runs with one recorded overlay patch (§5.1); the fork + checkout still carries the five-argument call and its fork test + `tests/model_executor/test_enabled_custom_ops.py::test_topk_softmax_wrapper_forwards_renormalize`. +- C6 was measured with a completion-only probe because the boundary seam is + on PR 35; the PR 35 composition check is pending in the parent task. From f4f12a0beac5614e9bc2894fe8a6811577086cdd Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 10:47:08 +0800 Subject: [PATCH 43/80] docs(step9): drop the moving PR 36 head from the W9-01 note --- task_memory/task_2026-09-21_issue26_correctness_pr/issues.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md index 593dc15d..15e8c219 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md @@ -124,10 +124,10 @@ Option 2 was taken. The fix lives on its own branch and PR: | Item | Value | | --- | --- | -| Branch / PR | `fix/stage-admission-ordering`, draft https://github.com/NetX-lab/Frontier/pull/36, head `4bcd616` | +| Branch / PR | `fix/stage-admission-ordering`, draft https://github.com/NetX-lab/Frontier/pull/36 | | Rule commit | `dac4e69`: `StageExecutionContext.try_acquire` refuses a full-stage ticket only when an EP wave is queued ahead of it. EP waves keep the strict FIFO-head rule. | | Acceptance rules | `aeeca93` (plan D-9) | -| Records | `fc34341`, `4bcd616`; copies in `w9_01_stage_admission_ordering/` (`summary.md`, `test_report_2026-09-23_stage_admission_ordering.md`) | +| Records | `fc34341`, `4bcd616`, `ecff89a`; copies in `w9_01_stage_admission_ordering/` (`summary.md`, `test_report_2026-09-23_stage_admission_ordering.md`) | Observed on that branch (details in the copied test report): From 887d34b121ce7260efed75a5b012056bb275311e Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 12:01:42 +0800 Subject: [PATCH 44/80] docs(step9): record the PR 36 round-2 remediation for W9-01 PR 36 applied its round-2 review (rule refactor 1661bf1, PDD/online/PD-AF matrix groups a8e8d8a, comparison tools e35242f, records 7a7c22e). The composition check after the merge-forward now also reruns the online groups G9 and G10, whose Poisson cells reach every lane only with this branch's W2 lane rotation. The summary and test report copies are refreshed. --- .../issues.md | 18 +- .../progress.md | 3 +- .../w9_01_stage_admission_ordering/summary.md | 32 +++- ...ort_2026-09-23_stage_admission_ordering.md | 160 +++++++++++++++--- 4 files changed, 178 insertions(+), 35 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md index 15e8c219..6f817879 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | W9-01: PR 36 round-2 review remediation recorded; the composition check now also reruns PR 36 groups G9 and G10. | | 2026-09-23 | W9-01: fixed on `fix/stage-admission-ordering` (draft PR 36) under option 2; resolution recorded, summary and test report copied to `w9_01_stage_admission_ordering/`. | | 2026-09-22 | Created; recorded W9-01 (stage-admission deadlock at PP>1 with attn_dp>1) and W9-02 (PP=3 x attn_dp=2 topology rejection) found during Step 9 P1(b). | @@ -127,7 +128,8 @@ Option 2 was taken. The fix lives on its own branch and PR: | Branch / PR | `fix/stage-admission-ordering`, draft https://github.com/NetX-lab/Frontier/pull/36 | | Rule commit | `dac4e69`: `StageExecutionContext.try_acquire` refuses a full-stage ticket only when an EP wave is queued ahead of it. EP waves keep the strict FIFO-head rule. | | Acceptance rules | `aeeca93` (plan D-9) | -| Records | `fc34341`, `4bcd616`, `ecff89a`; copies in `w9_01_stage_admission_ordering/` (`summary.md`, `test_report_2026-09-23_stage_admission_ordering.md`) | +| Round-2 review fixes | `1661bf1` (rule refactor: an active ticket is refused), `a8e8d8a` (PDD, online and PD-AF matrix groups G8–G11), `e35242f` (comparison tools) | +| Records | `fc34341`, `4bcd616`, `ecff89a`, `1218ba6`, `7a7c22e`; copies in `w9_01_stage_admission_ordering/` (`summary.md`, `test_report_2026-09-23_stage_admission_ordering.md`) | Observed on that branch (details in the copied test report): @@ -137,14 +139,20 @@ Observed on that branch (details in the copied test report): - G2 shows no regressions. - The Step 9 probe shape, MoE `attn_dp=2, moe_ep=2, PP=2`, completes 6/6. - vLLM DP=2/PP=2 on 4×H800 gives 50 MATCH, 0 MISMATCH and 2 INFORMATIONAL - (dense V5, per D-9). + (dense V5, per D-9); the 4 base negative-control rows hold. +- Round 2 added PDD offline and online, co-location online and PD-AF + `PREFILL_PP=2` cells: 12 more base deadlocks complete, 0 STOP. On `main` + the online Poisson cells of MONOLITHIC and PREFILL run on lane 0 only, + because `main` lacks this branch's W2 lane rotation; the online multi-lane + coverage there comes from burst cells. Remaining here, in order: 1. After PR 36 merges into `main`, merge `main` forward into this branch. -2. Rerun the PR 36 matrix group G3b (mixed prefill/decode, `attn_dp > 1`, - `PP > 1`) on the merged tree as the composition check with W3. -3. If G3b passes, resume Step 9 P1(b) and the design checkpoint D9-2. +2. Rerun the PR 36 matrix groups on the merged tree as the composition check: + G3b (mixed prefill/decode, `attn_dp > 1`, `PP > 1`) with W3, and G9 and + G10 (online), whose Poisson cells reach every lane only with W2. +3. If they pass, resume Step 9 P1(b) and the design checkpoint D9-2. Step 9 C1's PP3 row stays on `attn_dp=1` (W9-02). diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index cfeb7cca..7a4f202e 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | PR 36 round-2 review remediation recorded; the W9-01 composition check extended to PR 36 groups G9 and G10. | | 2026-09-23 | W9-01 fixed on `fix/stage-admission-ordering` (draft PR 36); Step 9 PP>1 packages stay paused until it merges forward and passes G3b. | | 2026-09-22 | Step 9 execution started: P1(a) oracle completed, P1(b) blocked by W9-01, design checkpoint partially settled, G1 ground-truth instrumentation completed and case binding written. | | 2026-09-22 | FP8 native rerun PASS (`exp-0922-202645-561899`, 8 passed, exit 0). Second Step 9 plan review (plan §18.12, R9-01..R9-08) recorded; Step 9 still not started. | @@ -224,7 +225,7 @@ User start signal: "开始执行step9", with the quality gates repeated (readabi W9-01 is not caused by this PR: `stage_execution_context.py`, `replica_stage_schduler.py` and `stage_contexts.py` are byte-identical to `main`. It is unobserved because every Simulator-level test with `attn_dp > 1` uses `num_pipeline_stages = 1` and no shipped example sets `attn_dp > 1`. Scope decision requested from the user; recommendation is to fix it as a separate correctness item rather than inside this feature branch. -Scope decision (2026-09-22): "采纳你的推荐，继续" — option 2, a separate correctness item. Fixed on `fix/stage-admission-ordering`, draft PR 36, rule commit `dac4e69`, validated against vLLM DP=2/PP=2 on 4×H800. Branch records are copied to `w9_01_stage_admission_ordering/`. Resume order: PR 36 merges, `main` is merged forward here, G3b reruns as the composition check with W3, then P1(b) and D9-2 (`issues.md` W9-01, Resolution). +Scope decision (2026-09-22): "采纳你的推荐，继续" — option 2, a separate correctness item. Fixed on `fix/stage-admission-ordering`, draft PR 36, rule commit `dac4e69`, validated against vLLM DP=2/PP=2 on 4×H800. Branch records are copied to `w9_01_stage_admission_ordering/`. Resume order: PR 36 merges, `main` is merged forward here, G3b (with W3) and the online groups G9 and G10 (with W2) rerun as the composition check, then P1(b) and D9-2 (`issues.md` W9-01, Resolution). PR 36 round-2 review fixes: `1661bf1`, `a8e8d8a`, `e35242f`, records `7a7c22e`. W9-02: `attn_dp=2, moe_ep=2, PP=3` is rejected at construction (6 devices against node size 4). Plan C1's PP3 row amended to `attn_dp=1`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md index 13979b22..2a56ada2 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-10: round-2 review remediation (14 findings applied, R2-06 pending as P6); PDD, online and PD-AF cells added. | | 2026-09-23 | Created at P4: fix, tests, P0–P3 and the vLLM comparison complete under D-9. | ## Overview @@ -15,19 +16,20 @@ runs completed but started the lanes one forward apart. The fix (plan D-1, option B) changes one predicate in `StageExecutionContext.try_acquire`. A full-stage ticket is refused only by an -EP wave queued ahead of it. EP waves keep the strict FIFO-head rule. The -admitted ticket leaves the FIFO by `remove(ticket)`. +EP wave queued ahead of it. EP waves keep the strict FIFO-head rule. A +ticket that is already active is refused (round 2, R2-01). ## Deliverables | Item | Path / commit | | --- | --- | +| Rule refactor (round 2) | `1661bf1`: one branch per scope; an active ticket is refused; docstring on FIFO meaning | | Rule and P2 tests | `dac4e69`: `frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`, `tests/unit/test_stage_execution_context.py`, `tests/unit/test_shared_forward_group_admission.py`, `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`, `tests/integration/test_stage_admission_pipeline_lanes.py` | -| Case matrix | `tests/e2e/stage_admission_matrix.py` (`a054d87`, `5ade853`, `aeeca93`) | -| vLLM comparison | `tests/comparison/stage_admission_pp/{vllm_burst_driver.py,run_vllm_worker.sh,compare_lanes.py}` (`799ccb4`, `a1b9819`, `aeeca93`) | +| Case matrix | `tests/e2e/stage_admission_matrix.py` (`a054d87`, `5ade853`, `aeeca93`; round 2 `a8e8d8a`: PDD, online and PD-AF groups G8–G11, case timeout, set lock), 148 cases | +| vLLM comparison | `tests/comparison/stage_admission_pp/{vllm_burst_driver.py,run_vllm_worker.sh,compare_lanes.py}` (`799ccb4`, `a1b9819`, `aeeca93`; round 2 `e35242f`: negative-control rows, placement, overlay and patch fixes) and `tests/unit/test_stage_admission_pp_tools.py` | | Test report | `test_report_2026-09-23_stage_admission_ordering.md` | | Calibration case | `calibration/stage_admission_case_001/` (manifest, inputs incl. `groundtruth_overlay.patch`, two vLLM runs, `analysis/`) | -| Evidence | `evidence/` (base negative controls, G2 comparisons, path-T explanation, Step 9 probe, co-execution decomposition script) | +| Evidence | `evidence/` (base negative controls, G2 comparisons, path-T explanations for rounds 1 and 2, tool tests on the old tools, Step 9 probe, co-execution decomposition script) | | Branch / PR | `fix/stage-admission-ordering`, draft PR https://github.com/NetX-lab/Frontier/pull/36 | ## Validation (observed) @@ -42,15 +44,33 @@ admitted ticket leaves the FIFO by `remove(ticket)`. | C6 | Step 9 probe shape MoE `attn_dp=2, moe_ep=2, PP=2` completes 6/6 (base drains) | | C7 | vLLM DP=2/PP=2 on 4×H800, run `sa-pp-20260923b`: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5, D-9). MoE: 26/26 rows MATCH. Dense: V1–V4 MATCH in every round. The base fails its negative controls: MoE deadlock, dense pairing and co-start. | +Round 2 (R-10, test report §8): + +| Check | Result | +| --- | --- | +| Rule refactor | `after` vs `after-r2`: 98/98 cases byte-identical | +| New groups G8–G11 (50) | PDD offline and online, co-location online, PD-AF `PREFILL_PP=2`: U 18 PASS, L 12 PASS (base deadlocks, conserved), T 12 PASS and 4 EXPLAIN (start times only); 0 STOP | +| G2 | no regression, no new failure, skips unchanged | +| C7 rerun (run b vs `after-r2`) | 56 rows: 50 MATCH, 2 INFORMATIONAL, 4 negative controls HOLDS, 0 MISMATCH | +| Harness and tools | timeout, lock, placement, overlay and patch checks verified; tool tests fail 7/9 on the old tools | + Decisions taken during execution: | Id | Decision | | --- | --- | | R-7 | The vLLM ground truth uses the four-argument `topk_softmax`, applied as a recorded overlay patch. The checkout is unchanged. | -| R-8 / D-9 | Witnesses are judged by co-execution fraction. V5 gates MoE only. | +| R-8 / D-9 | Witnesses are judged by co-execution fraction. V5 gates MoE only. Dense rationale restated at R-10: pre-exchange wait and duration variance, neither admission. | +| R-10 | Round-2 fixes as recommended; PDD and online cells added, plus PD-AF `PP > 1` controls. | ## Open and deferred work +- R2-06 / P6: before PR 36 merges, drop the `.gitignore` exception and + untrack this task directory (the parent task keeps copies). It deletes + tracked records, so it waits for the owner's go-ahead. +- On this branch, online Poisson arrivals reach only lane 0 of MONOLITHIC and + PREFILL contexts (PR 35 W2). PR 35's composition check after the + merge-forward reruns G9 and G10 with lane rotation, as well as G3b. + - PR 35 (`fix/issue26-correctness-pr`) merges this branch forward after it lands and reruns G3b as the composition check with W3. Only then does Step 9 resume (C6). - `PP=3` with `attn_dp=2` stays rejected by the node-size rule on the default backends (W9-02), outside this fix. - vLLM-BS: the fork's Python `topk_softmax` still passes five arguments. So does its test `tests/model_executor/test_enabled_custom_ops.py::test_topk_softmax_wrapper_forwards_renormalize`. The four-argument form was applied only as this case's overlay patch. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md index 239ff593..267c850f 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-10 round-2 remediation: §8 added (rule refactor, PDD/online/PD-AF cells, tool fixes); §5.2 and §5.3 restated for the N rows and the barrier-aligned M5. | | 2026-09-23 | D-9 adopted ("采纳你的推荐，继续"): C3 witnesses judged by co-execution fraction, V5 gated on MoE only. Both comparisons rerun (`aeeca93`); all criteria pass. | | 2026-09-23 | Created. P0–P3 and P5 executed; two plan stop conditions reached (C3 witness metric at `attn_dp=4`, C7 V5 on the dense shape). P4 push held for the user's decision. | @@ -19,6 +20,11 @@ | C6 Step 9 probe | Informational: MoE `attn_dp=2, moe_ep=2, PP=2` completes 6/6 (base drains). `PP=3` stops on the known W9-02 node-size rejection. | §4.5 | | C7 vLLM comparison | PASS (D-9): 50 rows MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5). MoE matches on all 26 rows, V5 included; dense matches V1–V4 in every round. The negative controls fail on the base as planned. The first comparison stopped on dense V5; see §5.3. | §5 | +Round 2 (R-10, §8): the rule refactor keeps all 98 cases byte-identical; +50 new PDD, online and PD-AF cells pass (12 more base deadlocks repaired, +0 STOP); G2 shows no regression; the vLLM comparison rerun passes with the +negative controls as separate rows that hold. + Observed facts are separated from inferences. Inferences are marked "Inference". @@ -193,7 +199,8 @@ misplaced; `num_gpu_blocks` 600666 (MoE) and 304854 (dense). | Check | MoE n8 | MoE n16 | Dense n8 | Dense n16 | | --- | --- | --- | --- | --- | -| V1 completion | 3/3 MATCH; base `admission_deadlock` | 3/3 MATCH; base `admission_deadlock` | 3/3 MATCH | 3/3 MATCH | +| V1 completion | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | +| N1 / N4 base control (R-10) | N1 HOLDS: base `admission_deadlock` | N1 HOLDS: base `admission_deadlock` | N4 HOLDS: base co-start 1.0 | N4 HOLDS: base co-start 1.0 | | V2 lane sequences | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | | V3 stage-0 pairing | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH; base pairs 0↔3, 2↔5, …, 6↔none | 3/3 MATCH; base shifted by one forward | | V4 co-start (vLLM / after / base) | 0.009–0.063 / 0.0 / — | 0.005–0.024 / 0.0 / — | 0.008–0.248 / 0.0 / 1.0 | 0.046–0.171 / 0.0 / 1.0 | @@ -202,18 +209,36 @@ misplaced; `num_gpu_blocks` 600666 (MoE) and 304854 (dense). Run a (dense only, same scripts): V1–V4 all MATCH; V5 vLLM mean 0.714 (n8) and 0.685 (n16): MISMATCH under the first rule, INFORMATIONAL under D-9. +Until R-10 the base controls were folded into V1 and dense V4, so a base +without the defect would have turned those vLLM rows into MISMATCH (R2-05). +The rerun at R-10 (`--after after-r2`) writes them as rows N1 and N4: 56 rows, +50 MATCH, 2 INFORMATIONAL, 4 HOLDS, 0 MISMATCH; status PASS, +`negative_control_holds = true`, placement ok with 0 unseen requests. + ### 5.3 V5 on the dense shape `evidence/decompose_co_execution.py` splits the stage-0 non-overlap of each -M3 pair into `|Δstart| + |Δend|` -(`analysis/co_execution_decomposition_sa-pp-20260923{a,b}.json`). - -| Shape (run b) | vLLM M5 per round | Σ start offsets (ms) | Σ end offsets (ms) | stage-0 duration median (ms), CV | -| --- | --- | --- | --- | --- | -| MoE n8 | 0.977, 0.974, 0.977 | 0.32–0.54 | 0.19–0.20 | 5.3–8.5, 0.07–0.09 | -| MoE n16 | 0.978, 0.937, 0.928 | 0.70–3.83 | 0.26–0.69 | 5.3–7.5, 0.05–0.07 | -| Dense n8 | 0.657, 0.851, 0.609 | 1.49–2.63 | 0.26–4.40 | 3.0–3.8, 0.13–0.29 | -| Dense n16 | 0.926, 0.833, 0.837 | 1.02–3.60 | 0.60–2.81 | 2.6–2.7, 0.10–0.19 | +pair of overlapping forwards into `|Δstart| + |Δend|` +(`analysis/co_execution_decomposition_sa-pp-20260923{a,b}.json`). The identity +holds only for overlapping pairs; at R-10 the script counts disjoint pairs and +checks its pairing against M3. In every round of runs a and b there is no +disjoint pair and no unpaired forward, and the pairing equals M3. The last +column sets both starts of each pair to the later one (R-10, R2-02): vLLM +0.10.2 without CUDA graphs runs the per-forward DP metadata all-reduce inside +`set_forward_context`, after `forward_start_ts`, so neither rank computes +before the later one arrives. The traces carry no timestamp after that +exchange, so this column is derived, not measured. + +| Shape (run b) | vLLM M5 per round | Σ start offsets (ms) | Σ end offsets (ms) | stage-0 duration median (ms), CV | M5, starts aligned to the later one (derived) | +| --- | --- | --- | --- | --- | --- | +| MoE n8 | 0.977, 0.974, 0.977 | 0.32–0.54 | 0.19–0.20 | 5.3–8.5, 0.07–0.09 | 0.991, 0.991, 0.994 | +| MoE n16 | 0.978, 0.937, 0.928 | 0.70–3.83 | 0.26–0.69 | 5.3–7.5, 0.05–0.07 | 0.994, 0.988, 0.989 | +| Dense n8 | 0.657, 0.851, 0.609 | 1.49–2.63 | 0.26–4.40 | 3.0–3.8, 0.13–0.29 | 0.722, 0.977, 0.713 | +| Dense n16 | 0.926, 0.833, 0.837 | 1.02–3.60 | 0.60–2.81 | 2.6–2.7, 0.10–0.19 | 0.970, 0.972, 0.883 | + +Run a, dense, same columns: observed 0.642, 0.739, 0.760 (n8) and 0.537, +0.752, 0.767 (n16); starts aligned 0.739, 0.843, 0.782 and 0.656, 0.945, +0.950. Observed: @@ -222,19 +247,24 @@ Observed: differ by 0.18. - In every vLLM round the pairing (V3) and the one-to-one lane sequences (V2) match the after revision, and the first forwards co-start (V4). -- The non-overlap consists of per-pair start offsets of up to about 1.5 ms - (`forward_start_ts` is taken before the per-forward DP metadata exchange) - and end offsets from per-rank duration variation. +- The non-overlap consists of per-pair start offsets and end offsets. Start + offsets exceed end offsets in 3 of the 6 dense rounds of run b. +- With both starts aligned to the later one, M5 rises in every round (dense + 0.66–0.98, MoE 0.988–0.994). What remains in dense is the end offsets. - MoE ends align within 0.2–0.7 ms in total. -Inference: in MoE the EP collectives inside each forward hold the two ranks -together, so vLLM's co-execution is close to Frontier's 1.0. The dense ranks -meet once per forward and then run host-bound forwards of about 3 ms whose -durations vary per rank. The dummy predictor gives both lanes the same -duration, so Frontier's co-execution is exactly 1.0 whenever the lanes -co-start. The residual is a duration-variance property of the ground truth -that the dummy predictor does not model. It is not an admission difference: -admission is what V1–V4 measure, and they match. The dense base (0.600, +Inference (restated at R-10, R2-02): in MoE the EP collectives inside each +forward hold the two ranks together, so vLLM's co-execution is close to +Frontier's 1.0. The dense ranks meet once per forward, in the DP all-reduce. +The dense non-overlap has two sources. First, the rank that reaches the +all-reduce first records its wait as busy time, because `forward_start_ts` +precedes the exchange. Second, the host-bound forwards of about 3 ms vary in +duration per rank. Neither is an admission difference: both ranks enter the +same forward, which is what V1–V4 measure, and they match. The dummy +predictor models neither the wait nor the variation, so Frontier's +co-execution is exactly 1.0 whenever the lanes co-start. The first version of +this paragraph named only the duration variation; the start part was there +too. The dense base (0.600, 0.778) is numerically closer to vLLM only because base serialization removes overlap; its pairing (V3) and co-start (V4) are wrong in every round. @@ -243,8 +273,8 @@ overlap; its pairing (V3) and co-start (V4) are wrong in every round. the execution-time model instead. The first comparison stopped here with nothing adjusted. Under D-9 dense V5 is reported, not gated; the rerun gives `workflow_gap_status.json` status PASS with 0 mismatches. The P5a synthetic -check (`analysis/synthetic_check.py`) still flags its planted dummy-shifted -dense round through V3 and V4. +check, now `tests/unit/test_stage_admission_pp_tools.py` (R-10), still flags +a planted late-lane dense round through V3 and V4. ## 6. Decisions @@ -270,3 +300,87 @@ Both stops were resolved by D-9 (`plan.md`), adopted by the user on `tests/model_executor/test_enabled_custom_ops.py::test_topk_softmax_wrapper_forwards_renormalize`. - C6 was measured with a completion-only probe because the boundary seam is on PR 35; the PR 35 composition check is pending in the parent task. +- Round 2 (§8): on this branch, online Poisson arrivals reach only lane 0 of + MONOLITHIC and PREFILL contexts (PR 35's W2 defect on `main`); online + multi-lane coverage here comes from the `-burst` cells. The derived + barrier-aligned M5 of §5.3 is not a measurement. + +## 8. Round-2 remediation (R-10) + +Scope: `plan.md` §7; findings in `review.md` Round 2. Owner instruction: +"确认，执行上上述修复； R2-02 采纳你的推荐；R2-03需要补充 PDD+online（如果你认为pd-af+online有必要，请一并补充）". + +### 8.1 Commits and commands + +| Commit | Content | +| --- | --- | +| `1661bf1` | R2-01, R2-11, R2-13: `try_acquire` refactor, class docstring, unit test | +| `a8e8d8a` | R2-03, R2-07, R2-08: matrix groups G8–G11, cluster-keyed drain report, case timeout, set lock | +| `e35242f` | R2-02, R2-04, R2-05, R2-09, R2-10, R2-14, R2-15: comparison tools, tool unit tests, evidence scripts | + +```bash +# base for the new groups at a8e8d8a, with stage_execution_context.py replaced by +# its 1f694f7 version for the run (run.json: status "M frontier/.../stage_execution_context.py"), then restored +python -m tests.e2e.stage_admission_matrix run --set base --group G8 --group G9 --group G10 --group G11 --jobs 16 +python -m tests.e2e.stage_admission_matrix run --set after-r2 --jobs 16 # a8e8d8a, clean outside task_memory +python -m tests.e2e.stage_admission_matrix compare --before base --after after-r2 \ + --output /data/ycfeng/tmp/stage_admission_ordering/compare_base_after-r2.json +python task_memory/.../evidence/explain_t_path.py after-r2 /compare_base_after-r2.json \ + task_memory/.../evidence/r2_t_path_explanation.json +python -m pytest tests/ -q -p no:cacheprovider --continue-on-collection-errors \ + --junitxml=/after-r2-pytest/.xml +python -m tests.comparison.stage_admission_pp.compare_lanes \ + --vllm-run calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b \ + --before base --after after-r2 --output calibration/stage_admission_case_001/analysis +python task_memory/.../evidence/decompose_co_execution.py dense # and moe dense +``` + +Environment as §2; interpreter digest `ecd50ea8…` for every set. + +### 8.2 Results per finding + +| Finding | Check | Expected | Observed | Result | +| --- | --- | --- | --- | --- | +| R2-01, R2-11 | `try_acquire` on an active full-stage ticket, capacity 2 | `False`, context unchanged | base rule `False`; `dac4e69` raises `ValueError` ("not in deque"); `1661bf1` `False`, ticket still active, FIFO unchanged | PASS | +| R2-01, R2-11 | the three context unit files | pass, no assertion change | 181 passed | PASS | +| R2-01, R2-11 | set `after` vs `after-r2`, 98 cases | byte-identical | 97 success hash files identical; the configuration rejection has the same error (`identity_after_after-r2.json`) | PASS | +| R2-03 | groups G8–G11, 50 cases (§8.3) | §4.4 paths | U 18 PASS, L 12 PASS, T 16: 12 PASS, 4 EXPLAIN; 0 STOP | PASS | +| R2-07 | `run --case-timeout 2` on a recipe case | `other_failure`, set completes | `"case timeout after 2 s"` after 2 s; no child or simulator process left | PASS | +| R2-08 | a second `run` while one is running | fails at once | `RuntimeError: another set is running under …; sets share work/ and run one at a time`, exit 1 | PASS | +| R2-04, R2-05, R2-09, R2-10, R2-15 | `tests/unit/test_stage_admission_pp_tools.py` | pass; the ecff89a tools fail the new checks | 9 passed; on the ecff89a tools 7 failed, 2 passed (the late-lane round and the unexpected-file rejection, which the old tools already handled) (`evidence/r2_tool_tests_on_ecff89a.txt`) | PASS | +| R2-05 | `compare_lanes` rerun on run b vs `after-r2` | PASS with controls holding | 56 rows: 50 MATCH, 2 INFORMATIONAL, 4 HOLDS, 0 MISMATCH; `negative_control_holds = true` (§5.2) | PASS | +| R2-02 | decomposition rerun on runs a and b | identity stated for overlapping pairs; derived aligned M5 | no disjoint or unpaired forward in any round; pairing equals M3; existing fields unchanged; aligned M5 in §5.3 | done | +| R2-14 | `probe_completion.py` with `PYTHONPATH` only | C6 shapes complete | `moe_dp2_pp2` 6/6, `dense_dp1_pp2` 6/6; the probe's resolved config and `R0-moe-dp2-pp2-n6`'s differ only in `metrics_config` | PASS | +| G2 | `tests/unit`, `tests/integration` vs `base-pytest` | no regression | unit 84 failed / 3660 passed / 49 skipped / 10 errors (base 84 / 3644 / 49 / 10); integration 14 passed / 21 skipped / 5 errors (base 11 / 21 / 5). 0 regressions, 0 new failures, 0 skip changes; new node ids only: 16 unit, 3 integration (`evidence/r2_g2_*_compare.json`) | PASS | +| R2-12, R2-13 | documentation | contract stated | `design.md` round-2 note; PR body | done | +| R2-06 | pre-merge step P6 | recorded, not executed | `plan.md` §7 | open | + +### 8.3 New groups (R2-03) + +| Group | Cases | Base | `after-r2` | Paths | +| --- | --- | --- | --- | --- | +| G8 PDD offline | MoE `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}`, dense `dp1-pp2`, `n=8`, prefill 16 / decode 3 | 4 `admission_deadlock` (MoE `PP > 1`), 3 success | 7 success | U 3 PASS; L 4 PASS | +| G9 PDD online | the G8 shapes at Poisson 20/s; MoE `dp{2,4}-pp{2,3}` burst | 4 `admission_deadlock` (the burst cells), 7 success | 11 success | U 3 PASS; T 4 PASS, identical; L 4 PASS | +| G10 co-location online | MoE (prefill-only) and dense, `dp{2,4}-pp{1,2,3}`, Poisson 20/s and burst; `dp2-pp2` at 5/s and 80/s | 4 `admission_deadlock` (MoE burst `PP > 1`), 24 success | 28 success | U 8 PASS; T 12 PASS, identical, 4 EXPLAIN (dense burst `PP > 1`); L 4 PASS | +| G11 PD-AF | dense and MoE recipes, offline and online, `PREFILL_PP=2` | 4 success | 4 success | U 4 PASS | + +Every L case conserves requests and tokens (8 requests, 128 prefill tokens; +24 decode tokens for PD cells, 8 for prefill-only cells). The 4 EXPLAIN cases +(`evidence/r2_t_path_explanation.json`) run the same batches with the same +component durations as the base; only start times differ. Their co-execution +fraction goes from 0.818, 0.375, 0.846 and 0.846 to 1.0, `peak_lanes ≤ +attn_dp`, and no lane overlaps itself. These are the same values as the +offline G4 cells of the same shapes. + +Observed while running: in the Poisson online cells, every MONOLITHIC and +PREFILL forward runs on lane 0. PDD DECODE lanes all run. `_schedule_batch_mode` +numbers lanes from 0 within each scheduling call, and online arrivals come one +per call. That is PR 35's W2 defect on `main` (fixed on PR 35, not here). The +burst cells deliver all requests at `t=0` in online mode and reach every lane. +They carry the online L and T coverage on this branch. The Poisson cells show +that a multi-lane context with one live lane is unchanged. + +PD-AF: `DECODE_ATTN` requires `attn_dp = 1`, and PD-AF has one `attn_dp` for +every role, so no PD-AF context has more than one lane. The G11 cells are +capacity-1 `PREFILL` contexts at `PP = 2`, where R2-12's contract change +applies. They are byte-identical, offline and online. From 8315d9b149eef86e40a0a3ad27a57c2abbabf5b8 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 13:30:19 +0800 Subject: [PATCH 45/80] docs(step9): record the PR 36 pre-merge untrack for W9-01 PR 36 dropped its task_memory exception and untracked its task directory at 4d08c5d (its plan step P6), so the W9-01 copies kept here are now the published records of that task. Refresh the summary and test report copies and note the step in issues.md and progress.md. --- .../task_2026-09-21_issue26_correctness_pr/issues.md | 2 ++ .../task_2026-09-21_issue26_correctness_pr/progress.md | 3 ++- .../w9_01_stage_admission_ordering/summary.md | 6 +++--- .../test_report_2026-09-23_stage_admission_ordering.md | 3 ++- 4 files changed, 9 insertions(+), 5 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md index 6f817879..663b71be 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | W9-01: PR 36 ran its pre-merge untrack (P6, `4d08c5d`); the copies here are now the only published records of that task. | | 2026-09-23 | W9-01: PR 36 round-2 review remediation recorded; the composition check now also reruns PR 36 groups G9 and G10. | | 2026-09-23 | W9-01: fixed on `fix/stage-admission-ordering` (draft PR 36) under option 2; resolution recorded, summary and test report copied to `w9_01_stage_admission_ordering/`. | | 2026-09-22 | Created; recorded W9-01 (stage-admission deadlock at PP>1 with attn_dp>1) and W9-02 (PP=3 x attn_dp=2 topology rejection) found during Step 9 P1(b). | @@ -130,6 +131,7 @@ Option 2 was taken. The fix lives on its own branch and PR: | Acceptance rules | `aeeca93` (plan D-9) | | Round-2 review fixes | `1661bf1` (rule refactor: an active ticket is refused), `a8e8d8a` (PDD, online and PD-AF matrix groups G8–G11), `e35242f` (comparison tools) | | Records | `fc34341`, `4bcd616`, `ecff89a`, `1218ba6`, `7a7c22e`; copies in `w9_01_stage_admission_ordering/` (`summary.md`, `test_report_2026-09-23_stage_admission_ordering.md`) | +| Pre-merge untrack | `4d08c5d` (PR 36 plan P6): the branch restores `main`'s `.gitignore` and no longer tracks its task directory, so the copies here are the published records | Observed on that branch (details in the copied test report): diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 7a4f202e..986c71a3 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | PR 36 pre-merge untrack (P6, `4d08c5d`) done; W9-01 copies refreshed. Merge-forward waits for the PR 36 merge. | | 2026-09-23 | PR 36 round-2 review remediation recorded; the W9-01 composition check extended to PR 36 groups G9 and G10. | | 2026-09-23 | W9-01 fixed on `fix/stage-admission-ordering` (draft PR 36); Step 9 PP>1 packages stay paused until it merges forward and passes G3b. | | 2026-09-22 | Step 9 execution started: P1(a) oracle completed, P1(b) blocked by W9-01, design checkpoint partially settled, G1 ground-truth instrumentation completed and case binding written. | @@ -225,7 +226,7 @@ User start signal: "开始执行step9", with the quality gates repeated (readabi W9-01 is not caused by this PR: `stage_execution_context.py`, `replica_stage_schduler.py` and `stage_contexts.py` are byte-identical to `main`. It is unobserved because every Simulator-level test with `attn_dp > 1` uses `num_pipeline_stages = 1` and no shipped example sets `attn_dp > 1`. Scope decision requested from the user; recommendation is to fix it as a separate correctness item rather than inside this feature branch. -Scope decision (2026-09-22): "采纳你的推荐，继续" — option 2, a separate correctness item. Fixed on `fix/stage-admission-ordering`, draft PR 36, rule commit `dac4e69`, validated against vLLM DP=2/PP=2 on 4×H800. Branch records are copied to `w9_01_stage_admission_ordering/`. Resume order: PR 36 merges, `main` is merged forward here, G3b (with W3) and the online groups G9 and G10 (with W2) rerun as the composition check, then P1(b) and D9-2 (`issues.md` W9-01, Resolution). PR 36 round-2 review fixes: `1661bf1`, `a8e8d8a`, `e35242f`, records `7a7c22e`. +Scope decision (2026-09-22): "采纳你的推荐，继续" — option 2, a separate correctness item. Fixed on `fix/stage-admission-ordering`, draft PR 36, rule commit `dac4e69`, validated against vLLM DP=2/PP=2 on 4×H800. Branch records are copied to `w9_01_stage_admission_ordering/`. Resume order: PR 36 merges, `main` is merged forward here, G3b (with W3) and the online groups G9 and G10 (with W2) rerun as the composition check, then P1(b) and D9-2 (`issues.md` W9-01, Resolution). PR 36 round-2 review fixes: `1661bf1`, `a8e8d8a`, `e35242f`, records `7a7c22e`. PR 36 untracked its task directory before merge at `4d08c5d` (R-11); the copies here are its published records. W9-02: `attn_dp=2, moe_ep=2, PP=3` is rejected at construction (6 devices against node size 4). Plan C1's PP3 row amended to `attn_dp=1`. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md index 2a56ada2..0eb144c6 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/summary.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-11: P6 executed; the directory is untracked on the branch and archived in the parent task. | | 2026-09-23 | R-10: round-2 review remediation (14 findings applied, R2-06 pending as P6); PDD, online and PD-AF cells added. | | 2026-09-23 | Created at P4: fix, tests, P0–P3 and the vLLM comparison complete under D-9. | @@ -64,9 +65,8 @@ Decisions taken during execution: ## Open and deferred work -- R2-06 / P6: before PR 36 merges, drop the `.gitignore` exception and - untrack this task directory (the parent task keeps copies). It deletes - tracked records, so it waits for the owner's go-ahead. +- R2-06 / P6 was executed at R-11. This directory is no longer tracked on + the branch; the parent task keeps the archive copies. - On this branch, online Poisson arrivals reach only lane 0 of MONOLITHIC and PREFILL contexts (PR 35 W2). PR 35's composition check after the merge-forward reruns G9 and G10 with lane rotation, as well as G3b. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md index 267c850f..85e7a9a4 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-11: P6 executed (`4d08c5d`); R2-06 row closed. | | 2026-09-23 | R-10 round-2 remediation: §8 added (rule refactor, PDD/online/PD-AF cells, tool fixes); §5.2 and §5.3 restated for the N rows and the barrier-aligned M5. | | 2026-09-23 | D-9 adopted ("采纳你的推荐，继续"): C3 witnesses judged by co-execution fraction, V5 gated on MoE only. Both comparisons rerun (`aeeca93`); all criteria pass. | | 2026-09-23 | Created. P0–P3 and P5 executed; two plan stop conditions reached (C3 witness metric at `attn_dp=4`, C7 V5 on the dense shape). P4 push held for the user's decision. | @@ -353,7 +354,7 @@ Environment as §2; interpreter digest `ecd50ea8…` for every set. | R2-14 | `probe_completion.py` with `PYTHONPATH` only | C6 shapes complete | `moe_dp2_pp2` 6/6, `dense_dp1_pp2` 6/6; the probe's resolved config and `R0-moe-dp2-pp2-n6`'s differ only in `metrics_config` | PASS | | G2 | `tests/unit`, `tests/integration` vs `base-pytest` | no regression | unit 84 failed / 3660 passed / 49 skipped / 10 errors (base 84 / 3644 / 49 / 10); integration 14 passed / 21 skipped / 5 errors (base 11 / 21 / 5). 0 regressions, 0 new failures, 0 skip changes; new node ids only: 16 unit, 3 integration (`evidence/r2_g2_*_compare.json`) | PASS | | R2-12, R2-13 | documentation | contract stated | `design.md` round-2 note; PR body | done | -| R2-06 | pre-merge step P6 | recorded, not executed | `plan.md` §7 | open | +| R2-06 | P6 at `4d08c5d`: `git diff origin/main --name-only`, `git diff origin/main -- .gitignore`, `git ls-files task_memory`; affected tests rerun | no `task_memory` path, `.gitignore` equal to `main`, nothing tracked | 0 `task_memory` paths among 10 changed files; empty `.gitignore` diff; 0 tracked files; the tool, context and pipeline-lane tests 41 passed | PASS | ### 8.3 New groups (R2-03) From 4ab19644658b54c65606753d746bae3e7d2fd76b Mon Sep 17 00:00:00 2001 From: YICHENG FENG <50061432+fwyc0573@users.noreply.github.com> Date: Wed, 23 Sep 2026 15:02:59 +0800 Subject: [PATCH 46/80] fix(scheduler): stage admission deadlock with attention-DP lanes under pipeline parallelism (#36) * docs(stage-admission): plan and design for the pipeline-lane admission deadlock A MoE replica with more than one attention-DP lane and more than one pipeline stage drains its event queue with requests unfinished. Stage admission mints a ticket per arriving batch and admits only the strict FIFO head; at PP>1 a lane holds several queued tickets while consuming one, so the head can belong to a lane that is busy inside the sync room, and the room in turn waits for the lane the FIFO refuses. Dense completes but serializes its lanes. PP=1 and capacity-1 contexts are unaffected. The records diagnose this from source on main 1f694f7, compare four options, and recommend ordering only exclusive operations: a full-stage ticket waits only for an EP wave queued ahead of it. The lane-aware alternative was rejected because the DES wakes sibling lanes at release, not at a peer's acquisition. No source change in this commit. The task records are tracked through the same narrow .gitignore exception the Issue 26 branches use, so they can be reviewed on the remote. * docs(stage-admission): record PR 36 and the reviewer resume prompt * docs(stage-admission): apply the PR 36 round-1 plan review Each of the ten review findings was re-checked against main 1f694f7 and its disposition recorded in review.md. The records are corrected; nothing was executed and no source file changed. - design.md: cite the MONOLITHIC/PREFILL admission loop; state the drain as a queued-ticket arrangement rather than a shape; note that queued EP waves exist only on DECODE_FFN; remove the admitted ticket with remove(ticket); keep option A rejected on design grounds with its stall trace marked unverified; restate the capacity-1 and PP=1 expectations as caller-level conditions; narrow the queue bound; add the mixed-phase scope boundary; withdraw the unmeasured "lanes serialized" label. - plan.md: C1 targets confirmed deadlock witnesses from a prefill-only, phase-controlled group; C3 uses the stage-batch ledger with half-open overlap duration; C4 takes the reviewer's wording; P0 lists its artifacts and outcome classes; P2 covers both sides of the EP boundary, a DECODE_FFN dense-group control and a second admission round; P3 has separate unchanged, liveness and timing paths; the case list is published on the analytical backend and keeps attn_dp=2, PP=3 (decisions D-6, D-7). - requirements.md, progress.md, review_prompt.md: record the request, the state, and the round-2 reviewer prompt. * tests: add the stage admission case matrix for attention-DP lanes under PP Runs the case list of the stage-admission-ordering plan (section 4) one child process per case: synthetic MoE/dense shapes, the release recipes, and the vLLM-aligned shapes. Each case records its inputs, run provenance and one outcome artifact (metrics hashes, a live drain-state report, or the error). The drain classifier checks the admission-deadlock signature from the live stage contexts, lane queues and sync rooms. The compare command applies the unchanged / repaired-liveness / timing paths, using the stage-ledger lane-overlap metric. * tests: keep per-stage lane views as a list in the admission state report JSON turns integer dict keys into strings, so a state_report.json read back from disk could not be indexed by stage id the way the in-process report is. A list indexed by stage id has the same shape before and after the round trip. * tests: compare DP-lane stage admission with vLLM under pipeline parallelism vllm_burst_driver.py runs inside the vllm-openai v0.10.2 image: it builds the instrumented overlay from the vLLM-BS checkout, accepting it only when the files that differ from the image are exactly the fork's own changes, then drives DP=2, PP=2 bursts of prefill-only requests pinned to rank i mod 2. run_vllm_worker.sh is the 4-GPU worker entry point for the MoE and dense scenarios. compare_lanes.py turns the vLLM per-forward traces and the Frontier G7 stage ledgers into the same lane metrics (completion, per-lane sequences, stage-0 pairing, co-start and co-execution) and writes the workflow-gap table for plan criterion C7. * fix(scheduler): order full-stage admission only behind queued EP waves StageExecutionContext admitted a ticket only from the head of its ready FIFO. Under pipeline parallelism an attention-DP lane that is still busy can queue its next batch at a shared stage, and that ticket then refuses the idle lane's queued batch. With every lane waiting on the head, the sequential run ends with work left (admission deadlock), and dense runs serialize the lanes instead of overlapping them. A full-stage ticket may now be admitted ahead of earlier queued full-stage tickets, but never ahead of an EP wave queued before it. EP waves keep the strict head rule, so they still wait for every operation queued earlier, and the forward-group seal is unchanged. Tests: contract tests for the bypass, both sides of the EP boundary and capacity 1; a DECODE_FFN control that keeps dense groups in counter order around a queued EP wave; a two-lane drain through promotion and restore; and simulator cases that complete the MoE admission-deadlock witnesses and start both dense lanes in the first forward. On the base revision the bypass, capacity-1, drain and simulator tests fail as recorded in the task's negative controls. * tests: apply a recorded patch to the accepted vLLM overlay The ground-truth checkout passes a fifth renormalize argument to _moe_C::topk_softmax, while both the checkout's own csrc and the v0.10.2 image declare the four-argument op, so the MoE scenario failed in profile_run. The overlay step now takes an optional unified diff, applied after the fork-change check accepts the overlay, and records its SHA-256, the files it touched, and whether each now equals the image's copy. Hunks are applied as exact text replacements because the worker image need not carry patch or git. * docs(stage-admission): record P0-P3 and the vLLM comparison Adds the test report, the calibration case for the vLLM comparison (two GPU runs, the recorded four-argument topk_softmax overlay patch, and the workflow-gap analysis), and the selected evidence: base negative controls, G2 identity comparisons, the path-T explanation and the Step 9 probe. Progress, requirements (R-7) and plan are updated. Two plan stop conditions are open for the user: the C3 witness rule at attn_dp=4, and V5 co-execution on the dense vLLM shape. * tests: judge admission witnesses and dense co-execution per plan D-9 Contention witnesses now pass on a strictly larger co-execution fraction (multi_lane_busy_time / busy_time). At attn_dp=4 the fix admits all lanes together and shortens the busy period, so the absolute overlap time fell while the overlap became complete. V5 of the vLLM comparison gates the MoE shape only. vLLM's dense DP ranks meet once per forward and vary in duration per rank (co-execution 0.54-0.93 across rounds), which the dummy predictor does not model; the dense value is reported as INFORMATIONAL and the status counts only MISMATCH rows. * docs(stage-admission): adopt D-9 and close the comparisons Records the user's decision to judge contention witnesses by the co-execution fraction and to gate V5 on the MoE shape only (plan D-9, requirements R-8), the reruns of both comparisons under those rules (no STOP, no MISMATCH), the updated test report and calibration case, and the completion summary. * docs(stage-admission): track the base negative-control log The test report cites evidence/base_negative_controls.log, but the repository-wide *.log ignore rule kept it out of the tree. * docs(stage-admission): record P4 completion * docs(stage-admission): record the round-2 code review of PR 36 Fifteen findings on ecff89a, each re-checked against source and posted to PR 36 as inline comments. Fixes are deferred by the owner. * fix(stage-admission): refuse an active ticket in try_acquire Round-2 review R2-01: after dac4e69, try_acquire on a full-stage ticket that is already active ran off the FIFO scan and remove() raised ValueError; the base rule returned False. Each scope now has one branch. An EP wave leaves the FIFO by popleft. A full-stage ticket is found in one pass and deleted by position, and a ticket that is not queued is refused (R2-11). The class docstring now states that full-stage order comes from the lane stage schedulers and that admission_seq records enqueue order only (R2-13). A unit test re-acquires an active ticket and checks the context is unchanged. * tests: add PDD, online and PD-AF cells to the stage-admission matrix Round-2 review R2-03, R2-07 and R2-08. - Cases carry sys_arch, simulation_mode, a Poisson rate and recipe environment overrides. New groups: G8 PDD offline, G9 PDD online, G10 co-location online, G11 PD-AF recipes with PREFILL_PP=2. Dense PDD requires attn_dp=1 and PD-AF DECODE_ATTN requires attn_dp=1, so those cells are unchanged-path controls. On main, MONOLITHIC and PREFILL place incremental online arrivals on lane 0, so G9 and G10 add burst cells that reach several lanes. - The drain state report and the deadlock signature read every cluster type, keyed by cluster. - Each child runs in its own session under --case-timeout (default 600 s); a timed-out session is killed and recorded as other_failure. - run holds an exclusive lock on the matrix root, because every set shares work/ for byte-identical outputs. * tests: separate negative controls and harden the vLLM comparison tools Round-2 review R2-02, R2-04, R2-05, R2-09, R2-10, R2-14 and R2-15. - compare_lanes: V1 and V4 compare vLLM with the after revision only. The base negative controls are their own rows, N1 (MoE base admission_deadlock) and N4 (dense base co-start >= 0.5), reported as HOLDS or LOST, with negative_control_holds in the status. Placement is ok only when no request is misplaced or unseen. The D-9 comment names both sources of dense non-overlap. - vllm_burst_driver: overlay acceptance compares file sets. apply_patch reads an empty hunk line as a trimmed context line, rejects any other unknown hunk line, and keeps every section of a file that appears more than once. - tests/unit/test_stage_admission_pp_tools.py replaces the task-local synthetic_check.py and needs no scratch data. On the ecff89a tools 7 of its 9 tests fail. - decompose_co_execution.py states its identity for overlapping pairs only, counts disjoint and unpaired forwards, checks the pairing against M3, and adds M5 with both starts of a pair set to the later one (derived from where vLLM's DP all-reduce sits). Reran on runs a and b: no disjoint pair; existing fields unchanged. - explain_t_path.py takes the after set as an argument. - probe_main.py no longer puts a hard-coded worktree on sys.path. * docs(stage-admission): record the round-2 remediation (R-10) Plan section 7, the D-9 (b) rationale restated with both sources of dense non-overlap, test report section 8, dispositions of R2-01..R2-15, the capacity-1 contract and FIFO notes in design.md, the calibration rerun with separate negative-control rows, and the round-2 evidence (path-T explanation, G2 comparisons, tool tests on the old tools). R2-06 is recorded as pre-merge step P6, not executed. * chore: stop tracking the stage-admission task records before merge Restore the repository-wide task_memory/ ignore rule from main and untrack task_memory/task_2026-09-22_stage_admission_ordering, so that no task record reaches main (review finding R2-06, plan step P6). The parent task on fix/issue26-correctness-pr keeps the archive copies of the summary and the test report. The matrix module docstring now points at build_cases instead of the untracked plan file. --- .../stage_execution_context.py | 50 +- .../stage_admission_pp/compare_lanes.py | 272 ++++++ .../stage_admission_pp/run_vllm_worker.sh | 104 +++ .../stage_admission_pp/vllm_burst_driver.py | 281 ++++++ tests/e2e/stage_admission_matrix.py | 833 ++++++++++++++++++ .../test_stage_admission_pipeline_lanes.py | 65 ++ .../test_mixed_layer_decode_ffn_scheduling.py | 69 ++ .../test_shared_forward_group_admission.py | 37 + tests/unit/test_stage_admission_pp_tools.py | 237 +++++ tests/unit/test_stage_execution_context.py | 68 ++ 10 files changed, 2001 insertions(+), 15 deletions(-) create mode 100644 tests/comparison/stage_admission_pp/compare_lanes.py create mode 100644 tests/comparison/stage_admission_pp/run_vllm_worker.sh create mode 100644 tests/comparison/stage_admission_pp/vllm_burst_driver.py create mode 100644 tests/e2e/stage_admission_matrix.py create mode 100644 tests/integration/test_stage_admission_pipeline_lanes.py create mode 100644 tests/unit/test_stage_admission_pp_tools.py diff --git a/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py b/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py index e92a4edc..1dcda539 100644 --- a/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py +++ b/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py @@ -35,9 +35,14 @@ class StageExecutionContext: The context is intentionally independent of event timing and child lane queues. A complete operation first enters the ready FIFO, then the owner - admits it atomically. EP child schedulers may start only after their - wave's ticket has been acquired, and the ticket remains active through the - wave-level combine/cleanup boundary. + admits it atomically. An EP wave is admitted only from the FIFO head, so + it waits for every operation queued before it. A full-stage operation may + be admitted ahead of earlier queued full-stage operations, but never ahead + of an EP wave queued before it. Full-stage operations are therefore + admitted in the order their lane stage schedulers present them, and + ``admission_seq`` records enqueue order only. EP child schedulers may + start only after their wave's ticket has been acquired, and the ticket + remains active through the wave-level combine/cleanup boundary. """ def __init__( @@ -320,24 +325,39 @@ def _validate_ticket(self, ticket: StageAdmissionTicket) -> None: ) def try_acquire(self, ticket: StageAdmissionTicket) -> bool: - """Acquire the FIFO-head ticket if this stage is currently idle.""" + """Acquire ``ticket`` if the stage can admit it now. + + An EP wave must be the FIFO head of an idle stage. A full-stage ticket + must have no EP wave queued ahead of it. A ticket that is already + active is not queued and is refused. + """ self._validate_ticket(ticket) if ticket.scope == EP_WAVE: - if self._active_ep_ticket is not None or self._active_full_stage_tickets: + if ( + self._active_ep_ticket is not None + or self._active_full_stage_tickets + or not self._ready_fifo + or self._ready_fifo[0] != ticket + ): return False - elif self._active_ep_ticket is not None: - return False - elif len(self._active_full_stage_tickets) >= self._full_stage_capacity: - return False - elif self._forward_group_sealed: - return False - if not self._ready_fifo or self._ready_fifo[0] != ticket: - return False - self._ready_fifo.popleft() - if ticket.scope == EP_WAVE: + self._ready_fifo.popleft() self._active_ep_ticket = ticket else: + if ( + self._active_ep_ticket is not None + or len(self._active_full_stage_tickets) >= self._full_stage_capacity + or self._forward_group_sealed + ): + return False + for position, queued in enumerate(self._ready_fifo): + if queued == ticket: + break + if queued.scope == EP_WAVE: + return False + else: + return False + del self._ready_fifo[position] self._active_full_stage_tickets.add(ticket) self._refresh_active_ticket_view() return True diff --git a/tests/comparison/stage_admission_pp/compare_lanes.py b/tests/comparison/stage_admission_pp/compare_lanes.py new file mode 100644 index 00000000..1d16ecc3 --- /dev/null +++ b/tests/comparison/stage_admission_pp/compare_lanes.py @@ -0,0 +1,272 @@ +#!/usr/bin/env python3 +"""Compare DP-lane stage admission between vLLM traces and Frontier ledgers. + +Reads the vLLM ground truth written by ``run_vllm_worker.sh`` and the G7 cases +of ``tests.e2e.stage_admission_matrix`` for a set run before the change and one +run after it, computes the per-forward lane metrics of the stage-admission plan +(§4.7, M1–M5) on both sides with the same definitions, and writes the +workflow-gap table, summary and status for the calibration case. + +A vLLM forward row is one ``pp_boundary`` record: its lane is the DP rank the +driver pinned its requests to, and its stage is ``pp_rank``. Stage-0 +intervals end at ``send_start_ts``, taken after the post-forward synchronize; +last-stage intervals end at the record's wall-clock ``timestamp`` converted to +the monotonic clock with the offset the driver sampled around the round. A +Frontier forward row is one ``ATTN_DP_LANE`` ledger row. +""" + +from __future__ import annotations + +import argparse +import csv +import json +import statistics +from collections import defaultdict +from pathlib import Path + +from tests.e2e.stage_admission_matrix import ( + ADMISSION_DEADLOCK, + ATTN_DP_LANE, + SUCCESS, + interval_overlap, + matrix_root, + read_ledger, +) + +MODELS = ("moe", "dense") +BURSTS = (8, 16) +CO_START_BOUND = 0.5 +CO_EXECUTION_BOUND = 0.10 +# Dense DP ranks meet once per forward, in the DP metadata all-reduce that +# vLLM runs after ``forward_start_ts``. A rank's recorded interval therefore +# includes its wait for the other rank (start offsets) as well as its own +# duration variation (end offsets); the dummy predictor models neither, so +# dense co-execution is reported, not gated (plan §4.7 V5, D-9). MoE ranks +# stay aligned by EP collectives. +CO_EXECUTION_GATED = {"moe": True, "dense": False} +INFORMATIONAL = "INFORMATIONAL" +HOLDS, LOST = "HOLDS", "LOST" +FRONTIER_OWNER = "frontier/scheduler/replica_stage_scheduler/stage_execution_context.py" + + +def _forward(lane: int, stage: int, start: float, end: float, indices) -> dict: + return {"lane": lane, "stage": stage, "start": start, "end": end, + "indices": tuple(sorted(indices))} + + +def vllm_forwards(scenario_dir: Path) -> dict[tuple[int, int], dict]: + """Formal vLLM forwards keyed by ``(burst size, round)``.""" + requests = {row["request_id"]: row for row in map(json.loads, (scenario_dir / "requests.jsonl").read_text().splitlines())} + summary = json.loads((scenario_dir / "summary.json").read_text()) + offsets = { + entry["label"]: (entry["wall_minus_monotonic_before"] + entry["wall_minus_monotonic_after"]) / 2 + for entry in summary["rounds"] + } + runs: dict[tuple[int, int], dict] = {} + for line in (scenario_dir / "pp_boundary.jsonl").read_text().splitlines(): + record = json.loads(line) + members = [requests[request_id] for request_id in record["request_ids"]] + labels = {member["burst"] for member in members} + ranks = {member["rank"] for member in members} + if len(labels) != 1 or len(ranks) != 1: + raise ValueError(f"forward mixes bursts or ranks: {record['request_ids']}") + label = labels.pop() + if label == "warmup": + continue + if record["is_last_rank"]: + end = record["timestamp"] - offsets[label] + else: + end = record["send_start_ts"] + member = members[0] + run = runs.setdefault( + (int(label.split("-")[0][1:]), member["round"]), + {"forwards": [], "requests": [row for row in requests.values() if row["burst"] == label]}, + ) + run["forwards"].append(_forward(ranks.pop(), record["pp_rank"], record["forward_start_ts"], end, + (m["index"] for m in members))) + for run in runs.values(): + rows = run.pop("requests") + run["submitted"] = len(rows) + run["completed"] = sum(1 for row in rows if row["num_output_tokens"] == 1) + return runs + + +def vllm_placement(scenario_dir: Path) -> dict: + """Check from engine iterations that each formal request ran on its pinned rank.""" + pinned = {row["request_id"]: row["rank"] for row in map(json.loads, (scenario_dir / "requests.jsonl").read_text().splitlines())} + scheduled_by = defaultdict(set) + for path in sorted((scenario_dir / "dp_placement").glob("*.jsonl")): + for record in map(json.loads, path.read_text().splitlines()): + if record["kind"] == "engine_iteration": + for request_id in record["scheduled_new_req_ids"]: + scheduled_by[request_id].add(record["engine"]) + misplaced = sorted(rid for rid, rank in pinned.items() if scheduled_by.get(rid, {rank}) != {rank}) + unseen = sorted(rid for rid in pinned if rid not in scheduled_by) + return {"requests": len(pinned), "misplaced": misplaced, "unseen": unseen, + "ok": not misplaced and not unseen} + + +def frontier_run(set_dir: Path, case_id: str) -> dict: + case_dir = set_dir / case_id + run = json.loads((case_dir / "run.json").read_text()) + case = json.loads((case_dir / "case.json").read_text()) + result = {"outcome": run["outcome"], "submitted": case["num_requests"], "forwards": []} + if run["outcome"] != SUCCESS: + result["completed"] = None + return result + result["completed"] = case["num_requests"] + for row in read_ledger(case_dir / "metrics"): + if row["execution_scope"] != ATTN_DP_LANE: + continue + result["forwards"].append(_forward(row["replica_local_id"], row["stage_id"], row["stage_start_ts"], + row["stage_end_ts"], (int(rid) for rid in row["request_ids"]))) + lanes_match_index = all(index % 2 == forward["lane"] for forward in result["forwards"] + for index in forward["indices"]) + result["placement_ok"] = lanes_match_index + return result + + +def lane_metrics(forwards: list[dict]) -> dict: + """M2–M5 of plan §4.7 from one run's forwards.""" + by_lane_stage = defaultdict(list) + for forward in forwards: + by_lane_stage[(forward["lane"], forward["stage"])].append(forward) + sequences = { + f"lane{lane}/stage{stage}": [list(f["indices"]) for f in sorted(rows, key=lambda f: f["start"])] + for (lane, stage), rows in sorted(by_lane_stage.items()) + } + metrics = {"M2_sequences": sequences} + for stage in sorted({forward["stage"] for forward in forwards}): + lane0 = sorted(by_lane_stage[(0, stage)], key=lambda f: f["start"]) + lane1 = sorted(by_lane_stage[(1, stage)], key=lambda f: f["start"]) + pairing = [] + for forward in lane0: + overlaps = [(min(forward["end"], other["end"]) - max(forward["start"], other["start"]), other) + for other in lane1] + overlap, partner = max(overlaps, key=lambda item: item[0], default=(0.0, None)) + pairing.append([list(forward["indices"]), list(partner["indices"]) if partner and overlap > 0 else None]) + durations = [f["end"] - f["start"] for f in lane0 + lane1] + skew = abs(lane0[0]["start"] - lane1[0]["start"]) / statistics.median(durations) if lane0 and lane1 else None + overlap = interval_overlap([(f["start"], f["end"], f["lane"]) for f in lane0 + lane1]) + metrics[f"stage{stage}"] = { + "M3_pairing": pairing, + "M4_co_start": skew, + "M5_co_execution": overlap["multi_lane_busy_time"] / overlap["busy_time"] if overlap["busy_time"] else None, + "median_forward_duration": statistics.median(durations) if durations else None, + "self_overlap": overlap["self_overlap"], + } + return metrics + + +def _row(check, model, burst, round_index, metric, groundtruth, after, base, status, note=""): + return {"check": check, "model": model, "burst": burst, "round": round_index, "metric": metric, + "groundtruth": json.dumps(groundtruth), "frontier_after": json.dumps(after), + "frontier_base": json.dumps(base), "status": status, + "frontier_owner": FRONTIER_OWNER if status == "MISMATCH" else "", "note": note} + + +def compare(vllm_run: Path, frontier_root: Path, before: str, after: str) -> tuple[list[dict], dict]: + rows, details = [], {"placement": {}, "runs": {}} + for model in MODELS: + scenario_dir = vllm_run / "runs" / model + vllm_runs = vllm_forwards(scenario_dir) + details["placement"][model] = vllm_placement(scenario_dir) + for burst in BURSTS: + case_id = f"G7-{model}-dp2-pp2-n{burst}" + base = frontier_run(frontier_root / before, case_id) + new = frontier_run(frontier_root / after, case_id) + base_metrics = lane_metrics(base["forwards"]) if base["outcome"] == SUCCESS else None + new_metrics = lane_metrics(new["forwards"]) if new["outcome"] == SUCCESS else None + rounds = sorted(r for (b, r) in vllm_runs if b == burst) + vllm_metrics = {r: lane_metrics(vllm_runs[(burst, r)]["forwards"]) for r in rounds} + details["runs"][case_id] = {"frontier_base": base_metrics, "frontier_after": new_metrics, + "frontier_base_outcome": base["outcome"], + "frontier_after_outcome": new["outcome"], + "frontier_after_placement_ok": new.get("placement_ok"), + "vllm": vllm_metrics} + base_m4 = base_metrics["stage0"]["M4_co_start"] if base_metrics else None + # Negative controls: the base rule deadlocks MoE and starts dense + # lanes one forward apart. They describe the base, not vLLM. + if model == "moe": + rows.append(_row("N1", model, burst, "base", "base outcome", None, new["outcome"], + base["outcome"], HOLDS if base["outcome"] == ADMISSION_DEADLOCK else LOST, + note=f"expected {ADMISSION_DEADLOCK}")) + else: + rows.append(_row("N4", model, burst, "base", "M4 stage-0 co-start", None, + new_metrics["stage0"]["M4_co_start"] if new_metrics else None, base_m4, + HOLDS if base_m4 is not None and base_m4 >= CO_START_BOUND else LOST, + note=f"expected >= {CO_START_BOUND}")) + for r in rounds: + run = vllm_runs[(burst, r)] + completed = run["completed"] == run["submitted"] == burst + status = "MATCH" if completed and new["outcome"] == SUCCESS else "MISMATCH" + rows.append(_row("V1", model, burst, r, "M1 completion", + f"{run['completed']}/{run['submitted']}", new["outcome"], base["outcome"], status)) + gt = vllm_metrics[r] + after_m2 = new_metrics["M2_sequences"] if new_metrics else None + rows.append(_row("V2", model, burst, r, "M2 lane sequences", gt["M2_sequences"], after_m2, + base_metrics["M2_sequences"] if base_metrics else None, + "MATCH" if after_m2 == gt["M2_sequences"] else "MISMATCH")) + after_m3 = new_metrics["stage0"]["M3_pairing"] if new_metrics else None + rows.append(_row("V3", model, burst, r, "M3 stage-0 pairing", gt["stage0"]["M3_pairing"], after_m3, + base_metrics["stage0"]["M3_pairing"] if base_metrics else None, + "MATCH" if after_m3 == gt["stage0"]["M3_pairing"] else "MISMATCH")) + after_m4 = new_metrics["stage0"]["M4_co_start"] if new_metrics else None + m4_ok = (gt["stage0"]["M4_co_start"] < CO_START_BOUND and after_m4 is not None + and after_m4 < CO_START_BOUND) + rows.append(_row("V4", model, burst, r, "M4 stage-0 co-start", gt["stage0"]["M4_co_start"], + after_m4, base_m4, "MATCH" if m4_ok else "MISMATCH")) + gt_m5 = statistics.mean(vllm_metrics[r]["stage0"]["M5_co_execution"] for r in rounds) + after_m5 = new_metrics["stage0"]["M5_co_execution"] if new_metrics else None + base_m5 = base_metrics["stage0"]["M5_co_execution"] if base_metrics else None + if CO_EXECUTION_GATED[model]: + m5_ok = after_m5 is not None and abs(after_m5 - gt_m5) <= CO_EXECUTION_BOUND + m5_status = "MATCH" if m5_ok else "MISMATCH" + else: + m5_status = INFORMATIONAL + rows.append(_row("V5", model, burst, "mean", "M5 stage-0 co-execution", gt_m5, after_m5, base_m5, + m5_status)) + return rows, details + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0]) + parser.add_argument("--vllm-run", type=Path, required=True, help="evidence directory of one worker run") + parser.add_argument("--before", default="base") + parser.add_argument("--after", default="after") + parser.add_argument("--output", type=Path, required=True, help="case analysis directory") + args = parser.parse_args(argv) + + rows, details = compare(args.vllm_run, matrix_root(), args.before, args.after) + args.output.mkdir(parents=True, exist_ok=True) + with (args.output / "workflow_gap_table.csv").open("w", newline="") as handle: + writer = csv.DictWriter(handle, fieldnames=list(rows[0])) + writer.writeheader() + writer.writerows(rows) + (args.output / "lane_metrics.json").write_text(json.dumps(details, indent=1, sort_keys=True)) + mismatches = [row for row in rows if row["status"] == "MISMATCH"] + placement_ok = all(p["ok"] for p in details["placement"].values()) + placement_unseen = sum(len(p["unseen"]) for p in details["placement"].values()) + controls = [row for row in rows if row["status"] in (HOLDS, LOST)] + status = { + "analysis_state": "COMPLETE", + "status": "PASS" if not mismatches and placement_ok else "FAIL", + "correction_state": "not_applicable", + "rows": len(rows), + "mismatches": len(mismatches), + "vllm_placement_ok": placement_ok, + "vllm_placement_unseen_requests": placement_unseen, + "negative_control_holds": all(row["status"] == HOLDS for row in controls), + "next_action": ("record C7 in the test report" if not mismatches and placement_ok + else "report each MISMATCH row with its cause before P4; adjust nothing"), + } + (args.output / "workflow_gap_status.json").write_text(json.dumps(status, indent=1)) + for row in rows: + print(f"{row['check']} {row['model']:<5} n{row['burst']:<3} r{row['round']!s:<5} {row['status']:<9} " + f"gt={row['groundtruth'][:40]} after={row['frontier_after'][:40]} base={row['frontier_base'][:40]}") + print(json.dumps(status)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/comparison/stage_admission_pp/run_vllm_worker.sh b/tests/comparison/stage_admission_pp/run_vllm_worker.sh new file mode 100644 index 00000000..e7dea8ff --- /dev/null +++ b/tests/comparison/stage_admission_pp/run_vllm_worker.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# Worker entry point for the stage-admission vLLM ground truth (4 GPUs, +# vllm/vllm-openai:v0.10.2). Builds the instrumented overlay, runs the MoE and +# dense bursts, and publishes the traces to the cloud-volume archive and to the +# calibration case directory on the mounted workspace. +# +# Required environment: +# RUN_TAG identifier of this run +# FRONTIER_TREE Frontier worktree on the mounted workspace +# GROUNDTRUTH vLLM-BS checkout on the mounted workspace +# CASE_DIR calibration case directory on the mounted workspace; reads +# inputs/, writes runs/vllm-instrumented// +# ARCHIVE_DIR cloud-volume directory for this run +# Optional: +# OVERLAY_PATCH recorded unified diff applied to the accepted overlay +set -euo pipefail +set +x +: "${RUN_TAG:?}" "${FRONTIER_TREE:?}" "${GROUNDTRUTH:?}" "${CASE_DIR:?}" "${ARCHIVE_DIR:?}" +EVIDENCE_DIR="$CASE_DIR/runs/vllm-instrumented/$RUN_TAG" + +for d in /usr/local/nvidia/lib64 /usr/local/nvidia/lib /usr/lib/x86_64-linux-gnu; do + if [ -e "$d/libcuda.so.1" ]; then + export LD_LIBRARY_PATH="$d${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" + break + fi +done + +PY=python3 +SCRIPT_DIR="$FRONTIER_TREE/tests/comparison/stage_admission_pp" +WORK=/tmp/stage_admission_pp/$RUN_TAG +mkdir -p "$WORK/runs" +export VLLM_CACHE_ROOT="$WORK/vllm_cache" HF_HOME="$WORK/hf" HF_HUB_OFFLINE=1 \ + TRANSFORMERS_OFFLINE=1 VLLM_NO_USAGE_STATS=1 DO_NOT_TRACK=1 + +publish() { + local target="$1" + mkdir -p "$target" + for item in runs overlay_report.json worker_env.json vllm_import.txt; do + if [ -e "$WORK/$item" ]; then cp -r "$WORK/$item" "$target/"; fi + done + echo "status=$status" > "$target/COMPLETE" +} +# The worker writes the mounted workspace as root; hand the evidence back to +# the owner of the case directory. +publish_evidence() { + publish "$EVIDENCE_DIR" + chown -R "$(stat -c %u:%g "$CASE_DIR")" "$EVIDENCE_DIR" +} +status=0 +"$PY" - <<'PY' | tee "$WORK/worker_env.json" +import json, platform, torch +print(json.dumps({ + "python": platform.python_version(), + "torch": torch.__version__, + "cuda_available": torch.cuda.is_available(), + "device_count": torch.cuda.device_count(), + "devices": [torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())], +})) +PY + +SITE_VLLM=$("$PY" -c 'import importlib.util, os; print(os.path.dirname(importlib.util.find_spec("vllm").origin))') +"$PY" "$SCRIPT_DIR/vllm_burst_driver.py" overlay \ + --site-vllm "$SITE_VLLM" --checkout "$GROUNDTRUTH" --destination "$WORK/overlay" \ + --expected-changes "$CASE_DIR/inputs/fork_changed_files.txt" \ + --report "$WORK/overlay_report.json" ${OVERLAY_PATCH:+--patch "$OVERLAY_PATCH"} || status=3 +if [ "$status" -ne 0 ]; then + publish "$ARCHIVE_DIR"; publish_evidence + echo "WORKER_STATUS=$status overlay rejected" + exit "$status" +fi +export PYTHONPATH="$WORK/overlay" +"$PY" -c 'import vllm, vllm.v1.frontier_trace as t; print("VLLM_IMPORT", vllm.__version__, vllm.__file__, t.__file__)' \ + | tee "$WORK/vllm_import.txt" + +run_scenario() { + local name="$1"; shift + local out="$WORK/runs/$name" + mkdir -p "$out" + # Engine cores write their placement records at interpreter exit, which a + # forked multiprocessing child skips; spawned children run it. + if VLLM_FRONTIER_INSTRUMENTATION=1 VLLM_WORKER_MULTIPROC_METHOD=spawn \ + VLLM_FRONTIER_PP_BOUNDARY_LOG_PATH="$out/pp_boundary.jsonl" \ + VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR="$out/dp_placement" \ + timeout 1500 "$PY" "$SCRIPT_DIR/vllm_burst_driver.py" run --output-dir "$out" "$@" \ + > "$out/driver.log" 2>&1; then + echo "SCENARIO_PASS $name" + else + echo "SCENARIO_FAIL $name exit=$?" + tail -n 60 "$out/driver.log" + status=1 + fi +} +run_scenario moe --model-config "$FRONTIER_TREE/data/config/models/Qwen3-30B-A3B-tiny.json" --enable-expert-parallel +run_scenario dense --model-config "$FRONTIER_TREE/data/config/models/Llama-3.2-1B-Instruct.json" + +publish "$ARCHIVE_DIR" +publish_evidence + +for name in moe dense; do + grep -h "DRIVER_DONE" "$WORK/runs/$name/driver.log" | sed "s/^/$name /" || true + wc -l "$WORK/runs/$name/pp_boundary.jsonl" 2>/dev/null || true +done +echo "WORKER_STATUS=$status RUN_TAG=$RUN_TAG" +exit "$status" diff --git a/tests/comparison/stage_admission_pp/vllm_burst_driver.py b/tests/comparison/stage_admission_pp/vllm_burst_driver.py new file mode 100644 index 00000000..21a04109 --- /dev/null +++ b/tests/comparison/stage_admission_pp/vllm_burst_driver.py @@ -0,0 +1,281 @@ +#!/usr/bin/env python3 +"""vLLM ground truth for stage admission of DP ranks under pipeline parallelism. + +Runs inside the ``vllm/vllm-openai:v0.10.2`` image on one 4-GPU worker. Two +subcommands: + +``overlay`` + Build the instrumented vLLM package: copy the image's installed ``vllm`` + package (which carries the compiled extensions) and copy every + ``vllm/**/*.py`` of the ground-truth checkout over it. The overlay is + accepted only when the files where the image and the checkout differ are + exactly the checkout's own changes over its upstream base, listed in + ``--expected-changes``. An accepted overlay may then take one recorded + ``--patch`` (a unified diff), whose SHA-256 and files enter the report. + +``run`` + Start one ``AsyncLLM`` with DP=2, PP=2, TP=1 (EP for the MoE model), run + warmup requests, then bursts of prefill-only requests pinned to rank + ``i mod 2``. Every request of a burst is added before any output is + awaited. The engines are idle between rounds. Per-forward traces come + from the checkout's own instrumentation; this script records each + request's rank, submit and finish times, and the wall/monotonic clock + offset around each round. + +The script imports vLLM only inside ``run`` so that ``overlay`` never loads +the package it is building. +""" + +from __future__ import annotations + +import argparse +import asyncio +import filecmp +import hashlib +import json +import os +import random +import re +import shutil +import sys +import time +from pathlib import Path + + +def build_overlay(site_vllm: Path, checkout: Path, destination: Path, expected_changes: Path) -> dict: + target = destination / "vllm" + if target.exists(): + shutil.rmtree(target) + shutil.copytree(site_vllm, target, symlinks=True) + differing = [] + for source in sorted((checkout / "vllm").rglob("*.py")): + relative = source.relative_to(checkout) + installed = site_vllm.parent / relative + if not installed.exists() or not filecmp.cmp(source, installed, shallow=False): + differing.append(str(relative)) + copy_target = destination / relative + copy_target.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, copy_target) + expected = sorted( + line.strip() for line in expected_changes.read_text().splitlines() + if line.strip().endswith(".py") + ) + return { + "site_vllm": str(site_vllm), + "checkout": str(checkout), + "overlay": str(target), + "differing_py_files": differing, + "expected_py_changes": expected, + "unexpected": sorted(set(differing) - set(expected)), + "missing": sorted(set(expected) - set(differing)), + "accepted": set(differing) == set(expected), + } + + +def apply_patch(patch: Path, root: Path) -> list[str]: + """Apply the unified diff ``patch`` to files under ``root``. + + The worker image need not carry ``patch`` or ``git``, so hunks are applied + here as text replacements; each hunk must match its file exactly once. + An empty hunk line is a context line whose leading space was trimmed. + """ + lines = patch.read_text().splitlines(keepends=True) + hunks: dict[str, list[tuple[str, str]]] = {} + index = 0 + while index < len(lines): + line = lines[index] + index += 1 + if line.startswith("+++ "): + target = line[4:].strip().removeprefix("b/") + hunks.setdefault(target, []) + elif line.startswith("@@ "): + header = re.match(r"@@ -\d+(?:,(\d+))? \+\d+(?:,(\d+))? @@", line) + old_count, new_count = (int(count or 1) for count in header.groups()) + old, new = [], [] + while old_count or new_count: + tag, text = lines[index][0], lines[index][1:] + index += 1 + if tag == "\n": + tag, text = " ", "\n" + if tag not in " -+": + raise ValueError(f"{patch}: unexpected hunk line {lines[index - 1]!r}") + if tag in " -": + old.append(text) + old_count -= 1 + if tag in " +": + new.append(text) + new_count -= 1 + hunks[target].append(("".join(old), "".join(new))) + for relative, edits in hunks.items(): + path = root / relative + text = path.read_text() + for old, new in edits: + if text.count(old) != 1: + raise ValueError(f"{patch}: a hunk does not match {relative} exactly once") + text = text.replace(old, new) + path.write_text(text) + return sorted(hunks) + + +def write_model_dir(model_config: Path, model_dir: Path) -> dict: + config = json.loads(model_config.read_text()) + model_dir.mkdir(parents=True, exist_ok=True) + (model_dir / "config.json").write_text(json.dumps(config, indent=1)) + return config + + +def prompt_token_ids(request_id: str, length: int, vocab_size: int) -> list[int]: + generator = random.Random(request_id) + return [generator.randrange(100, vocab_size - 100) for _ in range(length)] + + +async def run_bursts(args: argparse.Namespace) -> dict: + from vllm import SamplingParams + from vllm.engine.arg_utils import AsyncEngineArgs + from vllm.inputs import TokensPrompt + from vllm.sampling_params import RequestOutputKind + from vllm.v1.engine.async_llm import AsyncLLM + + output_dir = Path(args.output_dir) + model_config = write_model_dir(Path(args.model_config), output_dir / "model") + engine_args = AsyncEngineArgs( + model=str(output_dir / "model"), + load_format="dummy", + skip_tokenizer_init=True, + dtype="bfloat16", + tensor_parallel_size=1, + pipeline_parallel_size=args.pipeline_parallel_size, + data_parallel_size=args.data_parallel_size, + enable_expert_parallel=args.enable_expert_parallel, + enforce_eager=True, + enable_prefix_caching=False, + enable_chunked_prefill=True, + max_num_batched_tokens=args.prompt_tokens, + max_num_seqs=args.max_num_seqs, + block_size=16, + max_model_len=args.max_model_len, + gpu_memory_utilization=args.gpu_memory_utilization, + seed=0, + disable_log_stats=True, + ) + engine = AsyncLLM.from_engine_args(engine_args) + sampling = SamplingParams( + max_tokens=1, ignore_eos=True, temperature=0.0, detokenize=False, + output_kind=RequestOutputKind.FINAL_ONLY, + ) + vocab_size = int(model_config["vocab_size"]) + records: list[dict] = [] + + async def wait_until_idle() -> float: + started = time.monotonic() + while engine.engine_core.dp_engines_running(): + if time.monotonic() - started > args.idle_timeout_s: + raise RuntimeError("DP engines did not pause between rounds") + await asyncio.sleep(0.05) + await asyncio.sleep(args.idle_gap_s) + return time.monotonic() - started + + async def burst(label: str, round_index: int, num_requests: int) -> dict: + offset_before = time.time() - time.monotonic() + queues = [] + for index in range(num_requests): + request_id = f"{label}-q{index}" + rank = index % args.data_parallel_size + prompt = TokensPrompt( + prompt_token_ids=prompt_token_ids(request_id, args.prompt_tokens, vocab_size) + ) + submitted = time.monotonic() + queue = await engine.add_request(request_id, prompt, sampling, data_parallel_rank=rank) + queues.append((request_id, index, rank, submitted, queue)) + for request_id, index, rank, submitted, queue in queues: + output = await queue.get() + while not output.finished: + output = await queue.get() + records.append({ + "request_id": request_id, "burst": label, "round": round_index, + "index": index, "rank": rank, "submit_monotonic": submitted, + "finish_monotonic": time.monotonic(), + "num_prompt_tokens": len(output.prompt_token_ids), + "num_output_tokens": len(output.outputs[0].token_ids), + "finish_reason": output.outputs[0].finish_reason, + }) + offset_after = time.time() - time.monotonic() + return {"label": label, "round": round_index, "num_requests": num_requests, + "wall_minus_monotonic_before": offset_before, + "wall_minus_monotonic_after": offset_after, + "idle_wait_s": await wait_until_idle()} + + rounds = [] + try: + rounds.append(await burst("warmup", 0, args.warmups)) + for num_requests in args.bursts: + for round_index in range(args.rounds): + rounds.append(await burst(f"b{num_requests}-r{round_index}", round_index, num_requests)) + cache_config = engine.vllm_config.cache_config + summary = { + "model_config": args.model_config, + "num_gpu_blocks": cache_config.num_gpu_blocks, + "block_size": cache_config.block_size, + "engine_args": {key: value for key, value in vars(engine_args).items() + if isinstance(value, (bool, int, float, str, type(None)))}, + "rounds": rounds, + } + finally: + engine.shutdown() + (output_dir / "requests.jsonl").write_text("".join(json.dumps(row) + "\n" for row in records)) + return summary + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0]) + commands = parser.add_subparsers(dest="command", required=True) + overlay = commands.add_parser("overlay") + overlay.add_argument("--site-vllm", type=Path, required=True) + overlay.add_argument("--checkout", type=Path, required=True) + overlay.add_argument("--destination", type=Path, required=True) + overlay.add_argument("--expected-changes", type=Path, required=True) + overlay.add_argument("--report", type=Path, required=True) + overlay.add_argument("--patch", type=Path) + run = commands.add_parser("run") + run.add_argument("--model-config", required=True) + run.add_argument("--output-dir", required=True) + run.add_argument("--enable-expert-parallel", action="store_true") + run.add_argument("--data-parallel-size", type=int, default=2) + run.add_argument("--pipeline-parallel-size", type=int, default=2) + run.add_argument("--prompt-tokens", type=int, default=256) + run.add_argument("--max-num-seqs", type=int, default=4) + run.add_argument("--max-model-len", type=int, default=512) + run.add_argument("--gpu-memory-utilization", type=float, default=0.5) + run.add_argument("--bursts", type=int, nargs="+", default=[8, 16]) + run.add_argument("--rounds", type=int, default=3) + run.add_argument("--warmups", type=int, default=4) + run.add_argument("--idle-gap-s", type=float, default=1.0) + run.add_argument("--idle-timeout-s", type=float, default=60.0) + args = parser.parse_args(argv) + + if args.command == "overlay": + report = build_overlay(args.site_vllm, args.checkout, args.destination, args.expected_changes) + if report["accepted"] and args.patch is not None: + files = apply_patch(args.patch, args.destination) + report["patch"] = { + "path": str(args.patch), + "sha256": hashlib.sha256(args.patch.read_bytes()).hexdigest(), + "files": files, + "equal_to_image_after_patch": { + name: filecmp.cmp(args.destination / name, args.site_vllm.parent / name, shallow=False) + for name in files + }, + } + args.report.write_text(json.dumps(report, indent=1)) + print(json.dumps({key: report[key] for key in ("accepted", "unexpected", "missing")})) + return 0 if report["accepted"] else 3 + Path(args.output_dir).mkdir(parents=True, exist_ok=True) + summary = asyncio.run(run_bursts(args)) + Path(args.output_dir, "summary.json").write_text(json.dumps(summary, indent=1)) + print("DRIVER_DONE", json.dumps({"rounds": len(summary["rounds"]), + "num_gpu_blocks": summary["num_gpu_blocks"]})) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/e2e/stage_admission_matrix.py b/tests/e2e/stage_admission_matrix.py new file mode 100644 index 00000000..c872b011 --- /dev/null +++ b/tests/e2e/stage_admission_matrix.py @@ -0,0 +1,833 @@ +#!/usr/bin/env python3 +"""Case matrix for stage admission of attention-DP lanes under pipeline parallelism. + +Runs the case list of ``build_cases`` on the current source tree and writes, +for each case, its inputs (``case.json``), its run provenance (``run.json``) +and one outcome artifact: + +* ``success``: ``sha256sums.txt`` over the copied metrics tree; +* ``admission_deadlock``: ``state_report.json`` read from the live scheduler + objects after the sequential run ends with work left; +* ``configuration_rejection`` / ``other_failure``: ``error.txt``. + +Each case runs in its own child process because ``IS_MOE`` is process-global, +and each child runs in its own session so a case that exceeds +``--case-timeout`` is killed with everything it started. Every child writes +its simulator output under ``/work/``, a path shared by all +sets, so that files embedding the output path compare byte for byte between a +set run before a change and one run after it. Sets therefore run one at a +time: ``run`` holds an exclusive lock on the matrix root. + +Usage:: + + python -m tests.e2e.stage_admission_matrix run --set base [--group G3a] + python -m tests.e2e.stage_admission_matrix compare --before base --after after +""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import fcntl +import os +import shutil +import signal +import subprocess +import sys +import time +import traceback +from collections import defaultdict +from concurrent.futures import ThreadPoolExecutor +from dataclasses import asdict, dataclass +from importlib import metadata as importlib_metadata +from pathlib import Path +from typing import Iterable, Sequence + +from tests.scratch_root import resolve_scratch_root + + +REPO_ROOT = Path(__file__).resolve().parents[2] +MATRIX_DIR_NAME = "stage_admission_ordering" +DRAIN_MESSAGE = "Sequential simulation ended with non-empty scheduler state" +ATTN_DP_LANE = "ATTN_DP_LANE" + +SUCCESS = "success" +ADMISSION_DEADLOCK = "admission_deadlock" +CONFIGURATION_REJECTION = "configuration_rejection" +OTHER_FAILURE = "other_failure" + +# Plan §4.1 synthetic fixture and §4.7 vLLM-aligned fixture. +SYNTHETIC = "synthetic" +VLLM_ALIGNED = "vllm_aligned" +VLLM_ALIGNED_MODELS = {True: "Qwen3-30B-A3B-tiny", False: "Llama-3.2-1B-Instruct"} + +PREFILL_ONLY = (16, 1) +PREFILL_DECODE = (16, 3) +VLLM_ALIGNED_PREFILL_ONLY = (256, 1) +ONLINE_QPS = 20.0 +ONLINE_QPS_SWEEP = (5.0, 80.0) +DEFAULT_CASE_TIMEOUT_S = 600 + + +@dataclass(frozen=True) +class Case: + case_id: str + group: str + fixture: str = SYNTHETIC + is_moe: bool = False + attn_dp: int = 1 + stages: int = 1 + num_requests: int = 0 + prefill_tokens: int = 0 + decode_tokens: int = 0 + arrival: str = "static" + qps: float = 1e6 + sys_arch: str = "co-location" + simulation_mode: str = "offline" + cc_backend: str = "analytical" + recipe: str | None = None + recipe_env: tuple[tuple[str, str], ...] = () + contention_witness: bool = False + + @property + def path(self) -> str: + """Plan §4.2 path before P0 classification refines L/T.""" + if self.recipe is not None or self.stages == 1 or self.attn_dp == 1: + return "U" + return "LT" + + +def _shape_id(group: str, is_moe: bool, attn_dp: int, stages: int, num_requests: int) -> str: + kind = "moe" if is_moe else "dense" + return f"{group}-{kind}-dp{attn_dp}-pp{stages}-n{num_requests}" + + +def _synthetic(group: str, is_moe: bool, attn_dp: int, stages: int, num_requests: int, + lengths: tuple[int, int], suffix: str = "", **fields) -> Case: + return Case( + case_id=_shape_id(group, is_moe, attn_dp, stages, num_requests) + suffix, + group=group, + is_moe=is_moe, + attn_dp=attn_dp, + stages=stages, + num_requests=num_requests, + prefill_tokens=lengths[0], + decode_tokens=lengths[1], + **fields, + ) + + +def _release_recipes() -> list[Case]: + architecture_root = REPO_ROOT / "examples" / "architecture" + recipes = [] + for architecture in ("co-location", "pdd", "pd-af-disagg"): + for mode in ("offline", "online"): + for script in sorted((architecture_root / architecture / mode).glob("*.sh")): + recipes.append( + Case( + case_id=f"G1-{architecture}-{mode}-{script.stem}", + group="G1", + recipe=str(script.relative_to(REPO_ROOT)), + ) + ) + return recipes + + +def build_cases() -> list[Case]: + """Return the plan §4.2 case list in a fixed order.""" + cases: list[Case] = [] + # R0 reproduces the author-reported shapes of design.md with their inputs: + # Poisson arrivals, the default CC backend and prefill 16 / decode 3. + r0_shapes = [ + (True, 2, 2, 3), (True, 2, 2, 4), (True, 2, 2, 6), (True, 4, 2, 8), + (True, 2, 1, 6), (True, 2, 1, 12), (True, 4, 1, 8), (True, 4, 1, 12), + (True, 1, 2, 6), (True, 1, 3, 6), + (False, 2, 2, 6), (False, 4, 2, 8), (False, 2, 1, 6), (False, 4, 1, 8), + (False, 1, 2, 6), (True, 2, 3, 6), + ] + for is_moe, attn_dp, stages, num_requests in r0_shapes: + cases.append( + _synthetic("R0", is_moe, attn_dp, stages, num_requests, PREFILL_DECODE, + arrival="poisson", cc_backend="default") + ) + cases.extend(_release_recipes()) + for attn_dp in (2, 4): + for stages in (1, 2, 3): + for num_requests in (4, 8, 12): + cases.append(_synthetic("G3a", True, attn_dp, stages, num_requests, PREFILL_ONLY)) + for attn_dp in (2, 4): + for stages in (1, 2, 3): + for num_requests in (4, 8): + cases.append(_synthetic("G3b", True, attn_dp, stages, num_requests, PREFILL_DECODE)) + for attn_dp in (2, 4): + for stages in (1, 2, 3): + for num_requests in (4, 8): + cases.append( + _synthetic("G4", False, attn_dp, stages, num_requests, PREFILL_DECODE, + contention_witness=stages > 1 and num_requests == 8) + ) + for is_moe in (True, False): + for stages in (1, 2, 3): + cases.append(_synthetic("G5", is_moe, 1, stages, 6, PREFILL_DECODE)) + for is_moe in (True, False): + for num_requests in (8, 16): + cases.append( + _synthetic("G7", is_moe, 2, 2, num_requests, VLLM_ALIGNED_PREFILL_ONLY, + fixture=VLLM_ALIGNED) + ) + # Plan §7 (R2-03). PDD admits multi-lane contexts only for MoE: dense PDD + # requires attn_dp == 1. Online cells use Poisson arrivals at ONLINE_QPS. + # On main, MONOLITHIC and PREFILL place every request of one scheduling + # call from lane 0 on, so incremental online arrivals all land on lane 0; + # the "-burst" cells deliver all requests at t=0 to reach several lanes. + pdd_shapes = [(True, attn_dp, stages) for attn_dp in (2, 4) for stages in (1, 2, 3)] + pdd_shapes.append((False, 1, 2)) + online = dict(simulation_mode="online", arrival="poisson", qps=ONLINE_QPS) + burst = dict(simulation_mode="online", suffix="-burst") + for group, timing in (("G8", {}), ("G9", online)): + for is_moe, attn_dp, stages in pdd_shapes: + cases.append( + _synthetic(group, is_moe, attn_dp, stages, 8, PREFILL_DECODE, + sys_arch="pd-disaggregation", **timing) + ) + for attn_dp in (2, 4): + for stages in (2, 3): + cases.append( + _synthetic("G9", True, attn_dp, stages, 8, PREFILL_DECODE, + sys_arch="pd-disaggregation", **burst) + ) + for is_moe, lengths in ((True, PREFILL_ONLY), (False, PREFILL_DECODE)): + for attn_dp in (2, 4): + for stages in (1, 2, 3): + cases.append(_synthetic("G10", is_moe, attn_dp, stages, 8, lengths, **online)) + cases.append(_synthetic("G10", is_moe, attn_dp, stages, 8, lengths, **burst)) + for qps in ONLINE_QPS_SWEEP: + cases.append( + _synthetic("G10", is_moe, 2, 2, 8, lengths, suffix=f"-q{qps:g}", + **dict(online, qps=qps)) + ) + # PD-AF contexts have capacity 1 (DECODE_ATTN requires attn_dp == 1), so + # these are unchanged controls for PREFILL pipeline stages. + for mode in ("offline", "online"): + suffix = "_online" if mode == "online" else "" + for stem in ("dense_model_basic", "moe_model_basic"): + script = f"examples/architecture/pd-af-disagg/{mode}/{stem}{suffix}.sh" + cases.append( + Case(case_id=f"G11-pd-af-disagg-{mode}-{stem}-pp2", group="G11", + recipe=script, recipe_env=(("PREFILL_PP", "2"),)) + ) + return cases + + +# --------------------------------------------------------------------------- +# Fixture (runs inside the child process) +# --------------------------------------------------------------------------- + + +def _synthetic_model(is_moe: bool): + from frontier.config import BaseModelConfig + from frontier.types import ActivationType, NormType + + model = BaseModelConfig( + num_layers=6, num_q_heads=4, num_kv_heads=2, embedding_dim=256, + mlp_hidden_dim=64, max_position_embeddings=4096, use_gated_mlp=True, + use_bias=False, use_qkv_bias=False, activation=ActivationType.SILU, + norm=NormType.RMS_NORM, post_attn_norm=True, vocab_size=1024, + is_moe=is_moe, num_experts=8 if is_moe else 0, + num_experts_per_tok=2 if is_moe else 0, torch_dtype="bfloat16", + ) + model._model_name = f"stage_admission_{'moe' if is_moe else 'dense'}" + registered = BaseModelConfig.create_from_name.__func__ + BaseModelConfig.create_from_name = classmethod( + lambda cls, name: model if name == model._model_name else registered(cls, name) + ) + return model._model_name + + +def build_config(case: Case, output_dir: Path, cache_dir: Path): + """Build the SimulationConfig of one synthetic or vLLM-aligned case.""" + from frontier.cc_backend.cc_backend_config import AnalyticalCCBackendConfig + from frontier.config import ( + ClusterConfig, FixedRequestLengthGeneratorConfig, MetricsConfig, + PoissonRequestIntervalGeneratorConfig, RandomForrestExecutionTimePredictorConfig, + ReplicaConfig, RoundRobinClusterSchedulerConfig, SimulationConfig, + StaticRequestIntervalGeneratorConfig, SyntheticRequestGeneratorConfig, + VllmV1SchedulerConfig, + ) + + if case.fixture == SYNTHETIC: + model_name = _synthetic_model(case.is_moe) + device, network_device = "a100", "a100_pairwise_nvlink" + scheduler = VllmV1SchedulerConfig( + num_blocks=128, block_size=16, batch_size_cap=4, + max_tokens_in_batch=16, enable_chunked_prefill=True, + ) + elif case.fixture == VLLM_ALIGNED: + model_name = VLLM_ALIGNED_MODELS[case.is_moe] + device, network_device = "h800", "h800_dgx" + scheduler = VllmV1SchedulerConfig( + num_blocks=1024, block_size=16, batch_size_cap=4, + max_tokens_in_batch=case.prefill_tokens, enable_chunked_prefill=True, + ) + else: + raise ValueError(f"unknown fixture {case.fixture!r}") + + moe_fields = ( + dict(moe_tensor_parallel_size=1, moe_expert_parallel_size=case.attn_dp) + if case.is_moe else {} + ) + replica = ReplicaConfig( + model_name=model_name, device=device, network_device=network_device, + num_pipeline_stages=case.stages, attn_tensor_parallel_size=1, + attn_dp=case.attn_dp, memory_margin_fraction=0.1, **moe_fields, + ) + cluster_fields = {} + if case.cc_backend == "analytical": + cluster_fields["cc_backend_config"] = AnalyticalCCBackendConfig() + elif case.cc_backend != "default": + raise ValueError(f"unknown CC backend selector {case.cc_backend!r}") + if case.sys_arch == "pd-disaggregation": + # One Replica per role; both roles take the fixture's replica config. + cluster_fields.update(prefill_cluster_num_replicas=1, decode_cluster_num_replicas=1) + cluster = ClusterConfig( + replica_config=replica, + replica_scheduler_config=scheduler, + cluster_scheduler_config=RoundRobinClusterSchedulerConfig(), + execution_time_predictor_config=RandomForrestExecutionTimePredictorConfig( + enable_dummy_mode=True + ), + **cluster_fields, + ) + if case.arrival == "static": + interval = StaticRequestIntervalGeneratorConfig() + elif case.arrival == "poisson": + interval = PoissonRequestIntervalGeneratorConfig(qps=case.qps) + else: + raise ValueError(f"unknown arrival process {case.arrival!r}") + return SimulationConfig( + simulation_mode=case.simulation_mode, sys_arch=case.sys_arch, + enable_parallel_clusters=False, decode_cuda_graph_mode="none", + cluster_config=cluster, + metrics_config=MetricsConfig( + output_dir=str(output_dir), cache_dir=str(cache_dir), + run_id=case.case_id, write_metrics=True, store_request_metrics=True, + store_plots=False, enable_chrome_trace=False, write_json_trace=False, + ), + request_generator_config=SyntheticRequestGeneratorConfig( + num_requests=case.num_requests, + length_generator_config=FixedRequestLengthGeneratorConfig( + prefill_tokens=case.prefill_tokens, decode_tokens=case.decode_tokens, + ), + interval_generator_config=interval, + ), + ) + + +# --------------------------------------------------------------------------- +# Drain state and outcome classification (child process) +# --------------------------------------------------------------------------- + + +def _ticket_view(ticket) -> dict: + return { + "admission_seq": ticket.admission_seq, + "operation_id": str(ticket.operation_id), + "scope": ticket.scope, + } + + +def build_state_report(simulator) -> dict: + """Read stage contexts, lane queues and sync rooms of every cluster after a drain.""" + lanes = {} + contexts = [] + rooms = [] + for cluster_type, cluster_scheduler in simulator.scheduler._cluster_schedulers.items(): + _read_cluster_state(cluster_type.name, cluster_scheduler, lanes, contexts, rooms) + return { + "simulation_time": simulator._time, + "contexts": contexts, + "lanes": lanes, + "sync_rooms": rooms, + } + + +def _read_cluster_state(cluster: str, cluster_scheduler, lanes: dict, + contexts: list, rooms: list) -> None: + queued_owner = {} + for (replica_id, lane_id), replica_scheduler in sorted( + cluster_scheduler._replica_schedulers.items(), key=lambda item: str(item[0]) + ): + stage_views = [] + for stage_id in range(replica_scheduler._num_stages): + stage = replica_scheduler.get_replica_stage_scheduler(stage_id) + heap = [] + for batch in stage.get_queue_batches(): + ticket = batch._stage_admission_ticket + queued_owner[(replica_id, stage_id, ticket.admission_seq)] = lane_id + heap.append({"batch_id": batch.id, "global_id": batch.global_id, + **_ticket_view(ticket)}) + stage_views.append({"busy": stage.is_busy, "heap": heap}) + lanes[f"{cluster}/{replica_id}/{lane_id}"] = { + "cluster": cluster, "replica_id": replica_id, "lane": lane_id, + "stages": stage_views, + } + + for (replica_id, stage_id), context in sorted(cluster_scheduler._stage_execution_contexts.items()): + contexts.append({ + "cluster": cluster, + "replica_id": replica_id, + "stage_id": stage_id, + "capacity": context.full_stage_capacity, + "sealed": context.forward_group_sealed, + "bound_group": context._forward_group_id, + "ep_wave_active": context._active_ep_ticket is not None, + "active_full_stage": sorted( + (_ticket_view(ticket) for ticket in context._active_full_stage_tickets), + key=lambda view: view["admission_seq"], + ), + "fifo": [ + {**_ticket_view(ticket), + "lane": queued_owner.get((replica_id, stage_id, ticket.admission_seq))} + for ticket in context.queued_tickets + ], + }) + + for room_name in ("_prefill_sync_waiting_room", "_decode_sync_waiting_room"): + by_replica = getattr(cluster_scheduler, room_name) or {} + for replica_id, by_stage in by_replica.items(): + for stage_id, by_step in by_stage.items(): + for step, by_layer in by_step.items(): + for layer, by_sync in by_layer.items(): + for sync_stage, room in by_sync.items(): + if not room["batches"]: + continue + rooms.append({ + "cluster": cluster, "room": room_name.strip("_"), + "replica_id": replica_id, "stage_id": stage_id, + "step": step, "layer": layer, "sync_stage": str(sync_stage), + "lanes_present": sorted(room["batches"]), + }) + + +def has_admission_deadlock_signature(report: dict) -> bool: + """Plan §4.3: a busy lane's queued ticket heads the FIFO while an idle lane + with queued work, needed by that lane's sync room, is refused behind it.""" + lanes = report["lanes"] + for context in report["contexts"]: + if (context["ep_wave_active"] or context["sealed"] or not context["fifo"] + or len(context["active_full_stage"]) >= context["capacity"]): + continue + head = context["fifo"][0] + if head["scope"] != "FULL_STAGE_WORLD" or head["lane"] is None: + continue + cluster, replica_id, stage_id = context["cluster"], context["replica_id"], context["stage_id"] + head_stage = lanes[f"{cluster}/{replica_id}/{head['lane']}"]["stages"][stage_id] + if not head_stage["busy"]: + continue + for lane in lanes.values(): + if ((lane["cluster"], lane["replica_id"]) != (cluster, replica_id) + or lane["lane"] == head["lane"]): + continue + stage = lane["stages"][stage_id] + if stage["busy"] or not stage["heap"]: + continue + if stage["heap"][0]["admission_seq"] <= head["admission_seq"]: + continue + for room in report["sync_rooms"]: + if ((room["cluster"], room["replica_id"], room["stage_id"]) + == (cluster, replica_id, stage_id) + and head["lane"] in room["lanes_present"] + and lane["lane"] not in room["lanes_present"]): + return True + return False + + +def _run_simulator_case(case: Case, work_dir: Path, case_dir: Path) -> dict: + output_root = work_dir / "metrics" + try: + config = build_config(case, output_root, work_dir / "cache") + from frontier.simulator import Simulator + + simulator = Simulator(config) + except ValueError as exc: + (case_dir / "error.txt").write_text(traceback.format_exc()) + return {"outcome": CONFIGURATION_REJECTION, "exception": repr(exc)} + resolved_config = json.loads( + (Path(config.metrics_config.output_dir) / "config.json").read_text() + ) + (case_dir / "resolved_config.json").write_text(json.dumps(resolved_config, indent=1, sort_keys=True)) + try: + simulator.run() + except RuntimeError as exc: + if not str(exc).startswith(DRAIN_MESSAGE): + (case_dir / "error.txt").write_text(traceback.format_exc()) + return {"outcome": OTHER_FAILURE, "exception": repr(exc)[:2000]} + report = build_state_report(simulator) + (case_dir / "state_report.json").write_text(json.dumps(report, indent=1, sort_keys=True)) + outcome = ADMISSION_DEADLOCK if has_admission_deadlock_signature(report) else OTHER_FAILURE + return {"outcome": outcome, "exception": DRAIN_MESSAGE, + "simulation_time": report["simulation_time"]} + except Exception as exc: # classified, never converted into success + (case_dir / "error.txt").write_text(traceback.format_exc()) + return {"outcome": OTHER_FAILURE, "exception": repr(exc)[:2000]} + requests = list(simulator._all_requests) + completed = sum(1 for request in requests if request.completed) + if completed != len(requests): + (case_dir / "error.txt").write_text( + f"run returned with {completed} of {len(requests)} requests completed\n" + ) + return {"outcome": OTHER_FAILURE, "exception": "incomplete requests"} + return {"outcome": SUCCESS, "completed_requests": completed, + "metrics_run_dir": str(Path(config.metrics_config.output_dir).relative_to(work_dir))} + + +def _run_recipe_case(case: Case, work_dir: Path, case_dir: Path) -> dict: + env = dict(os.environ) + env.update({ + "PYTHON_BIN": sys.executable, + "METRICS_OUTPUT_DIR": str(work_dir / "metrics"), + "RUN_ID": case.case_id, + **dict(case.recipe_env), + }) + result = subprocess.run( + ["bash", str(REPO_ROOT / case.recipe)], cwd=REPO_ROOT, env=env, + capture_output=True, text=True, + ) + (case_dir / "stdout.log").write_text(result.stdout[-200_000:] + result.stderr[-200_000:]) + if result.returncode != 0: + (case_dir / "error.txt").write_text(result.stderr[-50_000:]) + return {"outcome": OTHER_FAILURE, "exception": f"exit code {result.returncode}"} + return {"outcome": SUCCESS} + + +def run_case_in_child(case: Case, root: Path, set_name: str) -> None: + """Child entry point: run one case, then publish its artifacts.""" + case_dir = root / set_name / case.case_id + work_dir = root / "work" / case.case_id + for directory in (case_dir, work_dir): + if directory.exists(): + shutil.rmtree(directory) + directory.mkdir(parents=True) + started = time.time() + if case.recipe is None: + result = _run_simulator_case(case, work_dir, case_dir) + else: + result = _run_recipe_case(case, work_dir, case_dir) + result["wall_start"] = started + result["wall_end"] = time.time() + if result["outcome"] == SUCCESS: + shutil.copytree(work_dir / "metrics", case_dir / "metrics") + (case_dir / "sha256sums.txt").write_text(sha256_lines(case_dir / "metrics")) + shutil.rmtree(work_dir) + (case_dir / "outcome.json").write_text(json.dumps(result, indent=1, sort_keys=True)) + + +def sha256_lines(directory: Path) -> str: + lines = [] + for path in sorted(p for p in directory.rglob("*") if p.is_file()): + digest = hashlib.sha256(path.read_bytes()).hexdigest() + lines.append(f"{digest} {path.relative_to(directory)}") + return "\n".join(lines) + "\n" + + +# --------------------------------------------------------------------------- +# Lane-overlap metric (plan §4.5) +# --------------------------------------------------------------------------- + + +def interval_overlap(intervals: Iterable[tuple[float, float, object]]) -> dict: + """Half-open ``[start, end)`` intervals keyed by lane. + + Returns the time with at least one open lane, the time with at least two + distinct open lanes, the peak number of distinct open lanes, the latest + end, and whether any lane overlaps itself. + """ + intervals = list(intervals) + events = [] + for start, end, lane in intervals: + if end > start: + events.append((start, 1, lane)) + events.append((end, -1, lane)) + # Ends sort before starts at the same instant: touching intervals do not overlap. + events.sort(key=lambda event: (event[0], event[1])) + open_by_lane: dict[object, int] = defaultdict(int) + busy_time = multi_lane_time = 0.0 + peak_lanes = 0 + self_overlap = False + previous_time = None + for event_time, delta, lane in events: + if previous_time is not None: + open_lanes = sum(1 for count in open_by_lane.values() if count > 0) + if open_lanes >= 1: + busy_time += event_time - previous_time + if open_lanes >= 2: + multi_lane_time += event_time - previous_time + open_by_lane[lane] += delta + if open_by_lane[lane] > 1: + self_overlap = True + peak_lanes = max(peak_lanes, sum(1 for count in open_by_lane.values() if count > 0)) + previous_time = event_time + return { + "busy_time": busy_time, + "multi_lane_busy_time": multi_lane_time, + "peak_lanes": peak_lanes, + "makespan": max((end for _, end, _ in intervals), default=0.0), + "self_overlap": self_overlap, + } + + +def read_ledger(metrics_dir: Path) -> list[dict]: + paths = sorted(metrics_dir.rglob("frontier_stage_batch_ledger.jsonl")) + if len(paths) != 1: + raise ValueError(f"expected one stage ledger under {metrics_dir}, found {len(paths)}") + return [json.loads(line) for line in paths[0].read_text().splitlines() if line.strip()] + + +def lane_intervals(rows: Sequence[dict]) -> dict[tuple, list[tuple[float, float, int]]]: + """``ATTN_DP_LANE`` ledger intervals per physical stage.""" + by_stage: dict[tuple, list[tuple[float, float, int]]] = defaultdict(list) + for row in rows: + if row["execution_scope"] != ATTN_DP_LANE: + continue + key = (row["cluster_type"], row["replica_id"], row["stage_id"]) + by_stage[key].append((row["stage_start_ts"], row["stage_end_ts"], row["replica_local_id"])) + return dict(by_stage) + + +def ledger_lane_metric(metrics_dir: Path) -> dict: + """Plan §4.5 metric per physical stage, keyed ``cluster/replica/stage``.""" + stages = {} + for (cluster_type, replica_id, stage_id), intervals in sorted(lane_intervals(read_ledger(metrics_dir)).items()): + metric = interval_overlap(intervals) + metric["lanes"] = sorted({lane for _, _, lane in intervals}) + stages[f"{cluster_type}/{replica_id}/{stage_id}"] = metric + return stages + + +# --------------------------------------------------------------------------- +# Parent: set runner and provenance +# --------------------------------------------------------------------------- + + +def _git(*args: str) -> str: + return subprocess.run(["git", "-C", str(REPO_ROOT), *args], check=True, + capture_output=True, text=True).stdout.strip() + + +def set_provenance() -> dict: + distributions = sorted( + f"{dist.metadata['Name']}=={dist.version}" for dist in importlib_metadata.distributions() + ) + status = _git("status", "--porcelain", "--", ".", ":!task_memory") + return { + "interpreter": sys.executable, + "python_vv": subprocess.run([sys.executable, "-VV"], check=True, + capture_output=True, text=True).stdout.strip(), + "distributions_sha256": hashlib.sha256("\n".join(distributions).encode()).hexdigest(), + "git_head": _git("rev-parse", "HEAD"), + "clean_outside_task_memory": status == "", + "status_outside_task_memory": status, + } + + +def matrix_root() -> Path: + return resolve_scratch_root() / MATRIX_DIR_NAME + + +def _run_one(case: Case, root: Path, set_name: str, provenance: dict, + case_timeout: float) -> dict: + command = [sys.executable, "-m", "tests.e2e.stage_admission_matrix", "child", + "--set", set_name, "--case", case.case_id] + env = dict(os.environ, PYTHONPATH=str(REPO_ROOT), WANDB_DISABLED="true", + VIDUR_DISABLE_WANDB="1") + child = subprocess.Popen(command, cwd=REPO_ROOT, env=env, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, text=True, start_new_session=True) + try: + stdout, stderr = child.communicate(timeout=case_timeout) + failure = f"child exit code {child.returncode}" + except subprocess.TimeoutExpired: + os.killpg(child.pid, signal.SIGKILL) + stdout, stderr = child.communicate() + failure = f"case timeout after {case_timeout:g} s" + case_dir = root / set_name / case.case_id + case_dir.mkdir(parents=True, exist_ok=True) + outcome_path = case_dir / "outcome.json" + if child.returncode != 0 or not outcome_path.exists(): + (case_dir / "error.txt").write_text(stdout[-50_000:] + stderr[-50_000:]) + outcome = {"outcome": OTHER_FAILURE, "exception": failure} + else: + outcome = json.loads(outcome_path.read_text()) + (case_dir / "case.json").write_text(json.dumps(asdict(case), indent=1, sort_keys=True)) + run = {"command": command, **provenance, **outcome} + (case_dir / "run.json").write_text(json.dumps(run, indent=1, sort_keys=True)) + return {"case_id": case.case_id, "group": case.group, "outcome": outcome["outcome"], + "exception": outcome.get("exception")} + + +def run_set(set_name: str, cases: Sequence[Case], jobs: int, case_timeout: float) -> list[dict]: + root = matrix_root() + (root / set_name).mkdir(parents=True, exist_ok=True) + with (root / "run.lock").open("w") as lock: + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + raise RuntimeError( + f"another set is running under {root}; sets share work/ and run one at a time" + ) from None + return _run_locked_set(root, set_name, cases, jobs, case_timeout) + + +def _run_locked_set(root: Path, set_name: str, cases: Sequence[Case], jobs: int, + case_timeout: float) -> list[dict]: + provenance = set_provenance() + with ThreadPoolExecutor(max_workers=jobs) as pool: + rows = list(pool.map( + lambda case: _run_one(case, root, set_name, provenance, case_timeout), cases + )) + if set_provenance()["git_head"] != provenance["git_head"]: + raise RuntimeError("git HEAD changed while the set was running") + index = root / set_name / "cases.jsonl" + existing = {} + if index.exists(): + existing = {row["case_id"]: row for row in map(json.loads, index.read_text().splitlines())} + existing.update({row["case_id"]: row for row in rows}) + order = [case.case_id for case in build_cases()] + index.write_text("".join(json.dumps(existing[case_id]) + "\n" + for case_id in order if case_id in existing)) + return rows + + +# --------------------------------------------------------------------------- +# Parent: before/after comparison (plan §4.4) +# --------------------------------------------------------------------------- + + +def _case_state(root: Path, set_name: str, case_id: str) -> dict: + case_dir = root / set_name / case_id + run = json.loads((case_dir / "run.json").read_text()) + state = {"outcome": run["outcome"]} + if run["outcome"] == SUCCESS: + state["sha256sums"] = (case_dir / "sha256sums.txt").read_text() + if (case_dir / "metrics").exists(): + state["metrics_dir"] = case_dir / "metrics" + return state + + +def _conservation(case: Case, metrics_dir: Path) -> dict: + paths = sorted(metrics_dir.rglob("request_metrics.csv")) + if len(paths) != 1: + return {"ok": False, "reason": f"{len(paths)} request_metrics.csv files"} + with paths[0].open() as handle: + rows = list(csv.DictReader(handle)) + prefill = sum(int(float(row["request_num_prefill_tokens"])) for row in rows) + decode = sum(int(float(row["request_num_decode_tokens"])) for row in rows) + expected = (case.num_requests, case.num_requests * case.prefill_tokens, + case.num_requests * case.decode_tokens) + observed = (len(rows), prefill, decode) + return {"ok": observed == expected, "observed": observed, "expected": expected} + + +def _differing_files(before: str, after: str) -> list[str]: + def parse(text): + return dict(reversed(line.split(" ", 1)) for line in text.splitlines() if line) + left, right = parse(before), parse(after) + return sorted(name for name in set(left) | set(right) if left.get(name) != right.get(name)) + + +def compare_sets(before: str, after: str) -> list[dict]: + root = matrix_root() + rows = [] + for case in build_cases(): + if not (root / before / case.case_id / "run.json").exists(): + continue + base = _case_state(root, before, case.case_id) + new = _case_state(root, after, case.case_id) + row = {"case_id": case.case_id, "group": case.group, + "before": base["outcome"], "after": new["outcome"]} + if case.group == "R0": + row.update(path="R0", verdict="informational") + elif case.path == "U": + row["path"] = "U" + identical = base["outcome"] == new["outcome"] == SUCCESS and base["sha256sums"] == new["sha256sums"] + row["verdict"] = "PASS" if identical else "STOP" + if base["outcome"] == new["outcome"] == SUCCESS and not identical: + row["differing_files"] = _differing_files(base["sha256sums"], new["sha256sums"]) + elif base["outcome"] == ADMISSION_DEADLOCK: + row["path"] = "L" + if new["outcome"] == SUCCESS: + row["conservation"] = _conservation(case, new["metrics_dir"]) + row["verdict"] = "PASS" if row["conservation"]["ok"] else "STOP" + else: + row["verdict"] = "STOP" + elif base["outcome"] == SUCCESS: + row["path"] = "T" + if new["outcome"] != SUCCESS: + row["verdict"] = "STOP" + else: + before_metric = ledger_lane_metric(base["metrics_dir"]) + after_metric = ledger_lane_metric(new["metrics_dir"]) + row["lane_metric_before"] = before_metric + row["lane_metric_after"] = after_metric + checks_ok = all( + not stage["self_overlap"] and stage["peak_lanes"] <= case.attn_dp + for stage in after_metric.values() + ) + identical = base["sha256sums"] == new["sha256sums"] + if not identical: + row["differing_files"] = _differing_files(base["sha256sums"], new["sha256sums"]) + if case.contention_witness: + # The fraction, not the absolute overlap: admitting more lanes + # together also shortens the busy period. + fraction = lambda metric: (sum(stage["multi_lane_busy_time"] for stage in metric.values()) + / sum(stage["busy_time"] for stage in metric.values())) + row["witness_increase"] = fraction(after_metric) > fraction(before_metric) + checks_ok = checks_ok and row["witness_increase"] + row["verdict"] = ("PASS" if identical and checks_ok + else "EXPLAIN" if checks_ok else "STOP") + else: + row.update(path="class", verdict="STOP") + rows.append(row) + return rows + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0]) + commands = parser.add_subparsers(dest="command", required=True) + run_parser = commands.add_parser("run", help="run cases into one named set") + run_parser.add_argument("--set", required=True) + run_parser.add_argument("--group", action="append", default=[]) + run_parser.add_argument("--case", action="append", default=[]) + run_parser.add_argument("--jobs", type=int, default=8) + run_parser.add_argument("--case-timeout", type=float, default=DEFAULT_CASE_TIMEOUT_S, + help="seconds before a case's child session is killed") + child_parser = commands.add_parser("child", help=argparse.SUPPRESS) + child_parser.add_argument("--set", required=True) + child_parser.add_argument("--case", required=True) + compare_parser = commands.add_parser("compare", help="apply the plan §4.4 paths") + compare_parser.add_argument("--before", required=True) + compare_parser.add_argument("--after", required=True) + compare_parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args(argv) + + cases_by_id = {case.case_id: case for case in build_cases()} + if args.command == "child": + run_case_in_child(cases_by_id[args.case], matrix_root(), args.set) + return 0 + if args.command == "compare": + rows = compare_sets(args.before, args.after) + args.output.write_text(json.dumps(rows, indent=1, sort_keys=True, default=str)) + for row in rows: + print(f"{row['case_id']:<48} {row['path']:<5} {row['before']:<24} {row['after']:<24} {row['verdict']}") + return 0 + selected = [case for case in cases_by_id.values() + if (not args.group or case.group in args.group) + and (not args.case or case.case_id in args.case)] + for row in run_set(args.set, selected, args.jobs, args.case_timeout): + print(f"{row['case_id']:<48} {row['outcome']}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/integration/test_stage_admission_pipeline_lanes.py b/tests/integration/test_stage_admission_pipeline_lanes.py new file mode 100644 index 00000000..11970c18 --- /dev/null +++ b/tests/integration/test_stage_admission_pipeline_lanes.py @@ -0,0 +1,65 @@ +"""Simulator regression for attention-DP lanes sharing pipeline stages.""" + +import csv +import json +import os +from pathlib import Path +import subprocess +import sys + +import pytest + +from tests.e2e.stage_admission_matrix import ATTN_DP_LANE, SUCCESS, build_cases, read_ledger, run_case_in_child + +REPO_ROOT = Path(__file__).resolve().parents[2] +SET_NAME = "test" + + +def run_case(root, case_id): + """Run one matrix case in its own process, because ``IS_MOE`` is process-global.""" + result = subprocess.run( + [sys.executable, str(Path(__file__).resolve()), str(root), case_id], + cwd=REPO_ROOT, + env={**os.environ, "PYTHONPATH": str(REPO_ROOT), "WANDB_DISABLED": "true", + "VIDUR_DISABLE_WANDB": "1", "OMP_NUM_THREADS": "1", "OPENBLAS_NUM_THREADS": "1"}, + text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=300, + ) + (root / "run.log").write_text(result.stdout) + assert result.returncode == 0, result.stdout[-15000:] + case_dir = root / SET_NAME / case_id + outcome = json.loads((case_dir / "outcome.json").read_text()) + assert outcome["outcome"] == SUCCESS, outcome + return case_dir / "metrics" + + +@pytest.mark.parametrize("case_id, expected", [ + # (requests, prefill tokens, decode tokens): 16 prompt tokens and one output token each. + ("G3a-moe-dp2-pp2-n4", (4, 64, 4)), + ("G3a-moe-dp4-pp2-n8", (8, 128, 8)), +]) +def test_moe_lanes_complete_every_request(tmp_path, case_id, expected): + metrics_dir = run_case(tmp_path, case_id) + with next(metrics_dir.rglob("request_metrics.csv")).open() as handle: + rows = list(csv.DictReader(handle)) + observed = ( + len(rows), + sum(int(float(row["request_num_prefill_tokens"])) for row in rows), + sum(int(float(row["request_num_decode_tokens"])) for row in rows), + ) + assert observed == expected + + +def test_dense_lanes_start_in_the_same_first_forward(tmp_path): + """Every request arrives at t=0 and the stage has room for both lanes.""" + rows = read_ledger(run_case(tmp_path, "G4-dense-dp2-pp2-n8")) + first_start = {} + for row in sorted(rows, key=lambda row: row["stage_start_ts"]): + if row["execution_scope"] == ATTN_DP_LANE and row["stage_id"] == 0: + first_start.setdefault(row["replica_local_id"], row["stage_start_ts"]) + assert sorted(first_start) == [0, 1] + assert first_start[0] == first_start[1] + + +if __name__ == "__main__": + cases = {case.case_id: case for case in build_cases()} + run_case_in_child(cases[sys.argv[2]], Path(sys.argv[1]), SET_NAME) diff --git a/tests/unit/test_mixed_layer_decode_ffn_scheduling.py b/tests/unit/test_mixed_layer_decode_ffn_scheduling.py index 7f3db6fe..9c959682 100644 --- a/tests/unit/test_mixed_layer_decode_ffn_scheduling.py +++ b/tests/unit/test_mixed_layer_decode_ffn_scheduling.py @@ -799,6 +799,75 @@ def test_decode_ffn_wave_materialization_attaches_one_parent_ticket( assert context.queued_tickets == (tickets[0],) +def test_decode_ffn_dense_groups_keep_counter_order_around_a_queued_ep_wave( + mixed_model_config, +) -> None: + """Dense groups enter in group-counter order and never pass an EP wave queued ahead.""" + + scheduler, _, _, lane_sinks = _atomicity_scheduler( + mixed_model_config, + layer_id=4, + ep_size=2, + ) + full_stage_sink = _QueuedBatchSink() + scheduler._full_stage_replica_schedulers = {0: full_stage_sink} + scheduler.get_full_stage_replica_scheduler = Mock(return_value=full_stage_sink) + scheduler._m2n_ready_groups = deque( + [ + [(_source_batch(layer_id=layer_id), _transfer_info(layer_id=layer_id))] + for layer_id in (3, 4, 3) + ] + ) + context = scheduler.get_stage_execution_context(0, 2) + for _ in range(3): + scheduler.schedule_ffn_with_m2n_immediate() + + first_dense, second_dense = full_stage_sink._m2n_immediate_batch_queue + lane_batches = [lane_sinks[ep_id]._m2n_immediate_batch_queue[0] for ep_id in (0, 1)] + wave = lane_batches[0]._stage_admission_ticket + assert [first_dense.global_id, lane_batches[0].global_id, second_dense.global_id] == [0, 1, 2] + assert context.queued_tickets == ( + first_dense._stage_admission_ticket, + wave, + second_dense._stage_admission_ticket, + ) + + def stage_scheduler(replica_local_id): + return ReplicaStageScheduler( + replica_id=0, + stage_id=2, + is_last_stage=True, + is_moe=True, + execution_time_predictor=object(), + cluster_type=ClusterType.DECODE_FFN, + replica_local_id=replica_local_id, + stage_execution_context=context, + ) + + full_stage = stage_scheduler(None) + ep_lanes = [stage_scheduler(ep_id) for ep_id in (0, 1)] + full_stage.add_batch(first_dense) + full_stage.add_batch(second_dense) + for lane, batch in zip(ep_lanes, lane_batches): + lane.add_batch(batch) + assert full_stage.get_queue_batches() == [first_dense, second_dense] + + assert full_stage.pop_batch_if_not_busy() is first_dense + full_stage.on_stage_end() + context.release(first_dense._stage_admission_ticket) + assert full_stage.pop_batch_if_not_busy() is None + + assert ep_lanes[0].pop_batch_if_not_busy() is lane_batches[0] + assert ep_lanes[1].pop_batch_if_not_busy() is lane_batches[1] + assert full_stage.pop_batch_if_not_busy() is None + for lane in ep_lanes: + lane.on_stage_end() + context.release(wave) + + assert full_stage.pop_batch_if_not_busy() is second_dense + assert context.queued_tickets == () + + def _atomicity_snapshot( scheduler, source_batch, queue_sinks, *, include_entity_ids: bool = True ): diff --git a/tests/unit/test_shared_forward_group_admission.py b/tests/unit/test_shared_forward_group_admission.py index 7b5284f8..5d97ebc7 100644 --- a/tests/unit/test_shared_forward_group_admission.py +++ b/tests/unit/test_shared_forward_group_admission.py @@ -51,6 +51,43 @@ def test_admitted_lanes_share_identity_after_unequal_batch_histories(cluster_typ assert [batch.global_id for batch in batches] == [2, 1] +@pytest.mark.parametrize("first_lane", [0, 1]) +def test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket(first_lane): + """Under PP a busy lane can hold the FIFO head; the other lane must still join.""" + + other_lane = 1 - first_lane + context = StageExecutionContext(replica_id=0, stage_id=0, ep_size=2, full_stage_capacity=2) + stages = [make_stage(context, lane, ClusterType.MONOLITHIC) for lane in range(2)] + first_now, first_next = make_batch(first_lane, 0), make_batch(first_lane, 1) + # Distinct provisional ids, so that sharing a bound group is observable. + other_now, other_next = make_batch(other_lane, 5), make_batch(other_lane, 6) + stages[first_lane].add_batch(first_now) + assert stages[first_lane].pop_batch_if_not_busy() is first_now + stages[first_lane].add_batch(first_next) + stages[other_lane].add_batch(other_now) + stages[other_lane].add_batch(other_next) + assert context.queued_tickets[0] == first_next._stage_admission_ticket + + assert stages[other_lane].pop_batch_if_not_busy() is other_now + assert other_now._forward_cohort_provisional_id == first_now._forward_cohort_provisional_id + + wave = context.replace_full_stage_owners_with_ep_wave( + (first_now._stage_admission_ticket, other_now._stage_admission_ticket), + operation_id="wave", participant_ep_ids=(0, 1), + ) + owners = context.replace_ep_wave_with_full_stage_owners(wave, operation_ids=("restored0", "restored1")) + for owner in owners: + context.release(owner) + for stage in stages: + stage.on_stage_end() + + assert stages[first_lane].pop_batch_if_not_busy() is first_next + assert stages[other_lane].pop_batch_if_not_busy() is other_next + assert first_next._forward_cohort_provisional_id == other_next._forward_cohort_provisional_id + assert first_next._forward_cohort_provisional_id > first_now._forward_cohort_provisional_id + assert context.queued_tickets == () + + def test_started_group_blocks_new_lane_through_ep_restore_and_partial_release(): context = StageExecutionContext(replica_id=0, stage_id=0, ep_size=2, full_stage_capacity=4) first = context.enqueue_full_stage(operation_id="first") diff --git a/tests/unit/test_stage_admission_pp_tools.py b/tests/unit/test_stage_admission_pp_tools.py new file mode 100644 index 00000000..ea88f196 --- /dev/null +++ b/tests/unit/test_stage_admission_pp_tools.py @@ -0,0 +1,237 @@ +"""Synthetic checks of the stage-admission vLLM comparison tools. + +``compare_lanes`` is driven end to end on hand-built vLLM traces and Frontier +sets: ideal lane pairing on both sides, and the base rule's two negative +controls (a MoE admission deadlock, dense lanes one forward apart). +``vllm_burst_driver`` is checked for overlay acceptance and patch parsing. +""" + +from __future__ import annotations + +import csv +import json +from pathlib import Path + +import pytest + +from tests.comparison.stage_admission_pp import compare_lanes +from tests.comparison.stage_admission_pp.vllm_burst_driver import apply_patch, build_overlay +from tests.e2e.stage_admission_matrix import ADMISSION_DEADLOCK, MATRIX_DIR_NAME, SUCCESS + + +FORWARD = 0.12 +ROUNDS = 3 + + +def forwards(num_requests: int, late_lane: bool = False) -> list[tuple[int, int, float, float, int]]: + """Two-stage forwards: pair ``k`` runs stage 0 in ``[k*F, (k+1)*F)``. + + With ``late_lane`` lane 1 starts one forward after lane 0, as under the + base rule. + """ + rows = [] + for index in range(num_requests): + lane, slot = index % 2, index // 2 + start = (slot + (1 if late_lane and lane == 1 else 0)) * FORWARD + rows.append((lane, 0, start, start + FORWARD, index)) + rows.append((lane, 1, start + FORWARD, start + 2 * FORWARD, index)) + return rows + + +def write_jsonl(path: Path, records) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("".join(json.dumps(record) + "\n" for record in records)) + + +def write_vllm_scenario(run_dir: Path, model: str, late_round: int | None = None) -> None: + scenario = run_dir / "runs" / model + requests, boundaries, rounds = [], [], [] + placement = {0: [], 1: []} + wall_offset, origin = 1.7e9, 100.0 + labels = [("warmup", 0, 4)] + [ + (f"b{burst}-r{r}", r, burst) for burst in compare_lanes.BURSTS for r in range(ROUNDS) + ] + for label, round_index, num_requests in labels: + late = label != "warmup" and round_index == late_round + for lane, stage, start, end, index in forwards(num_requests, late): + request_id = f"{label}-q{index}" + boundaries.append({ + "request_ids": [request_id], "pp_rank": stage, "is_last_rank": stage == 1, + "forward_start_ts": origin + start, + "send_start_ts": None if stage else origin + end, + "timestamp": wall_offset + origin + end, + }) + if stage == 0: + placement[lane].append({"kind": "engine_iteration", "engine": lane, + "scheduled_new_req_ids": [request_id]}) + requests.extend( + {"request_id": f"{label}-q{index}", "burst": label, "round": round_index, + "index": index, "rank": index % 2, "num_output_tokens": 1} + for index in range(num_requests) + ) + rounds.append({"label": label, "wall_minus_monotonic_before": wall_offset, + "wall_minus_monotonic_after": wall_offset}) + origin += 10.0 + write_jsonl(scenario / "requests.jsonl", requests) + write_jsonl(scenario / "pp_boundary.jsonl", boundaries) + (scenario / "summary.json").write_text(json.dumps({"rounds": rounds})) + for engine, records in placement.items(): + write_jsonl(scenario / "dp_placement" / f"dp_placement_{engine}.jsonl", records) + + +def write_frontier_case(set_dir: Path, model: str, burst: int, outcome: str, + late_lane: bool = False) -> None: + case_dir = set_dir / f"G7-{model}-dp2-pp2-n{burst}" + case_dir.mkdir(parents=True) + (case_dir / "run.json").write_text(json.dumps({"outcome": outcome})) + (case_dir / "case.json").write_text(json.dumps({"num_requests": burst})) + if outcome == SUCCESS: + write_jsonl( + case_dir / "metrics" / "run" / "frontier_stage_batch_ledger.jsonl", + ({"execution_scope": "ATTN_DP_LANE", "replica_local_id": lane, "stage_id": stage, + "stage_start_ts": start, "stage_end_ts": end, "request_ids": [str(index)]} + for lane, stage, start, end, index in forwards(burst, late_lane)), + ) + + +@pytest.fixture +def workspace(tmp_path, monkeypatch): + monkeypatch.setenv("FRONTIER_TMP_ROOT", str(tmp_path / "scratch")) + return tmp_path + + +def run_compare(workspace: Path, *, fixed_base: bool = False, late_round: int | None = None, + drop_placement_engine: int | None = None) -> tuple[dict, list[dict]]: + vllm_run = workspace / "vllm" + for model in compare_lanes.MODELS: + write_vllm_scenario(vllm_run, model, late_round if model == "dense" else None) + if drop_placement_engine is not None: + (vllm_run / "runs" / "moe" / "dp_placement" + / f"dp_placement_{drop_placement_engine}.jsonl").unlink() + frontier = workspace / "scratch" / MATRIX_DIR_NAME + for burst in compare_lanes.BURSTS: + for model in compare_lanes.MODELS: + write_frontier_case(frontier / "after", model, burst, SUCCESS) + if fixed_base: + write_frontier_case(frontier / "base", model, burst, SUCCESS) + elif model == "moe": + write_frontier_case(frontier / "base", model, burst, ADMISSION_DEADLOCK) + else: + write_frontier_case(frontier / "base", model, burst, SUCCESS, late_lane=True) + output = workspace / "analysis" + compare_lanes.main(["--vllm-run", str(vllm_run), "--output", str(output)]) + status = json.loads((output / "workflow_gap_status.json").read_text()) + with (output / "workflow_gap_table.csv").open() as handle: + rows = list(csv.DictReader(handle)) + return status, rows + + +def statuses(rows: list[dict], checks: tuple[str, ...]) -> set[str]: + return {row["status"] for row in rows if row["check"] in checks} + + +def test_ideal_after_revision_matches_and_base_controls_hold(workspace) -> None: + status, rows = run_compare(workspace) + + assert status["status"] == "PASS" + assert status["mismatches"] == 0 + assert status["negative_control_holds"] is True + assert statuses(rows, ("V1", "V2", "V3", "V4")) == {"MATCH"} + assert {(row["model"], row["status"]) for row in rows if row["check"] == "V5"} == { + ("moe", "MATCH"), ("dense", compare_lanes.INFORMATIONAL) + } + assert sorted((row["check"], row["model"], row["status"]) for row in rows + if row["check"] in ("N1", "N4")) == [ + ("N1", "moe", "HOLDS"), ("N1", "moe", "HOLDS"), + ("N4", "dense", "HOLDS"), ("N4", "dense", "HOLDS"), + ] + + +def test_fixed_base_loses_the_controls_without_a_mismatch(workspace) -> None: + status, rows = run_compare(workspace, fixed_base=True) + + assert status["status"] == "PASS" + assert status["mismatches"] == 0 + assert status["negative_control_holds"] is False + assert statuses(rows, ("V1", "V2", "V3", "V4")) == {"MATCH"} + assert statuses(rows, ("N1", "N4")) == {"LOST"} + + +def test_vllm_round_with_a_late_lane_is_reported(workspace) -> None: + status, rows = run_compare(workspace, late_round=1) + + mismatched = sorted((row["check"], row["model"], row["burst"], row["round"]) + for row in rows if row["status"] == "MISMATCH") + assert mismatched == [ + ("V3", "dense", "16", "1"), ("V3", "dense", "8", "1"), + ("V4", "dense", "16", "1"), ("V4", "dense", "8", "1"), + ] + assert status["status"] == "FAIL" + + +def test_missing_placement_log_fails_the_comparison(workspace) -> None: + status, _ = run_compare(workspace, drop_placement_engine=1) + + assert status["vllm_placement_ok"] is False + assert status["vllm_placement_unseen_requests"] > 0 + assert status["status"] == "FAIL" + + +def write_tree(root: Path, files: dict[str, str]) -> None: + for relative, text in files.items(): + path = root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text) + + +@pytest.mark.parametrize("extra_change, accepted", [(False, True), (True, False)]) +def test_overlay_acceptance_compares_file_sets(tmp_path, extra_change, accepted) -> None: + # Path order puts vllm/a/c.py first; string order puts vllm/a-b.py first. + changed = ["vllm/a-b.py", "vllm/a/c.py"] + write_tree(tmp_path / "site", {"vllm/a-b.py": "old\n", "vllm/a/c.py": "old\n", "vllm/d.py": "same\n"}) + checkout_files = {"vllm/a-b.py": "new\n", "vllm/a/c.py": "new\n", "vllm/d.py": "same\n"} + if extra_change: + checkout_files["vllm/d.py"] = "changed\n" + write_tree(tmp_path / "checkout", checkout_files) + expected = tmp_path / "expected_changes.txt" + expected.write_text("".join(f"{name}\n" for name in sorted(changed))) + + report = build_overlay(tmp_path / "site" / "vllm", tmp_path / "checkout", + tmp_path / "overlay", expected) + + assert report["accepted"] is accepted + assert report["unexpected"] == ([] if accepted else ["vllm/d.py"]) + + +def test_apply_patch_keeps_every_section_of_a_repeated_file(tmp_path) -> None: + write_tree(tmp_path, {"pkg/mod.py": "a = 1\nb = 2\nc = 3\n"}) + patch = tmp_path / "change.patch" + patch.write_text( + "--- a/pkg/mod.py\n+++ b/pkg/mod.py\n@@ -1,1 +1,1 @@\n-a = 1\n+a = 10\n" + "--- a/pkg/mod.py\n+++ b/pkg/mod.py\n@@ -3,1 +3,1 @@\n-c = 3\n+c = 30\n" + ) + + assert apply_patch(patch, tmp_path) == ["pkg/mod.py"] + assert (tmp_path / "pkg" / "mod.py").read_text() == "a = 10\nb = 2\nc = 30\n" + + +def test_apply_patch_reads_a_trimmed_context_line(tmp_path) -> None: + write_tree(tmp_path, {"pkg/mod.py": "a = 1\n\nb = 2\n"}) + patch = tmp_path / "change.patch" + patch.write_text("--- a/pkg/mod.py\n+++ b/pkg/mod.py\n@@ -1,3 +1,3 @@\n a = 1\n\n-b = 2\n+b = 20\n") + + apply_patch(patch, tmp_path) + + assert (tmp_path / "pkg" / "mod.py").read_text() == "a = 1\n\nb = 20\n" + + +def test_apply_patch_rejects_an_unknown_hunk_line(tmp_path) -> None: + write_tree(tmp_path, {"pkg/mod.py": "a = 1\n"}) + patch = tmp_path / "change.patch" + patch.write_text( + "--- a/pkg/mod.py\n+++ b/pkg/mod.py\n@@ -1,1 +1,1 @@\n-a = 1\n" + "\\ No newline at end of file\n+a = 2\n" + ) + + with pytest.raises(ValueError, match="unexpected hunk line"): + apply_patch(patch, tmp_path) diff --git a/tests/unit/test_stage_execution_context.py b/tests/unit/test_stage_execution_context.py index 4eb05de0..0e471f5c 100644 --- a/tests/unit/test_stage_execution_context.py +++ b/tests/unit/test_stage_execution_context.py @@ -91,6 +91,74 @@ def test_admission_fifo_cannot_skip_an_earlier_ready_wave() -> None: context.release(second) +def _context_with_wave_between_full_stage_tickets(): + context = StageExecutionContext( + replica_id=0, + stage_id=0, + ep_size=2, + full_stage_capacity=2, + ) + full0 = context.enqueue_full_stage(operation_id="full0") + full1 = context.enqueue_full_stage(operation_id="full1") + wave0 = context.enqueue_ep_wave(operation_id="wave0", participant_ep_ids=(0, 1)) + full2 = context.enqueue_full_stage(operation_id="full2") + return context, full0, full1, wave0, full2 + + +def test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave() -> None: + context, full0, full1, wave0, full2 = _context_with_wave_between_full_stage_tickets() + + assert context.try_acquire(full1) is True + assert context.queued_tickets == (full0, wave0, full2) + # Capacity remains, but wave0 is queued ahead of full2. + assert context.try_acquire(full2) is False + assert context.try_acquire(full0) is True + + +def test_queued_ep_wave_orders_full_stage_work_on_both_sides() -> None: + context, full0, full1, wave0, full2 = _context_with_wave_between_full_stage_tickets() + + assert context.try_acquire(full0) is True + assert context.try_acquire(full1) is True + context.release(full1) + assert context.try_acquire(full2) is False + assert context.try_acquire(wave0) is False + context.release(full0) + assert context.try_acquire(wave0) is True + assert context.try_acquire(full2) is False + context.release(wave0) + assert context.try_acquire(full2) is True + context.release(full2) + assert context.is_idle + assert context.queued_tickets == () + + +def test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket() -> None: + context = StageExecutionContext(replica_id=0, stage_id=0, ep_size=1) + full0 = context.enqueue_full_stage(operation_id="full0") + full1 = context.enqueue_full_stage(operation_id="full1") + + assert context.try_acquire(full1) is True + assert context.queued_tickets == (full0,) + + +def test_active_full_stage_ticket_is_refused_without_changing_the_stage() -> None: + context = StageExecutionContext( + replica_id=0, + stage_id=0, + ep_size=1, + full_stage_capacity=2, + ) + first = context.enqueue_full_stage(operation_id=("lane", 0)) + second = context.enqueue_full_stage(operation_id=("lane", 1)) + assert context.try_acquire(first) is True + + assert context.try_acquire(first) is False + assert context.is_active(first) + assert context.queued_tickets == (second,) + assert context.try_acquire(second) is True + + def test_release_requires_the_active_operation_ticket() -> None: context = StageExecutionContext(replica_id=0, stage_id=0, ep_size=1) wave = context.enqueue_ep_wave(operation_id=30, participant_ep_ids=(0,)) From 03d5f24c7fd85f5d2b9ca0b2dbbb73e83e52aab3 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 15:11:26 +0800 Subject: [PATCH 47/80] test(stage-admission): skip dispatched sync rooms in the drain report enter_layer_sync pops the fields of a room it dispatches but keeps the room's key, so a run that drains after any dispatch leaves empty rooms behind. The drain reader indexed room["batches"] and raised KeyError on them, turning an admission deadlock into other_failure. This first showed on PDD online cells with PP=3 once W2 spreads Poisson arrivals over the lanes; both cells now classify as admission_deadlock under the pre-fix rule. --- tests/e2e/stage_admission_matrix.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/e2e/stage_admission_matrix.py b/tests/e2e/stage_admission_matrix.py index c872b011..aded6e60 100644 --- a/tests/e2e/stage_admission_matrix.py +++ b/tests/e2e/stage_admission_matrix.py @@ -401,7 +401,9 @@ def _read_cluster_state(cluster: str, cluster_scheduler, lanes: dict, for step, by_layer in by_step.items(): for layer, by_sync in by_layer.items(): for sync_stage, room in by_sync.items(): - if not room["batches"]: + # A dispatched room keeps its key with its fields + # popped (enter_layer_sync). + if not room.get("batches"): continue rooms.append({ "cluster": cluster, "room": room_name.strip("_"), From d3e6e785f92f571828e9ed0c2bb77f0491687fe2 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 15:20:47 +0800 Subject: [PATCH 48/80] docs(step9): record the W9-01 merge-forward and composition check main (PR 36, squash 4ab1964) is merged in as dd9b8d9. Plan section 18.14 fixes the composition check before measuring: G3b, G9 and G10 on the merged tree against the pre-merge rule, W2 lane coverage, and both test suites. K1 to K4 pass on 03d5f24. K2 is amended after measuring for online cells, where an earlier admission can change later batch contents on that lane; the one such cell is analyzed in the test report. --- .../issues.md | 6 + .../plan.md | 55 + .../progress.md | 19 +- .../requirements.md | 13 + ...port_2026-09-23_w9_01_composition_check.md | 139 ++ .../composition_check.py | 136 ++ .../composition_compare_junit.py | 41 + .../compare_c-pr35_c-merged.json | 1956 +++++++++++++++++ .../composition_check.json | 1576 +++++++++++++ .../integration_compare.json | 22 + .../composition_evidence/unit_compare.json | 132 ++ .../composition_run_suites.sh | 11 + 12 files changed, 4105 insertions(+), 1 deletion(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-23_w9_01_composition_check.md create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_check.py create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_compare_junit.py create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/compare_c-pr35_c-merged.json create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/composition_check.json create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/integration_compare.json create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/unit_compare.json create mode 100755 task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_run_suites.sh diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md index 663b71be..69462b90 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | W9-01: merged forward (`dd9b8d9`); composition check passes on `03d5f24`. | | 2026-09-23 | W9-01: PR 36 ran its pre-merge untrack (P6, `4d08c5d`); the copies here are now the only published records of that task. | | 2026-09-23 | W9-01: PR 36 round-2 review remediation recorded; the composition check now also reruns PR 36 groups G9 and G10. | | 2026-09-23 | W9-01: fixed on `fix/stage-admission-ordering` (draft PR 36) under option 2; resolution recorded, summary and test report copied to `w9_01_stage_admission_ordering/`. | @@ -151,10 +152,15 @@ Observed on that branch (details in the copied test report): Remaining here, in order: 1. After PR 36 merges into `main`, merge `main` forward into this branch. + Done: PR 36 squash `4ab1964`, merge `dd9b8d9`. 2. Rerun the PR 36 matrix groups on the merged tree as the composition check: G3b (mixed prefill/decode, `attn_dp > 1`, `PP > 1`) with W3, and G9 and G10 (online), whose Poisson cells reach every lane only with W2. + Done: K1–K4 pass on `03d5f24` (`test_report_2026-09-23_w9_01_composition_check.md`), + after a harness drain-reader fix. With W2 two Poisson PDD PP3 cells drain + under the pre-merge rule; both complete on the merged tree. 3. If they pass, resume Step 9 P1(b) and the design checkpoint D9-2. + In progress. Step 9 C1's PP3 row stays on `attn_dp=1` (W9-02). diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md index b862a338..719afa8c 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md @@ -4,6 +4,8 @@ | Date | Change | | --- | --- | +| 2026-09-23 | §18.14 results: K1–K4 pass on `03d5f24`; K2 amended after measuring for online cells (one cell's batches differ after an earlier admission). | +| 2026-09-23 | §18.14 added: W9-01 merge-forward and the composition check, with pass criteria fixed before measuring. | | 2026-09-22 | Step 9 execution started. §18.13 added: P1(a) oracle complete with the state-table evidence, P1(b) blocked by the pre-existing stage-admission deadlock W9-01 (`issues.md`), design checkpoint D9-2 left open because the candidate key fails invariant I5 and I1/I5 can only be settled on the deadlocking shape. C1 amended per W9-02: the PP3 row uses `attn_dp=1`. | | 2026-09-22 | §18 second review at the user's direction (codebase integration, readability, value, no hard-coding/patches/over-defense/redundancy, plain names): findings R9-01..R9-08 in §18.12; D9-1, D9-2, P1(a), §18.10 and the §18.11 representation column amended in place. Execution still not started. | | 2026-09-21 | Landed the execution specification verbatim (Section "Execution Specification" below) and recorded the amendments agreed with the user before Step 0. | @@ -1097,3 +1099,56 @@ Review question, as set by the user: is every planned change to Frontier's core Gate check of the plan after these amendments: no hard-coded constants beyond the cited reference values already in `vllm_dp_load_balancer.py`; no temporary patch (the guard is removed, not bypassed); no new defensive branches (the hook is unconditional on a constructor-required reference); no redundant state (no readiness classifier, no second coordinator model); names are the user's hook name and plain accessors. +### 18.14 W9-01 merge-forward and composition check (2026-09-23) + +PR 36 was squash-merged into `main` as `4ab1964`. Merging `origin/main` into +this branch (merge commit `dd9b8d9`) brings one source file, +`frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`, plus +PR 36's tests and harnesses. No file overlaps this branch's changes. The +composition check asks whether the admission rule and this branch's W2 (lane +rotation across scheduling calls) and W3 (one shared monolithic forward) +still behave as each did alone. It covers the PR 36 matrix groups G3b (MoE, +mixed prefill/decode, offline), G9 (PDD online) and G10 (co-location online), +51 cases. + +Sets, under the matrix root `/data/ycfeng/tmp/stage_admission_ordering`: + +| Set | Tree | +| --- | --- | +| `c-merged` | merged tree `dd9b8d9` | +| `c-pr35` | merged tree with the rule file taken from `1f694f7`: this branch's source before the merge | +| `base`, `after-r2` (existing) | `main` without and with the rule (PR 36 runs) | + +Pass criteria, fixed before measuring: + +| Id | Check | Pass | +| --- | --- | --- | +| K1 | Liveness on `c-merged` | all 51 cases succeed; requests and prefill/decode tokens equal the generated workload | +| K2 | The rule behaves on this branch as on `main`: `compare --before c-pr35 --after c-merged` | 0 STOP; path U byte-identical; path L complete and conserved; every EXPLAIN runs the same ordered batches with the same component durations on each (cluster, replica, stage, lane), so only start times differ | +| K3 | W2 still reaches every lane | every `c-merged` Poisson cell with `attn_dp > 1` places batches on all `attn_dp` lanes of each MONOLITHIC and PREFILL stage; `after-r2` is reported beside it | +| K4 | Tests on the merged tree | PR 36's tests, W2 and W3 tests and the forward-sync regression set pass; `tests/unit` and `tests/integration` show no regression against `8315d9b` | + +Not a criterion, reported: which Poisson cells deadlock on `c-pr35`. With W2 +the online arrivals reach several lanes, so the W9-01 defect becomes +reachable in cells that ran on lane 0 only on `main`. + +On a pass, Step 9 resumes at P1(b) with the fourth shape, MoE `attn_dp=2, +moe_ep=2, PP=2`, and then the design checkpoint D9-2. + +**Results (2026-09-23, `03d5f24`; `test_report_2026-09-23_w9_01_composition_check.md`).** +K1 51/51. K2 0 STOP (U 15, L 16, T 12 identical, 8 EXPLAIN). K3 22/22 cells +use every lane; `after-r2` uses lane 0 only. K4 0 regressions. The run found +and fixed a drain-reader defect in the harness (`03d5f24`: dispatched rooms +keep an empty entry). Two Poisson PDD cells, `G9-moe-dp{2,4}-pp3-n8`, drain +under the pre-merge rule but not on `main`, because W2 spreads their +arrivals. Both complete on the merged tree. + +**K2 amended after measuring.** One EXPLAIN cell, `G10-dense-dp2-pp3-n8` +(online), changes lane 1's batches (36 → 33 ledger rows) while lane 0 is +identical. The first divergence is the same batch on stage 0, lane 1, +admitted 23.4 ms earlier: the removed W9-01 coupling. Lane 1 then meets later +Poisson arrivals at other points of its schedule. "Only start times differ" +holds where batch contents are fixed at t=0 (offline and burst). For online +cells K2 now accepts a batch difference when the first divergence is an +identical batch admitted earlier on the same stage and lane. + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 986c71a3..883c979b 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | W9-01 merge-forward: `origin/main` merged (`dd9b8d9`); composition check K1–K4 pass on `03d5f24` (drain-reader fix); Step 9 P1(b) resumed. | | 2026-09-23 | PR 36 pre-merge untrack (P6, `4d08c5d`) done; W9-01 copies refreshed. Merge-forward waits for the PR 36 merge. | | 2026-09-23 | PR 36 round-2 review remediation recorded; the W9-01 composition check extended to PR 36 groups G9 and G10. | | 2026-09-23 | W9-01 fixed on `fix/stage-admission-ordering` (draft PR 36); Step 9 PP>1 packages stay paused until it merges forward and passes G3b. | @@ -220,7 +221,7 @@ User start signal: "开始执行step9", with the quality gates repeated (readabi | Package | State | Evidence | | --- | --- | --- | | P1(a) reference-loop oracle | completed | `tests/comparison/dp_placement_pp/reference_loop.py`; `tests/unit/test_dp_placement_reference_loop.py` (9 passed, 1.21 s, `frontier-py310`). §18.11 state table confirmed as written; PP=1 shown to degenerate to "every iteration schedules and applies"; depth 3 shown to allow two consecutive admission-only publications, which rules out any stride constant. | -| P1(b) Frontier boundary probe | blocked | Three shapes probed (`attn_dp=2 PP=1`, `attn_dp=1 PP=2`, `attn_dp=1 PP=3`), tables in `plan.md` §18.13. The fourth shape, MoE `attn_dp=2, moe_ep=2, PP=2`, drains the event queue with requests unfinished — pre-existing defect W9-01 in `issues.md`. | +| P1(b) Frontier boundary probe | blocked until 2026-09-23, resumed (see below) | Three shapes probed (`attn_dp=2 PP=1`, `attn_dp=1 PP=2`, `attn_dp=1 PP=3`), tables in `plan.md` §18.13. The fourth shape, MoE `attn_dp=2, moe_ep=2, PP=2`, drains the event queue with requests unfinished — pre-existing defect W9-01 in `issues.md`. | | Design checkpoint (D9-1, D9-2) | open | D9-1's payload is settled (the completion hook signature already carries lane, load and a key source). D9-2 is not: the candidate key `ForwardSyncState._next_step_id_by_replica` satisfies I1, I2, I3, I4 and I6 on the runnable shapes but fails I5, and no alternative can be checked against I1 without a running `attn_dp>1, PP>1` shape. | | P2–P6, G3–G5 | paused | All depend on the design checkpoint or on that shape. | @@ -230,6 +231,22 @@ Scope decision (2026-09-22): "采纳你的推荐，继续" — option 2, a separ W9-02: `attn_dp=2, moe_ep=2, PP=3` is rejected at construction (6 devices against node size 4). Plan C1's PP3 row amended to `attn_dp=1`. +### W9-01 merge-forward and composition check (2026-09-23) + +| Step | Command / action | Evidence | Result | +| --- | --- | --- | --- | +| Request | "我已经完成 PR 36 merge，把 origin/main merge 进 fix/issue26-correctness-pr（用 merge，不 rebase），重跑 G3b、G9、G10 作为 composition check。通过后恢复 Step 9 的 P1(b) 和 D9-2；暂不处理 pr34和35的 gitingore" | `requirements.md` | recorded | +| Merge | `git merge --no-ff origin/main` (PR 36 squash `4ab1964`) | `dd9b8d9` | no conflict; the only source file is the rule file | +| Criteria | K1–K4 written before measuring | `plan.md` §18.14 | — | +| First sets | `c-merged` at `dd9b8d9`; `c-pr35` with the `1f694f7` rule file | scratch `composition/` | `c-merged` 51/51 success; `c-pr35` two cells `other_failure` from `KeyError: 'batches'` in the drain reader | +| Diagnosis | exported tree, room dump | `debug_rooms.py` in scratch | dispatched rooms keep an empty entry (`sync_entry.py`, same on `main`); the reader indexed it | +| Harness fix | `if not room.get("batches")` | `03d5f24` | both cells classify as `admission_deadlock` | +| Sets rerun | both sets at `03d5f24`; rule file restored and equal to `origin/main` | scratch | `c-merged` 51/51; `c-pr35` 16 `admission_deadlock`, 35 success | +| K2 compare | harness `compare` + `composition_check.py` | `w9_01_stage_admission_ordering/composition_evidence/` | 0 STOP; 7/8 EXPLAIN start-times-only; `G10-dense-dp2-pp3-n8` first divergence = same batch admitted earlier; K2 amended (plan §18.14) | +| K3 | `composition_check.py` | same | 22/22 all lanes; `after-r2` lane 0 only | +| K4 | `composition_run_suites.sh` on both trees; `composition_compare_junit.py` | same | 0 regressions; export-only failures are git-metadata failures; 487 targeted tests pass | +| Report | — | `test_report_2026-09-23_w9_01_composition_check.md` | PASS | + ### G1 ground-truth instrumentation (2026-09-22, completed) Runs in parallel with P1 in the work graph and does not depend on W9-01. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md index 5ea099cf..72b64002 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md @@ -7,6 +7,7 @@ | 2026-09-21 | Recorded the original request, the specification hand-off, and the decisions from the planning interview. | | 2026-09-22 | Recorded the W6 artifact-identity decision and the native GPU validation instruction. | | 2026-09-22 | Recorded the PP>1 `vllm_load_balancing` request, the codesign-only GPU instruction, and the open Step 9 decisions. | +| 2026-09-23 | Recorded the W9-01 merge-forward request after PR 36 merged. | ## [Original Request] 2026-09-21 @@ -90,3 +91,15 @@ Reading: read the external review in full, verify each finding against the sourc | --- | --- | | FP8 native rerun | Authorized and executed: `exp-0922-202645-561899`, `codesign` / 1×H800, 8 passed in 14.27 s, exit 0 (W6 report §8). | | Plan §18 second review | Performed against `c231322` with the stated gates; findings R9-01..R9-08 in `plan.md` §18.12, amendments to D9-1, D9-2, P1, §18.10, §18.11 and `design.md`. Records only; no Step 9 source change; execution still awaits the user's start signal. | + +## [Original Request] 2026-09-23 — W9-01 merge-forward after PR 36 + +> 我已经完成 PR 36 merge，把 origin/main merge 进 fix/issue26-correctness-pr（用 merge，不 rebase），重跑 G3b、G9、G10 作为 composition check。通过后恢复 Step 9 的 P1(b) 和 D9-2；暂不处理 pr34和35的 gitingore（由我在merge前人工处理） + +| Item | Decision | +| --- | --- | +| Merge | PR 36 was squash-merged into `main` as `4ab1964` by the owner. `origin/main` is merged into this branch with a merge commit; no rebase. | +| Composition check | PR 36 matrix groups G3b, G9 and G10 rerun on the merged tree. | +| Next | On a pass, resume Step 9 P1(b) and the design checkpoint D9-2. | +| `.gitignore` | The `task_memory` exceptions of PR 34 and PR 35 stay as they are; the owner removes them before those merges. | + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-23_w9_01_composition_check.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-23_w9_01_composition_check.md new file mode 100644 index 00000000..7d1778f5 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-23_w9_01_composition_check.md @@ -0,0 +1,139 @@ +# Test report — W9-01 merge-forward composition check (2026-09-23) + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-23 | Created. K1–K4 of plan §18.14 on the merged tree `03d5f24`. | + +## Environment + +| Item | Value | +| --- | --- | +| Host | `kun-workspace-vgen2` | +| Worktree | `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`, branch `fix/issue26-correctness-pr` | +| Merge | `dd9b8d9` merges `origin/main` `4ab1964` (PR 36, squash) into `8315d9b`; no conflict, no overlapping file | +| Revision under test | `03d5f24` (the merge plus the drain-reader fix below) | +| Interpreter | `/data/ycfeng/envs/frontier-py310/bin/python`, Python 3.10.6; distributions SHA-256 `ecd50ea8…902620` | +| Environment variables | `PYTHONPATH=`, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`, `FRONTIER_TMP_ROOT=/data/ycfeng/tmp` | +| Matrix root | `/data/ycfeng/tmp/stage_admission_ordering` (sets `c-merged`, `c-pr35`; PR 36 sets `base`, `after-r2`) | +| Untracked, not ours | `outputs/metrics/meta_llama_llama_2_7b_hf/` (recorded in each set's provenance) | + +## Commands + +```bash +# c-merged: the merged tree, clean +python -m tests.e2e.stage_admission_matrix run --set c-merged --group G3b --group G9 --group G10 --jobs 16 +# c-pr35: the same tree with the rule file of 1f694f7, restored afterwards with git checkout +git show 1f694f7:frontier/scheduler/replica_stage_scheduler/stage_execution_context.py > frontier/scheduler/replica_stage_scheduler/stage_execution_context.py +python -m tests.e2e.stage_admission_matrix run --set c-pr35 --group G3b --group G9 --group G10 --jobs 16 +git checkout -- frontier/scheduler/replica_stage_scheduler/stage_execution_context.py +python -m tests.e2e.stage_admission_matrix compare --before c-pr35 --after c-merged --output /compare_c-pr35_c-merged.json +python task_memory/.../w9_01_stage_admission_ordering/composition_check.py c-pr35 c-merged after-r2 +# K4: both trees, full suites with JUnit output +w9_01_stage_admission_ordering/composition_run_suites.sh # merged tree +w9_01_stage_admission_ordering/composition_run_suites.sh # pre-merge tree +python w9_01_stage_admission_ordering/composition_compare_junit.py +``` + +`c-pr35` equals this branch's source before the merge. The merge brings one +source file, the rule file, so swapping it back reproduces `8315d9b`'s +`frontier/` tree. After the run the file equals `origin/main` again. + +## Harness fix found on the way (`03d5f24`) + +The first `c-pr35` run classified two drained cells, `G9-moe-dp2-pp3-n8` and +`G9-moe-dp4-pp3-n8`, as `other_failure`. The drain reader raised +`KeyError: 'batches'`: `enter_layer_sync` pops the fields of a room it +dispatches but keeps the room's key, and the reader indexed every room. The +same code is on `main` (`sync_entry.py:140`, `:277`); it was first reached +here, because W2 lets Poisson PDD cells drain after some dispatches. The +reader now skips rooms without batches. Both cells then classify as +`admission_deadlock` (probe in an exported tree, then the full reruns below). +Both sets were rerun at `03d5f24` so that they share one harness revision. + +## Results + +| Id | Check | Expected | Actual | Result | +| --- | --- | --- | --- | --- | +| K1 | `c-merged` outcomes and conservation, 51 cases | all succeed; requests and prefill/decode tokens equal the workload | 51/51 success (G3b 12, G9 11, G10 28); conservation holds in all 51 | PASS | +| K2 | `compare --before c-pr35 --after c-merged` | 0 STOP; U identical; L complete and conserved; each EXPLAIN only moves start times | U 15 PASS; L 16 PASS; T 12 PASS (identical) and 8 EXPLAIN; 0 STOP. EXPLAIN: 7 have the same ordered batches and component durations on every (cluster, replica, stage, lane); 1 (`G10-dense-dp2-pp3-n8`) does not — see below | PASS under the amended rule | +| K3 | lanes used by Poisson cells with `attn_dp > 1` | every MONOLITHIC/PREFILL stage uses all lanes in `c-merged` | 22/22 cells use all lanes on every stage; `after-r2` (main plus the rule, no W2) uses lane 0 only in all 22 | PASS | +| K4 | unit and integration suites, merged vs pre-merge tree | 0 regressions; targeted modules pass | unit: 0 regressions, 0 new failures, 0 skip changes; integration: same, 3 new tests pass; 25 targeted modules, 487 tests, all pass | PASS | + +Path L (16 cells that drain under the pre-merge rule): G3b 6, G9 burst 4, +G10 MoE burst 4, and two cells that do not drain on `main`, +`G9-moe-dp2-pp3-n8` and `G9-moe-dp4-pp3-n8`. W2 spreads their Poisson +arrivals over the lanes, which makes the W9-01 pattern reachable there. The +drained cluster is the unified DECODE (`simulation_time` 0.4161 s in both +cells). Every L cell completes on the merged tree with requests and tokens +conserved. + +### K2: the one EXPLAIN cell whose batches differ + +`G10-dense-dp2-pp3-n8` (online Poisson, dense, `attn_dp=2`, PP=3): + +| Item | Value | +| --- | --- | +| Identical lanes | lane 0 on stages 0, 1 and 2 | +| First divergence of the time-ordered ledger (position 12) | MONOLITHIC stage 0, lane 1, batch of request `1`, same duration and components, starts at 0.16027 s on the merged tree against 0.18366 s before | +| Why it waited before | request `1` left stage 2 at 0.16027 s and stage 0 of lane 1 was idle, but lane 0's queued ticket headed the FIFO while lane 0's stage 0 was busy (0.1477–0.1837 s): the W9-01 coupling | +| Consequence | lane 1 then forms later batches at different times, so Poisson arrivals `5` and `7` share batches (36 → 33 lane-1 ledger rows) | + +Lane 0 is byte-for-byte the same, and the first difference is the rule's +intended effect: an identical ready batch admitted earlier. The pre-fixed K2 +wording ("only start times differ") holds for offline and burst cells, where +batch contents are fixed at t=0, but not for an online cell. There, a lane +admitted earlier meets later arrivals at a different point in its schedule. +The check therefore accepts a batch difference when the first divergence is +the same batch on the same stage and lane, admitted earlier. This amendment +was made after measuring; it is recorded in plan §18.14 with this cell. + +### K4 detail + +| Suite | Pre-merge `8315d9b` (export) | Merged `03d5f24` (worktree) | +| --- | --- | --- | +| `tests/unit` | 172 failed, 3706 passed, 51 skipped, 10 errors | 84 failed, 3814 passed, 50 skipped, 10 errors | +| `tests/integration` | 16 passed, 22 skipped, 5 errors | 19 passed, 22 skipped, 5 errors | + +The 88 extra unit failures on the pre-merge side all come from the export, +which has no `.git`: `git rev-parse HEAD` and `git ls-files` fail in +`test_sim_walltime_scaling_sweep` (40), `test_moe_ep_non_dummy_matrix` (30), +`test_sim_walltime_scaling_run_case` (14), `test_moe_ep_h800_profile_backfill_script` +(3) and `test_pd_disaggregation_naming_guard` (1). They pass in the worktree. +The one skip difference is `test_collective_sim_zero_payload`: it is skipped +at module level in the export, which lacks the built collective-sim binary, +and its 4 tests run in the worktree. The 20 unit tests and 3 integration tests +that exist only on the merged side are PR 36's and the 4 zero-payload tests; +all pass. The 84 common failures and the 10 collection errors (no `torch` or +`matplotlib`) are the same set on both sides. The 5 integration errors are +the absent PD-AF Reference checkout. + +Targeted modules (all pass on the merged tree): PR 36's +`test_stage_execution_context`, `test_shared_forward_group_admission`, +`test_mixed_layer_decode_ffn_scheduling`, `test_stage_admission_pp_tools` and +`test_stage_admission_pipeline_lanes`; W2 `test_cluster_scheduler_dp_lanes`; +W3 `test_monolithic_mixed_forward_sync` and `test_monolithic_mixed_forward_runtime`; +the forward-sync regression set of the W3 report; Step 9's +`test_vllm_dp_load_balancer`, `test_dp_placement_reference_loop` and +`test_vllm_dp_placement_runtime`. + +## Evidence + +- `w9_01_stage_admission_ordering/composition_evidence/`: + - `compare_c-pr35_c-merged.json` (harness paths); + - `composition_check.json` (K1, K2 EXPLAIN detail, K3 lanes for both sets); + - `unit_compare.json`, `integration_compare.json`. +- Scripts: `composition_check.py`, `composition_compare_junit.py`, + `composition_run_suites.sh`. +- Raw sets and suite logs: `/data/ycfeng/tmp/stage_admission_ordering/{c-merged,c-pr35,composition}`. + +## Limits + +- The Frontier side uses the dummy predictor; the check concerns admission, + placement and completion, not latency. +- The pre-merge suite baseline ran from a `git archive` export. Its + git-dependent failures are attributed from their messages, not by a rerun + in a checkout. +- `c-pr35` reproduces the pre-merge `frontier/` tree by swapping the rule file + in the worktree. Its provenance records the modified file. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_check.py b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_check.py new file mode 100644 index 00000000..ee2befbb --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_check.py @@ -0,0 +1,136 @@ +"""Evaluate K1-K3 of the W9-01 composition check (plan §18.14). + +K1: every case of the composed set succeeds and conserves requests and tokens. +K2 (EXPLAIN rows only; the harness ``compare`` settles the rest): every + (cluster, replica, stage, lane) runs the same ordered batches with the same + component durations in both sets, so only start times differ. +K3: every Poisson cell with ``attn_dp > 1`` places batches on all lanes of each + MONOLITHIC and PREFILL stage; the reference set is reported beside it. + +Usage: python composition_check.py + +""" +import json +import sys +from collections import defaultdict +from pathlib import Path + +from tests.e2e.stage_admission_matrix import ( + ATTN_DP_LANE, SUCCESS, _conservation, build_cases, ledger_lane_metric, matrix_root, read_ledger, +) + +GROUPS = ("G3b", "G9", "G10") +LANE_CLUSTERS = ("MONOLITHIC", "PREFILL") + +before_set, composed_set, reference_set = sys.argv[1:4] +compare_rows = json.loads(Path(sys.argv[4]).read_text()) +output = Path(sys.argv[5]) +root = matrix_root() +cases = [case for case in build_cases() if case.group in GROUPS] + + +def metrics_dir(set_name, case_id): + return root / set_name / case_id / "metrics" + + +def outcome(set_name, case_id): + return json.loads((root / set_name / case_id / "run.json").read_text())["outcome"] + + +def lane_batches(set_name, case_id): + rows = defaultdict(list) + for row in read_ledger(metrics_dir(set_name, case_id)): + if row["execution_scope"] == ATTN_DP_LANE: + key = (row["cluster_type"], row["replica_id"], row["stage_id"], row["replica_local_id"]) + rows[key].append(row) + return { + str(key): [(tuple(row["request_ids"]), round(row["stage_end_ts"] - row["stage_start_ts"], 9), + json.dumps(row["execution_time"], sort_keys=True)) + for row in sorted(stage_rows, key=lambda row: row["stage_start_ts"])] + for key, stage_rows in rows.items() + } + + +k1 = {} +for case in cases: + state = outcome(composed_set, case.case_id) + k1[case.case_id] = {"outcome": state} + if state == SUCCESS: + k1[case.case_id]["conservation"] = _conservation(case, metrics_dir(composed_set, case.case_id)) +k1_pass = all(row["outcome"] == SUCCESS and row["conservation"]["ok"] for row in k1.values()) + +def first_divergence(case_id): + """First differing row of the two time-ordered ledgers. + + Passes when it is the same batch on the same stage and lane, admitted + earlier in the composed set: later batch differences on that lane then + follow from online arrivals meeting an earlier lane schedule. + """ + def ordered(set_name): + rows = [row for row in read_ledger(metrics_dir(set_name, case_id)) + if row["execution_scope"] == ATTN_DP_LANE] + rows.sort(key=lambda row: (row["stage_start_ts"], row["stage_id"], row["replica_local_id"])) + return [((row["cluster_type"], row["replica_id"], row["stage_id"], row["replica_local_id"], + tuple(row["request_ids"]), round(row["stage_end_ts"] - row["stage_start_ts"], 9), + json.dumps(row["execution_time"], sort_keys=True)), row["stage_start_ts"]) + for row in rows] + + before, after = ordered(before_set), ordered(composed_set) + index = next(i for i, pair in enumerate(zip(before, after)) if pair[0] != pair[1]) + first_after = after[index] + held = next((start for work, start in before[index:] if work == first_after[0]), None) + return { + "position": index, + "batch": list(first_after[0][:5]), + "start_composed": first_after[1], + "start_before": held, + "earlier_admission_of_same_batch": held is not None and first_after[1] < held, + } + + +k2 = {} +for row in compare_rows: + if row["verdict"] != "EXPLAIN": + continue + before = lane_batches(before_set, row["case_id"]) + after = lane_batches(composed_set, row["case_id"]) + entry = { + "same_batches_and_component_durations": before == after, + "identical_lanes": sorted(key for key in before if before[key] == after.get(key)), + } + if before != after: + entry["first_divergence"] = first_divergence(row["case_id"]) + entry["pass"] = (entry["same_batches_and_component_durations"] + or entry["first_divergence"]["earlier_admission_of_same_batch"]) + k2[row["case_id"]] = entry +k2_pass = all(row["pass"] for row in k2.values()) + + +def stage_lanes(set_name, case_id): + return {stage: metric["lanes"] for stage, metric in ledger_lane_metric(metrics_dir(set_name, case_id)).items()} + + +k3 = {} +for case in cases: + if case.arrival != "poisson" or case.attn_dp == 1: + continue + composed = stage_lanes(composed_set, case.case_id) + reference = (stage_lanes(reference_set, case.case_id) + if outcome(reference_set, case.case_id) == SUCCESS else None) + covered = all(lanes == list(range(case.attn_dp)) + for stage, lanes in composed.items() if stage.split("/")[0] in LANE_CLUSTERS) + k3[case.case_id] = {"all_lanes": covered, "composed": composed, "reference": reference} +k3_pass = all(row["all_lanes"] for row in k3.values()) + +report = { + "sets": {"before": before_set, "composed": composed_set, "reference": reference_set}, + "K1": {"pass": k1_pass, "cases": k1}, + "K2_explain": {"pass": k2_pass, "cases": k2}, + "K3": {"pass": k3_pass, "cases": k3}, +} +output.write_text(json.dumps(report, indent=1, sort_keys=True)) +print("K1", k1_pass, len(k1), "K2 explain", k2_pass, len(k2), "K3", k3_pass, len(k3)) +for case_id, row in k3.items(): + lane_view = lambda stages: ({stage: lanes for stage, lanes in stages.items() + if stage.split("/")[0] in LANE_CLUSTERS} if stages else stages) + print(case_id, "composed", lane_view(row["composed"]), "reference", lane_view(row["reference"])) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_compare_junit.py b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_compare_junit.py new file mode 100644 index 00000000..cdb251e5 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_compare_junit.py @@ -0,0 +1,41 @@ +"""Compare two JUnit reports by test id: regressions, new failures, skip changes. + +Usage: python compare_junit.py +""" +import json +import sys +import xml.etree.ElementTree as ET + + +def outcomes(path): + result = {} + for case in ET.parse(path).getroot().iter("testcase"): + test_id = f"{case.get('classname')}::{case.get('name')}" + tags = {child.tag for child in case} + if "failure" in tags: + state = "failed" + elif "error" in tags: + state = "error" + elif "skipped" in tags: + state = "skipped" + else: + state = "passed" + result[test_id] = state + return result + + +before, after = outcomes(sys.argv[1]), outcomes(sys.argv[2]) +common = before.keys() & after.keys() +report = { + "counts_before": {state: list(before.values()).count(state) for state in set(before.values())}, + "counts_after": {state: list(after.values()).count(state) for state in set(after.values())}, + "regressions": sorted(t for t in common if before[t] == "passed" and after[t] in ("failed", "error")), + "now_passing": sorted(t for t in common if before[t] in ("failed", "error") and after[t] == "passed"), + "skip_changes": sorted(t for t in common if (before[t] == "skipped") != (after[t] == "skipped")), + "new_failures": sorted(t for t in after.keys() - before.keys() if after[t] in ("failed", "error")), + "only_before": sorted(before.keys() - after.keys()), + "only_after": sorted(after.keys() - before.keys()), +} +with open(sys.argv[3], "w") as handle: + json.dump(report, handle, indent=1) +print({key: (len(value) if isinstance(value, list) else value) for key, value in report.items()}) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/compare_c-pr35_c-merged.json b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/compare_c-pr35_c-merged.json new file mode 100644 index 00000000..f900516e --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/compare_c-pr35_c-merged.json @@ -0,0 +1,1956 @@ +[ + { + "after": "success", + "before": "success", + "case_id": "G3b-moe-dp2-pp1-n4", + "group": "G3b", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G3b-moe-dp2-pp1-n8", + "group": "G3b", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G3b-moe-dp2-pp2-n4", + "conservation": { + "expected": [ + 4, + 64, + 12 + ], + "observed": [ + 4, + 64, + 12 + ], + "ok": true + }, + "group": "G3b", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G3b-moe-dp2-pp2-n8", + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "group": "G3b", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G3b-moe-dp2-pp3-n4", + "conservation": { + "expected": [ + 4, + 64, + 12 + ], + "observed": [ + 4, + 64, + 12 + ], + "ok": true + }, + "group": "G3b", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G3b-moe-dp2-pp3-n8", + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "group": "G3b", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G3b-moe-dp4-pp1-n4", + "group": "G3b", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G3b-moe-dp4-pp1-n8", + "group": "G3b", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G3b-moe-dp4-pp2-n4", + "group": "G3b", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.14400000000000007, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.24000000000000016, + "multi_lane_busy_time": 0.14400000000000007, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.14400000000000013, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.2880000000000002, + "multi_lane_busy_time": 0.14400000000000013, + "peak_lanes": 4, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.14400000000000007, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.24000000000000016, + "multi_lane_busy_time": 0.14400000000000007, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.14400000000000013, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.2880000000000002, + "multi_lane_busy_time": 0.14400000000000013, + "peak_lanes": 4, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G3b-moe-dp4-pp2-n8", + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "group": "G3b", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G3b-moe-dp4-pp3-n4", + "group": "G3b", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.10200000000000006, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.23800000000000016, + "multi_lane_busy_time": 0.10200000000000006, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.10200000000000006, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.2720000000000002, + "multi_lane_busy_time": 0.10200000000000006, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.10200000000000009, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.3060000000000002, + "multi_lane_busy_time": 0.10200000000000009, + "peak_lanes": 4, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.10200000000000006, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.23800000000000016, + "multi_lane_busy_time": 0.10200000000000006, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.10200000000000006, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.2720000000000002, + "multi_lane_busy_time": 0.10200000000000006, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.10200000000000009, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.3060000000000002, + "multi_lane_busy_time": 0.10200000000000009, + "peak_lanes": 4, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G3b-moe-dp4-pp3-n8", + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "group": "G3b", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G9-moe-dp2-pp1-n8", + "group": "G9", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G9-moe-dp2-pp2-n8", + "group": "G9", + "lane_metric_after": { + "DECODE/1/0": { + "busy_time": 0.4499999999999997, + "lanes": [ + 0, + 1 + ], + "makespan": 0.6373733884758764, + "multi_lane_busy_time": 0.4049999999999997, + "peak_lanes": 2, + "self_overlap": false + }, + "DECODE/1/1": { + "busy_time": 0.44999999999999934, + "lanes": [ + 0, + 1 + ], + "makespan": 0.6823733884758761, + "multi_lane_busy_time": 0.40499999999999964, + "peak_lanes": 2, + "self_overlap": false + }, + "PREFILL/0/0": { + "busy_time": 0.23081406218246542, + "lanes": [ + 0, + 1 + ], + "makespan": 0.3650302780913621, + "multi_lane_busy_time": 0.13500000000000012, + "peak_lanes": 2, + "self_overlap": false + }, + "PREFILL/0/1": { + "busy_time": 0.2250000000000002, + "lanes": [ + 0, + 1 + ], + "makespan": 0.41003027809136217, + "multi_lane_busy_time": 0.13500000000000012, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "DECODE/1/0": { + "busy_time": 0.4499999999999997, + "lanes": [ + 0, + 1 + ], + "makespan": 0.6373733884758764, + "multi_lane_busy_time": 0.4049999999999997, + "peak_lanes": 2, + "self_overlap": false + }, + "DECODE/1/1": { + "busy_time": 0.44999999999999934, + "lanes": [ + 0, + 1 + ], + "makespan": 0.6823733884758761, + "multi_lane_busy_time": 0.40499999999999964, + "peak_lanes": 2, + "self_overlap": false + }, + "PREFILL/0/0": { + "busy_time": 0.23081406218246542, + "lanes": [ + 0, + 1 + ], + "makespan": 0.3650302780913621, + "multi_lane_busy_time": 0.13500000000000012, + "peak_lanes": 2, + "self_overlap": false + }, + "PREFILL/0/1": { + "busy_time": 0.2250000000000002, + "lanes": [ + 0, + 1 + ], + "makespan": 0.41003027809136217, + "multi_lane_busy_time": 0.13500000000000012, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G9-moe-dp2-pp3-n8", + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "group": "G9", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G9-moe-dp4-pp1-n8", + "group": "G9", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G9-moe-dp4-pp2-n8", + "group": "G9", + "lane_metric_after": { + "DECODE/1/0": { + "busy_time": 0.4499999999999997, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.6373733884758764, + "multi_lane_busy_time": 0.4049999999999997, + "peak_lanes": 4, + "self_overlap": false + }, + "DECODE/1/1": { + "busy_time": 0.44999999999999934, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.6823733884758761, + "multi_lane_busy_time": 0.40499999999999964, + "peak_lanes": 4, + "self_overlap": false + }, + "PREFILL/0/0": { + "busy_time": 0.23081406218246542, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.3650302780913621, + "multi_lane_busy_time": 0.13500000000000012, + "peak_lanes": 2, + "self_overlap": false + }, + "PREFILL/0/1": { + "busy_time": 0.2250000000000002, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.41003027809136217, + "multi_lane_busy_time": 0.13500000000000012, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "DECODE/1/0": { + "busy_time": 0.4499999999999997, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.6373733884758764, + "multi_lane_busy_time": 0.4049999999999997, + "peak_lanes": 4, + "self_overlap": false + }, + "DECODE/1/1": { + "busy_time": 0.44999999999999934, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.6823733884758761, + "multi_lane_busy_time": 0.40499999999999964, + "peak_lanes": 4, + "self_overlap": false + }, + "PREFILL/0/0": { + "busy_time": 0.23081406218246542, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.3650302780913621, + "multi_lane_busy_time": 0.13500000000000012, + "peak_lanes": 2, + "self_overlap": false + }, + "PREFILL/0/1": { + "busy_time": 0.2250000000000002, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.41003027809136217, + "multi_lane_busy_time": 0.13500000000000012, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G9-moe-dp4-pp3-n8", + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "group": "G9", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G9-dense-dp1-pp2-n8", + "group": "G9", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G9-moe-dp2-pp2-n8-burst", + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "group": "G9", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G9-moe-dp2-pp3-n8-burst", + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "group": "G9", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G9-moe-dp4-pp2-n8-burst", + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "group": "G9", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G9-moe-dp4-pp3-n8-burst", + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "group": "G9", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-moe-dp2-pp1-n8", + "group": "G10", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-moe-dp2-pp1-n8-burst", + "group": "G10", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-moe-dp2-pp2-n8", + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.24581406218246543, + "lanes": [ + 0, + 1 + ], + "makespan": 0.36803027809136213, + "multi_lane_busy_time": 0.14400000000000013, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.2400000000000002, + "lanes": [ + 0, + 1 + ], + "makespan": 0.41603027809136217, + "multi_lane_busy_time": 0.14400000000000013, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.24581406218246543, + "lanes": [ + 0, + 1 + ], + "makespan": 0.36803027809136213, + "multi_lane_busy_time": 0.14400000000000013, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.2400000000000002, + "lanes": [ + 0, + 1 + ], + "makespan": 0.41603027809136217, + "multi_lane_busy_time": 0.14400000000000013, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G10-moe-dp2-pp2-n8-burst", + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "group": "G10", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-moe-dp2-pp3-n8", + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.17581406218246537, + "lanes": [ + 0, + 1 + ], + "makespan": 0.3540302780913621, + "multi_lane_busy_time": 0.10200000000000008, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.17000000000000015, + "lanes": [ + 0, + 1 + ], + "makespan": 0.38803027809136215, + "multi_lane_busy_time": 0.10200000000000009, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.17000000000000015, + "lanes": [ + 0, + 1 + ], + "makespan": 0.4220302780913622, + "multi_lane_busy_time": 0.10200000000000009, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.17581406218246537, + "lanes": [ + 0, + 1 + ], + "makespan": 0.3540302780913621, + "multi_lane_busy_time": 0.10200000000000008, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.17000000000000015, + "lanes": [ + 0, + 1 + ], + "makespan": 0.38803027809136215, + "multi_lane_busy_time": 0.10200000000000009, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.17000000000000015, + "lanes": [ + 0, + 1 + ], + "makespan": 0.4220302780913622, + "multi_lane_busy_time": 0.10200000000000009, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G10-moe-dp2-pp3-n8-burst", + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "group": "G10", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-moe-dp4-pp1-n8", + "group": "G10", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-moe-dp4-pp1-n8-burst", + "group": "G10", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-moe-dp4-pp2-n8", + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.24581406218246543, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.36803027809136213, + "multi_lane_busy_time": 0.14400000000000013, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.2400000000000002, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.41603027809136217, + "multi_lane_busy_time": 0.14400000000000013, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.24581406218246543, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.36803027809136213, + "multi_lane_busy_time": 0.14400000000000013, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.2400000000000002, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.41603027809136217, + "multi_lane_busy_time": 0.14400000000000013, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G10-moe-dp4-pp2-n8-burst", + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "group": "G10", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-moe-dp4-pp3-n8", + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.17581406218246537, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.3540302780913621, + "multi_lane_busy_time": 0.10200000000000008, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.17000000000000015, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.38803027809136215, + "multi_lane_busy_time": 0.10200000000000009, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.17000000000000015, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.4220302780913622, + "multi_lane_busy_time": 0.10200000000000009, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.17581406218246537, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.3540302780913621, + "multi_lane_busy_time": 0.10200000000000008, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.17000000000000015, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.38803027809136215, + "multi_lane_busy_time": 0.10200000000000009, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.17000000000000015, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.4220302780913622, + "multi_lane_busy_time": 0.10200000000000009, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + }, + { + "after": "success", + "before": "admission_deadlock", + "case_id": "G10-moe-dp4-pp3-n8-burst", + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "group": "G10", + "path": "L", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-moe-dp2-pp2-n8-q5", + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.3410657678085461, + "lanes": [ + 0, + 1 + ], + "makespan": 1.357930631444134, + "multi_lane_busy_time": 0.04800000000000004, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.3359999999999983, + "lanes": [ + 0, + 1 + ], + "makespan": 1.4059306314441333, + "multi_lane_busy_time": 0.04800000000000004, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.3410657678085461, + "lanes": [ + 0, + 1 + ], + "makespan": 1.357930631444134, + "multi_lane_busy_time": 0.04800000000000004, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.3359999999999983, + "lanes": [ + 0, + 1 + ], + "makespan": 1.4059306314441333, + "multi_lane_busy_time": 0.04800000000000004, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-moe-dp2-pp2-n8-q80", + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.19231661048803433, + "lanes": [ + 0, + 1 + ], + "makespan": 0.20506736407896936, + "multi_lane_busy_time": 0.19200000000000012, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.19200000000000014, + "lanes": [ + 0, + 1 + ], + "makespan": 0.2530673640789694, + "multi_lane_busy_time": 0.19200000000000014, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.19231661048803433, + "lanes": [ + 0, + 1 + ], + "makespan": 0.20506736407896936, + "multi_lane_busy_time": 0.19200000000000012, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.19200000000000014, + "lanes": [ + 0, + 1 + ], + "makespan": 0.2530673640789694, + "multi_lane_busy_time": 0.19200000000000014, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp2-pp1-n8", + "group": "G10", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp2-pp1-n8-burst", + "group": "G10", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp2-pp2-n8", + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.5050657678085477, + "lanes": [ + 0, + 1 + ], + "makespan": 0.7022694563158771, + "multi_lane_busy_time": 0.4949342321914525, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.5050657678085477, + "lanes": [ + 0, + 1 + ], + "makespan": 0.7522694563158772, + "multi_lane_busy_time": 0.49493423219145255, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.5050657678085477, + "lanes": [ + 0, + 1 + ], + "makespan": 0.7022694563158771, + "multi_lane_busy_time": 0.4949342321914525, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.5050657678085477, + "lanes": [ + 0, + 1 + ], + "makespan": 0.7522694563158772, + "multi_lane_busy_time": 0.49493423219145255, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp2-pp2-n8-burst", + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_burst/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_burst/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_burst/system_metrics.json" + ], + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.49999999999999994, + "lanes": [ + 0, + 1 + ], + "makespan": 0.49999999999999994, + "multi_lane_busy_time": 0.49999999999999994, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.49999999999999994, + "lanes": [ + 0, + 1 + ], + "makespan": 0.5499999999999999, + "multi_lane_busy_time": 0.49999999999999994, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.5499999999999999, + "lanes": [ + 0, + 1 + ], + "makespan": 0.5499999999999999, + "multi_lane_busy_time": 0.44999999999999996, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.55, + "lanes": [ + 0, + 1 + ], + "makespan": 0.6, + "multi_lane_busy_time": 0.4499999999999999, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "EXPLAIN" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp2-pp3-n8", + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8/system_metrics.json" + ], + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.471096003569163, + "lanes": [ + 0, + 1 + ], + "makespan": 0.7002694563158774, + "multi_lane_busy_time": 0.35690399643083753, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.47109600356916304, + "lanes": [ + 0, + 1 + ], + "makespan": 0.7362694563158775, + "multi_lane_busy_time": 0.35690399643083753, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.47109600356916304, + "lanes": [ + 0, + 1 + ], + "makespan": 0.7722694563158775, + "multi_lane_busy_time": 0.3569039964308376, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.46926644195213724, + "lanes": [ + 0, + 1 + ], + "makespan": 0.6876596024435355, + "multi_lane_busy_time": 0.3947335580478633, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.4692664419521373, + "lanes": [ + 0, + 1 + ], + "makespan": 0.7236596024435356, + "multi_lane_busy_time": 0.39473355804786336, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.4692664419521373, + "lanes": [ + 0, + 1 + ], + "makespan": 0.7596596024435356, + "multi_lane_busy_time": 0.39473355804786336, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "EXPLAIN" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp2-pp3-n8-burst", + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8_burst/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8_burst/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8_burst/system_metrics.json" + ], + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.39600000000000013, + "lanes": [ + 0, + 1 + ], + "makespan": 0.4680000000000002, + "multi_lane_busy_time": 0.39600000000000013, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.39600000000000013, + "lanes": [ + 0, + 1 + ], + "makespan": 0.5040000000000002, + "multi_lane_busy_time": 0.39600000000000013, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.3960000000000002, + "lanes": [ + 0, + 1 + ], + "makespan": 0.5400000000000003, + "multi_lane_busy_time": 0.3960000000000002, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.5760000000000003, + "lanes": [ + 0, + 1 + ], + "makespan": 0.6120000000000003, + "multi_lane_busy_time": 0.21600000000000008, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.5760000000000003, + "lanes": [ + 0, + 1 + ], + "makespan": 0.6480000000000004, + "multi_lane_busy_time": 0.21600000000000014, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.5760000000000003, + "lanes": [ + 0, + 1 + ], + "makespan": 0.6840000000000004, + "multi_lane_busy_time": 0.2160000000000001, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "EXPLAIN" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp4-pp1-n8", + "group": "G10", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp4-pp1-n8-burst", + "group": "G10", + "path": "U", + "verdict": "PASS" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp4-pp2-n8", + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8/system_metrics.json" + ], + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.46946797964256276, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.580979968803131, + "multi_lane_busy_time": 0.3821624064074965, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.4694679796425628, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.630979968803131, + "multi_lane_busy_time": 0.3821624064074965, + "peak_lanes": 4, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.4299769544393909, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.5976596024435351, + "multi_lane_busy_time": 0.36608120320374826, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.42997695443939093, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.6476596024435352, + "multi_lane_busy_time": 0.3660812032037483, + "peak_lanes": 4, + "self_overlap": false + } + }, + "path": "T", + "verdict": "EXPLAIN" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp4-pp2-n8-burst", + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8_burst/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8_burst/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8_burst/system_metrics.json" + ], + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.3, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.3, + "multi_lane_busy_time": 0.3, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.3, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.35, + "multi_lane_busy_time": 0.3, + "peak_lanes": 4, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.65, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.65, + "multi_lane_busy_time": 0.55, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.6500000000000001, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.7000000000000001, + "multi_lane_busy_time": 0.55, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "EXPLAIN" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp4-pp3-n8", + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8/system_metrics.json" + ], + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.40792663008249286, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.5849799688031313, + "multi_lane_busy_time": 0.2695639759708406, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.4079266300824929, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.6209799688031313, + "multi_lane_busy_time": 0.26956397597084064, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.40792663008249286, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.6569799688031314, + "multi_lane_busy_time": 0.26956397597084064, + "peak_lanes": 4, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.40061768715858287, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.5849799688031313, + "multi_lane_busy_time": 0.2374015695633443, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.4006176871585829, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.6209799688031313, + "multi_lane_busy_time": 0.2374015695633443, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.4006176871585829, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.6569799688031314, + "multi_lane_busy_time": 0.2374015695633443, + "peak_lanes": 4, + "self_overlap": false + } + }, + "path": "T", + "verdict": "EXPLAIN" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp4-pp3-n8-burst", + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8_burst/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8_burst/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8_burst/system_metrics.json" + ], + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.21600000000000003, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.28800000000000003, + "multi_lane_busy_time": 0.21600000000000003, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.21600000000000008, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.32400000000000007, + "multi_lane_busy_time": 0.21600000000000008, + "peak_lanes": 4, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.21600000000000005, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.3600000000000001, + "multi_lane_busy_time": 0.21600000000000005, + "peak_lanes": 4, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.4680000000000002, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.4680000000000002, + "multi_lane_busy_time": 0.39600000000000013, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.4680000000000002, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.5040000000000002, + "multi_lane_busy_time": 0.3960000000000002, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/2": { + "busy_time": 0.46800000000000025, + "lanes": [ + 0, + 1, + 2, + 3 + ], + "makespan": 0.5400000000000003, + "multi_lane_busy_time": 0.39600000000000024, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "EXPLAIN" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp2-pp2-n8-q5", + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_q5/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_q5/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_q5/system_metrics.json" + ], + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.8170323968428383, + "lanes": [ + 0, + 1 + ], + "makespan": 1.5301211123654486, + "multi_lane_busy_time": 0.3829676031571622, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.8170323968428383, + "lanes": [ + 0, + 1 + ], + "makespan": 1.5801211123654486, + "multi_lane_busy_time": 0.3829676031571623, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.8027075840278453, + "lanes": [ + 0, + 1 + ], + "makespan": 1.5301211123654486, + "multi_lane_busy_time": 0.3972924159721553, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.8027075840278453, + "lanes": [ + 0, + 1 + ], + "makespan": 1.5801211123654486, + "multi_lane_busy_time": 0.39729241597215537, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "EXPLAIN" + }, + { + "after": "success", + "before": "success", + "case_id": "G10-dense-dp2-pp2-n8-q80", + "group": "G10", + "lane_metric_after": { + "MONOLITHIC/0/0": { + "busy_time": 0.5003166104880342, + "lanes": [ + 0, + 1 + ], + "makespan": 0.5130673640789692, + "multi_lane_busy_time": 0.4996833895119658, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.5003166104880342, + "lanes": [ + 0, + 1 + ], + "makespan": 0.5630673640789693, + "multi_lane_busy_time": 0.49968338951196584, + "peak_lanes": 2, + "self_overlap": false + } + }, + "lane_metric_before": { + "MONOLITHIC/0/0": { + "busy_time": 0.5003166104880342, + "lanes": [ + 0, + 1 + ], + "makespan": 0.5130673640789692, + "multi_lane_busy_time": 0.4996833895119658, + "peak_lanes": 2, + "self_overlap": false + }, + "MONOLITHIC/0/1": { + "busy_time": 0.5003166104880342, + "lanes": [ + 0, + 1 + ], + "makespan": 0.5630673640789693, + "multi_lane_busy_time": 0.49968338951196584, + "peak_lanes": 2, + "self_overlap": false + } + }, + "path": "T", + "verdict": "PASS" + } +] \ No newline at end of file diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/composition_check.json b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/composition_check.json new file mode 100644 index 00000000..6e39452e --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/composition_check.json @@ -0,0 +1,1576 @@ +{ + "K1": { + "cases": { + "G10-dense-dp2-pp1-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp2-pp1-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp2-pp2-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp2-pp2-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp2-pp2-n8-q5": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp2-pp2-n8-q80": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp2-pp3-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp2-pp3-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp4-pp1-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp4-pp1-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp4-pp2-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp4-pp2-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp4-pp3-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-dense-dp4-pp3-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp2-pp1-n8": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp2-pp1-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp2-pp2-n8": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp2-pp2-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp2-pp2-n8-q5": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp2-pp2-n8-q80": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp2-pp3-n8": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp2-pp3-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp4-pp1-n8": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp4-pp1-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp4-pp2-n8": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp4-pp2-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp4-pp3-n8": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G10-moe-dp4-pp3-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 8 + ], + "observed": [ + 8, + 128, + 8 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp2-pp1-n4": { + "conservation": { + "expected": [ + 4, + 64, + 12 + ], + "observed": [ + 4, + 64, + 12 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp2-pp1-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp2-pp2-n4": { + "conservation": { + "expected": [ + 4, + 64, + 12 + ], + "observed": [ + 4, + 64, + 12 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp2-pp2-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp2-pp3-n4": { + "conservation": { + "expected": [ + 4, + 64, + 12 + ], + "observed": [ + 4, + 64, + 12 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp2-pp3-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp4-pp1-n4": { + "conservation": { + "expected": [ + 4, + 64, + 12 + ], + "observed": [ + 4, + 64, + 12 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp4-pp1-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp4-pp2-n4": { + "conservation": { + "expected": [ + 4, + 64, + 12 + ], + "observed": [ + 4, + 64, + 12 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp4-pp2-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp4-pp3-n4": { + "conservation": { + "expected": [ + 4, + 64, + 12 + ], + "observed": [ + 4, + 64, + 12 + ], + "ok": true + }, + "outcome": "success" + }, + "G3b-moe-dp4-pp3-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G9-dense-dp1-pp2-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G9-moe-dp2-pp1-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G9-moe-dp2-pp2-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G9-moe-dp2-pp2-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G9-moe-dp2-pp3-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G9-moe-dp2-pp3-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G9-moe-dp4-pp1-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G9-moe-dp4-pp2-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G9-moe-dp4-pp2-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G9-moe-dp4-pp3-n8": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + }, + "G9-moe-dp4-pp3-n8-burst": { + "conservation": { + "expected": [ + 8, + 128, + 24 + ], + "observed": [ + 8, + 128, + 24 + ], + "ok": true + }, + "outcome": "success" + } + }, + "pass": true + }, + "K2_explain": { + "cases": { + "G10-dense-dp2-pp2-n8-burst": { + "identical_lanes": [ + "('MONOLITHIC', 0, 0, 0)", + "('MONOLITHIC', 0, 0, 1)", + "('MONOLITHIC', 0, 1, 0)", + "('MONOLITHIC', 0, 1, 1)" + ], + "pass": true, + "same_batches_and_component_durations": true + }, + "G10-dense-dp2-pp2-n8-q5": { + "identical_lanes": [ + "('MONOLITHIC', 0, 0, 0)", + "('MONOLITHIC', 0, 0, 1)", + "('MONOLITHIC', 0, 1, 0)", + "('MONOLITHIC', 0, 1, 1)" + ], + "pass": true, + "same_batches_and_component_durations": true + }, + "G10-dense-dp2-pp3-n8": { + "first_divergence": { + "batch": [ + "MONOLITHIC", + 0, + 0, + 1, + [ + "1" + ] + ], + "earlier_admission_of_same_batch": true, + "position": 12, + "start_before": 0.1836596024435351, + "start_composed": 0.16026945631587702 + }, + "identical_lanes": [ + "('MONOLITHIC', 0, 0, 0)", + "('MONOLITHIC', 0, 1, 0)", + "('MONOLITHIC', 0, 2, 0)" + ], + "pass": true, + "same_batches_and_component_durations": false + }, + "G10-dense-dp2-pp3-n8-burst": { + "identical_lanes": [ + "('MONOLITHIC', 0, 0, 0)", + "('MONOLITHIC', 0, 0, 1)", + "('MONOLITHIC', 0, 1, 0)", + "('MONOLITHIC', 0, 1, 1)", + "('MONOLITHIC', 0, 2, 0)", + "('MONOLITHIC', 0, 2, 1)" + ], + "pass": true, + "same_batches_and_component_durations": true + }, + "G10-dense-dp4-pp2-n8": { + "identical_lanes": [ + "('MONOLITHIC', 0, 0, 0)", + "('MONOLITHIC', 0, 0, 1)", + "('MONOLITHIC', 0, 0, 2)", + "('MONOLITHIC', 0, 0, 3)", + "('MONOLITHIC', 0, 1, 0)", + "('MONOLITHIC', 0, 1, 1)", + "('MONOLITHIC', 0, 1, 2)", + "('MONOLITHIC', 0, 1, 3)" + ], + "pass": true, + "same_batches_and_component_durations": true + }, + "G10-dense-dp4-pp2-n8-burst": { + "identical_lanes": [ + "('MONOLITHIC', 0, 0, 0)", + "('MONOLITHIC', 0, 0, 1)", + "('MONOLITHIC', 0, 0, 2)", + "('MONOLITHIC', 0, 0, 3)", + "('MONOLITHIC', 0, 1, 0)", + "('MONOLITHIC', 0, 1, 1)", + "('MONOLITHIC', 0, 1, 2)", + "('MONOLITHIC', 0, 1, 3)" + ], + "pass": true, + "same_batches_and_component_durations": true + }, + "G10-dense-dp4-pp3-n8": { + "identical_lanes": [ + "('MONOLITHIC', 0, 0, 0)", + "('MONOLITHIC', 0, 0, 1)", + "('MONOLITHIC', 0, 0, 2)", + "('MONOLITHIC', 0, 0, 3)", + "('MONOLITHIC', 0, 1, 0)", + "('MONOLITHIC', 0, 1, 1)", + "('MONOLITHIC', 0, 1, 2)", + "('MONOLITHIC', 0, 1, 3)", + "('MONOLITHIC', 0, 2, 0)", + "('MONOLITHIC', 0, 2, 1)", + "('MONOLITHIC', 0, 2, 2)", + "('MONOLITHIC', 0, 2, 3)" + ], + "pass": true, + "same_batches_and_component_durations": true + }, + "G10-dense-dp4-pp3-n8-burst": { + "identical_lanes": [ + "('MONOLITHIC', 0, 0, 0)", + "('MONOLITHIC', 0, 0, 1)", + "('MONOLITHIC', 0, 0, 2)", + "('MONOLITHIC', 0, 0, 3)", + "('MONOLITHIC', 0, 1, 0)", + "('MONOLITHIC', 0, 1, 1)", + "('MONOLITHIC', 0, 1, 2)", + "('MONOLITHIC', 0, 1, 3)", + "('MONOLITHIC', 0, 2, 0)", + "('MONOLITHIC', 0, 2, 1)", + "('MONOLITHIC', 0, 2, 2)", + "('MONOLITHIC', 0, 2, 3)" + ], + "pass": true, + "same_batches_and_component_durations": true + } + }, + "pass": true + }, + "K3": { + "cases": { + "G10-dense-dp2-pp1-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ] + } + }, + "G10-dense-dp2-pp2-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1 + ], + "MONOLITHIC/0/1": [ + 0, + 1 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ] + } + }, + "G10-dense-dp2-pp2-n8-q5": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1 + ], + "MONOLITHIC/0/1": [ + 0, + 1 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ] + } + }, + "G10-dense-dp2-pp2-n8-q80": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1 + ], + "MONOLITHIC/0/1": [ + 0, + 1 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ] + } + }, + "G10-dense-dp2-pp3-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1 + ], + "MONOLITHIC/0/1": [ + 0, + 1 + ], + "MONOLITHIC/0/2": [ + 0, + 1 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ], + "MONOLITHIC/0/2": [ + 0 + ] + } + }, + "G10-dense-dp4-pp1-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1, + 2, + 3 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ] + } + }, + "G10-dense-dp4-pp2-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1, + 2, + 3 + ], + "MONOLITHIC/0/1": [ + 0, + 1, + 2, + 3 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ] + } + }, + "G10-dense-dp4-pp3-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1, + 2, + 3 + ], + "MONOLITHIC/0/1": [ + 0, + 1, + 2, + 3 + ], + "MONOLITHIC/0/2": [ + 0, + 1, + 2, + 3 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ], + "MONOLITHIC/0/2": [ + 0 + ] + } + }, + "G10-moe-dp2-pp1-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ] + } + }, + "G10-moe-dp2-pp2-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1 + ], + "MONOLITHIC/0/1": [ + 0, + 1 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ] + } + }, + "G10-moe-dp2-pp2-n8-q5": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1 + ], + "MONOLITHIC/0/1": [ + 0, + 1 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ] + } + }, + "G10-moe-dp2-pp2-n8-q80": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1 + ], + "MONOLITHIC/0/1": [ + 0, + 1 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ] + } + }, + "G10-moe-dp2-pp3-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1 + ], + "MONOLITHIC/0/1": [ + 0, + 1 + ], + "MONOLITHIC/0/2": [ + 0, + 1 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ], + "MONOLITHIC/0/2": [ + 0 + ] + } + }, + "G10-moe-dp4-pp1-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1, + 2, + 3 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ] + } + }, + "G10-moe-dp4-pp2-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1, + 2, + 3 + ], + "MONOLITHIC/0/1": [ + 0, + 1, + 2, + 3 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ] + } + }, + "G10-moe-dp4-pp3-n8": { + "all_lanes": true, + "composed": { + "MONOLITHIC/0/0": [ + 0, + 1, + 2, + 3 + ], + "MONOLITHIC/0/1": [ + 0, + 1, + 2, + 3 + ], + "MONOLITHIC/0/2": [ + 0, + 1, + 2, + 3 + ] + }, + "reference": { + "MONOLITHIC/0/0": [ + 0 + ], + "MONOLITHIC/0/1": [ + 0 + ], + "MONOLITHIC/0/2": [ + 0 + ] + } + }, + "G9-moe-dp2-pp1-n8": { + "all_lanes": true, + "composed": { + "DECODE/1/0": [ + 0, + 1 + ], + "PREFILL/0/0": [ + 0, + 1 + ] + }, + "reference": { + "DECODE/1/0": [ + 0, + 1 + ], + "PREFILL/0/0": [ + 0 + ] + } + }, + "G9-moe-dp2-pp2-n8": { + "all_lanes": true, + "composed": { + "DECODE/1/0": [ + 0, + 1 + ], + "DECODE/1/1": [ + 0, + 1 + ], + "PREFILL/0/0": [ + 0, + 1 + ], + "PREFILL/0/1": [ + 0, + 1 + ] + }, + "reference": { + "DECODE/1/0": [ + 0, + 1 + ], + "DECODE/1/1": [ + 0, + 1 + ], + "PREFILL/0/0": [ + 0 + ], + "PREFILL/0/1": [ + 0 + ] + } + }, + "G9-moe-dp2-pp3-n8": { + "all_lanes": true, + "composed": { + "DECODE/1/0": [ + 0, + 1 + ], + "DECODE/1/1": [ + 0, + 1 + ], + "DECODE/1/2": [ + 0, + 1 + ], + "PREFILL/0/0": [ + 0, + 1 + ], + "PREFILL/0/1": [ + 0, + 1 + ], + "PREFILL/0/2": [ + 0, + 1 + ] + }, + "reference": { + "DECODE/1/0": [ + 0, + 1 + ], + "DECODE/1/1": [ + 0, + 1 + ], + "DECODE/1/2": [ + 0, + 1 + ], + "PREFILL/0/0": [ + 0 + ], + "PREFILL/0/1": [ + 0 + ], + "PREFILL/0/2": [ + 0 + ] + } + }, + "G9-moe-dp4-pp1-n8": { + "all_lanes": true, + "composed": { + "DECODE/1/0": [ + 0, + 1, + 2, + 3 + ], + "PREFILL/0/0": [ + 0, + 1, + 2, + 3 + ] + }, + "reference": { + "DECODE/1/0": [ + 0, + 1, + 2, + 3 + ], + "PREFILL/0/0": [ + 0 + ] + } + }, + "G9-moe-dp4-pp2-n8": { + "all_lanes": true, + "composed": { + "DECODE/1/0": [ + 0, + 1, + 2, + 3 + ], + "DECODE/1/1": [ + 0, + 1, + 2, + 3 + ], + "PREFILL/0/0": [ + 0, + 1, + 2, + 3 + ], + "PREFILL/0/1": [ + 0, + 1, + 2, + 3 + ] + }, + "reference": { + "DECODE/1/0": [ + 0, + 1, + 2, + 3 + ], + "DECODE/1/1": [ + 0, + 1, + 2, + 3 + ], + "PREFILL/0/0": [ + 0 + ], + "PREFILL/0/1": [ + 0 + ] + } + }, + "G9-moe-dp4-pp3-n8": { + "all_lanes": true, + "composed": { + "DECODE/1/0": [ + 0, + 1, + 2, + 3 + ], + "DECODE/1/1": [ + 0, + 1, + 2, + 3 + ], + "DECODE/1/2": [ + 0, + 1, + 2, + 3 + ], + "PREFILL/0/0": [ + 0, + 1, + 2, + 3 + ], + "PREFILL/0/1": [ + 0, + 1, + 2, + 3 + ], + "PREFILL/0/2": [ + 0, + 1, + 2, + 3 + ] + }, + "reference": { + "DECODE/1/0": [ + 0, + 1, + 2, + 3 + ], + "DECODE/1/1": [ + 0, + 1, + 2, + 3 + ], + "DECODE/1/2": [ + 0, + 1, + 2, + 3 + ], + "PREFILL/0/0": [ + 0 + ], + "PREFILL/0/1": [ + 0 + ], + "PREFILL/0/2": [ + 0 + ] + } + } + }, + "pass": true + }, + "sets": { + "before": "c-pr35", + "composed": "c-merged", + "reference": "after-r2" + } +} \ No newline at end of file diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/integration_compare.json b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/integration_compare.json new file mode 100644 index 00000000..bd683d45 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/integration_compare.json @@ -0,0 +1,22 @@ +{ + "counts_before": { + "error": 5, + "skipped": 22, + "passed": 16 + }, + "counts_after": { + "error": 5, + "skipped": 22, + "passed": 19 + }, + "regressions": [], + "now_passing": [], + "skip_changes": [], + "new_failures": [], + "only_before": [], + "only_after": [ + "tests.integration.test_stage_admission_pipeline_lanes::test_dense_lanes_start_in_the_same_first_forward", + "tests.integration.test_stage_admission_pipeline_lanes::test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4-expected0]", + "tests.integration.test_stage_admission_pipeline_lanes::test_moe_lanes_complete_every_request[G3a-moe-dp4-pp2-n8-expected1]" + ] +} \ No newline at end of file diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/unit_compare.json b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/unit_compare.json new file mode 100644 index 00000000..76a85f2f --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_evidence/unit_compare.json @@ -0,0 +1,132 @@ +{ + "counts_before": { + "skipped": 51, + "error": 10, + "passed": 3706, + "failed": 172 + }, + "counts_after": { + "skipped": 50, + "error": 10, + "passed": 3814, + "failed": 84 + }, + "regressions": [], + "now_passing": [ + "tests.unit.test_moe_ep_h800_profile_backfill_script::test_profile_backfill_audit_only_skips_collection_and_merge", + "tests.unit.test_moe_ep_h800_profile_backfill_script::test_profile_backfill_dry_run_snapshots_explicit_git_base_ref", + "tests.unit.test_moe_ep_h800_profile_backfill_script::test_profile_backfill_step_tp1_scope_profiles_only_missing_rows", + "tests.unit.test_moe_ep_non_dummy_matrix::test_optimization_compare_cli_writes_json_csv_and_markdown", + "tests.unit.test_moe_ep_non_dummy_matrix::test_optimization_compare_rejects_mutated_persisted_pair_manifest", + "tests.unit.test_moe_ep_non_dummy_matrix::test_run_cases_bounds_parallelism_and_serializes_ledger_writes", + "tests.unit.test_moe_ep_non_dummy_matrix::test_run_cases_preflights_every_selected_case_before_launch", + "tests.unit.test_moe_ep_non_dummy_matrix::test_run_cases_uses_filesystem_freshness_marker", + "tests.unit.test_moe_ep_non_dummy_matrix::test_source_provenance_binds_campaign_manifest_contents", + "tests.unit.test_moe_ep_non_dummy_matrix::test_source_provenance_records_campaign_and_runner_identity", + "tests.unit.test_moe_ep_non_dummy_matrix::test_source_provenance_strict_gate_rejects_dirty_tree", + "tests.unit.test_moe_ep_non_dummy_matrix::test_source_provenance_strict_gate_rejects_missing_campaign_path", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_accepts_barrier_des_timestamp_equations", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_accepts_one_batched_wave_for_all_requests", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_binds_runtime_routing_snapshot_to_independent_sidecar", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_common_mode_identity_mutation[afd_stage_idx=-1-afd_stage_idx=4-afd_stage_idx]", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_common_mode_identity_mutation[iteration_ids=[0]-iteration_ids=[7]-iteration_ids]", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_common_mode_identity_mutation[operation_id=10-operation_id=999999-operation_id]", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_common_mode_identity_mutation[schedule_epoch=0-schedule_epoch=9-schedule_epoch]", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_cross_replica_wave_identity", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_duplicate_expected_wave", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_duplicate_iteration_identity", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_extra_expected_wave", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_missing_barrier_des_start_time", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_missing_expected_wave", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_missing_iteration_identity", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_missing_runtime_routing_snapshot", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_mutated_barrier_des_end_time", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_mutated_routing_details_snapshot", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_mutated_runtime_routing_snapshot", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_operation_kind_mismatch", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_rejects_self_consistent_but_wrong_runtime_token_ledger", + "tests.unit.test_moe_ep_non_dummy_matrix::test_strict_checker_uses_independent_request_token_oracle", + "tests.unit.test_pd_disaggregation_naming_guard::test_frontier_examples_tests_use_pd_disaggregation_naming", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_classifies_simulated_oom", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_classifies_unexpected_bug", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_counts_parallel_cluster_events", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_executes_real_tiny_sequential_simulation", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_honors_falsey_factory_callables", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_rejects_effective_mode_mismatch_before_run[parallel-False]", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_rejects_effective_mode_mismatch_before_run[sequential-True]", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_rejects_generated_request_count_mismatch", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_rejects_incomplete_normal_return", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_rejects_missing_effective_mode_before_run", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_rejects_missing_parallel_cluster_event_counter", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_rejects_non_boolean_effective_mode_before_run", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_rejects_success_without_processed_events", + "tests.unit.test_sim_walltime_scaling_run_case::test_run_case_validates_output_flags_before_simulator_init", + "tests.unit.test_sim_walltime_scaling_sweep::test_child_exit_without_a_result_is_recorded_as_bug[-15]", + "tests.unit.test_sim_walltime_scaling_sweep::test_child_exit_without_a_result_is_recorded_as_bug[0]", + "tests.unit.test_sim_walltime_scaling_sweep::test_child_exit_without_a_result_is_recorded_as_bug[1]", + "tests.unit.test_sim_walltime_scaling_sweep::test_child_exit_without_a_result_is_recorded_as_bug[2]", + "tests.unit.test_sim_walltime_scaling_sweep::test_child_produced_terminal_records_are_validated_and_kept[bug-1]", + "tests.unit.test_sim_walltime_scaling_sweep::test_child_produced_terminal_records_are_validated_and_kept[simulated-oom-2]", + "tests.unit.test_sim_walltime_scaling_sweep::test_child_produced_terminal_records_are_validated_and_kept[success-0]", + "tests.unit.test_sim_walltime_scaling_sweep::test_dense_master_wrapper_is_an_exact_argument_vector_prefix", + "tests.unit.test_sim_walltime_scaling_sweep::test_empty_simulated_oom_evidence_is_a_valid_terminal_record", + "tests.unit.test_sim_walltime_scaling_sweep::test_falsey_callable_injections_are_honored", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_child_success_record_is_rejected_without_replacement[completed_requests-63]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_child_success_record_is_rejected_without_replacement[effective_parallel_mode-True]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_child_success_record_is_rejected_without_replacement[event_count-0]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_child_success_record_is_rejected_without_replacement[events_per_s-0.0]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_child_success_record_is_rejected_without_replacement[runner_sha256-not-a-sha256]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_child_success_record_is_rejected_without_replacement[sim_wallclock_s-0.0]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[bug-exit_code-True]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[bug-failure_reason-None]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[bug-oom_evidence-value2]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[bug-signal-Signals.SIGTERM]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[bug-sim_wallclock_s-not-a-number]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[bug-total_proc_s-0.0]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[simulated-oom-exit_code-0]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[simulated-oom-failure_reason-None]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[simulated-oom-oom_evidence-None]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[simulated-oom-oom_evidence-value9]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[simulated-oom-signal-Signals.SIGTERM]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[simulated-oom-sim_wallclock_s-not-a-number]", + "tests.unit.test_sim_walltime_scaling_sweep::test_invalid_failure_terminal_record_is_rejected_without_resume[simulated-oom-total_proc_s-0.0]", + "tests.unit.test_sim_walltime_scaling_sweep::test_parent_never_clobbers_a_child_result", + "tests.unit.test_sim_walltime_scaling_sweep::test_relative_output_root_is_resolved_before_child_command", + "tests.unit.test_sim_walltime_scaling_sweep::test_run_sweep_uses_requested_bounded_concurrency", + "tests.unit.test_sim_walltime_scaling_sweep::test_sigkill_is_host_oom_only_with_explicit_evidence[-9---bug]", + "tests.unit.test_sim_walltime_scaling_sweep::test_sigkill_is_host_oom_only_with_explicit_evidence[-9--Out of memory: Killed process 1234 (python)-host-oom]", + "tests.unit.test_sim_walltime_scaling_sweep::test_sigkill_is_host_oom_only_with_explicit_evidence[-9--memory cgroup out of memory-host-oom]", + "tests.unit.test_sim_walltime_scaling_sweep::test_sigkill_is_host_oom_only_with_explicit_evidence[137-Killed--bug]", + "tests.unit.test_sim_walltime_scaling_sweep::test_sigkill_is_host_oom_only_with_explicit_evidence[137-memory.events:\\nlow 0\\noom 0\\noom_kill 0\\n--bug]", + "tests.unit.test_sim_walltime_scaling_sweep::test_sigkill_is_host_oom_only_with_explicit_evidence[137-memory.events:\\nlow 0\\noom 1\\noom_kill 1\\n--host-oom]", + "tests.unit.test_sim_walltime_scaling_sweep::test_sigkill_is_host_oom_only_with_explicit_evidence[137-oom-kill: constraint=CONSTRAINT_MEMCG--host-oom]", + "tests.unit.test_sim_walltime_scaling_sweep::test_timeout_creates_a_complete_parent_owned_terminal_result" + ], + "skip_changes": [], + "new_failures": [], + "only_before": [ + "::tests.unit.test_collective_sim_zero_payload" + ], + "only_after": [ + "tests.unit.test_collective_sim_zero_payload::test_a_negative_payload_is_still_rejected", + "tests.unit.test_collective_sim_zero_payload::test_a_populated_all_to_all_costs_more_than_an_empty_one", + "tests.unit.test_collective_sim_zero_payload::test_an_empty_all_to_all_keeps_its_synchronization_latency", + "tests.unit.test_collective_sim_zero_payload::test_an_empty_reduce_scatter_keeps_its_synchronization_latency", + "tests.unit.test_mixed_layer_decode_ffn_scheduling::test_decode_ffn_dense_groups_keep_counter_order_around_a_queued_ep_wave", + "tests.unit.test_shared_forward_group_admission::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0]", + "tests.unit.test_shared_forward_group_admission::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[1]", + "tests.unit.test_stage_admission_pp_tools::test_apply_patch_keeps_every_section_of_a_repeated_file", + "tests.unit.test_stage_admission_pp_tools::test_apply_patch_reads_a_trimmed_context_line", + "tests.unit.test_stage_admission_pp_tools::test_apply_patch_rejects_an_unknown_hunk_line", + "tests.unit.test_stage_admission_pp_tools::test_fixed_base_loses_the_controls_without_a_mismatch", + "tests.unit.test_stage_admission_pp_tools::test_ideal_after_revision_matches_and_base_controls_hold", + "tests.unit.test_stage_admission_pp_tools::test_missing_placement_log_fails_the_comparison", + "tests.unit.test_stage_admission_pp_tools::test_overlay_acceptance_compares_file_sets[False-True]", + "tests.unit.test_stage_admission_pp_tools::test_overlay_acceptance_compares_file_sets[True-False]", + "tests.unit.test_stage_admission_pp_tools::test_vllm_round_with_a_late_lane_is_reported", + "tests.unit.test_stage_execution_context::test_active_full_stage_ticket_is_refused_without_changing_the_stage", + "tests.unit.test_stage_execution_context::test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave", + "tests.unit.test_stage_execution_context::test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket", + "tests.unit.test_stage_execution_context::test_queued_ep_wave_orders_full_stage_work_on_both_sides" + ] +} \ No newline at end of file diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_run_suites.sh b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_run_suites.sh new file mode 100755 index 00000000..2d6b9238 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/w9_01_stage_admission_ordering/composition_run_suites.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +# Run tests/unit and tests/integration of one tree with JUnit output. +set -u +tree="$1"; out="$2" +cd "$tree" +export PYTHONPATH="$tree" WANDB_DISABLED=true VIDUR_DISABLE_WANDB=1 FRONTIER_TMP_ROOT=/data/ycfeng/tmp PYTHONDONTWRITEBYTECODE=1 +for suite in unit integration; do + timeout 1800 /data/ycfeng/envs/frontier-py310/bin/python -m pytest "tests/$suite" -q -p no:cacheprovider --continue-on-collection-errors \ + --basetemp="$out/tmp-$suite" --junitxml="$out/$suite.xml" > "$out/$suite.log" 2>&1 + echo "$suite exit $?" >> "$out/$suite.log" +done From 9b5d7be5c67230ece844d564c6e718cfc1f267c6 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 15:46:45 +0800 Subject: [PATCH 49/80] docs(step9): complete P1(b) and propose the D9-2 report key The W9-01 fix on main lets the multi-lane PP shapes run, so P1(b) now probes seven MoE shapes (attn_dp 1 and 2, PP 1-3, burst and staggered arrivals); all complete 6/6. A scorer derives each report's stage-0 forward group after the fact and scores candidate keys by pairwise order and by a VllmDPLoadBalancer replay. The pinned vLLM source shows DP engines pair their k-th forwards through the per-forward DP all-reduce, with a blocking dummy forward on idle iterations, so the reference key is the shared forward index. The ForwardSyncState id merges PP3 cold fills and a per-lane counter drifts under staggered arrivals; a group-anchored key, max(joinable stage-0 group, last admitted key + 1), matches every scored report. It is recorded as a proposal awaiting the user's decision, with its residuals and the new W9-03 observation (Frontier lanes do not model the reference's DP lockstep under PP). --- .../design.md | 161 + .../issues.md | 52 +- .../plan.md | 75 + .../progress.md | 17 +- .../step9_p1b/analyze_keys.py | 225 + .../step9_p1b/evidence/key_scores.json | 3715 +++++++++++++++++ .../step9_p1b/probe_boundaries.py | 165 + .../summary.md | 1 + ...st_report_2026-09-22_w9_pp_dp_placement.md | 58 +- 9 files changed, 4464 insertions(+), 5 deletions(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/analyze_keys.py create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/evidence/key_scores.json create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/probe_boundaries.py diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md index 4bfd2dc1..1e76a5b7 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md @@ -7,6 +7,7 @@ scope decisions and the pre-measurement expectation for that package. | Date | Change | | --- | --- | +| 2026-09-23 | Added "Design checkpoint D9-2: the key from the fourth shape": reference lockstep facts, seven-shape P1(b) scores, the proposed group-anchored key rule with its invariant argument and residuals, W9-03 pointer. Proposal only; awaits the user's decision. | | 2026-09-22 | Added the design checkpoint section: D9-1 payload settled from the P1 oracle; D9-2 key left open because the `ForwardSyncState` candidate fails invariant I5 at PP>1 and the I1/I5 trade-off is only observable on the shape blocked by W9-01. | | 2026-09-22 | W9 second review (user-directed quality gates): section "What the code already provides" added; planned-edits rows for the hook payload, the call site and the CPU oracle amended; plan §18.12 R9-01..R9-08. | | 2026-09-22 | Created. Source-backed design for Checkpoint D's W3 half, with the scope decisions and their evidence. | @@ -553,3 +554,163 @@ does can only be decided by observing a shape with `attn_dp > 1` **and** The checkpoint therefore closes with D9-1 fixed and D9-2 open. Implementing a key rule now would mean choosing between two invariants with no way to test the choice, which is the kind of unfalsifiable design the gates exclude. + +## Design checkpoint D9-2: the key from the fourth shape (2026-09-23) + +W9-01 is fixed on `main` (PR 36) and merged forward, so the shape that decides +D9-2 now runs. This section records what the probe shows and the proposed key +rule. The rule is a proposal: it needs the user's decision before P2. + +### The reference key is the shared forward index + +R9-08 said DP engines are not iteration-lockstep, because the unfinished-work +all-reduce runs only every 32 steps. The pinned source shows lockstep by +another route: + +| Fact | Source | +| --- | --- | +| An iteration that schedules no tokens enqueues the empty output, then blocks on the oldest queued output. | `core.py:364-420` (`step_with_batch_queue`) | +| After such an iteration the busy loop runs `execute_dummy_batch`, a blocking collective RPC to every worker, each of which runs `_dummy_run(1)`. | `core.py:1170-1195`; `multiproc_executor.py:197-199`; `gpu_worker.py:556-557` | +| An empty scheduler output runs no forward. | `gpu_model_runner.py:2250-2254` | +| Every real or dummy forward on every stage joins its peers' DP all-reduce in `get_dp_padding` (CUDA graphs enabled). MoE layers add EP collectives. | `gpu_model_runner.py:1904-1925`; `forward_context.py:72-85` | + +So each iteration of a running engine launches exactly one forward, real or +dummy, and on every stage the k-th forward of one engine pairs with the k-th +forward of each peer. The `step_counter` a report carries is therefore the +index of that shared forward. Peers reporting the same forward compare equal. +The next iteration is strictly greater. The 32-step all-reduce only decides +when a wave ends. + +Frontier's shared stage-0 forward group is the analog of that index. +`StageExecutionContext.bind_forward_group` gives every MoE lane batch that +starts one stage-0 forward the same id, and it hands out ids in forward order. +Idle lanes take part in the same forward's layer rooms. + +### What each report describes + +The D9-1 rule sorts every report into one of three kinds: + +| Kind | When | Forward it describes | +| --- | --- | --- | +| admission | admitted with `running_after < stages`, published on its own | the admitted batch's stage-0 group | +| completion_folded | completion whose lane holds an admission made with a full pipeline | the folded batch's group | +| completion_only | completion with no folded admission | the lane's next Frontier forward (the reference runs a dummy forward here) | + +The probe (`step9_p1b/probe_boundaries.py`) records these boundaries with the +stage-0 state at each one, and records every batch's stage-0 group and each +room's group. `step9_p1b/analyze_keys.py` derives each report's group after +the fact and scores the candidate keys against it in two ways: + +- pairwise order: peer splits (the same group gets different keys, I1); merges + (different groups get one key; on one lane this is I5); inversions (the + order is reversed); +- replay: the reports drive a fresh `VllmDPLoadBalancer`, and the score is the + milliseconds in which the frontend-visible counts differ from the replay + keyed by the groups. + +Shapes: 6-layer synthetic MoE model, analytical CC backend, round-robin +placement, dummy predictor, prefill 16 and decode 3 tokens, 6 requests. The +burst shapes are offline at 1e6 QPS; the staggered shapes are online Poisson at +20 QPS. Every shape completes 6/6. + +| Shape | Reports (adm / fold / only) | A: `ForwardSyncState` next id | Lane report counter | Group-anchored (proposed) | +| --- | --- | --- | --- | --- | +| MoE dp2 PP1 burst | 0 / 12 / 0 | 0 / 0 / 0, 0 ms | 0 / 0 / 0, 0 ms | 0 / 0 / 0, 0 ms | +| MoE dp1 PP2 burst | 1 / 13 / 1 | 0 / 0 / 0, 0 ms | 0 / 0 / 0, 0 ms | 0 / 0 / 0, 0 ms | +| MoE dp1 PP3 burst | 7 / 9 / 7 | 0 / 1 / 0, 400 ms | 5 / 0 / 0, 0 ms | 0 / 0 / 0, 0 ms | +| MoE dp2 PP2 burst | 4 / 12 / 4 | 0 / 0 / 0, 0 ms | 4 / 0 / 0, 0 ms | 0 / 0 / 0, 0 ms | +| MoE dp2 PP2 staggered | 5 / 12 / 5 | 0 / 0 / 0, 0 ms | 6 / 0 / 0, 0 ms | 0 / 0 / 0, 0 ms | +| MoE dp2 PP3 burst | 4 / 14 / 4 | 0 / 4 / 0, 148 ms | 0 / 0 / 0, 0 ms | 0 / 0 / 0, 0 ms | +| MoE dp2 PP3 staggered | 7 / 11 / 7 | 4 / 1 / 0, 200 ms | 15 / 7 / 5, 0 ms | 0 / 0 / 0, 0 ms | + +Cells are splits / merges / inversions, then the replay mismatch. The scored +reports are those whose forward exists in the run; 12 final drain completions +have no later forward and are left out. + +- **A** merges the PP3 cold fill. Several admissions on one lane read one + value, because the id advances when a room opens, not per iteration. The + same happens in the dp2 PP3 burst, where the two lanes' first and second + admissions all read 0. The replay differs for 148–400 ms. +- **The lane report counter** is exact whenever every lane runs the same + sequence of report kinds. With staggered arrivals the lanes run different + numbers of completion-only iterations and the counters drift apart. In dp2 + PP3 staggered, 6 splits, 5 merges and 5 inversions fall on reports that do + carry a real forward. Nothing brings the counters back together: in the + reference the dummy forwards keep lanes aligned, and Frontier has no such + forwards. A variant of the proposed rule that also advances on every + completion-only report drifts in the same way, with the same 6/5/5. +- **Group-anchored** matches every scored report in all seven shapes, and on + every report it equals `predicted_group`. That reference predictor needs + the count of the lane's admitted batches that have not started stage 0, + which a policy cannot read. + +### Proposed rule (D9-2) + +For lane `l` of the Replica, with `C` the Replica's stage-0 context: + +``` +key(l) = max(C.joinable_forward_group_id, last_admitted_key[l] + 1) +``` + +`joinable_forward_group_id` is the bound group's id while it is not sealed. +Otherwise it is the next id. A batch admitted now joins exactly that group, +unless its lane already has a batch in it, in which case the `+ 1` term takes +over. + +- At an admission: `last_admitted_key[l] = key(l)`. The admission is + reported now while `num_running_batches < num_pipeline_stages`. Otherwise + the key is held in `held_key[l]`. +- At a completion: report `held_key.pop(l)` if a key is held. Otherwise report + `key(l)` and store nothing. + +Against the invariants: + +| Invariant | How the rule meets it | Evidence | +| --- | --- | --- | +| I1 | Peers of one forward read one Replica-scoped group id. | 0 splits in 7 shapes. | +| I2 | Group ids only grow, and `+ 1` orders a lane's later forwards. | 0 inversions. | +| I3 | A folded admission reports once, with the completion it joins. | D9-1 folding. | +| I4 | Unchanged counts are still dropped inside `VllmDPLoadBalancer.report`. | Unchanged code. | +| I5 | Each consecutive cold-fill admission on a lane gets `+ 1`. No spacing constant. | dp1 and dp2 PP3 cold fills. | +| I6 | Two integers per lane, overwritten in place. | Bounded by `attn_dp`. | + +At PP=1 every admission is folded, and the held key is the group of the +forward that completes. Peers compare equal and later forwards compare +greater, so the comparisons match the current `ForwardSyncState.get_step_id` +key. C2's byte-identical PP=1 check tests exactly that. The policy's +`ForwardSyncState` import and use are then superseded and removed. + +New surface, compared with the planned edits above: one read-only property on +`StageExecutionContext` over two existing fields, in place of the planned +`ForwardSyncState` accessor, and two per-lane dicts in the policy scheduler. +The policy reaches the context through the base class's +`_stage_execution_contexts[(replica_id, 0)]`. For a dense model the group is +never bound (`bind_forward_group` is MoE-only), so the key falls back to the +lane's admission counter. The policy admits a dense model only at +`attn_dp=1`, where placement has one choice. + +### What the rule does not reproduce + +1. **The reference's dummy iterations.** A completion-only report shares its + key with the lane's next report whenever Frontier runs no forward between + them. The reference gives them keys `k` and `k+1`. There are 17 such pairs + in the seven shapes, 3 of them with changed counts (two in the dp2 PP3 + burst drain, one in dp1 PP3). The effect: the coordinator does not latch + between the two reports, so the frontend can see the later counts one + publication interval (100 ms) early. No Frontier forward exists to anchor + a separate key. Inventing one would bring back the drift measured above. +2. **The reference's forward pairing when lanes diverge.** In the reference, + an engine with nothing new to schedule blocks and then runs a dummy + forward, and its peers' next forward waits for it. Frontier lanes do not + block. In the dp2 PP3 staggered probe, lane 1's first batch joins the group + that lane 0 opened 1.3 ms earlier. At 154.3 ms lane 1's batch `b26` joins + lane 0's open group 2; there, the reference would pair engine 1's dummy + forward with that forward and `b26` with the next one. This is a + forward-model difference, not a key defect: the key follows Frontier's own + grouping. It is recorded as W9-03 in `issues.md` for a separate decision. + Only source reading supports it; G4's `engine_iteration` records would + measure it. + +The analytical backend runs MoE `attn_dp=2, PP=3`. W9-02's rejection is the +collective-sim topology rule. The P3/P4 matrix can therefore restore the +multi-lane PP3 row that I5 needs (C1 amendment, part of this proposal). diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md index 69462b90..a631a0d1 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | W9-01 remaining step 3 done (P1(b) complete, D9-2 proposed); W9-02 narrowed to the collective-sim backend; W9-03 recorded (reference DP lockstep under PP, observation). | | 2026-09-23 | W9-01: merged forward (`dd9b8d9`); composition check passes on `03d5f24`. | | 2026-09-23 | W9-01: PR 36 ran its pre-merge untrack (P6, `4d08c5d`); the copies here are now the only published records of that task. | | 2026-09-23 | W9-01: PR 36 round-2 review remediation recorded; the composition check now also reruns PR 36 groups G9 and G10. | @@ -160,7 +161,9 @@ Remaining here, in order: after a harness drain-reader fix. With W2 two Poisson PDD PP3 cells drain under the pre-merge rule; both complete on the merged tree. 3. If they pass, resume Step 9 P1(b) and the design checkpoint D9-2. - In progress. + Done: the fourth shape and three more multi-lane PP shapes complete 6/6; + the D9-2 proposal is in `design.md` ("Design checkpoint D9-2") and waits + for the user's decision. Step 9 C1's PP3 row stays on `attn_dp=1` (W9-02). @@ -168,6 +171,9 @@ Step 9 C1's PP3 row stays on `attn_dp=1` (W9-02). Status: open, expected behavior, affects plan wording only. Found: 2026-09-22, Step 9 package P1(b). +Narrowed 2026-09-23: the rejection is the collective-sim topology rule. With +the analytical CC backend, MoE `attn_dp=2, PP=3` constructs and completes 6/6 +(P1(b) probe, `moe_dp2_pp3_burst` and `moe_dp2_pp3_staggered`). Construction fails with: @@ -180,3 +186,47 @@ The rejection is correct: 2 lanes x 3 stages is 6 devices against a node size of 4. Plan acceptance criterion C1 lists MoE `attn_dp=2` at both PP2 and PP3; the PP3 row must be `attn_dp=1`, or must choose a device count that divides the node size. C1 was amended accordingly. + +## W9-03 Frontier does not model the reference's DP engine lockstep under PP + +Status: open, observation from source reading, not measured. Outside Step 9's +scope; decision pending with the user. +Found: 2026-09-23, Step 9 package P1(b), while deriving the D9-2 key. + +### Reference (pinned `.real-engine/vLLM-BS`) + +- An iteration that schedules no tokens enqueues the empty output. It then + blocks on the oldest queued output (`core.py:364-420`). +- The busy loop then runs `execute_dummy_batch` (`core.py:1170-1195`). This + is a blocking collective RPC: every worker runs `_dummy_run(1)` + (`multiproc_executor.py:197-199`, `gpu_worker.py:556-557`). +- Every real or dummy forward on every stage joins its peers' DP all-reduce + in `get_dp_padding` when CUDA graphs are enabled + (`gpu_model_runner.py:1904-1925`, `forward_context.py:72-85`). MoE layers + add EP collectives. + +The k-th forward of one engine therefore pairs with the k-th forward of every +peer, per stage. An engine that is only waiting for an output delays its +peers' next forward until its own wait and dummy forward finish. + +### Frontier (probe `moe_dp2_pp3_staggered`) + +- Lanes never block on their oldest output. +- A lane's batch can join a group that a peer's batch has already opened. + Lane 1's first batch arrives at 52.3 ms and joins group 0, which lane 0 + opened 1.3 ms earlier. +- At 154.3 ms, lane 1's `b26` joins lane 0's open group 2. The reference + would pair engine 1's dummy forward with that forward, and `b26` with the + next one. + +### Effect + +The difference affects the timing of MoE `attn_dp > 1, PP > 1` runs +(co-location and PDD) when lanes have unequal work. Its size is unknown. The +G4 trace would measure it: `engine_iteration` records carry +`(engine, wave, step)`. + +It is not a report-key defect. The proposed D9-2 key follows Frontier's own +grouping, and its residual 1 in `design.md` is the part of this difference +that reaches the report stream. + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md index 719afa8c..dea67ff1 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | §18.15 added: P1(b) completed on seven shapes; D9-2 proposal (group-anchored key) and the C1 PP3 amendment await the user's decision. §18.13 blocker marked resolved. | | 2026-09-23 | §18.14 results: K1–K4 pass on `03d5f24`; K2 amended after measuring for online cells (one cell's batches differ after an earlier admission). | | 2026-09-23 | §18.14 added: W9-01 merge-forward and the composition check, with pass criteria fixed before measuring. | | 2026-09-22 | Step 9 execution started. §18.13 added: P1(a) oracle complete with the state-table evidence, P1(b) blocked by the pre-existing stage-admission deadlock W9-01 (`issues.md`), design checkpoint D9-2 left open because the candidate key fails invariant I5 and I1/I5 can only be settled on the deadlocking shape. C1 amended per W9-02: the PP3 row uses `attn_dp=1`. | @@ -1082,6 +1083,8 @@ Start approval was given ("开始执行step9"). P1(a) is complete and P1(b) stop **Consequence.** Step 9 packages P2, P3, P4, G3, G4 and G5 all depend on the design checkpoint or on a running `attn_dp=2, PP=2` shape. They are paused pending the user's scope decision on W9-01. P1 and its records are complete. +**Resolved 2026-09-23.** W9-01 is fixed on `main` (PR 36) and merged forward (§18.14). P1(b) is complete and D9-2 is proposed in §18.15. + ### 18.12 Second review (2026-09-22, user-directed): codebase integration and quality gates Review question, as set by the user: is every planned change to Frontier's core modules grounded in the whole codebase, readable and maintainable, high-value (fidelity or simulation function, not replaceable), and free of hard-coding, temporary patches, over-defensive code, redundant design and vague names. Findings are against the code on `c231322`; each states the source it rests on and what it changes in this plan. None of them changes source now. @@ -1152,3 +1155,75 @@ holds where batch contents are fixed at t=0 (offline and burst). For online cells K2 now accepts a batch difference when the first divergence is an identical batch admitted earlier on the same stage and lane. +### 18.15 P1(b) completed and the D9-2 proposal (2026-09-23) + +The fourth shape runs after the merge-forward. P1(b) now covers seven shapes: + +- MoE `attn_dp` 1 and 2; +- PP 1, 2 and 3; +- offline bursts, plus online Poisson for `attn_dp=2` at PP 2 and 3. + +All seven complete 6/6. The probe and the scorer are +`step9_p1b/probe_boundaries.py` and `step9_p1b/analyze_keys.py`. The scores +are in `step9_p1b/evidence/key_scores.json`, and the full argument is in +`design.md` ("Design checkpoint D9-2: the key from the fourth shape"). + +**Reference correction (amends R9-08).** Peer step counters are aligned +through forward pairing. Each iteration launches exactly one forward, real or +the blocking dummy. On every stage, each forward joins the peers' DP +all-reduce and the MoE collectives. The report key is therefore the index of +the shared forward. Frontier's analog is the stage-0 forward-group id. + +**Scores.** Each candidate is compared with the report's actual stage-0 +group. + +| Candidate | Result | +| --- | --- | +| `ForwardSyncState` next id (A) | Merges the PP3 cold fill; 148–400 ms of frontend-visible mismatch in replay. | +| Lane report counter | Drifts under staggered arrivals: 6 splits, 5 merges and 5 inversions on real-forward reports. | +| Every-report variant of the proposal | Drifts in the same way. | +| Group-anchored (proposed) | 0 splits, merges or inversions and 0 ms mismatch in all seven shapes. | + +**Proposed D9-2 rule.** + +``` +key(l) = max(C.joinable_forward_group_id, last_admitted_key[l] + 1) +``` + +Here `C` is the Replica's stage-0 context and `joinable_forward_group_id` is +the bound group while it is unsealed, otherwise the next id. The rule updates +state as follows: + +- An admission stores `key(l)` as `last_admitted_key[l]`. +- It reports that key while the pipeline has room. Otherwise it holds the key + for the next completion. +- A completion reports the held key, or `key(l)` without storing it. + +This meets I1–I6, and at PP=1 it keeps today's comparisons (C2). + +New surface: + +- one read-only property on `StageExecutionContext`, replacing the planned + `ForwardSyncState` accessor; +- two per-lane dicts in the policy scheduler. + +The policy's `ForwardSyncState` key is removed. + +**Residuals (documented, not fixed).** + +1. A completion-only report shares its key with the lane's next report when + no Frontier forward runs between them. The reference gives them `k` and + `k+1`. This happens 17 times in the seven shapes, 3 with changed counts. At + worst the frontend sees the later counts one 100 ms publication early. +2. The reference's forward pairing when lanes diverge (W9-03) is a + forward-model difference and is outside Step 9. + +**C1 amendment (part of the proposal).** Restore a MoE `attn_dp=2, PP=3` row +that uses the analytical backend. W9-02 is specific to collective-sim, and I5 +needs a multi-lane PP3 case. + +**Decision needed before P2.** The user chooses the D9-2 rule. Once it is +chosen, P2 implements it with the D9-1 hook and the guard change. P3/P4 carry +the §18.11 matrix with the restored PP3 row. P2–P6 and G3–G5 stay paused until +then. + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 883c979b..092642c7 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | Step 9 P1(b) completed on seven shapes; D9-2 proposal recorded (design.md, plan §18.15); W9-03 observation; awaits the user's D9-2 decision. | | 2026-09-23 | W9-01 merge-forward: `origin/main` merged (`dd9b8d9`); composition check K1–K4 pass on `03d5f24` (drain-reader fix); Step 9 P1(b) resumed. | | 2026-09-23 | PR 36 pre-merge untrack (P6, `4d08c5d`) done; W9-01 copies refreshed. Merge-forward waits for the PR 36 merge. | | 2026-09-23 | PR 36 round-2 review remediation recorded; the W9-01 composition check extended to PR 36 groups G9 and G10. | @@ -221,8 +222,8 @@ User start signal: "开始执行step9", with the quality gates repeated (readabi | Package | State | Evidence | | --- | --- | --- | | P1(a) reference-loop oracle | completed | `tests/comparison/dp_placement_pp/reference_loop.py`; `tests/unit/test_dp_placement_reference_loop.py` (9 passed, 1.21 s, `frontier-py310`). §18.11 state table confirmed as written; PP=1 shown to degenerate to "every iteration schedules and applies"; depth 3 shown to allow two consecutive admission-only publications, which rules out any stride constant. | -| P1(b) Frontier boundary probe | blocked until 2026-09-23, resumed (see below) | Three shapes probed (`attn_dp=2 PP=1`, `attn_dp=1 PP=2`, `attn_dp=1 PP=3`), tables in `plan.md` §18.13. The fourth shape, MoE `attn_dp=2, moe_ep=2, PP=2`, drains the event queue with requests unfinished — pre-existing defect W9-01 in `issues.md`. | -| Design checkpoint (D9-1, D9-2) | open | D9-1's payload is settled (the completion hook signature already carries lane, load and a key source). D9-2 is not: the candidate key `ForwardSyncState._next_step_id_by_replica` satisfies I1, I2, I3, I4 and I6 on the runnable shapes but fails I5, and no alternative can be checked against I1 without a running `attn_dp>1, PP>1` shape. | +| P1(b) Frontier boundary probe | completed 2026-09-23 (see "Step 9 P1(b) and D9-2" below) | Three shapes probed (`attn_dp=2 PP=1`, `attn_dp=1 PP=2`, `attn_dp=1 PP=3`), tables in `plan.md` §18.13. The fourth shape, MoE `attn_dp=2, moe_ep=2, PP=2`, drains the event queue with requests unfinished — pre-existing defect W9-01 in `issues.md`. | +| Design checkpoint (D9-1, D9-2) | D9-1 settled; D9-2 proposed 2026-09-23, awaiting decision | D9-1's payload is settled (the completion hook signature already carries lane, load and a key source). D9-2 is not: the candidate key `ForwardSyncState._next_step_id_by_replica` satisfies I1, I2, I3, I4 and I6 on the runnable shapes but fails I5, and no alternative can be checked against I1 without a running `attn_dp>1, PP>1` shape. | | P2–P6, G3–G5 | paused | All depend on the design checkpoint or on that shape. | W9-01 is not caused by this PR: `stage_execution_context.py`, `replica_stage_schduler.py` and `stage_contexts.py` are byte-identical to `main`. It is unobserved because every Simulator-level test with `attn_dp > 1` uses `num_pipeline_stages = 1` and no shipped example sets `attn_dp > 1`. Scope decision requested from the user; recommendation is to fix it as a separate correctness item rather than inside this feature branch. @@ -247,6 +248,18 @@ W9-02: `attn_dp=2, moe_ep=2, PP=3` is rejected at construction (6 devices agains | K4 | `composition_run_suites.sh` on both trees; `composition_compare_junit.py` | same | 0 regressions; export-only failures are git-metadata failures; 487 targeted tests pass | | Report | — | `test_report_2026-09-23_w9_01_composition_check.md` | PASS | +### Step 9 P1(b) and D9-2 (2026-09-23) + +| Step | Command / action | Evidence | Result | +| --- | --- | --- | --- | +| Reference check | Read the busy loop, `step_with_batch_queue`, `execute_dummy_batch` and `get_dp_padding` in `.real-engine/vLLM-BS` | `design.md` "Design checkpoint D9-2", reference table | Each iteration launches one forward, real or the blocking dummy. Forwards pair per stage through the DP all-reduce, so the step key is the shared forward index. Amends R9-08. | +| Probe | `probe_boundaries.py ` for 7 shapes, one process each; `PYTHONPATH=`, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`, `FRONTIER_TMP_ROOT=/data/ycfeng/tmp`, `frontier-py310` | `/data/ycfeng/tmp/issue26-correctness-pr/step9_p1b//` | All 7 complete 6/6 (108–210 records). The extended fields are stage-0 sealed, lane busy, lane queue and room group. | +| Scoring | `analyze_keys.py /data/ycfeng/tmp/issue26-correctness-pr/step9_p1b ` | `step9_p1b/evidence/key_scores.json` | The group-anchored key has 0 splits, merges or inversions and 0 ms replay mismatch in all 7 shapes. A and the lane counter fail as tabulated in plan §18.15. | +| Variant | Every-report-advance variant, candidate `every_report_advances` in `analyze_keys.py` | `key_scores.json` | Drifts on dp2 PP3 staggered: 6/5/5 on real-forward reports. Rejected. | +| Residual count | Completion-only report followed by an equal-key report on the same lane | `key_scores.json` rows | 17 pairs, 3 with changed counts. | +| Records | `design.md`; plan §18.15 and the §18.13 resolution; `issues.md` (W9-01 step 3, W9-02 narrowed, W9-03); test report §2 addendum | — | Done | +| Decision | D9-2 rule and the C1 PP3 amendment | — | Awaiting the user | + ### G1 ground-truth instrumentation (2026-09-22, completed) Runs in parallel with P1 in the work graph and does not depend on W9-01. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/analyze_keys.py b/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/analyze_keys.py new file mode 100644 index 00000000..7c4890e6 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/analyze_keys.py @@ -0,0 +1,225 @@ +"""Step 9 P1(b): score report-key candidates against forward membership. + +Reads the probe files written by ``probe_boundaries.py``. For every report the +opt-in policy would emit under the settled D9-1 rule (an admission is +published on its own only while ``running_after < stages``; otherwise it is +folded into the lane's next completion report), it derives the report's +*target*: the stage-0 forward group of the iteration the report describes. + +* admission published on its own: the group the admitted batch ran in; +* completion with a folded admission: the folded batch's group; +* completion without one (``completion_only``): the lane launches no forward + of its own here (the reference runs a dummy forward), so the target is the + lane's next Frontier forward: the first group after the lane's newest batch + in which the lane takes part after the boundary, with a real or an idle + batch. + +Targets are Frontier's own forward grouping, read after the fact. Each +candidate key is read at the boundary and scored in two ways: + +1. Pairwise order against the target, split into peer splits (equal target, + different key: invariant I1), merges (different targets, equal key: + I5 is the one-lane case) and inversions. +2. Replay: the reports, keyed by the candidate, drive a fresh + ``VllmDPLoadBalancer`` and the frontend-visible counts are sampled every + millisecond; the score is the number of milliseconds in which they differ + from the replay keyed by the target. + +Candidates: + +* ``A``: ``ForwardSyncState._next_step_id_by_replica`` (the plan's first); +* ``lane_counter``: the lane's number of earlier reports; +* ``next_group``: the stage-0 context's next forward-group id; +* ``predicted_group``: the group a batch admitted now would join, from + stage-0 state and the lane's admitted batches that have not started stage 0 + (for a completion with a folded admission, the value captured at that + admission); +* ``group_anchored``: the same group from state a policy can hold: + ``max(open-or-next stage-0 group, lane's last admitted key + 1)``, where the + first term is the bound group while it is unsealed, else the next group id. + An admission stores its key as the lane's last admitted key; a completion + with a folded admission reports the key held for it; a completion without + one reads the formula and stores nothing; +* ``every_report_advances``: the same formula, but a completion without a + folded admission also stores its key (a per-iteration increment like the + reference's ``step_counter``). + +Usage: python analyze_keys.py +""" +from __future__ import annotations + +import copy +import json +import sys +from collections import defaultdict +from pathlib import Path + +from frontier.scheduler.request_load import RequestLoad +from frontier.scheduler.utils.vllm_dp_load_balancer import VllmDPLoadBalancer + +CANDIDATES = ("A", "lane_counter", "next_group", "predicted_group", "group_anchored", + "every_report_advances") + + +def stage0_membership(records): + group_of_batch = {} + rooms_by_lane = defaultdict(list) + for record in records: + if record["kind"] == "stage_start" and record["stage"] == 0: + group_of_batch[record["batch"]] = record["group"] + if record["kind"] == "room" and record["stage"] == 0 and record["group"] is not None: + rooms_by_lane[record["lane"]].append((record["seq"], record["group"])) + start_seq = {record["batch"]: record["seq"] for record in records + if record["kind"] == "stage_start" and record["stage"] == 0} + return group_of_batch, rooms_by_lane, start_seq + + +def open_or_next_group(record): + if record["stage0_bound_group"] is not None and not record["stage0_sealed"]: + return record["stage0_bound_group"] + return record["stage0_next_group"] + + +def predicted_group(record, admitted_by_lane, start_seq): + """Group a batch admitted at this boundary would join, from stage-0 state.""" + not_started_ahead = sum( + 1 for batch, _ in admitted_by_lane[record["lane"]] + if batch != record["batch"] and start_seq.get(batch, float("inf")) > record["seq"] + ) + open_group = (record["stage0_bound_group"] is not None + and not record["stage0_sealed"] and not record["lane_stage0_busy"]) + first = record["stage0_bound_group"] if open_group else record["stage0_next_group"] + return first + not_started_ahead + + +def reports(result): + records = result["records"] + group_of_batch, rooms_by_lane, start_seq = stage0_membership(records) + admitted_by_lane = defaultdict(list) + folded = {} + newest_group = defaultdict(lambda: -1) + last_admitted_key = defaultdict(lambda: -1) + last_report_key = defaultdict(lambda: -1) + count_by_lane = defaultdict(int) + rows = [] + for record in records: + if record["kind"] not in ("admit", "complete"): + continue + lane = record["lane"] + anchored = max(open_or_next_group(record), last_admitted_key[lane] + 1) + advanced = max(open_or_next_group(record), last_report_key[lane] + 1) + if record["kind"] == "admit": + admitted_by_lane[lane].append((record["batch"], record["seq"])) + key = predicted_group(record, admitted_by_lane, start_seq) + last_admitted_key[lane] = anchored + last_report_key[lane] = advanced + newest_group[lane] = max(newest_group[lane], group_of_batch.get(record["batch"], -1)) + if record["running_after"] < record["stages"]: + row = {"kind": "admission", "target": group_of_batch.get(record["batch"]), + "predicted_group": key, "group_anchored": anchored, + "every_report_advances": advanced} + else: + folded[lane] = (record["batch"], key, anchored, advanced) + continue + else: + if lane in folded: + batch, key, held, held_advanced = folded.pop(lane) + row = {"kind": "completion_folded", "target": group_of_batch.get(batch), + "predicted_group": key, "group_anchored": held, + "every_report_advances": held_advanced, "folded_batch": batch} + else: + later = [group for seq, group in rooms_by_lane.get(lane, []) + if seq > record["seq"] and group > newest_group[lane]] + last_report_key[lane] = advanced + row = {"kind": "completion_only", "target": min(later) if later else None, + "predicted_group": predicted_group(record, admitted_by_lane, start_seq), + "group_anchored": anchored, "every_report_advances": advanced} + row.update(seq=record["seq"], time=record["time"], lane=lane, batch=record["batch"], + load=record["load"], A=record["replica_forward_id"], + lane_counter=count_by_lane[lane], next_group=record["stage0_next_group"]) + count_by_lane[lane] += 1 + rows.append(row) + return rows + + +def pairwise(rows, name, kinds=("admission", "completion_folded", "completion_only")): + scored = [row for row in rows if row["target"] is not None and row["kind"] in kinds] + splits, merges, inversions, merge_same_lane = [], [], [], 0 + for i, first in enumerate(scored): + for second in scored[i + 1:]: + target = (first["target"] > second["target"]) - (first["target"] < second["target"]) + key = (first[name] > second[name]) - (first[name] < second[name]) + if target == key: + continue + pair = (first["seq"], second["seq"]) + if target == 0: + splits.append(pair) + elif key == 0: + merges.append(pair) + merge_same_lane += first["lane"] == second["lane"] + else: + inversions.append(pair) + return {"peer_splits": len(splits), "merges": len(merges), "merges_same_lane": merge_same_lane, + "inversions": len(inversions), "examples": {"peer_splits": splits[:3], "merges": merges[:3], + "inversions": inversions[:3]}} + + +def frontend_timeline(rows, name, num_lanes, end_ms): + balancer = VllmDPLoadBalancer(num_lanes) + timeline = [] + pending = [row for row in rows if row[name] is not None] + index = 0 + for now_ms in range(end_ms + 1): + while index < len(pending) and int(pending[index]["time"] * 1000) <= now_ms: + row = pending[index] + balancer.report(row["time"], row["lane"], row[name], RequestLoad(*row["load"])) + index += 1 + view = copy.deepcopy(balancer) + view._advance(now_ms / 1000) + timeline.append(tuple(view.frontend_counts)) + return timeline + + +def analyze(path: Path) -> dict: + result = json.loads(path.read_text()) + rows = reports(result) + num_lanes = result["case"]["attn_dp"] + end_ms = int(max(record["time"] for record in result["records"]) * 1000) + 200 + reference = frontend_timeline( + [dict(row, target_key=row["target"]) for row in rows if row["target"] is not None], + "target_key", num_lanes, end_ms) + scored_rows = [row for row in rows if row["target"] is not None] + summary = { + "shape": result["shape"], "completed": result["completed"], "requests": result["requests"], + "reports": {kind: sum(row["kind"] == kind for row in rows) + for kind in ("admission", "completion_folded", "completion_only")}, + "unscored_reports": len(rows) - len(scored_rows), + "candidates": {}, + } + for name in CANDIDATES: + timeline = frontend_timeline(scored_rows, name, num_lanes, end_ms) + forward_rows = pairwise(rows, name, kinds=("admission", "completion_folded")) + summary["candidates"][name] = { + **pairwise(rows, name), + "forward_reports": {key: forward_rows[key] for key in ("peer_splits", "merges", "inversions")}, + "equal_to_target": sum(row[name] == row["target"] for row in scored_rows), + "frontend_mismatch_ms": sum(a != b for a, b in zip(timeline, reference)), + } + summary["rows"] = rows + return summary + + +if __name__ == "__main__": + probe_dir, output = Path(sys.argv[1]), Path(sys.argv[2]) + summaries = [analyze(path) for path in sorted(probe_dir.glob("*/*.json"))] + output.write_text(json.dumps(summaries, indent=1)) + print(f"{'shape':22} {'adm/fold/only':>13} {'candidate':15} {'=target':>7} " + f"{'split/merge/inv':>15} {'forward rows':>12} {'fe_ms':>6}") + for summary in summaries: + counts = "/".join(str(value) for value in summary["reports"].values()) + for name, score in summary["candidates"].items(): + scored = len(summary["rows"]) - summary["unscored_reports"] + forward = "/".join(str(value) for value in score["forward_reports"].values()) + print(f"{summary['shape']:22} {counts:>13} {name:15} {score['equal_to_target']:>3}/{scored:<3} " + f"{score['peer_splits']:>5}/{score['merges']}/{score['inversions']:<5} {forward:>12} " + f"{score['frontend_mismatch_ms']:>6}") diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/evidence/key_scores.json b/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/evidence/key_scores.json new file mode 100644 index 00000000..2706dfca --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/evidence/key_scores.json @@ -0,0 +1,3715 @@ +[ + { + "shape": "moe_dp1_pp2_burst", + "completed": 6, + "requests": 6, + "reports": { + "admission": 1, + "completion_folded": 13, + "completion_only": 1 + }, + "unscored_reports": 1, + "candidates": { + "A": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 1, + "frontend_mismatch_ms": 0 + }, + "lane_counter": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 14, + "frontend_mismatch_ms": 0 + }, + "next_group": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 1, + "frontend_mismatch_ms": 0 + }, + "predicted_group": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 14, + "frontend_mismatch_ms": 0 + }, + "group_anchored": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 14, + "frontend_mismatch_ms": 0 + }, + "every_report_advances": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 14, + "frontend_mismatch_ms": 0 + } + }, + "rows": [ + { + "kind": "admission", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "seq": 0, + "time": 0, + "lane": 0, + "batch": 0, + "load": [ + 5, + 1 + ], + "A": 0, + "lane_counter": 0, + "next_group": 0 + }, + { + "kind": "completion_folded", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "folded_batch": 1, + "seq": 15, + "time": 0.08400000000000002, + "lane": 0, + "batch": 0, + "load": [ + 4, + 2 + ], + "A": 9, + "lane_counter": 1, + "next_group": 2 + }, + { + "kind": "completion_folded", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "folded_batch": 11, + "seq": 24, + "time": 0.12600000000000006, + "lane": 0, + "batch": 1, + "load": [ + 3, + 3 + ], + "A": 15, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "completion_folded", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 3, + "folded_batch": 18, + "seq": 34, + "time": 0.1680000000000001, + "lane": 0, + "batch": 11, + "load": [ + 2, + 4 + ], + "A": 21, + "lane_counter": 3, + "next_group": 4 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 4, + "folded_batch": 25, + "seq": 44, + "time": 0.21000000000000013, + "lane": 0, + "batch": 18, + "load": [ + 2, + 4 + ], + "A": 27, + "lane_counter": 4, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 5, + "folded_batch": 32, + "seq": 54, + "time": 0.25200000000000017, + "lane": 0, + "batch": 25, + "load": [ + 2, + 3 + ], + "A": 33, + "lane_counter": 5, + "next_group": 6 + }, + { + "kind": "completion_folded", + "target": 6, + "predicted_group": 6, + "group_anchored": 6, + "every_report_advances": 6, + "folded_batch": 39, + "seq": 64, + "time": 0.2940000000000002, + "lane": 0, + "batch": 32, + "load": [ + 1, + 3 + ], + "A": 39, + "lane_counter": 6, + "next_group": 7 + }, + { + "kind": "completion_folded", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 7, + "folded_batch": 46, + "seq": 74, + "time": 0.33600000000000024, + "lane": 0, + "batch": 39, + "load": [ + 0, + 4 + ], + "A": 45, + "lane_counter": 7, + "next_group": 8 + }, + { + "kind": "completion_folded", + "target": 8, + "predicted_group": 8, + "group_anchored": 8, + "every_report_advances": 8, + "folded_batch": 53, + "seq": 84, + "time": 0.3780000000000003, + "lane": 0, + "batch": 46, + "load": [ + 0, + 4 + ], + "A": 51, + "lane_counter": 8, + "next_group": 9 + }, + { + "kind": "completion_folded", + "target": 9, + "predicted_group": 9, + "group_anchored": 9, + "every_report_advances": 9, + "folded_batch": 60, + "seq": 94, + "time": 0.4200000000000003, + "lane": 0, + "batch": 53, + "load": [ + 0, + 3 + ], + "A": 57, + "lane_counter": 9, + "next_group": 10 + }, + { + "kind": "completion_folded", + "target": 10, + "predicted_group": 10, + "group_anchored": 10, + "every_report_advances": 10, + "folded_batch": 67, + "seq": 104, + "time": 0.46200000000000035, + "lane": 0, + "batch": 60, + "load": [ + 0, + 2 + ], + "A": 63, + "lane_counter": 10, + "next_group": 11 + }, + { + "kind": "completion_folded", + "target": 11, + "predicted_group": 11, + "group_anchored": 11, + "every_report_advances": 11, + "folded_batch": 74, + "seq": 114, + "time": 0.5040000000000003, + "lane": 0, + "batch": 67, + "load": [ + 0, + 2 + ], + "A": 69, + "lane_counter": 11, + "next_group": 12 + }, + { + "kind": "completion_folded", + "target": 12, + "predicted_group": 12, + "group_anchored": 12, + "every_report_advances": 12, + "folded_batch": 81, + "seq": 124, + "time": 0.546, + "lane": 0, + "batch": 74, + "load": [ + 0, + 2 + ], + "A": 75, + "lane_counter": 12, + "next_group": 13 + }, + { + "kind": "completion_folded", + "target": 13, + "predicted_group": 13, + "group_anchored": 13, + "every_report_advances": 13, + "folded_batch": 88, + "seq": 134, + "time": 0.5879999999999997, + "lane": 0, + "batch": 81, + "load": [ + 0, + 1 + ], + "A": 81, + "lane_counter": 13, + "next_group": 14 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 14, + "group_anchored": 14, + "every_report_advances": 14, + "seq": 139, + "time": 0.6299999999999994, + "lane": 0, + "batch": 88, + "load": [ + 0, + 0 + ], + "A": 84, + "lane_counter": 14, + "next_group": 14 + } + ] + }, + { + "shape": "moe_dp1_pp3_burst", + "completed": 6, + "requests": 6, + "reports": { + "admission": 7, + "completion_folded": 9, + "completion_only": 7 + }, + "unscored_reports": 2, + "candidates": { + "A": { + "peer_splits": 0, + "merges": 1, + "merges_same_lane": 1, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [ + [ + 0, + 1 + ] + ], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 1, + "inversions": 0 + }, + "equal_to_target": 1, + "frontend_mismatch_ms": 400 + }, + "lane_counter": { + "peer_splits": 5, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [ + [ + 116, + 118 + ], + [ + 124, + 126 + ], + [ + 138, + 140 + ] + ], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 12, + "frontend_mismatch_ms": 0 + }, + "next_group": { + "peer_splits": 0, + "merges": 3, + "merges_same_lane": 3, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [ + [ + 0, + 1 + ], + [ + 109, + 116 + ], + [ + 109, + 118 + ] + ], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 2, + "inversions": 0 + }, + "equal_to_target": 11, + "frontend_mismatch_ms": 400 + }, + "predicted_group": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 21, + "frontend_mismatch_ms": 0 + }, + "group_anchored": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 21, + "frontend_mismatch_ms": 0 + }, + "every_report_advances": { + "peer_splits": 5, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [ + [ + 116, + 118 + ], + [ + 124, + 126 + ], + [ + 138, + 140 + ] + ], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 12, + "frontend_mismatch_ms": 0 + } + }, + "rows": [ + { + "kind": "admission", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "seq": 0, + "time": 0, + "lane": 0, + "batch": 0, + "load": [ + 5, + 1 + ], + "A": 0, + "lane_counter": 0, + "next_group": 0 + }, + { + "kind": "admission", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "seq": 1, + "time": 0, + "lane": 0, + "batch": 1, + "load": [ + 4, + 2 + ], + "A": 0, + "lane_counter": 1, + "next_group": 0 + }, + { + "kind": "completion_folded", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "folded_batch": 2, + "seq": 23, + "time": 0.09000000000000004, + "lane": 0, + "batch": 0, + "load": [ + 3, + 3 + ], + "A": 12, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "completion_folded", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 3, + "folded_batch": 15, + "seq": 33, + "time": 0.12000000000000006, + "lane": 0, + "batch": 1, + "load": [ + 2, + 4 + ], + "A": 18, + "lane_counter": 3, + "next_group": 4 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 4, + "folded_batch": 22, + "seq": 43, + "time": 0.15000000000000008, + "lane": 0, + "batch": 2, + "load": [ + 2, + 4 + ], + "A": 24, + "lane_counter": 4, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 5, + "folded_batch": 29, + "seq": 54, + "time": 0.1800000000000001, + "lane": 0, + "batch": 15, + "load": [ + 2, + 4 + ], + "A": 30, + "lane_counter": 5, + "next_group": 6 + }, + { + "kind": "completion_folded", + "target": 6, + "predicted_group": 6, + "group_anchored": 6, + "every_report_advances": 6, + "folded_batch": 36, + "seq": 65, + "time": 0.21000000000000013, + "lane": 0, + "batch": 22, + "load": [ + 2, + 4 + ], + "A": 36, + "lane_counter": 6, + "next_group": 7 + }, + { + "kind": "completion_folded", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 7, + "folded_batch": 43, + "seq": 76, + "time": 0.24000000000000016, + "lane": 0, + "batch": 29, + "load": [ + 2, + 4 + ], + "A": 42, + "lane_counter": 7, + "next_group": 8 + }, + { + "kind": "completion_folded", + "target": 8, + "predicted_group": 8, + "group_anchored": 8, + "every_report_advances": 8, + "folded_batch": 50, + "seq": 87, + "time": 0.2700000000000002, + "lane": 0, + "batch": 36, + "load": [ + 2, + 3 + ], + "A": 48, + "lane_counter": 8, + "next_group": 9 + }, + { + "kind": "completion_folded", + "target": 9, + "predicted_group": 9, + "group_anchored": 9, + "every_report_advances": 9, + "folded_batch": 57, + "seq": 98, + "time": 0.3000000000000002, + "lane": 0, + "batch": 43, + "load": [ + 1, + 3 + ], + "A": 54, + "lane_counter": 9, + "next_group": 10 + }, + { + "kind": "completion_folded", + "target": 10, + "predicted_group": 10, + "group_anchored": 10, + "every_report_advances": 10, + "folded_batch": 64, + "seq": 109, + "time": 0.33000000000000024, + "lane": 0, + "batch": 50, + "load": [ + 0, + 3 + ], + "A": 60, + "lane_counter": 10, + "next_group": 11 + }, + { + "kind": "completion_only", + "target": 11, + "predicted_group": 11, + "group_anchored": 11, + "every_report_advances": 11, + "seq": 116, + "time": 0.36000000000000026, + "lane": 0, + "batch": 57, + "load": [ + 0, + 3 + ], + "A": 64, + "lane_counter": 11, + "next_group": 11 + }, + { + "kind": "admission", + "target": 11, + "predicted_group": 11, + "group_anchored": 11, + "every_report_advances": 12, + "seq": 118, + "time": 0.36000000000000026, + "lane": 0, + "batch": 75, + "load": [ + 0, + 3 + ], + "A": 64, + "lane_counter": 12, + "next_group": 11 + }, + { + "kind": "completion_only", + "target": 12, + "predicted_group": 12, + "group_anchored": 12, + "every_report_advances": 13, + "seq": 124, + "time": 0.3900000000000003, + "lane": 0, + "batch": 64, + "load": [ + 0, + 3 + ], + "A": 68, + "lane_counter": 13, + "next_group": 12 + }, + { + "kind": "admission", + "target": 12, + "predicted_group": 12, + "group_anchored": 12, + "every_report_advances": 14, + "seq": 126, + "time": 0.3900000000000003, + "lane": 0, + "batch": 80, + "load": [ + 0, + 3 + ], + "A": 68, + "lane_counter": 14, + "next_group": 12 + }, + { + "kind": "completion_only", + "target": 13, + "predicted_group": 13, + "group_anchored": 13, + "every_report_advances": 15, + "seq": 138, + "time": 0.45000000000000034, + "lane": 0, + "batch": 75, + "load": [ + 0, + 2 + ], + "A": 76, + "lane_counter": 15, + "next_group": 13 + }, + { + "kind": "admission", + "target": 13, + "predicted_group": 13, + "group_anchored": 13, + "every_report_advances": 16, + "seq": 140, + "time": 0.45000000000000034, + "lane": 0, + "batch": 89, + "load": [ + 0, + 2 + ], + "A": 76, + "lane_counter": 16, + "next_group": 13 + }, + { + "kind": "completion_only", + "target": 14, + "predicted_group": 14, + "group_anchored": 14, + "every_report_advances": 17, + "seq": 146, + "time": 0.48000000000000037, + "lane": 0, + "batch": 80, + "load": [ + 0, + 2 + ], + "A": 80, + "lane_counter": 17, + "next_group": 14 + }, + { + "kind": "admission", + "target": 14, + "predicted_group": 14, + "group_anchored": 14, + "every_report_advances": 18, + "seq": 148, + "time": 0.48000000000000037, + "lane": 0, + "batch": 94, + "load": [ + 0, + 2 + ], + "A": 80, + "lane_counter": 18, + "next_group": 14 + }, + { + "kind": "completion_only", + "target": 15, + "predicted_group": 15, + "group_anchored": 15, + "every_report_advances": 19, + "seq": 160, + "time": 0.5400000000000001, + "lane": 0, + "batch": 89, + "load": [ + 0, + 2 + ], + "A": 88, + "lane_counter": 19, + "next_group": 15 + }, + { + "kind": "admission", + "target": 15, + "predicted_group": 15, + "group_anchored": 15, + "every_report_advances": 20, + "seq": 162, + "time": 0.5400000000000001, + "lane": 0, + "batch": 103, + "load": [ + 0, + 2 + ], + "A": 88, + "lane_counter": 20, + "next_group": 15 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 16, + "group_anchored": 16, + "every_report_advances": 21, + "seq": 168, + "time": 0.57, + "lane": 0, + "batch": 94, + "load": [ + 0, + 1 + ], + "A": 92, + "lane_counter": 21, + "next_group": 16 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 16, + "group_anchored": 16, + "every_report_advances": 22, + "seq": 175, + "time": 0.6299999999999996, + "lane": 0, + "batch": 103, + "load": [ + 0, + 0 + ], + "A": 96, + "lane_counter": 22, + "next_group": 16 + } + ] + }, + { + "shape": "moe_dp2_pp1_burst", + "completed": 6, + "requests": 6, + "reports": { + "admission": 0, + "completion_folded": 12, + "completion_only": 0 + }, + "unscored_reports": 0, + "candidates": { + "A": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 0, + "frontend_mismatch_ms": 0 + }, + "lane_counter": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 12, + "frontend_mismatch_ms": 0 + }, + "next_group": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 0, + "frontend_mismatch_ms": 0 + }, + "predicted_group": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 12, + "frontend_mismatch_ms": 0 + }, + "group_anchored": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 12, + "frontend_mismatch_ms": 0 + }, + "every_report_advances": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 12, + "frontend_mismatch_ms": 0 + } + }, + "rows": [ + { + "kind": "completion_folded", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "folded_batch": 0, + "seq": 16, + "time": 0.09000000000000004, + "lane": 1, + "batch": 0, + "load": [ + 2, + 1 + ], + "A": 6, + "lane_counter": 0, + "next_group": 1 + }, + { + "kind": "completion_folded", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "folded_batch": 1, + "seq": 17, + "time": 0.09000000000000004, + "lane": 0, + "batch": 1, + "load": [ + 2, + 1 + ], + "A": 6, + "lane_counter": 0, + "next_group": 1 + }, + { + "kind": "completion_folded", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "folded_batch": 14, + "seq": 34, + "time": 0.1800000000000001, + "lane": 1, + "batch": 14, + "load": [ + 1, + 2 + ], + "A": 12, + "lane_counter": 1, + "next_group": 2 + }, + { + "kind": "completion_folded", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "folded_batch": 15, + "seq": 35, + "time": 0.1800000000000001, + "lane": 0, + "batch": 15, + "load": [ + 1, + 2 + ], + "A": 12, + "lane_counter": 1, + "next_group": 2 + }, + { + "kind": "completion_folded", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "folded_batch": 28, + "seq": 52, + "time": 0.2700000000000002, + "lane": 1, + "batch": 28, + "load": [ + 0, + 2 + ], + "A": 18, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "completion_folded", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "folded_batch": 29, + "seq": 53, + "time": 0.2700000000000002, + "lane": 0, + "batch": 29, + "load": [ + 0, + 2 + ], + "A": 18, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "completion_folded", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 3, + "folded_batch": 42, + "seq": 70, + "time": 0.36000000000000026, + "lane": 1, + "batch": 42, + "load": [ + 0, + 2 + ], + "A": 24, + "lane_counter": 3, + "next_group": 4 + }, + { + "kind": "completion_folded", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 3, + "folded_batch": 43, + "seq": 71, + "time": 0.36000000000000026, + "lane": 0, + "batch": 43, + "load": [ + 0, + 2 + ], + "A": 24, + "lane_counter": 3, + "next_group": 4 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 4, + "folded_batch": 56, + "seq": 88, + "time": 0.45000000000000034, + "lane": 1, + "batch": 56, + "load": [ + 0, + 1 + ], + "A": 30, + "lane_counter": 4, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 4, + "folded_batch": 57, + "seq": 89, + "time": 0.45000000000000034, + "lane": 0, + "batch": 57, + "load": [ + 0, + 1 + ], + "A": 30, + "lane_counter": 4, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 5, + "folded_batch": 70, + "seq": 106, + "time": 0.5400000000000001, + "lane": 1, + "batch": 70, + "load": [ + 0, + 0 + ], + "A": 36, + "lane_counter": 5, + "next_group": 6 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 5, + "folded_batch": 71, + "seq": 107, + "time": 0.5400000000000001, + "lane": 0, + "batch": 71, + "load": [ + 0, + 0 + ], + "A": 36, + "lane_counter": 5, + "next_group": 6 + } + ] + }, + { + "shape": "moe_dp2_pp2_burst", + "completed": 6, + "requests": 6, + "reports": { + "admission": 4, + "completion_folded": 12, + "completion_only": 4 + }, + "unscored_reports": 2, + "candidates": { + "A": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 2, + "frontend_mismatch_ms": 0 + }, + "lane_counter": { + "peer_splits": 4, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [ + [ + 138, + 140 + ], + [ + 138, + 141 + ], + [ + 139, + 140 + ] + ], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 16, + "frontend_mismatch_ms": 0 + }, + "next_group": { + "peer_splits": 0, + "merges": 8, + "merges_same_lane": 4, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [ + [ + 128, + 138 + ], + [ + 128, + 139 + ], + [ + 128, + 140 + ] + ], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 4, + "inversions": 0 + }, + "equal_to_target": 6, + "frontend_mismatch_ms": 0 + }, + "predicted_group": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 18, + "frontend_mismatch_ms": 0 + }, + "group_anchored": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 18, + "frontend_mismatch_ms": 0 + }, + "every_report_advances": { + "peer_splits": 4, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [ + [ + 138, + 140 + ], + [ + 138, + 141 + ], + [ + 139, + 140 + ] + ], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 16, + "frontend_mismatch_ms": 0 + } + }, + "rows": [ + { + "kind": "admission", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "seq": 0, + "time": 0, + "lane": 1, + "batch": 0, + "load": [ + 2, + 1 + ], + "A": 0, + "lane_counter": 0, + "next_group": 0 + }, + { + "kind": "admission", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "seq": 2, + "time": 0, + "lane": 0, + "batch": 2, + "load": [ + 2, + 1 + ], + "A": 0, + "lane_counter": 0, + "next_group": 0 + }, + { + "kind": "completion_folded", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "folded_batch": 1, + "seq": 30, + "time": 0.09600000000000004, + "lane": 1, + "batch": 0, + "load": [ + 1, + 2 + ], + "A": 9, + "lane_counter": 1, + "next_group": 2 + }, + { + "kind": "completion_folded", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "folded_batch": 3, + "seq": 31, + "time": 0.09600000000000004, + "lane": 0, + "batch": 2, + "load": [ + 1, + 2 + ], + "A": 9, + "lane_counter": 1, + "next_group": 2 + }, + { + "kind": "completion_folded", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "folded_batch": 22, + "seq": 48, + "time": 0.14400000000000007, + "lane": 1, + "batch": 1, + "load": [ + 0, + 3 + ], + "A": 15, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "completion_folded", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "folded_batch": 23, + "seq": 49, + "time": 0.14400000000000007, + "lane": 0, + "batch": 3, + "load": [ + 0, + 3 + ], + "A": 15, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "completion_folded", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 3, + "folded_batch": 36, + "seq": 68, + "time": 0.19200000000000012, + "lane": 1, + "batch": 22, + "load": [ + 0, + 3 + ], + "A": 21, + "lane_counter": 3, + "next_group": 4 + }, + { + "kind": "completion_folded", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 3, + "folded_batch": 37, + "seq": 69, + "time": 0.19200000000000012, + "lane": 0, + "batch": 23, + "load": [ + 0, + 3 + ], + "A": 21, + "lane_counter": 3, + "next_group": 4 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 4, + "folded_batch": 50, + "seq": 88, + "time": 0.24000000000000016, + "lane": 1, + "batch": 36, + "load": [ + 0, + 3 + ], + "A": 27, + "lane_counter": 4, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 4, + "folded_batch": 51, + "seq": 89, + "time": 0.24000000000000016, + "lane": 0, + "batch": 37, + "load": [ + 0, + 3 + ], + "A": 27, + "lane_counter": 4, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 5, + "folded_batch": 64, + "seq": 108, + "time": 0.2880000000000002, + "lane": 1, + "batch": 50, + "load": [ + 0, + 2 + ], + "A": 33, + "lane_counter": 5, + "next_group": 6 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 5, + "folded_batch": 65, + "seq": 109, + "time": 0.2880000000000002, + "lane": 0, + "batch": 51, + "load": [ + 0, + 2 + ], + "A": 33, + "lane_counter": 5, + "next_group": 6 + }, + { + "kind": "completion_folded", + "target": 6, + "predicted_group": 6, + "group_anchored": 6, + "every_report_advances": 6, + "folded_batch": 78, + "seq": 128, + "time": 0.33600000000000024, + "lane": 1, + "batch": 64, + "load": [ + 0, + 1 + ], + "A": 39, + "lane_counter": 6, + "next_group": 7 + }, + { + "kind": "completion_folded", + "target": 6, + "predicted_group": 6, + "group_anchored": 6, + "every_report_advances": 6, + "folded_batch": 79, + "seq": 129, + "time": 0.33600000000000024, + "lane": 0, + "batch": 65, + "load": [ + 0, + 1 + ], + "A": 39, + "lane_counter": 6, + "next_group": 7 + }, + { + "kind": "completion_only", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 7, + "seq": 138, + "time": 0.3840000000000003, + "lane": 1, + "batch": 78, + "load": [ + 0, + 1 + ], + "A": 42, + "lane_counter": 7, + "next_group": 7 + }, + { + "kind": "completion_only", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 7, + "seq": 139, + "time": 0.3840000000000003, + "lane": 0, + "batch": 79, + "load": [ + 0, + 1 + ], + "A": 42, + "lane_counter": 7, + "next_group": 7 + }, + { + "kind": "admission", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 8, + "seq": 140, + "time": 0.3840000000000003, + "lane": 1, + "batch": 98, + "load": [ + 0, + 1 + ], + "A": 42, + "lane_counter": 8, + "next_group": 7 + }, + { + "kind": "admission", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 8, + "seq": 141, + "time": 0.3840000000000003, + "lane": 0, + "batch": 99, + "load": [ + 0, + 1 + ], + "A": 42, + "lane_counter": 8, + "next_group": 7 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 8, + "group_anchored": 8, + "every_report_advances": 9, + "seq": 158, + "time": 0.48000000000000037, + "lane": 1, + "batch": 98, + "load": [ + 0, + 0 + ], + "A": 48, + "lane_counter": 9, + "next_group": 8 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 8, + "group_anchored": 8, + "every_report_advances": 9, + "seq": 159, + "time": 0.48000000000000037, + "lane": 0, + "batch": 99, + "load": [ + 0, + 0 + ], + "A": 48, + "lane_counter": 9, + "next_group": 8 + } + ] + }, + { + "shape": "moe_dp2_pp2_staggered", + "completed": 6, + "requests": 6, + "reports": { + "admission": 5, + "completion_folded": 12, + "completion_only": 5 + }, + "unscored_reports": 1, + "candidates": { + "A": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 2, + "frontend_mismatch_ms": 0 + }, + "lane_counter": { + "peer_splits": 6, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [ + [ + 138, + 140 + ], + [ + 138, + 141 + ], + [ + 139, + 140 + ] + ], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 16, + "frontend_mismatch_ms": 0 + }, + "next_group": { + "peer_splits": 1, + "merges": 8, + "merges_same_lane": 4, + "inversions": 0, + "examples": { + "peer_splits": [ + [ + 0, + 2 + ] + ], + "merges": [ + [ + 128, + 138 + ], + [ + 128, + 139 + ], + [ + 128, + 140 + ] + ], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 1, + "merges": 4, + "inversions": 0 + }, + "equal_to_target": 8, + "frontend_mismatch_ms": 0 + }, + "predicted_group": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 21, + "frontend_mismatch_ms": 0 + }, + "group_anchored": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 21, + "frontend_mismatch_ms": 0 + }, + "every_report_advances": { + "peer_splits": 6, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [ + [ + 138, + 140 + ], + [ + 138, + 141 + ], + [ + 139, + 140 + ] + ], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 16, + "frontend_mismatch_ms": 0 + } + }, + "rows": [ + { + "kind": "admission", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "seq": 0, + "time": 0.05100301436374005, + "lane": 0, + "batch": 0, + "load": [ + 0, + 1 + ], + "A": 0, + "lane_counter": 0, + "next_group": 0 + }, + { + "kind": "admission", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "seq": 2, + "time": 0.052269456315877, + "lane": 1, + "batch": 1, + "load": [ + 0, + 1 + ], + "A": 0, + "lane_counter": 0, + "next_group": 1 + }, + { + "kind": "completion_folded", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "folded_batch": 4, + "seq": 30, + "time": 0.14826945631587707, + "lane": 0, + "batch": 0, + "load": [ + 1, + 2 + ], + "A": 9, + "lane_counter": 1, + "next_group": 2 + }, + { + "kind": "completion_folded", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "folded_batch": 7, + "seq": 31, + "time": 0.14826945631587707, + "lane": 1, + "batch": 1, + "load": [ + 0, + 2 + ], + "A": 9, + "lane_counter": 1, + "next_group": 2 + }, + { + "kind": "completion_folded", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "folded_batch": 22, + "seq": 48, + "time": 0.1962694563158771, + "lane": 0, + "batch": 4, + "load": [ + 0, + 3 + ], + "A": 15, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "completion_folded", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "folded_batch": 23, + "seq": 49, + "time": 0.1962694563158771, + "lane": 1, + "batch": 7, + "load": [ + 0, + 2 + ], + "A": 15, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "completion_folded", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 3, + "folded_batch": 36, + "seq": 68, + "time": 0.24426945631587715, + "lane": 0, + "batch": 22, + "load": [ + 0, + 3 + ], + "A": 21, + "lane_counter": 3, + "next_group": 4 + }, + { + "kind": "completion_folded", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 3, + "folded_batch": 37, + "seq": 69, + "time": 0.24426945631587715, + "lane": 1, + "batch": 23, + "load": [ + 1, + 2 + ], + "A": 21, + "lane_counter": 3, + "next_group": 4 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 4, + "folded_batch": 50, + "seq": 88, + "time": 0.2922694563158772, + "lane": 0, + "batch": 36, + "load": [ + 0, + 3 + ], + "A": 27, + "lane_counter": 4, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 4, + "folded_batch": 51, + "seq": 89, + "time": 0.2922694563158772, + "lane": 1, + "batch": 37, + "load": [ + 0, + 3 + ], + "A": 27, + "lane_counter": 4, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 5, + "folded_batch": 64, + "seq": 108, + "time": 0.34026945631587724, + "lane": 0, + "batch": 50, + "load": [ + 0, + 2 + ], + "A": 33, + "lane_counter": 5, + "next_group": 6 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 5, + "folded_batch": 65, + "seq": 109, + "time": 0.34026945631587724, + "lane": 1, + "batch": 51, + "load": [ + 0, + 2 + ], + "A": 33, + "lane_counter": 5, + "next_group": 6 + }, + { + "kind": "completion_folded", + "target": 6, + "predicted_group": 6, + "group_anchored": 6, + "every_report_advances": 6, + "folded_batch": 78, + "seq": 128, + "time": 0.3882694563158773, + "lane": 0, + "batch": 64, + "load": [ + 0, + 1 + ], + "A": 39, + "lane_counter": 6, + "next_group": 7 + }, + { + "kind": "completion_folded", + "target": 6, + "predicted_group": 6, + "group_anchored": 6, + "every_report_advances": 6, + "folded_batch": 79, + "seq": 129, + "time": 0.3882694563158773, + "lane": 1, + "batch": 65, + "load": [ + 0, + 1 + ], + "A": 39, + "lane_counter": 6, + "next_group": 7 + }, + { + "kind": "completion_only", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 7, + "seq": 138, + "time": 0.4362694563158773, + "lane": 0, + "batch": 78, + "load": [ + 0, + 1 + ], + "A": 42, + "lane_counter": 7, + "next_group": 7 + }, + { + "kind": "completion_only", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 7, + "seq": 139, + "time": 0.4362694563158773, + "lane": 1, + "batch": 79, + "load": [ + 0, + 1 + ], + "A": 42, + "lane_counter": 7, + "next_group": 7 + }, + { + "kind": "admission", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 8, + "seq": 140, + "time": 0.4362694563158773, + "lane": 0, + "batch": 98, + "load": [ + 0, + 1 + ], + "A": 42, + "lane_counter": 8, + "next_group": 7 + }, + { + "kind": "admission", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 8, + "seq": 141, + "time": 0.4362694563158773, + "lane": 1, + "batch": 99, + "load": [ + 0, + 1 + ], + "A": 42, + "lane_counter": 8, + "next_group": 7 + }, + { + "kind": "completion_only", + "target": 8, + "predicted_group": 8, + "group_anchored": 8, + "every_report_advances": 9, + "seq": 158, + "time": 0.5322694563158772, + "lane": 0, + "batch": 98, + "load": [ + 0, + 0 + ], + "A": 48, + "lane_counter": 9, + "next_group": 8 + }, + { + "kind": "completion_only", + "target": 8, + "predicted_group": 8, + "group_anchored": 8, + "every_report_advances": 9, + "seq": 159, + "time": 0.5322694563158772, + "lane": 1, + "batch": 99, + "load": [ + 0, + 1 + ], + "A": 48, + "lane_counter": 9, + "next_group": 8 + }, + { + "kind": "admission", + "target": 8, + "predicted_group": 8, + "group_anchored": 8, + "every_report_advances": 10, + "seq": 160, + "time": 0.5322694563158772, + "lane": 1, + "batch": 112, + "load": [ + 0, + 1 + ], + "A": 48, + "lane_counter": 10, + "next_group": 8 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 9, + "group_anchored": 9, + "every_report_advances": 11, + "seq": 175, + "time": 0.6282694563158766, + "lane": 1, + "batch": 112, + "load": [ + 0, + 0 + ], + "A": 54, + "lane_counter": 11, + "next_group": 9 + } + ] + }, + { + "shape": "moe_dp2_pp3_burst", + "completed": 6, + "requests": 6, + "reports": { + "admission": 4, + "completion_folded": 14, + "completion_only": 4 + }, + "unscored_reports": 4, + "candidates": { + "A": { + "peer_splits": 0, + "merges": 4, + "merges_same_lane": 2, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [ + [ + 0, + 1 + ], + [ + 0, + 4 + ], + [ + 1, + 3 + ] + ], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 4, + "inversions": 0 + }, + "equal_to_target": 2, + "frontend_mismatch_ms": 148 + }, + "lane_counter": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 18, + "frontend_mismatch_ms": 0 + }, + "next_group": { + "peer_splits": 0, + "merges": 4, + "merges_same_lane": 2, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [ + [ + 0, + 1 + ], + [ + 0, + 4 + ], + [ + 1, + 3 + ] + ], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 4, + "inversions": 0 + }, + "equal_to_target": 2, + "frontend_mismatch_ms": 148 + }, + "predicted_group": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 18, + "frontend_mismatch_ms": 0 + }, + "group_anchored": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 18, + "frontend_mismatch_ms": 0 + }, + "every_report_advances": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 18, + "frontend_mismatch_ms": 0 + } + }, + "rows": [ + { + "kind": "admission", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "seq": 0, + "time": 0, + "lane": 1, + "batch": 0, + "load": [ + 2, + 1 + ], + "A": 0, + "lane_counter": 0, + "next_group": 0 + }, + { + "kind": "admission", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "seq": 1, + "time": 0, + "lane": 1, + "batch": 1, + "load": [ + 1, + 2 + ], + "A": 0, + "lane_counter": 1, + "next_group": 0 + }, + { + "kind": "admission", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "seq": 3, + "time": 0, + "lane": 0, + "batch": 3, + "load": [ + 2, + 1 + ], + "A": 0, + "lane_counter": 0, + "next_group": 0 + }, + { + "kind": "admission", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "seq": 4, + "time": 0, + "lane": 0, + "batch": 4, + "load": [ + 1, + 2 + ], + "A": 0, + "lane_counter": 1, + "next_group": 0 + }, + { + "kind": "completion_folded", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "folded_batch": 2, + "seq": 46, + "time": 0.10200000000000004, + "lane": 1, + "batch": 0, + "load": [ + 0, + 3 + ], + "A": 12, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "completion_folded", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "folded_batch": 5, + "seq": 47, + "time": 0.10200000000000004, + "lane": 0, + "batch": 3, + "load": [ + 0, + 3 + ], + "A": 12, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "completion_folded", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 3, + "folded_batch": 30, + "seq": 66, + "time": 0.13600000000000007, + "lane": 1, + "batch": 1, + "load": [ + 0, + 3 + ], + "A": 18, + "lane_counter": 3, + "next_group": 4 + }, + { + "kind": "completion_folded", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 3, + "folded_batch": 31, + "seq": 67, + "time": 0.13600000000000007, + "lane": 0, + "batch": 4, + "load": [ + 0, + 3 + ], + "A": 18, + "lane_counter": 3, + "next_group": 4 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 4, + "folded_batch": 44, + "seq": 86, + "time": 0.1700000000000001, + "lane": 1, + "batch": 2, + "load": [ + 0, + 3 + ], + "A": 24, + "lane_counter": 4, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 4, + "folded_batch": 45, + "seq": 87, + "time": 0.1700000000000001, + "lane": 0, + "batch": 5, + "load": [ + 0, + 3 + ], + "A": 24, + "lane_counter": 4, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 5, + "folded_batch": 58, + "seq": 108, + "time": 0.20400000000000013, + "lane": 1, + "batch": 30, + "load": [ + 0, + 3 + ], + "A": 30, + "lane_counter": 5, + "next_group": 6 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 5, + "folded_batch": 59, + "seq": 109, + "time": 0.20400000000000013, + "lane": 0, + "batch": 31, + "load": [ + 0, + 3 + ], + "A": 30, + "lane_counter": 5, + "next_group": 6 + }, + { + "kind": "completion_folded", + "target": 6, + "predicted_group": 6, + "group_anchored": 6, + "every_report_advances": 6, + "folded_batch": 72, + "seq": 130, + "time": 0.23800000000000016, + "lane": 1, + "batch": 44, + "load": [ + 0, + 3 + ], + "A": 36, + "lane_counter": 6, + "next_group": 7 + }, + { + "kind": "completion_folded", + "target": 6, + "predicted_group": 6, + "group_anchored": 6, + "every_report_advances": 6, + "folded_batch": 73, + "seq": 131, + "time": 0.23800000000000016, + "lane": 0, + "batch": 45, + "load": [ + 0, + 3 + ], + "A": 36, + "lane_counter": 6, + "next_group": 7 + }, + { + "kind": "completion_folded", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 7, + "folded_batch": 86, + "seq": 152, + "time": 0.2720000000000002, + "lane": 1, + "batch": 58, + "load": [ + 0, + 3 + ], + "A": 42, + "lane_counter": 7, + "next_group": 8 + }, + { + "kind": "completion_folded", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 7, + "folded_batch": 87, + "seq": 153, + "time": 0.2720000000000002, + "lane": 0, + "batch": 59, + "load": [ + 0, + 3 + ], + "A": 42, + "lane_counter": 7, + "next_group": 8 + }, + { + "kind": "completion_folded", + "target": 8, + "predicted_group": 8, + "group_anchored": 8, + "every_report_advances": 8, + "folded_batch": 100, + "seq": 174, + "time": 0.3060000000000002, + "lane": 1, + "batch": 72, + "load": [ + 0, + 2 + ], + "A": 48, + "lane_counter": 8, + "next_group": 9 + }, + { + "kind": "completion_folded", + "target": 8, + "predicted_group": 8, + "group_anchored": 8, + "every_report_advances": 8, + "folded_batch": 101, + "seq": 175, + "time": 0.3060000000000002, + "lane": 0, + "batch": 73, + "load": [ + 0, + 2 + ], + "A": 48, + "lane_counter": 8, + "next_group": 9 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 9, + "group_anchored": 9, + "every_report_advances": 9, + "seq": 188, + "time": 0.34000000000000025, + "lane": 1, + "batch": 86, + "load": [ + 0, + 1 + ], + "A": 52, + "lane_counter": 9, + "next_group": 9 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 9, + "group_anchored": 9, + "every_report_advances": 9, + "seq": 189, + "time": 0.34000000000000025, + "lane": 0, + "batch": 87, + "load": [ + 0, + 1 + ], + "A": 52, + "lane_counter": 9, + "next_group": 9 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 9, + "group_anchored": 9, + "every_report_advances": 10, + "seq": 196, + "time": 0.3740000000000003, + "lane": 1, + "batch": 100, + "load": [ + 0, + 0 + ], + "A": 54, + "lane_counter": 10, + "next_group": 9 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 9, + "group_anchored": 9, + "every_report_advances": 10, + "seq": 197, + "time": 0.3740000000000003, + "lane": 0, + "batch": 101, + "load": [ + 0, + 0 + ], + "A": 54, + "lane_counter": 10, + "next_group": 9 + } + ] + }, + { + "shape": "moe_dp2_pp3_staggered", + "completed": 6, + "requests": 6, + "reports": { + "admission": 7, + "completion_folded": 11, + "completion_only": 7 + }, + "unscored_reports": 2, + "candidates": { + "A": { + "peer_splits": 4, + "merges": 1, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [ + [ + 6, + 9 + ], + [ + 173, + 185 + ], + [ + 173, + 186 + ] + ], + "merges": [ + [ + 172, + 173 + ] + ], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 1, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 3, + "frontend_mismatch_ms": 200 + }, + "lane_counter": { + "peer_splits": 15, + "merges": 7, + "merges_same_lane": 0, + "inversions": 5, + "examples": { + "peer_splits": [ + [ + 38, + 41 + ], + [ + 39, + 41 + ], + [ + 52, + 53 + ] + ], + "merges": [ + [ + 41, + 52 + ], + [ + 53, + 88 + ], + [ + 57, + 110 + ] + ], + "inversions": [ + [ + 57, + 88 + ], + [ + 89, + 110 + ], + [ + 111, + 130 + ] + ] + }, + "forward_reports": { + "peer_splits": 6, + "merges": 5, + "inversions": 5 + }, + "equal_to_target": 13, + "frontend_mismatch_ms": 0 + }, + "next_group": { + "peer_splits": 1, + "merges": 6, + "merges_same_lane": 2, + "inversions": 0, + "examples": { + "peer_splits": [ + [ + 0, + 2 + ] + ], + "merges": [ + [ + 2, + 6 + ], + [ + 2, + 9 + ], + [ + 172, + 173 + ] + ], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 1, + "merges": 3, + "inversions": 0 + }, + "equal_to_target": 7, + "frontend_mismatch_ms": 200 + }, + "predicted_group": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 23, + "frontend_mismatch_ms": 0 + }, + "group_anchored": { + "peer_splits": 0, + "merges": 0, + "merges_same_lane": 0, + "inversions": 0, + "examples": { + "peer_splits": [], + "merges": [], + "inversions": [] + }, + "forward_reports": { + "peer_splits": 0, + "merges": 0, + "inversions": 0 + }, + "equal_to_target": 23, + "frontend_mismatch_ms": 0 + }, + "every_report_advances": { + "peer_splits": 15, + "merges": 7, + "merges_same_lane": 0, + "inversions": 5, + "examples": { + "peer_splits": [ + [ + 38, + 41 + ], + [ + 39, + 41 + ], + [ + 52, + 53 + ] + ], + "merges": [ + [ + 41, + 52 + ], + [ + 53, + 88 + ], + [ + 57, + 110 + ] + ], + "inversions": [ + [ + 57, + 88 + ], + [ + 89, + 110 + ], + [ + 111, + 130 + ] + ] + }, + "forward_reports": { + "peer_splits": 6, + "merges": 5, + "inversions": 5 + }, + "equal_to_target": 13, + "frontend_mismatch_ms": 0 + } + }, + "rows": [ + { + "kind": "admission", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "seq": 0, + "time": 0.05100301436374005, + "lane": 0, + "batch": 0, + "load": [ + 0, + 1 + ], + "A": 0, + "lane_counter": 0, + "next_group": 0 + }, + { + "kind": "admission", + "target": 0, + "predicted_group": 0, + "group_anchored": 0, + "every_report_advances": 0, + "seq": 2, + "time": 0.052269456315877, + "lane": 1, + "batch": 1, + "load": [ + 0, + 1 + ], + "A": 0, + "lane_counter": 0, + "next_group": 1 + }, + { + "kind": "admission", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "seq": 6, + "time": 0.06835065951962527, + "lane": 0, + "batch": 4, + "load": [ + 0, + 2 + ], + "A": 1, + "lane_counter": 1, + "next_group": 1 + }, + { + "kind": "admission", + "target": 1, + "predicted_group": 1, + "group_anchored": 1, + "every_report_advances": 1, + "seq": 9, + "time": 0.08097996880313095, + "lane": 1, + "batch": 7, + "load": [ + 0, + 2 + ], + "A": 2, + "lane_counter": 1, + "next_group": 1 + }, + { + "kind": "completion_folded", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "folded_batch": 24, + "seq": 38, + "time": 0.15426945631587707, + "lane": 0, + "batch": 0, + "load": [ + 0, + 3 + ], + "A": 10, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "completion_only", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 2, + "seq": 39, + "time": 0.15426945631587707, + "lane": 1, + "batch": 1, + "load": [ + 0, + 2 + ], + "A": 10, + "lane_counter": 2, + "next_group": 3 + }, + { + "kind": "admission", + "target": 2, + "predicted_group": 2, + "group_anchored": 2, + "every_report_advances": 3, + "seq": 41, + "time": 0.15426945631587707, + "lane": 1, + "batch": 26, + "load": [ + 0, + 2 + ], + "A": 10, + "lane_counter": 3, + "next_group": 3 + }, + { + "kind": "completion_folded", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 3, + "folded_batch": 25, + "seq": 52, + "time": 0.1882694563158771, + "lane": 0, + "batch": 4, + "load": [ + 0, + 3 + ], + "A": 14, + "lane_counter": 3, + "next_group": 4 + }, + { + "kind": "completion_only", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 4, + "seq": 53, + "time": 0.1882694563158771, + "lane": 1, + "batch": 7, + "load": [ + 0, + 2 + ], + "A": 14, + "lane_counter": 4, + "next_group": 4 + }, + { + "kind": "admission", + "target": 3, + "predicted_group": 3, + "group_anchored": 3, + "every_report_advances": 5, + "seq": 57, + "time": 0.1882694563158771, + "lane": 1, + "batch": 36, + "load": [ + 0, + 2 + ], + "A": 14, + "lane_counter": 5, + "next_group": 4 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 4, + "folded_batch": 35, + "seq": 88, + "time": 0.25626945631587716, + "lane": 0, + "batch": 24, + "load": [ + 0, + 3 + ], + "A": 24, + "lane_counter": 4, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 4, + "predicted_group": 4, + "group_anchored": 4, + "every_report_advances": 6, + "folded_batch": 41, + "seq": 89, + "time": 0.25626945631587716, + "lane": 1, + "batch": 26, + "load": [ + 0, + 3 + ], + "A": 24, + "lane_counter": 6, + "next_group": 5 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 5, + "folded_batch": 58, + "seq": 110, + "time": 0.2902694563158772, + "lane": 0, + "batch": 25, + "load": [ + 0, + 3 + ], + "A": 30, + "lane_counter": 5, + "next_group": 6 + }, + { + "kind": "completion_folded", + "target": 5, + "predicted_group": 5, + "group_anchored": 5, + "every_report_advances": 7, + "folded_batch": 59, + "seq": 111, + "time": 0.2902694563158772, + "lane": 1, + "batch": 36, + "load": [ + 0, + 3 + ], + "A": 30, + "lane_counter": 7, + "next_group": 6 + }, + { + "kind": "completion_folded", + "target": 6, + "predicted_group": 6, + "group_anchored": 6, + "every_report_advances": 6, + "folded_batch": 72, + "seq": 130, + "time": 0.3242694563158772, + "lane": 0, + "batch": 35, + "load": [ + 0, + 3 + ], + "A": 36, + "lane_counter": 6, + "next_group": 7 + }, + { + "kind": "completion_folded", + "target": 6, + "predicted_group": 6, + "group_anchored": 6, + "every_report_advances": 8, + "folded_batch": 73, + "seq": 131, + "time": 0.3242694563158772, + "lane": 1, + "batch": 41, + "load": [ + 0, + 3 + ], + "A": 36, + "lane_counter": 8, + "next_group": 7 + }, + { + "kind": "completion_folded", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 7, + "folded_batch": 86, + "seq": 152, + "time": 0.35826945631587725, + "lane": 0, + "batch": 58, + "load": [ + 0, + 3 + ], + "A": 42, + "lane_counter": 7, + "next_group": 8 + }, + { + "kind": "completion_folded", + "target": 7, + "predicted_group": 7, + "group_anchored": 7, + "every_report_advances": 9, + "folded_batch": 87, + "seq": 153, + "time": 0.35826945631587725, + "lane": 1, + "batch": 59, + "load": [ + 0, + 2 + ], + "A": 42, + "lane_counter": 9, + "next_group": 8 + }, + { + "kind": "completion_folded", + "target": 8, + "predicted_group": 8, + "group_anchored": 8, + "every_report_advances": 8, + "folded_batch": 100, + "seq": 172, + "time": 0.3922694563158773, + "lane": 0, + "batch": 72, + "load": [ + 0, + 2 + ], + "A": 48, + "lane_counter": 8, + "next_group": 9 + }, + { + "kind": "completion_only", + "target": 9, + "predicted_group": 9, + "group_anchored": 9, + "every_report_advances": 10, + "seq": 173, + "time": 0.3922694563158773, + "lane": 1, + "batch": 73, + "load": [ + 0, + 1 + ], + "A": 48, + "lane_counter": 10, + "next_group": 9 + }, + { + "kind": "completion_only", + "target": 9, + "predicted_group": 9, + "group_anchored": 9, + "every_report_advances": 9, + "seq": 185, + "time": 0.4262694563158773, + "lane": 0, + "batch": 86, + "load": [ + 0, + 1 + ], + "A": 52, + "lane_counter": 9, + "next_group": 9 + }, + { + "kind": "completion_only", + "target": 9, + "predicted_group": 9, + "group_anchored": 9, + "every_report_advances": 11, + "seq": 186, + "time": 0.4262694563158773, + "lane": 1, + "batch": 87, + "load": [ + 0, + 1 + ], + "A": 52, + "lane_counter": 11, + "next_group": 9 + }, + { + "kind": "admission", + "target": 9, + "predicted_group": 9, + "group_anchored": 9, + "every_report_advances": 12, + "seq": 188, + "time": 0.4262694563158773, + "lane": 1, + "batch": 125, + "load": [ + 0, + 1 + ], + "A": 52, + "lane_counter": 12, + "next_group": 9 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 10, + "group_anchored": 10, + "every_report_advances": 10, + "seq": 198, + "time": 0.46026945631587735, + "lane": 0, + "batch": 100, + "load": [ + 0, + 0 + ], + "A": 56, + "lane_counter": 10, + "next_group": 10 + }, + { + "kind": "completion_only", + "target": null, + "predicted_group": 10, + "group_anchored": 10, + "every_report_advances": 13, + "seq": 209, + "time": 0.5282694563158772, + "lane": 1, + "batch": 125, + "load": [ + 0, + 0 + ], + "A": 60, + "lane_counter": 13, + "next_group": 10 + } + ] + } +] \ No newline at end of file diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/probe_boundaries.py b/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/probe_boundaries.py new file mode 100644 index 00000000..c57df394 --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p1b/probe_boundaries.py @@ -0,0 +1,165 @@ +"""Step 9 P1(b): Frontier report boundaries and forward membership under PP. + +No source change. One shape per process (``IS_MOE`` is process-global). The +probe builds the shape with the stage-admission matrix fixture, wraps four +existing seams and writes every record with a global sequence number: + +* ``admit``: ``VLLMv1EngineReplicaScheduler._get_next_batch`` returned a + batch (the admission; ``_running_requests`` has already grown). +* ``complete``: ``BaseClusterScheduler.on_replica_batch_end``, the existing + report boundary. +* ``stage_start``: ``ReplicaStageScheduler.pop_batch_if_not_busy`` returned a + batch; for a MoE lane its ``_forward_cohort_provisional_id`` is then the + stage's shared forward-group id (``StageExecutionContext.bind_forward_group``). +* ``room``: ``ForwardSyncState.resolve_step`` placed a lane batch, real or + idle, in a layer room. + +At ``admit`` and ``complete`` it reads three candidate keys: + +* ``replica_forward_id``: ``ForwardSyncState._next_step_id_by_replica`` + (candidate A, the plan's first candidate); +* ``stage0_next_group``: the stage-0 context's next forward-group id; +* ``stage0_bound_group``: the stage-0 forward group currently bound, if any; + +and the stage-0 state a derived key would need: whether the bound group is +sealed, whether this lane's stage 0 is busy, and its stage-0 queue length. +A ``room`` record also carries the batch's forward-group id, so an idle lane's +participation can be placed in a group. + +Usage: python probe_boundaries.py +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +from tests.e2e.stage_admission_matrix import PREFILL_DECODE, Case, build_config + +SHAPES = { + "moe_dp2_pp1_burst": dict(is_moe=True, attn_dp=2, stages=1), + "moe_dp2_pp2_burst": dict(is_moe=True, attn_dp=2, stages=2), + "moe_dp2_pp3_burst": dict(is_moe=True, attn_dp=2, stages=3), + "moe_dp2_pp2_staggered": dict(is_moe=True, attn_dp=2, stages=2, staggered=True), + "moe_dp2_pp3_staggered": dict(is_moe=True, attn_dp=2, stages=3, staggered=True), + "moe_dp1_pp2_burst": dict(is_moe=True, attn_dp=1, stages=2), + "moe_dp1_pp3_burst": dict(is_moe=True, attn_dp=1, stages=3), +} +NUM_REQUESTS = 6 +BURST_QPS = 1e6 +STAGGERED_QPS = 20.0 + + +def build_case(label: str) -> Case: + shape = dict(SHAPES[label]) + staggered = shape.pop("staggered", False) + return Case( + case_id=f"P1b-{label}", group="P1b", num_requests=NUM_REQUESTS, + prefill_tokens=PREFILL_DECODE[0], decode_tokens=PREFILL_DECODE[1], + arrival="poisson", qps=STAGGERED_QPS if staggered else BURST_QPS, + simulation_mode="online" if staggered else "offline", **shape, + ) + + +def run(label: str, output: Path) -> dict: + from frontier.scheduler.cluster_scheduler.base_cluster_scheduler import BaseClusterScheduler + from frontier.scheduler.replica_scheduler.vllm_v1_engine_replica_scheduler import ( + VLLMv1EngineReplicaScheduler, + ) + from frontier.scheduler.replica_stage_scheduler.replica_stage_schduler import ( + ReplicaStageScheduler, + ) + from frontier.scheduler.utils.forward_sync_state import ForwardSyncState + from frontier.simulator import Simulator + + case = build_case(label) + records: list[dict] = [] + held: dict = {} + + def emit(kind: str, **fields) -> None: + records.append({"seq": len(records), "kind": kind, "time": held["sim"]._time, **fields}) + + def keys(cluster_scheduler, replica_id: int, lane_scheduler) -> dict: + context = cluster_scheduler._stage_execution_contexts[(replica_id, 0)] + stage0 = lane_scheduler.get_replica_stage_scheduler(0) + return { + "replica_forward_id": int( + cluster_scheduler._forward_sync_state._next_step_id_by_replica.get(replica_id, 0) + ), + "stage0_next_group": context._next_forward_group_id, + "stage0_bound_group": context._forward_group_id, + "stage0_sealed": context._forward_group_sealed, + "lane_stage0_busy": stage0._is_busy, + "lane_stage0_queue": len(stage0._batch_queue), + } + + original_next_batch = VLLMv1EngineReplicaScheduler._get_next_batch + original_batch_end = BaseClusterScheduler.on_replica_batch_end + original_pop = ReplicaStageScheduler.pop_batch_if_not_busy + original_resolve = ForwardSyncState.resolve_step + + def observed_next_batch(self, is_micro_batch=False): + batch = original_next_batch(self, is_micro_batch=is_micro_batch) + if batch is not None: + emit("admit", lane=self._replica_local_id, batch=batch.id, + running_after=self._num_running_batches + 1, stages=self._num_stages, + load=list(self.get_request_load()), + **keys(self._cluster_scheduler, self._replica_id, self)) + return batch + + def observed_batch_end(self, time, replica_id, replica_local_id, batch): + result = original_batch_end(self, time, replica_id, replica_local_id, batch) + lane = self.get_replica_scheduler(replica_id, replica_local_id) + emit("complete", lane=replica_local_id, batch=batch.id, + running_after=lane.num_running_batches, load=list(lane.get_request_load()), + **keys(self, replica_id, lane)) + return result + + def observed_pop(self): + batch = original_pop(self) + if batch is not None: + emit("stage_start", lane=self._replica_local_id, stage=self._stage_id, + batch=batch.id, group=getattr(batch, "_forward_cohort_provisional_id", None)) + return batch + + def observed_resolve(self, **kwargs): + step = original_resolve(self, **kwargs) + batch = kwargs["batch"] + emit("room", lane=kwargs["lane_id"], stage=kwargs["stage_id"], layer=kwargs["layer_id"], + sync_stage=kwargs["sync_stage"], batch=batch.id, idle=bool(batch.is_idle), step=step, + group=getattr(batch, "_forward_cohort_provisional_id", None)) + return step + + VLLMv1EngineReplicaScheduler._get_next_batch = observed_next_batch + BaseClusterScheduler.on_replica_batch_end = observed_batch_end + ReplicaStageScheduler.pop_batch_if_not_busy = observed_pop + ForwardSyncState.resolve_step = observed_resolve + try: + config = build_config(case, output / "metrics", output / "cache") + simulator = Simulator(config) + held["sim"] = simulator + simulator.run() + requests = list(simulator._all_requests) + finally: + VLLMv1EngineReplicaScheduler._get_next_batch = original_next_batch + BaseClusterScheduler.on_replica_batch_end = original_batch_end + ReplicaStageScheduler.pop_batch_if_not_busy = original_pop + ForwardSyncState.resolve_step = original_resolve + return { + "shape": label, + "case": {field: getattr(case, field) for field in ( + "is_moe", "attn_dp", "stages", "num_requests", "prefill_tokens", + "decode_tokens", "arrival", "qps", "simulation_mode", "cc_backend")}, + "completed": sum(1 for request in requests if request.completed), + "requests": len(requests), + "records": records, + } + + +if __name__ == "__main__": + label, output = sys.argv[1], Path(sys.argv[2]) + output.mkdir(parents=True, exist_ok=True) + result = run(label, output) + (output / f"{label}.json").write_text(json.dumps(result, indent=1)) + print(label, f"{result['completed']}/{result['requests']} completed,", + len(result["records"]), "records") diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md b/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md index 64bcf94e..c0349654 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/summary.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | Status only: Step 9 in progress. W9-01 fixed via PR 36 and merged forward (composition check PASS); P1 complete; D9-2 proposed, awaiting the user's decision (`progress.md`, plan §18.15). The Step 0–8 archive below is unchanged. | | 2026-09-22 | FP8 native rerun with the corrected `block_shape` wiring PASS (`exp-0922-202645-561899`, 8 passed). Step 9 plan reviewed a second time against the user's quality gates (plan §18.12); still not started. | | 2026-09-21 | Placeholder created at Step 0. | | 2026-09-22 | Completion archive written at Step 8. | diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w9_pp_dp_placement.md b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w9_pp_dp_placement.md index 241da2fd..93d08550 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w9_pp_dp_placement.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/test_report_2026-09-22_w9_pp_dp_placement.md @@ -5,6 +5,7 @@ | Date | Change | | --- | --- | | 2026-09-22 | Created. Covers packages P1(a), P1(b) and G1. | +| 2026-09-23 | §2 addendum: P1(b) rerun after the W9-01 merge-forward on seven shapes, with key scoring. §4 updated. | Environment for every CPU check below: @@ -96,6 +97,59 @@ Both are recorded in `issues.md`. The three source files involved in W9-01 are byte-identical to `origin/main`, verified with `git rev-parse HEAD:` against `origin/main:`. +### 2.1 Addendum (2026-09-23): the multi-lane PP shapes after W9-01 + +Revision: `d3e6e78` plus the untracked probe files. The rule file equals +`origin/main`, PR 36 `4ab1964`. + +Driver: `step9_p1b/probe_boundaries.py`. It builds each shape with the +`tests.e2e.stage_admission_matrix` fixture: a 6-layer synthetic model, +analytical CC backend, round-robin placement, dummy predictor, prefill 16 and +decode 3 tokens, 6 requests. The driver records admissions, completions, +stage-0 starts with their forward group, and layer rooms, together with the +stage-0 state at each boundary. The scorer is `step9_p1b/analyze_keys.py`, +with method and targets in its docstring. + +Commands, with `PYTHONPATH=`, `WANDB_DISABLED=true`, +`VIDUR_DISABLE_WANDB=1`, `FRONTIER_TMP_ROOT=/data/ycfeng/tmp` and +`/data/ycfeng/envs/frontier-py310/bin/python`: + +```bash +python task_memory/.../step9_p1b/probe_boundaries.py /data/ycfeng/tmp/issue26-correctness-pr/step9_p1b/ +python task_memory/.../step9_p1b/analyze_keys.py /data/ycfeng/tmp/issue26-correctness-pr/step9_p1b +``` + +Expected results: + +1. Every shape completes 6/6. +2. For the key to be accepted, it has 0 peer splits, 0 merges and 0 + inversions against each report's actual stage-0 group, and 0 ms of replay + mismatch. + +| Shape | Completed | Records | A (split/merge/inv, ms) | Lane counter | Group-anchored | Verdict | +| --- | --- | --- | --- | --- | --- | --- | +| MoE dp2 PP1 burst | 6/6 | 108 | 0/0/0, 0 | 0/0/0, 0 | 0/0/0, 0 | PASS | +| MoE dp1 PP2 burst | 6/6 | 140 | 0/0/0, 0 | 0/0/0, 0 | 0/0/0, 0 | PASS | +| MoE dp1 PP3 burst | 6/6 | 176 | 0/1/0, 400 | 5/0/0, 0 | 0/0/0, 0 | PASS for group-anchored | +| MoE dp2 PP2 burst (the W9-01 shape) | 6/6 | 160 | 0/0/0, 0 | 4/0/0, 0 | 0/0/0, 0 | PASS for group-anchored; was FAIL (W9-01) | +| MoE dp2 PP2 staggered | 6/6 | 176 | 0/0/0, 0 | 6/0/0, 0 | 0/0/0, 0 | PASS for group-anchored | +| MoE dp2 PP3 burst | 6/6 | 198 | 0/4/0, 148 | 0/0/0, 0 | 0/0/0, 0 | PASS for group-anchored; was FAIL (W9-02, collective-sim only) | +| MoE dp2 PP3 staggered | 6/6 | 210 | 4/1/0, 200 | 15/7/5, 0 | 0/0/0, 0 | PASS for group-anchored | + +Split/merge counts for the lane counter on burst shapes fall on +completion-only rows. There the Frontier target is the lane's next forward, +while the reference would count a separate dummy iteration. The +real-forward-only counts are in `key_scores.json` (`forward_reports`). The +group-anchored key equals the stage-0 predictor on every report: 139 of 139. + +Limits: + +- Targets are Frontier's own forward grouping, not vLLM measurements. +- 12 final drain completions have no later forward and are unscored. +- The replay metric depends on this small workload's timing. The pairwise + counts are the primary evidence. +- W9-03 (reference lockstep) is a source-reading observation. + ## 3. G1 — ground-truth instrumentation Checkout `/data/ycfeng/Frontier/.real-engine/vLLM-BS`, local branch @@ -149,7 +203,7 @@ source reading, not from a run. | Package | Verdict | | --- | --- | | P1(a) | PASS | -| P1(b) | Evidence collected on three shapes; BLOCKED on the fourth by W9-01 | +| P1(b) | PASS on seven shapes after the W9-01 merge-forward (§2.1) | | Design checkpoint D9-1 | Settled | -| Design checkpoint D9-2 | Open; the candidate key fails invariant I5 and the alternatives cannot be tested until W9-01 is resolved | +| Design checkpoint D9-2 | Proposed (group-anchored key); awaiting the user's decision | | G1 | PASS for what is testable without a GPU | From d1a2a065a8b605f05954ec543adf2e7f391b157c Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 16:03:36 +0800 Subject: [PATCH 50/80] docs(step9): record the D9-2 decision and the C1 PP3 amendment The user chose the group-anchored report key. Record the decision, restore the multi-lane PP3 row on the analytical backend in C1, and align the design text with the implemented per-lane lists and the stage-context accessor. Note the one out-of-order arrival the rule produces in the seven probe shapes and why it mirrors a race in the reference. --- .../design.md | 19 +++++++++++++++---- .../plan.md | 10 ++++++++-- .../requirements.md | 12 ++++++++++++ 3 files changed, 35 insertions(+), 6 deletions(-) diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md index 1e76a5b7..e3962cc6 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md @@ -7,6 +7,7 @@ scope decisions and the pre-measurement expectation for that package. | Date | Change | | --- | --- | +| 2026-09-23 | D9-2 decided by the user: the group-anchored rule. | | 2026-09-23 | Added "Design checkpoint D9-2: the key from the fourth shape": reference lockstep facts, seven-shape P1(b) scores, the proposed group-anchored key rule with its invariant argument and residuals, W9-03 pointer. Proposal only; awaits the user's decision. | | 2026-09-22 | Added the design checkpoint section: D9-1 payload settled from the P1 oracle; D9-2 key left open because the `ForwardSyncState` candidate fails invariant I5 at PP>1 and the I1/I5 trade-off is only observable on the shape blocked by W9-01. | | 2026-09-22 | W9 second review (user-directed quality gates): section "What the code already provides" added; planned-edits rows for the hook payload, the call site and the CPU oracle amended; plan §18.12 R9-01..R9-08. | @@ -559,7 +560,7 @@ choice, which is the kind of unfalsifiable design the gates exclude. W9-01 is fixed on `main` (PR 36) and merged forward, so the shape that decides D9-2 now runs. This section records what the probe shows and the proposed key -rule. The rule is a proposal: it needs the user's decision before P2. +rule. The user chose this rule on 2026-09-23 (`requirements.md`). ### The reference key is the shared forward index @@ -660,7 +661,7 @@ over. - At an admission: `last_admitted_key[l] = key(l)`. The admission is reported now while `num_running_batches < num_pipeline_stages`. Otherwise the key is held in `held_key[l]`. -- At a completion: report `held_key.pop(l)` if a key is held. Otherwise report +- At a completion: report `held_key[l]` and clear it if a key is held. Otherwise report `key(l)` and store nothing. Against the invariants: @@ -674,6 +675,16 @@ Against the invariants: | I5 | Each consecutive cold-fill admission on a lane gets `+ 1`. No spacing constant. | dp1 and dp2 PP3 cold fills. | | I6 | Two integers per lane, overwritten in place. | Bounded by `attn_dp`. | +"0 inversions" compares keys with forward order. Arrival order at the balancer +is a separate matter. In the dp2 PP3 burst, lane 1's `on_schedule` admits two +batches (keys 0 and 1) before lane 0's `on_schedule` admits its first (key 0), +so one report in the seven shapes arrives with a smaller key than the last one. +`VllmDPLoadBalancer.report` then logs the reference's out-of-order warning and +applies the counts; the replay mismatch stays 0 ms. The reference has the same +race: a step whose batch queue still has room publishes without waiting for +its forward, so engine 1's step 1 can reach the coordinator before engine 0's +step 0. + At PP=1 every admission is folded, and the held key is the group of the forward that completes. Peers compare equal and later forwards compare greater, so the comparisons match the current `ForwardSyncState.get_step_id` @@ -682,9 +693,9 @@ key. C2's byte-identical PP=1 check tests exactly that. The policy's New surface, compared with the planned edits above: one read-only property on `StageExecutionContext` over two existing fields, in place of the planned -`ForwardSyncState` accessor, and two per-lane dicts in the policy scheduler. +`ForwardSyncState` accessor, and two per-lane lists in the policy scheduler. The policy reaches the context through the base class's -`_stage_execution_contexts[(replica_id, 0)]`. For a dense model the group is +`get_stage_execution_context(replica_id, 0)`. For a dense model the group is never bound (`bind_forward_group` is MoE-only), so the key falls back to the lane's admission counter. The policy admits a dense model only at `attn_dp=1`, where placement has one choice. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md index dea67ff1..5d6b3896 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | D9-2 decided by the user: group-anchored key (§18.15, `requirements.md`); C1 PP3 row amended; P2 started. | | 2026-09-23 | §18.15 added: P1(b) completed on seven shapes; D9-2 proposal (group-anchored key) and the C1 PP3 amendment await the user's decision. §18.13 blocker marked resolved. | | 2026-09-23 | §18.14 results: K1–K4 pass on `03d5f24`; K2 amended after measuring for online cells (one cell's batches differ after an earlier admission). | | 2026-09-23 | §18.14 added: W9-01 merge-forward and the composition check, with pass criteria fixed before measuring. | @@ -860,7 +861,7 @@ Under [R1]: `frontier/profiling/moe/moe_vllm_kernel.py`, `frontier/entities/stag | # | Criterion | Evidence | | --- | --- | --- | -| C1 | Valid PP2 and PP3 configurations (layer count divisible by PP; `MONOLITHIC`, one Replica, `vllm_v1`, MoE or `attn_dp == 1` — the PP1 clause is the only guard removed) complete every request with request/token/owner conservation, for dense `attn_dp=1` and MoE `attn_dp=2` at PP2. **Amended 2026-09-22 (W9-02):** the PP3 row uses `attn_dp=1`; `attn_dp=2, moe_ep=2, PP=3` is rejected at construction because 6 devices do not divide the node size of 4. PP3 uses a separate CPU fixture with a valid layer count (6 or 12); the native PP2 model stays the approved 8-layer tiny Qwen3-MoE. | P4 real-loop PP2 and PP3 cases; §18.11 behavioral matrix. | +| C1 | Valid PP2 and PP3 configurations (layer count divisible by PP; `MONOLITHIC`, one Replica, `vllm_v1`, MoE or `attn_dp == 1` — the PP1 clause is the only guard removed) complete every request with request/token/owner conservation, for dense `attn_dp=1` and MoE `attn_dp=2` at PP2. **Amended 2026-09-22 (W9-02):** the PP3 row uses `attn_dp=1`; `attn_dp=2, moe_ep=2, PP=3` is rejected at construction because 6 devices do not divide the node size of 4. **Amended 2026-09-23 (D9-2 decision):** that rejection is the collective-sim topology rule; a MoE `attn_dp=2, PP=3` row on the analytical backend is restored, because I5 needs a multi-lane PP3 case. PP3 uses a separate CPU fixture with a valid layer count (6 or 12); the native PP2 model stays the approved 8-layer tiny Qwen3-MoE. | P4 real-loop PP2 and PP3 cases; §18.11 behavioral matrix. | | C2 | Previously supported behavior is unchanged under the stated comparison contract: every existing PP1 `vllm_load_balancing` scenario has value-identical `request_metrics.csv` and identical `system_metrics.json` (timestamps/run ids removed, the Q11 rule), with no additional admission-only report; every other cluster scheduler, including the supported disaggregated paths, has identical event outcomes (the hook is inert for them). | P5 byte comparison; Step 8 regression set rerun. | | C3 | For a controlled or demonstrably matched iteration history, the emitted loads, the equality/order relation of logical-iteration keys, the coordinator snapshots and the frontend-visible counts agree with the reference. Natural-history divergence is classified by first cause (arrival/delivery order, batch composition, output readiness, count calculation, key grouping, snapshot publication, frontend selection), not hidden by re-indexing. Boundary-index comparison alone is not an alignment method. | CPU reference-loop oracle (P1) + causal join of the G4 trace (§18.11 instrumentation chain) + `workflow-gap-analysis`. | | C4 | In a trace-qualified native discriminating slice (§18.6, qualified per §18.11: the intended snapshot was applied at the frontend before the probe was routed), the corrected placement matches the reference and the explicit test-only completion-reporting control fails for the expected reason. The actual unmodified PP2 baseline is reported as rejected by its constructor, not as a placement. Otherwise the slice is `SCENARIO_NOT_REACHED` with the failed precondition named. | §18.6 comparison table with the control column. | @@ -1205,7 +1206,7 @@ New surface: - one read-only property on `StageExecutionContext`, replacing the planned `ForwardSyncState` accessor; -- two per-lane dicts in the policy scheduler. +- two per-lane lists in the policy scheduler. The policy's `ForwardSyncState` key is removed. @@ -1227,3 +1228,8 @@ chosen, P2 implements it with the D9-1 hook and the guard change. P3/P4 carry the §18.11 matrix with the restored PP3 row. P2–P6 and G3–G5 stay paused until then. +**Decided 2026-09-23.** The user chose the group-anchored rule, with the C1 +amendment (`requirements.md`, "[Decision] 2026-09-23 — D9-2 report key"). +P2–P5 proceed on the CPU. G3 and G4 need GPU runs whose authorization is still +BLOCKED in the case manifest, so G3–G5 wait for a separate go. + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md index 72b64002..8d29de56 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/requirements.md @@ -8,6 +8,7 @@ | 2026-09-22 | Recorded the W6 artifact-identity decision and the native GPU validation instruction. | | 2026-09-22 | Recorded the PP>1 `vllm_load_balancing` request, the codesign-only GPU instruction, and the open Step 9 decisions. | | 2026-09-23 | Recorded the W9-01 merge-forward request after PR 36 merged. | +| 2026-09-23 | Recorded the D9-2 decision (group-anchored report key). | ## [Original Request] 2026-09-21 @@ -103,3 +104,14 @@ Reading: read the external review in full, verify each finding against the sourc | Next | On a pass, resume Step 9 P1(b) and the design checkpoint D9-2. | | `.gitignore` | The `task_memory` exceptions of PR 34 and PR 35 stay as they are; the owner removes them before those merges. | +## [Decision] 2026-09-23 — D9-2 report key + +Question (AskUserQuestion): "D9-2：P2 应实现哪条 report key 规则？" + +Answer: "Group-anchored (Recommended)", selected together with its preview. + +| Item | Decision | +| --- | --- | +| Key rule | `key(l) = max(C.joinable_forward_group_id, last_admitted_key[l] + 1)`. An admission stores its key and reports it while the pipeline has room; otherwise the key is held. A completion reports the held key, or `key(l)` without storing it (`design.md` "Design checkpoint D9-2", plan §18.15). | +| Included with the option | A read-only `StageExecutionContext` property and two per-lane dicts in the policy scheduler. The policy's use of the `ForwardSyncState` key is removed. C1 regains the MoE `attn_dp=2, PP=3` row on the analytical backend. | +| Not decided | W9-03 (reference DP lockstep under PP) stays an observation outside Step 9. GPU runs G3/G4 still need their own authorization (manifest BLOCKED). | From 2ffe78d2a211033940a2375688063457ee9f7804 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 16:03:48 +0800 Subject: [PATCH 51/80] feat(dp-placement): publish vLLM DP loads per engine iteration under PP Under pipeline parallelism a vLLM engine publishes its request counts after every scheduling iteration, including iterations that only admit a batch while the batch queue still has room. The policy reported only at batch completion, so it could not show the state after a cold-fill admission and was limited to one pipeline stage. Add an inert on_replica_batch_scheduled seam, called from the MONOLITHIC and PREFILL admission loop after each admission. The vllm_load_balancing policy publishes an admission while the lane's pipeline has room and otherwise holds its key for the completion the engine waits on. Reports are keyed by the stage-0 forward group the lane's next batch joins, which peer lanes share, advanced per lane for consecutive admissions before a batch starts stage 0. At PP=1 every admission is held and the comparisons match the previous key. The PP=1 guard clause is removed; the other guards are unchanged. Unit tests cover PP1, PP2 cold fill in both lane orders, PP3 admission-only iterations, a full pipeline, the combined completion-admission iteration, drain to zero, the real admission loop, and both inert seams. --- .../base_cluster_scheduler.py | 15 + .../vllm_load_balancing_cluster_scheduler.py | 102 ++++-- .../base_replica_scheduler.py | 6 + .../stage_execution_context.py | 7 + .../progress.md | 17 +- .../test_shared_forward_group_admission.py | 20 ++ tests/unit/test_vllm_dp_load_balancer.py | 329 +++++++++++++++++- 7 files changed, 452 insertions(+), 44 deletions(-) diff --git a/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py b/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py index c9494f87..44b5a6e4 100644 --- a/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py +++ b/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py @@ -438,6 +438,21 @@ def schedule_at(self, time: float) -> List[Tuple[int, int, Request]]: return self.schedule() + def on_replica_batch_scheduled( + self, + time: float, + replica_id: int, + replica_local_id: int | None, + batch: Batch, + ) -> None: + """Observe one batch admitted by a MONOLITHIC or PREFILL lane. Inert by default. + + Called after the lane counts the batch as running, so a policy that + reads lane populations here sees the post-admission state. + """ + + return None + def on_replica_batch_end( self, time: float, diff --git a/frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py b/frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py index 1c059a1c..bd66dbb9 100644 --- a/frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py +++ b/frontier/scheduler/cluster_scheduler/vllm_load_balancing_cluster_scheduler.py @@ -6,7 +6,6 @@ from frontier.scheduler.cluster_scheduler.base_cluster_scheduler import ( BaseClusterScheduler, ) -from frontier.scheduler.utils.forward_sync_state import ForwardSyncState from frontier.scheduler.utils.vllm_dp_load_balancer import VllmDPLoadBalancer from frontier.types import ClusterType, ReplicaSchedulerType @@ -15,10 +14,10 @@ class VllmLoadBalancingClusterScheduler(BaseClusterScheduler): """Model single-frontend vLLM V1 DP routing with delayed load snapshots. Supported scope, and nothing wider: one co-location Replica, one modeled - frontend, the `vllm_v1` replica scheduler, one pipeline stage, and a report - key whose ordering actually holds -- a MoE model, whose attention-DP lanes - share one forward step identity, or a single lane, where the question does - not arise. Everything else is rejected in the constructor. + frontend, the `vllm_v1` replica scheduler, and a report key whose ordering + actually holds -- a MoE model, whose attention-DP lanes share each stage-0 + forward, or a single lane, where the question does not arise. Everything + else is rejected in the constructor. No placement or timing equivalence with a real vLLM deployment is claimed. """ @@ -28,24 +27,21 @@ def __init__(self, *args, **kwargs): if ( self._cluster_type is not ClusterType.MONOLITHIC or self._num_replicas != 1 - or self._config.replica_config.num_pipeline_stages != 1 or self._replica_scheduler_type is not ReplicaSchedulerType.VLLM_V1 ): raise ValueError( "vllm_load_balancing supports one co-location Replica with " - "vllm_v1 and PP1, got " + "vllm_v1, got " f"cluster_type={self._cluster_type.name}, " f"num_replicas={self._num_replicas}, " - f"num_pipeline_stages=" - f"{self._config.replica_config.num_pipeline_stages}, " f"replica_scheduler={self._replica_scheduler_type}" ) - # The load report is ordered by the forward step identity. A MoE - # Replica resolves one shared identity for every lane of one forward, so - # the key is monotonic per Replica. A dense Replica has no per-forward - # collective across its attention-DP lanes, so each lane keeps its own - # creation counter and two lanes report interleaved keys against one - # shared ordering scalar. One lane is safe either way. + # The load report is keyed by the Replica's stage-0 forward group. A + # MoE Replica binds one group for every lane of one forward, so peer + # lanes share the key. A dense Replica has no per-forward collective + # across its attention-DP lanes and binds no group, so two lanes would + # order their reports only by their own admission counts. One lane is + # safe either way. if not self._config.replica_config.model_config.is_moe: if self._replica_dp_size != 1: raise ValueError( @@ -55,7 +51,10 @@ def __init__(self, *args, **kwargs): f"attn_dp={self._replica_dp_size}" ) self._serving_replica_id = next(iter(self._cluster.replicas)) + self._num_pipeline_stages = self._config.replica_config.num_pipeline_stages self._load_balancer = VllmDPLoadBalancer(self._replica_dp_size) + self._last_admitted_key = [-1] * self._replica_dp_size + self._held_key: list[int | None] = [None] * self._replica_dp_size def schedule_at(self, time: float) -> List[Tuple[int, int, Request]]: """Route every queued request at this simulation time. @@ -81,6 +80,31 @@ def schedule(self) -> List[Tuple[int, int, Request]]: "route through schedule_at(time), which ClusterScheduleEvent calls" ) + def on_replica_batch_scheduled( + self, + time: float, + replica_id: int, + replica_local_id: int | None, + batch: Batch, + ) -> None: + """Report the lane's post-admission load while its pipeline has room. + + A vLLM engine whose batch queue still has room after scheduling + publishes at once; otherwise it waits for its oldest batch and + publishes after applying that output. That filled-pipeline admission + is therefore reported with the lane's next completion, under the key + taken here. + """ + + lane_id = self._lane_index(replica_local_id) + key = self._next_report_key(replica_id, lane_id) + self._last_admitted_key[lane_id] = key + lane = self.get_replica_scheduler(replica_id, lane_id) + if lane.num_running_batches < self._num_pipeline_stages: + self._load_balancer.report(time, lane_id, key, lane.get_request_load()) + else: + self._held_key[lane_id] = key + def on_replica_batch_end( self, time: float, @@ -88,25 +112,47 @@ def on_replica_batch_end( replica_local_id: int | None, batch: Batch, ) -> None: - """Report the lane's post-step load, keyed by the forward identity. + """Report the lane's post-step load under the iteration that applied it. + + That iteration is the one whose admission filled the pipeline, when + there is one. Otherwise the lane launches no forward here, and the key + is that of the lane's next forward. + """ + + lane_id = self._lane_index(replica_local_id) + key = self._held_key[lane_id] + if key is None: + key = self._next_report_key(replica_id, lane_id) + else: + self._held_key[lane_id] = None + lane = self.get_replica_scheduler(replica_id, lane_id) + self._load_balancer.report(time, lane_id, key, lane.get_request_load()) - The key's numeric value is not a vLLM step counter: it advances once per - layer, so consecutive forwards are roughly `num_layers` apart. Only its - ordering and equality are used, which is all the reference coordinator - uses its `(wave, step)` pair for. Frontier has no wave reset, so a - Replica-scoped monotonic counter collapses that pair to one scalar. + def _next_report_key(self, replica_id: int, lane_id: int) -> int: + """Key of the stage-0 forward that the lane's next batch joins. + + vLLM keys a report by the engine's step counter. Each step launches + one forward, real or dummy, that pairs with the peers' forward of the + same index, so the key names a forward shared by the peer engines. The + Replica's stage-0 forward group is that shared forward here. A lane's + batches admitted before they start stage 0 take consecutive groups, + which the `+ 1` term provides. Only comparisons of keys are used, as the + reference coordinator uses its `(wave, step)` pair; Frontier has no wave + reset, so the group id collapses that pair to one scalar. """ + context = self.get_stage_execution_context(replica_id, 0) + return max( + context.joinable_forward_group_id, + self._last_admitted_key[lane_id] + 1, + ) + + @staticmethod + def _lane_index(replica_local_id: int | None) -> int: if type(replica_local_id) is not int: raise ValueError( "vllm_load_balancing reports load per attention-DP lane and " "needs an exact lane index, got " f"replica_local_id={replica_local_id!r}" ) - lane = self.get_replica_scheduler(replica_id, replica_local_id) - self._load_balancer.report( - time, - replica_local_id, - ForwardSyncState.get_step_id(batch), - lane.get_request_load(), - ) + return replica_local_id diff --git a/frontier/scheduler/replica_scheduler/base_replica_scheduler.py b/frontier/scheduler/replica_scheduler/base_replica_scheduler.py index 30498599..4c68ceaf 100644 --- a/frontier/scheduler/replica_scheduler/base_replica_scheduler.py +++ b/frontier/scheduler/replica_scheduler/base_replica_scheduler.py @@ -1055,6 +1055,12 @@ def on_schedule(self, time: float = 0.0) -> List[Batch]: break scheduled_batches.append(batch) self._num_running_batches += 1 + # Reported per admission here, not after the call returns: the + # lane state after each of several admissions is visible only + # inside this loop. + self._cluster_scheduler.on_replica_batch_scheduled( + time, self._replica_id, self._replica_local_id, batch + ) if ( hasattr(self, "_has_monolithic_pp_mtp_output_wait") and self._has_monolithic_pp_mtp_output_wait() diff --git a/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py b/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py index 1dcda539..81d2ff1a 100644 --- a/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py +++ b/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py @@ -91,6 +91,13 @@ def forward_group_sealed(self) -> bool: """Whether new attention lanes must wait for the next stage forward.""" return self._forward_group_sealed + @property + def joinable_forward_group_id(self) -> int: + """Id of the stage forward that a lane not already in it would join now.""" + if self._forward_group_id is not None and not self._forward_group_sealed: + return self._forward_group_id + return self._next_forward_group_id + @property def replica_id(self) -> int: return self._replica_id diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 092642c7..0fb80f69 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | D9-2 decided (group-anchored key); P2 implemented and P3 unit tests added (132 targeted tests pass; the 19 new or changed cases fail on the pre-P2 tree). | | 2026-09-23 | Step 9 P1(b) completed on seven shapes; D9-2 proposal recorded (design.md, plan §18.15); W9-03 observation; awaits the user's D9-2 decision. | | 2026-09-23 | W9-01 merge-forward: `origin/main` merged (`dd9b8d9`); composition check K1–K4 pass on `03d5f24` (drain-reader fix); Step 9 P1(b) resumed. | | 2026-09-23 | PR 36 pre-merge untrack (P6, `4d08c5d`) done; W9-01 copies refreshed. Merge-forward waits for the PR 36 merge. | @@ -223,8 +224,10 @@ User start signal: "开始执行step9", with the quality gates repeated (readabi | --- | --- | --- | | P1(a) reference-loop oracle | completed | `tests/comparison/dp_placement_pp/reference_loop.py`; `tests/unit/test_dp_placement_reference_loop.py` (9 passed, 1.21 s, `frontier-py310`). §18.11 state table confirmed as written; PP=1 shown to degenerate to "every iteration schedules and applies"; depth 3 shown to allow two consecutive admission-only publications, which rules out any stride constant. | | P1(b) Frontier boundary probe | completed 2026-09-23 (see "Step 9 P1(b) and D9-2" below) | Three shapes probed (`attn_dp=2 PP=1`, `attn_dp=1 PP=2`, `attn_dp=1 PP=3`), tables in `plan.md` §18.13. The fourth shape, MoE `attn_dp=2, moe_ep=2, PP=2`, drains the event queue with requests unfinished — pre-existing defect W9-01 in `issues.md`. | -| Design checkpoint (D9-1, D9-2) | D9-1 settled; D9-2 proposed 2026-09-23, awaiting decision | D9-1's payload is settled (the completion hook signature already carries lane, load and a key source). D9-2 is not: the candidate key `ForwardSyncState._next_step_id_by_replica` satisfies I1, I2, I3, I4 and I6 on the runnable shapes but fails I5, and no alternative can be checked against I1 without a running `attn_dp>1, PP>1` shape. | -| P2–P6, G3–G5 | paused | All depend on the design checkpoint or on that shape. | +| Design checkpoint (D9-1, D9-2) | settled 2026-09-23 | D9-1's payload was settled from the P1 oracle. D9-2 is the group-anchored key, exact on all seven P1(b) shapes (`design.md` "Design checkpoint D9-2"); the user chose it on 2026-09-23. The first candidate, `ForwardSyncState._next_step_id_by_replica`, failed I5. | +| P2 implementation, P3 unit tests | completed 2026-09-23 | See "Step 9 P2 and P3" below. | +| P4, P5, P6 | pending | CPU only. | +| G3–G5 | blocked | GPU authorization is `BLOCKED` in the case manifest. | W9-01 is not caused by this PR: `stage_execution_context.py`, `replica_stage_schduler.py` and `stage_contexts.py` are byte-identical to `main`. It is unobserved because every Simulator-level test with `attn_dp > 1` uses `num_pipeline_stages = 1` and no shipped example sets `attn_dp > 1`. Scope decision requested from the user; recommendation is to fix it as a separate correctness item rather than inside this feature branch. @@ -258,7 +261,15 @@ W9-02: `attn_dp=2, moe_ep=2, PP=3` is rejected at construction (6 devices agains | Variant | Every-report-advance variant, candidate `every_report_advances` in `analyze_keys.py` | `key_scores.json` | Drifts on dp2 PP3 staggered: 6/5/5 on real-forward reports. Rejected. | | Residual count | Completion-only report followed by an equal-key report on the same lane | `key_scores.json` rows | 17 pairs, 3 with changed counts. | | Records | `design.md`; plan §18.15 and the §18.13 resolution; `issues.md` (W9-01 step 3, W9-02 narrowed, W9-03); test report §2 addendum | — | Done | -| Decision | D9-2 rule and the C1 PP3 amendment | — | Awaiting the user | +| Decision | D9-2 rule and the C1 PP3 amendment | `requirements.md` | User chose the group-anchored rule; C1 amended | + +### Step 9 P2 and P3 (2026-09-23) + +| Step | Change / command | Reason and expectation | Result | +| --- | --- | --- | --- | +| P2 source | `StageExecutionContext.joinable_forward_group_id`; inert `BaseClusterScheduler.on_replica_batch_scheduled`; its call in the MONOLITHIC/PREFILL admission loop of `BaseReplicaScheduler.on_schedule`, after `_num_running_batches += 1`; `VllmLoadBalancingClusterScheduler`: PP1 guard clause removed, group-anchored key with a held key per lane, `ForwardSyncState` use removed | D9-1..D9-3 as decided. Expect only the intended guard case to fail in the existing suites. | Existing suites: 1 failed (the `pipeline_parallel` rejection case), 112 passed. | +| P3 tests | `tests/unit/test_vllm_dp_load_balancer.py`: guard case inverted into PP2/PP3 construct cases (8 shapes) plus dense multi-lane PP2 and uneven-partition rejections; `_ScriptedReplica` drives scripted lane readings and real stage-0 forward groups; cases for PP1 (no schedule-time report), PP2 cold fill in both lane orders (no partial latch), PP3 two admission-only iterations, a full pipeline (MoE and dense), a completion plus the admission it makes room for (one key, no intermediate latch), drain to zero and new work, and the real admission loop at PP2; both seams in the inert and unknown-lane tests. `tests/unit/test_shared_forward_group_admission.py`: `joinable_forward_group_id` across bind, seal and release. | Expected reports written from the reference iteration, before running. Each new case must fail on the pre-P2 tree. | `pytest -q -p no:cacheprovider tests/unit/test_vllm_dp_load_balancer.py tests/integration/test_vllm_dp_placement_runtime.py tests/unit/test_dp_placement_reference_loop.py tests/unit/test_stage_execution_context.py tests/unit/test_shared_forward_group_admission.py`: 132 passed, 7.07 s. The same tests on `git archive HEAD` (pre-P2) plus the new test files: 19 failed, 72 passed; the 19 are exactly the new or changed cases. | +| Arrival-order check | Replay of the group-anchored keys from `key_scores.json` in report order, counting keys smaller than the last applied one | Recorded so the balancer's warning is not mistaken for a key defect later. | 1 report in 7 shapes (dp2 PP3 burst, t=0: lane 0 key 0 after lane 1 key 1); the reference has the same race. Noted in `design.md`. | ### G1 ground-truth instrumentation (2026-09-22, completed) diff --git a/tests/unit/test_shared_forward_group_admission.py b/tests/unit/test_shared_forward_group_admission.py index 5d97ebc7..15edc0ec 100644 --- a/tests/unit/test_shared_forward_group_admission.py +++ b/tests/unit/test_shared_forward_group_admission.py @@ -114,6 +114,26 @@ def test_started_group_blocks_new_lane_through_ep_restore_and_partial_release(): assert context.queued_tickets == () +def test_a_new_lane_joins_the_bound_group_only_until_it_is_sealed(): + context = StageExecutionContext(replica_id=0, stage_id=0, ep_size=2, full_stage_capacity=2) + assert context.joinable_forward_group_id == 0 + first = context.enqueue_full_stage(operation_id="first") + assert context.try_acquire(first) + group = context.bind_forward_group(first) + assert context.joinable_forward_group_id == group + wave = context.replace_full_stage_owners_with_ep_wave( + (first,), operation_id="wave", participant_ep_ids=(0, 1), + ) + assert context.joinable_forward_group_id == group + 1 + (owner,) = context.replace_ep_wave_with_full_stage_owners(wave, operation_ids=("next",)) + context.release(owner) + assert context.is_idle + assert context.joinable_forward_group_id == group + 1 + later = context.enqueue_full_stage(operation_id="later") + assert context.try_acquire(later) + assert context.bind_forward_group(later) == group + 1 + + @pytest.mark.parametrize("sync_kind", ["prefill", "decode"]) def test_next_group_queue_does_not_block_current_group_idle_participation(sync_kind): context = StageExecutionContext(replica_id=0, stage_id=0, ep_size=2, full_stage_capacity=2) diff --git a/tests/unit/test_vllm_dp_load_balancer.py b/tests/unit/test_vllm_dp_load_balancer.py index e8c6c682..88902251 100644 --- a/tests/unit/test_vllm_dp_load_balancer.py +++ b/tests/unit/test_vllm_dp_load_balancer.py @@ -361,9 +361,9 @@ def test_the_engine_count_must_be_a_positive_int(num_engines) -> None: # -------------------------------------------------------------------------- -def _model(*, is_moe: bool) -> BaseModelConfig: +def _model(*, is_moe: bool, num_layers: int = 4) -> BaseModelConfig: model = BaseModelConfig( - num_layers=4, + num_layers=num_layers, num_q_heads=4, num_kv_heads=2, embedding_dim=256, @@ -381,7 +381,7 @@ def _model(*, is_moe: bool) -> BaseModelConfig: num_experts_per_tok=2 if is_moe else 0, torch_dtype="bfloat16", ) - model._model_name = f"w4_dp_{'moe' if is_moe else 'dense'}" + model._model_name = f"w4_dp_{'moe' if is_moe else 'dense'}_{num_layers}l" return model @@ -393,13 +393,14 @@ def _policy_scheduler( moe_ep: int = 2, num_replicas: int = 1, num_pipeline_stages: int = 1, + num_layers: int = 4, cluster_type: ClusterType = ClusterType.MONOLITHIC, replica_scheduler_config=None, cluster_scheduler_config=None, ): """Build a real cluster scheduler through the real constructor path.""" - model = _model(is_moe=is_moe) + model = _model(is_moe=is_moe, num_layers=num_layers) original = BaseModelConfig.create_from_name patch.setattr( BaseModelConfig, @@ -535,7 +536,6 @@ def on_batch_end(self, *args, **kwargs): ("label", "kwargs", "message"), [ ("two_replicas", dict(num_replicas=2), "one co-location Replica"), - ("pipeline_parallel", dict(num_pipeline_stages=2), "one co-location Replica"), ( "wrong_replica_scheduler", dict(replica_scheduler_config=SarathiSchedulerConfig( @@ -549,6 +549,16 @@ def on_batch_end(self, *args, **kwargs): dict(is_moe=False, attn_dp=2), "monotonic per Replica", ), + ( + "dense_multi_lane_pipeline_parallel", + dict(is_moe=False, attn_dp=2, num_pipeline_stages=2), + "monotonic per Replica", + ), + ( + "uneven_layer_partition", + dict(num_pipeline_stages=3), + "evenly divisible", + ), ], ) def test_each_unsupported_topology_is_rejected_at_construction( @@ -560,13 +570,29 @@ def test_each_unsupported_topology_is_rejected_at_construction( @pytest.mark.parametrize( - ("is_moe", "attn_dp", "moe_ep"), - [(True, 2, 2), (True, 1, 1), (False, 1, 1)], + ("is_moe", "attn_dp", "moe_ep", "num_pipeline_stages", "num_layers"), + [ + (True, 2, 2, 1, 4), + (True, 1, 1, 1, 4), + (False, 1, 1, 1, 4), + (True, 2, 2, 2, 4), + (True, 1, 1, 2, 4), + (False, 1, 1, 2, 4), + (True, 2, 2, 3, 6), + (True, 1, 1, 3, 6), + ], ) -def test_the_supported_shapes_construct(is_moe, attn_dp, moe_ep) -> None: +def test_the_supported_shapes_construct( + is_moe, attn_dp, moe_ep, num_pipeline_stages, num_layers +) -> None: with pytest.MonkeyPatch.context() as patch: scheduler = _policy_scheduler( - patch, is_moe=is_moe, attn_dp=attn_dp, moe_ep=moe_ep + patch, + is_moe=is_moe, + attn_dp=attn_dp, + moe_ep=moe_ep, + num_pipeline_stages=num_pipeline_stages, + num_layers=num_layers, ) assert scheduler._load_balancer is not None @@ -605,12 +631,286 @@ def test_routing_places_every_queued_request_on_the_serving_replica() -> None: assert scheduler._request_queue == [] -def test_an_unknown_lane_identity_is_rejected_at_the_report_boundary() -> None: +@pytest.mark.parametrize( + "seam", ["on_replica_batch_scheduled", "on_replica_batch_end"] +) +def test_an_unknown_lane_identity_is_rejected_at_the_report_boundary(seam) -> None: with pytest.MonkeyPatch.context() as patch: scheduler = _policy_scheduler(patch) with pytest.raises(ValueError, match="exact lane index"): - scheduler.on_replica_batch_end(0.0, scheduler._serving_replica_id, None, None) + getattr(scheduler, seam)(0.0, scheduler._serving_replica_id, None, None) + + +# -------------------------------------------------------------------------- +# Report timing and keys under pipeline parallelism +# +# Expected reports are derived from the reference engine iteration +# (`core.py`, `step_with_batch_queue`): an iteration that schedules while its +# batch queue still has room publishes at once, one that fills the queue +# publishes after applying its oldest output, and the key is the index of the +# forward the iteration launches, shared by peer engines. The lane readings +# are scripted; the stage-0 forward groups go through the real context. +# -------------------------------------------------------------------------- + + +class _ScriptedLane: + """The two lane readings the policy takes, set by the test.""" + + def __init__(self, waiting: int): + self.num_running_batches = 0 + self.waiting = waiting + self.running = 0 + + def get_request_load(self) -> RequestLoad: + return RequestLoad(self.waiting, self.running) + + +class _ScriptedReplica: + """Drive one policy scheduler as its lanes and stage 0 would.""" + + def __init__(self, patch, *, waiting: list[int], **shape): + self.scheduler = _policy_scheduler(patch, attn_dp=len(waiting), **shape) + self.replica_id = self.scheduler._serving_replica_id + self.lanes = [_ScriptedLane(count) for count in waiting] + for lane_id, lane in enumerate(self.lanes): + self.scheduler._replica_schedulers[(self.replica_id, lane_id)] = lane + self.stage0 = self.scheduler.get_stage_execution_context(self.replica_id, 0) + self._stage0_owners = [] + self._operations = 0 + self.reports: list[tuple[int, int, RequestLoad]] = [] + balancer_report = self.scheduler._load_balancer.report + + def record(time, engine, step, load): + self.reports.append((engine, step, load)) + balancer_report(time, engine, step, load) + + patch.setattr(self.scheduler._load_balancer, "report", record) + + def admit(self, lane_id: int, time: float = 0.0) -> None: + """Admit one single-request batch, as the lane's admission loop does.""" + + lane = self.lanes[lane_id] + lane.num_running_batches += 1 + lane.waiting -= 1 + lane.running += 1 + self.scheduler.on_replica_batch_scheduled(time, self.replica_id, lane_id, None) + + def complete(self, lane_id: int, time: float, *, finished: int = 0) -> None: + """Finish one batch on the last stage, as `GlobalBatchEndEvent` does.""" + + lane = self.lanes[lane_id] + lane.num_running_batches -= 1 + lane.running -= finished + self.scheduler.on_replica_batch_end(time, self.replica_id, lane_id, None) + + def run_stage0_forward(self, num_lanes: int) -> int: + """Start one shared stage-0 forward, run it to its end, return its group.""" + + for _ in range(num_lanes): + ticket = self.stage0.enqueue_full_stage(operation_id=self._operations) + self._operations += 1 + assert self.stage0.try_acquire(ticket) + group = self.stage0.bind_forward_group(ticket) + self._stage0_owners.append(ticket) + for ticket in self._stage0_owners: + self.stage0.release(ticket) + self._stage0_owners.clear() + return group + + +def test_pp1_admissions_are_reported_only_with_the_completion_they_join() -> None: + with pytest.MonkeyPatch.context() as patch: + replica = _ScriptedReplica(patch, waiting=[3, 3]) + for iteration in range(3): + for lane_id in (0, 1): + replica.admit(lane_id, time=0.1 * iteration) + # The single pipeline slot is full after every admission. + assert len(replica.reports) == 2 * iteration + assert replica.run_stage0_forward(2) == iteration + for lane_id in (0, 1): + replica.complete( + lane_id, 0.1 * iteration + 0.05, finished=min(iteration, 1) + ) + + # One report per lane per iteration, each keyed by the forward that + # completed, as the reference's atomic PP=1 step publishes. + assert replica.reports == [ + (0, 0, RequestLoad(2, 1)), + (1, 0, RequestLoad(2, 1)), + (0, 1, RequestLoad(1, 1)), + (1, 1, RequestLoad(1, 1)), + (0, 2, RequestLoad(0, 1)), + (1, 2, RequestLoad(0, 1)), + ] + + +@pytest.mark.parametrize("lane_order", [(0, 1), (1, 0)]) +def test_a_pp2_cold_fill_keys_peer_lanes_by_the_forward_they_share( + lane_order, +) -> None: + with pytest.MonkeyPatch.context() as patch: + replica = _ScriptedReplica(patch, waiting=[3, 3], num_pipeline_stages=2) + for lane_id in lane_order: + replica.admit(lane_id) + replica.admit(lane_id) + # The first admission leaves a slot and is published; the second + # fills the pipeline and waits for the first completion. + assert replica.reports == [ + (lane_id, 0, RequestLoad(2, 1)) for lane_id in lane_order + ] + assert replica.run_stage0_forward(2) == 0 + assert replica.run_stage0_forward(2) == 1 + # Inside the first collection wait, so both lanes' new counts are + # still unpublished when forward 1 is reported. + for lane_id in lane_order: + replica.complete(lane_id, 0.02) + + assert replica.reports[2:] == [ + (lane_id, 1, RequestLoad(1, 2)) for lane_id in lane_order + ] + # The first report of forward 1 latched forward 0 for both lanes; the + # peer's report of the same forward latched nothing partial. + assert replica.scheduler._load_balancer.last_step_counts == [ + RequestLoad(2, 1), + RequestLoad(2, 1), + ] + + +@pytest.mark.parametrize("lane_order", [(0, 1), (1, 0)]) +def test_pp3_publishes_two_admission_only_iterations_before_any_completion( + lane_order, +) -> None: + with pytest.MonkeyPatch.context() as patch: + replica = _ScriptedReplica( + patch, waiting=[4, 4], num_pipeline_stages=3, num_layers=6 + ) + for lane_id in lane_order: + for _ in range(3): + replica.admit(lane_id) + assert replica.reports == [ + report + for lane_id in lane_order + for report in ( + (lane_id, 0, RequestLoad(3, 1)), + (lane_id, 1, RequestLoad(2, 2)), + ) + ] + assert [replica.run_stage0_forward(2) for _ in range(3)] == [0, 1, 2] + for lane_id in lane_order: + replica.complete(lane_id, 0.3) + + # The third admission filled the pipeline in forward 2. + assert replica.reports[4:] == [ + (lane_id, 2, RequestLoad(1, 3)) for lane_id in lane_order + ] + + +@pytest.mark.parametrize("is_moe", [True, False]) +def test_a_full_pipeline_makes_one_report_per_iteration(is_moe) -> None: + with pytest.MonkeyPatch.context() as patch: + replica = _ScriptedReplica( + patch, waiting=[3], is_moe=is_moe, moe_ep=1, num_pipeline_stages=2 + ) + + def forward(group: int) -> None: + # A dense Replica binds no forward group; its key is the lane's + # admission count. + if is_moe: + assert replica.run_stage0_forward(1) == group + + replica.admit(0) + replica.admit(0) + forward(0) + forward(1) + replica.complete(0, 0.2, finished=1) + replica.admit(0, 0.2) + forward(2) + replica.complete(0, 0.3, finished=1) + # Nothing left to admit: the next completion is the lane's next + # iteration on its own. + replica.complete(0, 0.4, finished=1) + + assert replica.reports == [ + (0, 0, RequestLoad(2, 1)), + (0, 1, RequestLoad(1, 1)), + (0, 2, RequestLoad(0, 1)), + (0, 3, RequestLoad(0, 0)), + ] + assert replica.scheduler._held_key == [None] + + +def test_a_completion_and_the_admission_it_makes_room_for_share_one_key() -> None: + with pytest.MonkeyPatch.context() as patch: + replica = _ScriptedReplica( + patch, waiting=[1], moe_ep=1, num_pipeline_stages=2 + ) + replica.admit(0) + assert replica.run_stage0_forward(1) == 0 + replica.lanes[0].waiting += 1 + replica.complete(0, 0.2) + replica.admit(0, 0.2) + balancer = replica.scheduler._load_balancer + + # The reference publishes this iteration once, after scheduling the new + # request and applying the ready output. Two reports under one key leave + # the coordinator the same state: nothing latched in between. + assert replica.reports == [ + (0, 0, RequestLoad(0, 1)), + (0, 1, RequestLoad(1, 1)), + (0, 1, RequestLoad(0, 2)), + ] + assert balancer.last_step_counts is None + assert balancer.engine_counts == [RequestLoad(0, 2)] + + +def test_a_lane_that_drains_to_zero_reports_it_and_new_work_keys_coherently() -> None: + with pytest.MonkeyPatch.context() as patch: + replica = _ScriptedReplica(patch, waiting=[1, 0], num_pipeline_stages=2) + replica.admit(0) + assert replica.run_stage0_forward(1) == 0 + replica.complete(0, 0.2, finished=1) + balancer = replica.scheduler._load_balancer + assert balancer.engine_counts == [RequestLoad(0, 0), RequestLoad(0, 0)] + + # New work on both lanes after a quiet interval joins one forward. + for lane_id in (1, 0): + replica.lanes[lane_id].waiting += 1 + replica.admit(lane_id, 10.0) + assert replica.run_stage0_forward(2) == 1 + + assert replica.reports == [ + (0, 0, RequestLoad(0, 1)), + (0, 1, RequestLoad(0, 0)), + (1, 1, RequestLoad(0, 1)), + (0, 1, RequestLoad(0, 1)), + ] + + +def test_the_admission_loop_reports_the_state_after_each_admission() -> None: + with pytest.MonkeyPatch.context() as patch: + scheduler = _policy_scheduler(patch, num_pipeline_stages=2) + reports = [] + balancer_report = scheduler._load_balancer.report + patch.setattr( + scheduler._load_balancer, + "report", + lambda time, engine, step, load: ( + reports.append((engine, step, load)), + balancer_report(time, engine, step, load), + ), + ) + lane = scheduler.get_replica_scheduler(scheduler._serving_replica_id, 0) + # Each request fills the 16-token budget, so each batch holds one. + for _ in range(3): + lane.add_request(_request(tokens=16)) + + batches = lane.on_schedule(0.0) + + assert len(batches) == 2 + # One call admitted two batches; only the first left a slot, and it is + # reported with the state after it alone. + assert reports == [(0, 0, RequestLoad(2, 1))] # -------------------------------------------------------------------------- @@ -676,7 +976,10 @@ def test_every_existing_policy_routes_identically_through_schedule_at( ] -def test_the_default_batch_end_seam_is_inert() -> None: +@pytest.mark.parametrize( + "seam", ["on_replica_batch_scheduled", "on_replica_batch_end"] +) +def test_the_default_batch_seams_are_inert(seam) -> None: class _Bare(BaseClusterScheduler): def schedule(self): return [] @@ -684,7 +987,7 @@ def schedule(self): bare = _Bare.__new__(_Bare) before = dict(vars(bare)) - assert bare.on_replica_batch_end(1.0, 0, 0, None) is None + assert getattr(bare, seam)(1.0, 0, 0, None) is None assert dict(vars(bare)) == before From bacdbb423a0c8cffa0ee60397203f2364246a809 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 16:16:02 +0800 Subject: [PATCH 52/80] test(dp-placement): cover schedule-time reports in the real loop under PP Record every seam call, its report and each stage-0 forward start, and check in the parent that each report stands for one reference engine iteration: an admission that leaves a pipeline slot is reported under its own forward, one that fills the pipeline is reported with the next completion under that forward, and a completion with nothing new is keyed after the lane's last forward. Run it on dense PP2, MoE dp2 PP2 offline and online, and MoE dp1 and dp2 PP3 (six layers, analytical backend), with conservation and released ownership, and on the existing PP1 cases. Add the plan's discriminating case at PP2: a burst routed from reservations, then a probe after the admission publish and before any completion. The policy sends it to lane 0 from the published counts; the completion-reporting control, which differs only in its two seams, still holds its reservations and sends it to lane 1. Carry the hybrid-layer decode credit case into two stages under the policy. Its per-source prediction window now ends at the source's own next wave, since another stage's wave can come in between; at one stage this is the same window. --- .../test_monolithic_mixed_forward_runtime.py | 90 ++- .../test_vllm_dp_placement_runtime.py | 558 ++++++++++++++---- 2 files changed, 513 insertions(+), 135 deletions(-) diff --git a/tests/integration/test_monolithic_mixed_forward_runtime.py b/tests/integration/test_monolithic_mixed_forward_runtime.py index 56be25c9..74ede5aa 100644 --- a/tests/integration/test_monolithic_mixed_forward_runtime.py +++ b/tests/integration/test_monolithic_mixed_forward_runtime.py @@ -72,13 +72,34 @@ def test_a_mixed_batch_crossing_a_dense_layer_is_credited_every_layer(evidence): assert hybrid["layer_credit_peaks"] == {str(hybrid["num_layers"]): hybrid["decode_tokens_credited"]}, hybrid +def test_the_dense_layer_credit_holds_with_pipeline_stages_and_dp_placement(evidence): + """The same hybrid model split over two stages and routed by the vLLM DP + placement policy, whose schedule-time reports run inside admission. + + The dense layer now sits on stage 0 while two routed layers follow on + stage 1, so a mixed batch crosses the stage boundary between the credit + paths. The run's own assertions cover conservation and released + ownership; this checks that the shape was reached. + """ + + hybrid = evidence["hybrid_layers_pp2_dp_placement"] + assert hybrid["num_pipeline_stages"] == 2, hybrid + assert hybrid["dense_layers"] == [1], hybrid + assert hybrid["mixed_dense_completions"] > 0, hybrid + assert hybrid["layer_credit_peaks"] == {str(hybrid["num_layers"]): hybrid["decode_tokens_credited"]}, hybrid + + # Requests chosen so that chunked prefill leaves one lane prefilling while the # other has already started decoding: unequal prefill lengths, unequal decode # budgets, all arriving at once. REQUEST_SHAPES = ((32, 4), (16, 4), (24, 3), (16, 3)) +# With two stages, each lane keeps two batches in flight and skips requests +# already in one, so two requests per lane never share a batch. A third lets +# a decoding request join another request's prefill chunk. +PIPELINED_REQUEST_SHAPES = ((32, 4), (16, 4), (24, 3), (16, 3), (16, 3), (24, 4)) -def _build_config(root, patch, *, moe_layers_enum=None): +def _build_config(root, patch, *, moe_layers_enum=None, num_pipeline_stages=1): import pandas as pd from frontier.config import ( @@ -167,7 +188,7 @@ def _build_config(root, patch, *, moe_layers_enum=None): model_name=model._model_name, device="a100", network_device="a100_pairwise_nvlink", - num_pipeline_stages=1, + num_pipeline_stages=num_pipeline_stages, attn_tensor_parallel_size=1, attn_dp=2, moe_tensor_parallel_size=1, @@ -210,7 +231,15 @@ def _drained(room) -> bool: return True -def run_case(root: Path, *, reporting: bool, moe_layers_enum: str | None = None): +def run_case( + root: Path, + *, + reporting: bool, + moe_layers_enum: str | None = None, + num_pipeline_stages: int = 1, + cluster_scheduler_config=None, + request_shapes=REQUEST_SHAPES, +): from frontier.entities import Request from frontier.request_generator.synthetic_request_generator import ( SyntheticRequestGenerator, @@ -221,7 +250,10 @@ def run_case(root: Path, *, reporting: bool, moe_layers_enum: str | None = None) with pytest.MonkeyPatch.context() as patch: model, predictor_config, replica, classes = _build_config( - root, patch, moe_layers_enum=moe_layers_enum + root, + patch, + moe_layers_enum=moe_layers_enum, + num_pipeline_stages=num_pipeline_stages, ) ( VllmV1SchedulerConfig, @@ -242,6 +274,11 @@ def run_case(root: Path, *, reporting: bool, moe_layers_enum: str | None = None) enable_chunked_prefill=True, ), execution_time_predictor_config=predictor_config, + **( + {} + if cluster_scheduler_config is None + else dict(cluster_scheduler_config=cluster_scheduler_config) + ), ) config = SimulationConfig( simulation_mode="offline", @@ -263,7 +300,7 @@ def run_case(root: Path, *, reporting: bool, moe_layers_enum: str | None = None) write_json_trace=False, ), request_generator_config=SyntheticRequestGeneratorConfig( - num_requests=len(REQUEST_SHAPES), + num_requests=len(request_shapes), length_generator_config=FixedRequestLengthGeneratorConfig( prefill_tokens=16, decode_tokens=3 ), @@ -273,7 +310,7 @@ def run_case(root: Path, *, reporting: bool, moe_layers_enum: str | None = None) ), ) requests = [ - Request(0.0, prefill, decode) for prefill, decode in REQUEST_SHAPES + Request(0.0, prefill, decode) for prefill, decode in request_shapes ] patch.setattr( SyntheticRequestGenerator, "generate", lambda self: list(requests) @@ -379,6 +416,7 @@ def observe_predict(batch, stage_id, cluster_type=None, **kwargs): evidence = { "reporting": reporting, "num_layers": model.num_layers, + "num_pipeline_stages": num_pipeline_stages, "dense_layers": [ layer for layer in range(model.num_layers) if not model.is_moe_layer(layer) @@ -402,7 +440,7 @@ def observe_predict(batch, stage_id, cluster_type=None, **kwargs): # Every request finishes, exactly once, with every token accounted for. assert all(request.completed for request in requests), evidence - for request, (prefill, decode) in zip(requests, REQUEST_SHAPES): + for request, (prefill, decode) in zip(requests, request_shapes): assert request.num_prefill_tokens == prefill assert request.num_decode_tokens == decode assert request.num_processed_tokens == prefill + decode @@ -410,18 +448,27 @@ def observe_predict(batch, stage_id, cluster_type=None, **kwargs): assert len(rows) == len(requests) assert len({row["Request Id"] for row in rows}) == len(requests) assert sum(int(float(row["request_num_tokens"])) for row in rows) == sum( - prefill + decode for prefill, decode in REQUEST_SHAPES + prefill + decode for prefill, decode in request_shapes ) - # Each live source in a mixed cohort continued on its own prediction. - # A cohort's completion runs after its own wave and before the next - # wave is scheduled, so that slice of the prediction log belongs to it. - boundaries = [cohort["first_call"] for cohort in cohorts] + [len(predictions)] + # Each live source in a mixed cohort continued on its own prediction, + # made after the cohort's wave and before that source's next wave. + # With pipeline stages another stage's wave can come in between, so + # each source's window ends at its own next wave. for index, cohort in enumerate(cohorts): if len(set(cohort["members"].values())) < 2: continue - window = set(predictions[cohort["first_call"] : boundaries[index + 1]]) - assert set(cohort["members"]) <= window, (cohort, sorted(window)) + for member in cohort["members"]: + end = next( + ( + later["first_call"] + for later in cohorts[index + 1 :] + if member in later["members"] + ), + len(predictions), + ) + window = predictions[cohort["first_call"] : end] + assert member in window, (cohort, member, window) # Nothing is stranded: no waiting room holds a batch and no stage # execution context still owns or queues a ticket. @@ -462,10 +509,23 @@ def main(root: Path) -> None: hybrid_root.mkdir(parents=True, exist_ok=True) hybrid = run_case(hybrid_root, reporting=False, moe_layers_enum="0,2,3") assert hybrid["mixed_dense_completions"] > 0, hybrid + from frontier.config import VllmLoadBalancingClusterSchedulerConfig + + staged_root = root / "hybrid_layers_pp2_dp_placement" + staged_root.mkdir(parents=True, exist_ok=True) + staged = run_case( + staged_root, + reporting=False, + moe_layers_enum="0,2,3", + num_pipeline_stages=2, + cluster_scheduler_config=VllmLoadBalancingClusterSchedulerConfig(), + request_shapes=PIPELINED_REQUEST_SHAPES, + ) merged = dict(evidence["off"]) merged["reporting_variants"] = evidence merged["hybrid_layers"] = hybrid + merged["hybrid_layers_pp2_dp_placement"] = staged (root / "shared_forward_evidence.json").write_text( json.dumps(merged, indent=2) + "\n" ) @@ -473,6 +533,8 @@ def main(root: Path) -> None: print("completed_requests:", merged["completed_requests"]) print("hybrid mixed_dense_completions:", hybrid["mixed_dense_completions"]) print("hybrid layer_credit_peaks:", hybrid["layer_credit_peaks"]) + print("pp2 mixed_dense_completions:", staged["mixed_dense_completions"]) + print("pp2 layer_credit_peaks:", staged["layer_credit_peaks"]) if __name__ == "__main__": diff --git a/tests/integration/test_vllm_dp_placement_runtime.py b/tests/integration/test_vllm_dp_placement_runtime.py index 6825c21e..e626581e 100644 --- a/tests/integration/test_vllm_dp_placement_runtime.py +++ b/tests/integration/test_vllm_dp_placement_runtime.py @@ -3,18 +3,21 @@ The state machine itself is covered by `tests/unit/test_vllm_dp_load_balancer.py` against reference-derived expectations. What can only be shown by running the simulator is the wiring: that `ClusterScheduleEvent` is what supplies the -routing time, that `GlobalBatchEndEvent` reports the lane's **post**-step load, -that the report key is ordered the way the policy's guard assumes, and that the -policy introduces no event that keeps a drained run alive. - -The child process runs two configurations that differ only in the cluster -scheduler policy, so the comparison isolates the policy. Execution time comes +routing time, that each report stands for one reference engine iteration -- +an admission while the pipeline has room, or a completion -- with the lane's +post-step load and the key of the forward it describes, and that the policy +introduces no event that keeps a drained run alive. + +The child process runs each configuration twice, once with the policy and once +with a comparison run that differs only in the cluster scheduler policy or, for +the discriminating case, only in how the policy reports. Execution time comes from the dummy predictor: placement here is decided by the balancer, not by -latency realism, and both policies see the same durations. +latency realism, and both runs see the same durations. """ from __future__ import annotations +from collections import defaultdict import json import os from pathlib import Path @@ -46,6 +49,100 @@ def _run_child(tmp_path: Path, case: str) -> dict: return json.loads((tmp_path / f"{case}_evidence.json").read_text()) +def _assert_reports_follow_engine_iterations(run: dict, num_pipeline_stages: int) -> int: + """Check every report against the reference engine iteration it stands for. + + Reference `step_with_batch_queue`: an iteration that schedules a batch and + still has pipeline room publishes at once; one that fills the pipeline + publishes after applying its oldest output; one with nothing new to + schedule applies the oldest output by itself. The key is the forward the + iteration launches, which peer engines share. Returns the number of + admissions published on their own. + """ + + records = run["records"] + forward: dict[int, int] = {} + starts_by_lane: dict[int, int] = defaultdict(int) + for record in records: + if record["kind"] != "stage0": + continue + if record["group"] is None: + # A dense Replica shares no forward; a lane's forwards are its own + # stage-0 starts. + forward[record["batch"]] = starts_by_lane[record["lane"]] + starts_by_lane[record["lane"]] += 1 + else: + forward[record["batch"]] = record["group"] + + held: dict[int, int] = {} + last_forward: dict[int, int] = defaultdict(lambda: -1) + last_key: dict[int, int] = defaultdict(lambda: -1) + admission_only = 0 + for record in records: + if record["kind"] == "stage0": + continue + lane = record["lane"] + if record["kind"] == "scheduled": + last_forward[lane] = forward[record["batch"]] + if record["running_after"] >= num_pipeline_stages: + assert record["report"] is None, record + held[lane] = record["batch"] + continue + assert record["report"] is not None, record + admission_only += 1 + expected = forward[record["batch"]] + else: + assert record["report"] is not None, record + expected = forward[held.pop(lane)] if lane in held else None + key = record["report"][0] + if expected is None: + # An iteration after the lane's last forward, with nothing new. + assert key > last_forward[lane], (record, last_forward[lane]) + else: + assert key == expected, (record, expected) + assert key >= last_key[lane], (record, last_key[lane]) + last_key[lane] = key + assert not held, held + return admission_only + + +def _assert_completions_report_post_step_load(run: dict) -> None: + """The lane's own release is bracketed, so pre- and post-step load differ + for some batches; every completion report carries the post-step one.""" + + completions = [record for record in run["records"] if record["kind"] == "end"] + assert completions + assert all(record["released"] for record in completions) + assert any(record["pre_step"] != record["post_step"] for record in completions) + assert all(record["report"][1] == record["post_step"] for record in completions) + assert not all(record["report"][1] == record["pre_step"] for record in completions) + + +def _assert_pp1_keys_are_ordered_forwards(run: dict) -> None: + """At PP=1 the key is ordered as the capability guard assumes: never + decreasing, and equal only for peer lanes of one shared forward.""" + + reports = [ + record + for record in run["records"] + if record["kind"] != "stage0" and record["report"] is not None + ] + keys = [record["report"][0] for record in reports] + assert keys == sorted(keys) + lanes_by_key: dict[int, list[int]] = defaultdict(list) + for record in reports: + lanes_by_key[record["report"][0]].append(record["lane"]) + for key, lanes in lanes_by_key.items(): + assert len(lanes) == len(set(lanes)), (key, lanes) + + +def _assert_run_conserves_work(run: dict) -> None: + assert run["completed_requests"] == run["num_requests"] > 0 + assert run["tokens_conserved"] + assert run["lanes_released"] + assert run["stage_contexts_released"] + + def test_dp_placement_runs_and_reports_post_step_load(tmp_path): evidence = _run_child(tmp_path, "moe_dp2") policy = evidence["vllm_load_balancing"] @@ -53,7 +150,7 @@ def test_dp_placement_runs_and_reports_post_step_load(tmp_path): # The run has to reach the shape under test. assert policy["num_lanes"] == 2 - assert policy["completed_requests"] == policy["num_requests"] + _assert_run_conserves_work(policy) # 1. The routing time comes from ClusterScheduleEvent, not retained state. # Every recorded routing time is one of the cluster schedule times, and @@ -66,35 +163,27 @@ def test_dp_placement_runs_and_reports_post_step_load(tmp_path): # the local reservation moves the choice. assert policy["first_four_lanes"] == [0, 1, 0, 1] - # 3. The report key is ordered as the capability guard assumes: never - # decreasing, and equal only for peer lanes of one shared forward. - keys = policy["report_keys"] - assert keys == sorted(keys) - for key, lanes in policy["lanes_by_key"].items(): - assert len(lanes) == len(set(lanes)), (key, lanes) + # 3. At PP=1 no admission leaves a pipeline slot, so every report is a + # completion keyed by the forward that completed. + assert _assert_reports_follow_engine_iterations(policy, 1) == 0 + _assert_pp1_keys_are_ordered_forwards(policy) - # 4. The reported load is the post-step state. The lane's own release is - # bracketed, so pre- and post-step load are distinct values for at least - # some batches; every report matches the post-step one and the reports are - # not merely the pre-step values. - assert policy["reports_after_the_lane_released_the_batch"] == policy["num_reports"] - assert policy["reports_where_the_release_changed_the_load"] > 0 - assert policy["reports_matching_post_step"] == policy["num_reports"] - assert policy["reports_matching_pre_step"] < policy["num_reports"] + # 4. The reported load is the post-step state. + _assert_completions_report_post_step_load(policy) # 5. No event type is introduced, and the run drains rather than being kept # alive by a heartbeat. assert set(policy["event_types"]) == set(baseline["event_types"]) assert policy["makespan"] > 0 - assert baseline["completed_requests"] == baseline["num_requests"] + _assert_run_conserves_work(baseline) def test_placement_follows_published_load_where_round_robin_cannot(tmp_path): """Spread the arrivals so snapshots land between them, and skew the load. - This is the discriminating case: the two policies see identical arrivals, - identical durations and identical lane capacity, so any difference in - placement comes from reading the published load. + This is the discriminating case at PP=1: the two policies see identical + arrivals, identical durations and identical lane capacity, so any + difference in placement comes from reading the published load. """ evidence = _run_child(tmp_path, "moe_dp2_online") @@ -120,17 +209,12 @@ def test_placement_follows_published_load_where_round_robin_cannot(tmp_path): # 3. The report key stays ordered across a much longer run, and peer lanes of # one shared forward remain the only source of equal keys. - keys = policy["report_keys"] - assert keys == sorted(keys) - assert len(keys) > 10 - for key, lanes in policy["lanes_by_key"].items(): - assert len(lanes) == len(set(lanes)), (key, lanes) + assert _assert_reports_follow_engine_iterations(policy, 1) == 0 + _assert_pp1_keys_are_ordered_forwards(policy) + assert sum(1 for record in policy["records"] if record["kind"] == "end") > 10 # 4. Still the post-step load, and still no new event type. - assert policy["reports_after_the_lane_released_the_batch"] == policy["num_reports"] - assert policy["reports_where_the_release_changed_the_load"] > 0 - assert policy["reports_matching_post_step"] == policy["num_reports"] - assert policy["reports_matching_pre_step"] < policy["num_reports"] + _assert_completions_report_post_step_load(policy) assert set(policy["event_types"]) == set(baseline["event_types"]) @@ -139,8 +223,99 @@ def test_a_single_lane_shape_routes_everything_to_lane_zero(tmp_path): policy = evidence["vllm_load_balancing"] assert policy["num_lanes"] == 1 - assert policy["completed_requests"] == policy["num_requests"] + _assert_run_conserves_work(policy) assert set(policy["first_four_lanes"]) == {0} + assert _assert_reports_follow_engine_iterations(policy, 1) == 0 + + +@pytest.mark.parametrize( + ("case", "num_pipeline_stages"), + [ + ("dense_dp1_pp2", 2), + ("moe_dp2_pp2", 2), + ("moe_dp2_pp2_online", 2), + ("moe_dp1_pp3", 3), + ("moe_dp2_pp3", 3), + ], +) +def test_pipeline_parallel_shapes_report_once_per_engine_iteration( + tmp_path, case, num_pipeline_stages +): + evidence = _run_child(tmp_path, case) + policy = evidence["vllm_load_balancing"] + baseline = evidence["round_robin"] + + # 1. Every request completes, every token is processed once, and no lane + # or stage context still owns work. + _assert_run_conserves_work(policy) + _assert_run_conserves_work(baseline) + + # 2. The routing time is still the cluster schedule time. + assert policy["routing_times"] == policy["cluster_schedule_times"] + + # 3. Each report stands for one engine iteration under its forward's key, + # and every lane's cold fill publishes an admission before anything + # completes. + admission_only = _assert_reports_follow_engine_iterations( + policy, num_pipeline_stages + ) + assert admission_only >= policy["num_lanes"] + + # 4. Completions still carry the post-step load, and the new seam adds no + # event type. + _assert_completions_report_post_step_load(policy) + assert set(policy["event_types"]) == set(baseline["event_types"]) + + +def test_schedule_time_reports_decide_a_probe_that_completion_reports_cannot( + tmp_path, +): + """Plan §18.6 under PP=2, against the completion-reporting control. + + The control differs from the policy only in its two seams, so the probe's + placement differs only because of what was published before it arrived. + """ + + evidence = _run_child(tmp_path, "moe_dp2_pp2_discriminating") + fixed = evidence["vllm_load_balancing"] + control = evidence["completion_reporting_control"] + + for run in (fixed, control): + _assert_run_conserves_work(run) + # Premise: the burst is routed from reservations alone, after the + # first collection publish of empty counts ... + assert run["selections"][0]["snapshot"] == [[0, 0], [0, 0]] + assert run["placements"][:5] == [0, 1, 0, 1, 0] + # ... and nothing completes before the probe arrives. + first_completion = min( + record["time"] for record in run["records"] if record["kind"] == "end" + ) + assert first_completion > 1.1 + + probe = {name: run["selections"][5] for name, run in evidence.items()} + assert probe["vllm_load_balancing"]["time"] == pytest.approx(1.1) + assert probe["completion_reporting_control"]["time"] == pytest.approx(1.1) + + # The policy published each lane's first admission: lane 0 runs its three + # short requests (score 3); lane 1 runs one chunk of the long prompt while + # the short request waits (score 4 + 1). The probe goes to lane 0. + first_admissions = {} + for record in fixed["records"]: + if record["kind"] == "scheduled" and record["report"] is not None: + first_admissions.setdefault(record["lane"], record["report"][1]) + assert first_admissions == {0: [0, 3], 1: [1, 1]} + assert probe["vllm_load_balancing"]["snapshot"] == [[0, 3], [1, 1]] + assert probe["vllm_load_balancing"]["engine"] == 0 + + # The control reported nothing yet, so the frontend still holds its own + # reservations (score 12 against 8) and sends the probe to lane 1. + assert all( + record["report"] is None + for record in control["records"] + if record["kind"] == "scheduled" + ) + assert probe["completion_reporting_control"]["snapshot"] == [[3, 0], [2, 0]] + assert probe["completion_reporting_control"]["engine"] == 1 # --------------------------------------------------------------------------- @@ -150,12 +325,12 @@ def test_a_single_lane_shape_routes_everything_to_lane_zero(tmp_path): REQUEST_SHAPES = 4 -def _model(*, is_moe: bool): +def _model(*, is_moe: bool, num_layers: int): from frontier.config import BaseModelConfig from frontier.types import ActivationType, NormType model = BaseModelConfig( - num_layers=4, + num_layers=num_layers, num_q_heads=4, num_kv_heads=2, embedding_dim=256, @@ -173,7 +348,7 @@ def _model(*, is_moe: bool): num_experts_per_tok=2 if is_moe else 0, torch_dtype="bfloat16", ) - model._model_name = f"w4_runtime_{'moe' if is_moe else 'dense'}" + model._model_name = f"w4_runtime_{'moe' if is_moe else 'dense'}_{num_layers}l" return model @@ -186,7 +361,12 @@ def _config( moe_ep: int, policy, trace: str | None = None, + num_pipeline_stages: int = 1, + num_layers: int = 4, + analytical_backend: bool = False, + dummy_execution_time_ms: float | None = None, ): + from frontier.cc_backend.cc_backend_config import AnalyticalCCBackendConfig from frontier.config import ( BaseModelConfig, ClusterConfig, @@ -201,7 +381,7 @@ def _config( VllmV1SchedulerConfig, ) - model = _model(is_moe=is_moe) + model = _model(is_moe=is_moe, num_layers=num_layers) original = BaseModelConfig.create_from_name patch.setattr( BaseModelConfig, @@ -224,12 +404,28 @@ def _config( model_name=model._model_name, device="a100", network_device="a100_pairwise_nvlink", - num_pipeline_stages=1, + num_pipeline_stages=num_pipeline_stages, attn_tensor_parallel_size=1, attn_dp=attn_dp, memory_margin_fraction=0.1, **moe_fields, ) + # Placement here is decided by the balancer, not by latency realism, so the + # predictor only has to be deterministic and identical across the policies + # being compared. + predictor = RandomForrestExecutionTimePredictorConfig( + enable_dummy_mode=True, + **( + {} + if dummy_execution_time_ms is None + else dict(dummy_execution_time_ms=dummy_execution_time_ms) + ), + ) + backend = ( + dict(cc_backend_config=AnalyticalCCBackendConfig()) + if analytical_backend + else {} + ) cluster = ClusterConfig( replica_config=replica, replica_scheduler_config=VllmV1SchedulerConfig( @@ -240,12 +436,8 @@ def _config( enable_chunked_prefill=True, ), cluster_scheduler_config=policy(), - # Placement here is decided by the balancer, not by latency realism, so - # the predictor only has to be deterministic and identical across the - # two policies being compared. - execution_time_predictor_config=RandomForrestExecutionTimePredictorConfig( - enable_dummy_mode=True - ), + execution_time_predictor_config=predictor, + **backend, ) generator = ( TraceRequestGeneratorConfig(trace_file=trace) @@ -284,13 +476,17 @@ def _config( def run_case( root: Path, *, - is_moe: bool, - attn_dp: int, - moe_ep: int, policy_name: str, - trace: str | None = None, + completion_reporting_control: bool = False, + **shape, ): - """Run one configuration and return what only the event loop can show.""" + """Run one configuration and return what only the event loop can show. + + `completion_reporting_control` swaps the policy's two seams for the + reporting it had before schedule-time reports: nothing at admission, and + each completion keyed by `ForwardSyncState.get_step_id`. It exists only + here, as the control the discriminating case is measured against. + """ from frontier.config import ( RoundRobinClusterSchedulerConfig, @@ -307,8 +503,13 @@ def run_case( from frontier.scheduler.replica_scheduler.vllm_v1_engine_replica_scheduler import ( # noqa: E501 VLLMv1EngineReplicaScheduler, ) + from frontier.scheduler.replica_stage_scheduler.replica_stage_schduler import ( + ReplicaStageScheduler, + ) from frontier.scheduler.utils.forward_sync_state import ForwardSyncState + from frontier.scheduler.utils.vllm_dp_load_balancer import VllmDPLoadBalancer from frontier.simulator import Simulator + from frontier.types import ClusterType policy = { "vllm_load_balancing": VllmLoadBalancingClusterSchedulerConfig, @@ -318,12 +519,18 @@ def run_case( cluster_schedule_times: list[float] = [] routing_times: list[float] = [] placements: list[int] = [] - reports: list[dict] = [] releases: dict[int, dict] = {} event_types: set[str] = set() + # Seam calls, their reports, and stage-0 forward starts, in event order. + records: list[dict] = [] + reports: list[list] = [] + selections: list[dict] = [] original_cluster_schedule = ClusterScheduleEvent.handle_event original_batch_end = GlobalBatchEndEvent.handle_event + original_report = VllmDPLoadBalancer.report + original_select = VllmDPLoadBalancer.select + original_stage_pop = ReplicaStageScheduler.pop_batch_if_not_busy def observed_cluster_schedule(self, scheduler, metrics_store): cluster_schedule_times.append(float(self.time)) @@ -354,43 +561,77 @@ def observed(self, batch): return observed - def observing_hook(original): + def observed_report(self, time, engine, step, load): + reports.append([step, list(load)]) + return original_report(self, time, engine, step, load) + + def observed_select(self, time): + # Applying the passed deadlines first changes nothing: `select` does + # the same before choosing. It exposes the snapshot the choice reads. + self._advance(time) + snapshot = [list(load) for load in self.frontend_counts] + engine = original_select(self, time) + selections.append({"time": float(time), "snapshot": snapshot, "engine": engine}) + return engine + + def observed_stage_pop(self): + batch = original_stage_pop(self) + if batch is not None and self._stage_id == 0: + records.append( + { + "kind": "stage0", + "lane": self._replica_local_id, + "batch": batch.id, + # A MoE lane batch carries the shared forward group bound + # at this start; a dense stage binds none. + "group": batch._forward_cohort_provisional_id + if self._is_moe + else None, + } + ) + return batch + + def observing_seam(kind, original): def observed(self, time, replica_id, replica_local_id, batch): + before = len(reports) result = original(self, time, replica_id, replica_local_id, batch) - balancer = getattr(self, "_load_balancer", None) - if balancer is not None and type(replica_local_id) is int: - # `None` means the lane had not yet released this batch when the - # hook ran, which is itself the ordering evidence. - release = releases.get(batch.id) - reports.append( - { - "lane": replica_local_id, - "key": ForwardSyncState.get_step_id(batch), - "released": release is not None, - "pre_step": release["before"] if release else None, - "post_step": release["after"] if release else None, - "reported": list(balancer.engine_counts[replica_local_id]), - } - ) + made = reports[before:] + assert len(made) <= 1, made + lane = self.get_replica_scheduler(replica_id, replica_local_id) + release = releases.get(batch.id) + records.append( + { + "kind": kind, + "time": float(time), + "lane": replica_local_id, + "batch": batch.id, + "running_after": lane.num_running_batches, + "report": made[0] if made else None, + "released": release is not None, + "pre_step": release["before"] if release else None, + "post_step": release["after"] if release else None, + } + ) return result return observed + def completion_reporting(self, time, replica_id, replica_local_id, batch): + lane = self.get_replica_scheduler(replica_id, replica_local_id) + self._load_balancer.report( + time, + replica_local_id, + ForwardSyncState.get_step_id(batch), + lane.get_request_load(), + ) + def observed_batch_end(self, scheduler, metrics_store): events = original_batch_end(self, scheduler, metrics_store) event_types.update(type(event).__name__ for event in events or []) return events with pytest.MonkeyPatch.context() as patch: - config = _config( - root, - patch, - is_moe=is_moe, - attn_dp=attn_dp, - moe_ep=moe_ep, - policy=policy, - trace=trace, - ) + config = _config(root, patch, policy=policy, **shape) patch.setattr(ClusterScheduleEvent, "handle_event", observed_cluster_schedule) patch.setattr(GlobalBatchEndEvent, "handle_event", observed_batch_end) patch.setattr( @@ -398,9 +639,12 @@ def observed_batch_end(self, scheduler, metrics_store): "on_batch_end", observing_release(vars(VLLMv1EngineReplicaScheduler)["on_batch_end"]), ) + patch.setattr(VllmDPLoadBalancer, "report", observed_report) + patch.setattr(VllmDPLoadBalancer, "select", observed_select) + patch.setattr(ReplicaStageScheduler, "pop_batch_if_not_busy", observed_stage_pop) # Both the inert base seam and the policy's override have to be - # wrapped: patching only the base would silently observe nothing on the - # very policy under test. + # wrapped for routing: patching only the base would silently observe + # nothing on the very policy under test. for owner in (BaseClusterScheduler, VllmLoadBalancingClusterScheduler): if "schedule_at" in vars(owner): patch.setattr( @@ -408,44 +652,66 @@ def observed_batch_end(self, scheduler, metrics_store): "schedule_at", observing_schedule_at(vars(owner)["schedule_at"]), ) - if "on_replica_batch_end" in vars(owner): - patch.setattr( - owner, - "on_replica_batch_end", - observing_hook(vars(owner)["on_replica_batch_end"]), - ) + seams = { + "on_replica_batch_scheduled": vars(VllmLoadBalancingClusterScheduler)[ + "on_replica_batch_scheduled" + ], + "on_replica_batch_end": vars(VllmLoadBalancingClusterScheduler)[ + "on_replica_batch_end" + ], + } + if completion_reporting_control: + seams = { + "on_replica_batch_scheduled": vars(BaseClusterScheduler)[ + "on_replica_batch_scheduled" + ], + "on_replica_batch_end": completion_reporting, + } + for name, kind in ( + ("on_replica_batch_scheduled", "scheduled"), + ("on_replica_batch_end", "end"), + ): + patch.setattr( + VllmLoadBalancingClusterScheduler, + name, + observing_seam(kind, seams[name]), + ) simulator = Simulator(config) simulator.run() requests = list(simulator._all_requests) - - lanes_by_key: dict[str, list[int]] = {} - for report in reports: - lanes_by_key.setdefault(str(report["key"]), []).append(report["lane"]) + cluster_scheduler = simulator._global_scheduler.get_cluster_scheduler( + ClusterType.MONOLITHIC + ) + lanes = [ + cluster_scheduler.get_replica_scheduler(replica_id, lane_id) + for replica_id in cluster_scheduler._cluster.replicas + for lane_id in range(shape["attn_dp"]) + ] + contexts = list(cluster_scheduler._stage_execution_contexts.values()) return { - "num_lanes": attn_dp, + "num_lanes": shape["attn_dp"], "num_requests": len(requests), "completed_requests": sum(1 for request in requests if request.completed), + "tokens_conserved": all( + request.num_processed_tokens + == request.num_prefill_tokens + request.num_decode_tokens + for request in requests + ), + "lanes_released": all( + lane.num_running_batches == 0 and not lane._running_requests + for lane in lanes + ), + "stage_contexts_released": all( + context.is_idle and context.queued_tickets == () for context in contexts + ), "makespan": max((request.completed_at for request in requests), default=0.0), "cluster_schedule_times": cluster_schedule_times, "routing_times": routing_times, "first_four_lanes": placements[:4], "placements": placements, - "report_keys": [report["key"] for report in reports], - "lanes_by_key": lanes_by_key, - "num_reports": len(reports), - "reports_after_the_lane_released_the_batch": sum( - 1 for report in reports if report["released"] - ), - "reports_matching_post_step": sum( - 1 for report in reports if report["post_step"] == report["reported"] - ), - "reports_matching_pre_step": sum( - 1 for report in reports if report["pre_step"] == report["reported"] - ), - "reports_where_the_release_changed_the_load": sum( - 1 for report in reports if report["pre_step"] != report["post_step"] - ), + "records": records, + "selections": selections, "event_types": sorted(event_types), } @@ -462,30 +728,80 @@ def observed_batch_end(self, scheduler, metrics_store): 1.0,16,1 """ +# Plan §18.6. A burst after the first collection publish, so the frontend +# routes it from local reservations: lanes 0, 1, 0, 1, 0. Lane 0's three short +# prompts fit one batch; lane 1's long prompt takes the whole token budget, so +# its short request waits. A probe then arrives after the publication of those +# admissions and before any batch completes. +DISCRIMINATING_TRACE = """arrived_at,num_prefill_tokens,num_decode_tokens +1.0,4,8 +1.0,32,8 +1.0,4,8 +1.0,4,8 +1.0,4,8 +1.1,4,1 +""" -def _trace_file(root: Path) -> str: - path = root / "asymmetric_arrivals.csv" - path.write_text(ASYMMETRIC_TRACE) - return str(path) +TRACES = {"asymmetric": ASYMMETRIC_TRACE, "discriminating": DISCRIMINATING_TRACE} + +CASES = { + "moe_dp2": dict(is_moe=True, attn_dp=2, moe_ep=2), + "dense_dp1": dict(is_moe=False, attn_dp=1, moe_ep=1), + "moe_dp2_online": dict(is_moe=True, attn_dp=2, moe_ep=2, trace="asymmetric"), + "dense_dp1_pp2": dict( + is_moe=False, attn_dp=1, moe_ep=1, num_pipeline_stages=2, + analytical_backend=True, + ), + "moe_dp2_pp2": dict( + is_moe=True, attn_dp=2, moe_ep=2, num_pipeline_stages=2, + analytical_backend=True, + ), + "moe_dp2_pp2_online": dict( + is_moe=True, attn_dp=2, moe_ep=2, num_pipeline_stages=2, + analytical_backend=True, trace="asymmetric", + ), + "moe_dp1_pp3": dict( + is_moe=True, attn_dp=1, moe_ep=1, num_pipeline_stages=3, num_layers=6, + analytical_backend=True, + ), + "moe_dp2_pp3": dict( + is_moe=True, attn_dp=2, moe_ep=2, num_pipeline_stages=3, num_layers=6, + analytical_backend=True, + ), + # Stages long enough that no batch completes before the probe (plan D-e). + "moe_dp2_pp2_discriminating": dict( + is_moe=True, attn_dp=2, moe_ep=2, num_pipeline_stages=2, + analytical_backend=True, trace="discriminating", + dummy_execution_time_ms=10.0, + ), +} + +# The policy run's comparison run: a load-blind baseline, or for the +# discriminating case the completion-reporting control. +COMPARISONS = {"moe_dp2_pp2_discriminating": "completion_reporting_control"} if __name__ == "__main__": root = Path(sys.argv[1]) case = sys.argv[2] - shape = { - "moe_dp2": dict(is_moe=True, attn_dp=2, moe_ep=2), - "dense_dp1": dict(is_moe=False, attn_dp=1, moe_ep=1), - "moe_dp2_online": dict(is_moe=True, attn_dp=2, moe_ep=2, trace=True), - }[case] + shape = CASES[case] + comparison = COMPARISONS.get(case, "round_robin") evidence = {} - for policy_name in ("vllm_load_balancing", "round_robin"): - case_root = root / case / policy_name + for run_name in ("vllm_load_balancing", comparison): + case_root = root / case / run_name case_root.mkdir(parents=True, exist_ok=True) arguments = dict(shape) - if arguments.pop("trace", False): - arguments["trace"] = _trace_file(case_root) - evidence[policy_name] = run_case( - case_root, policy_name=policy_name, **arguments + if "trace" in arguments: + trace_path = case_root / f"{arguments['trace']}_arrivals.csv" + trace_path.write_text(TRACES[arguments["trace"]]) + arguments["trace"] = str(trace_path) + evidence[run_name] = run_case( + case_root, + policy_name="round_robin" + if run_name == "round_robin" + else "vllm_load_balancing", + completion_reporting_control=run_name == "completion_reporting_control", + **arguments, ) (root / f"{case}_evidence.json").write_text(json.dumps(evidence, indent=1)) print(json.dumps({k: v["completed_requests"] for k, v in evidence.items()})) From c1a570de9e7ed6f059100a078d2adecbf375f123 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 20:06:31 +0800 Subject: [PATCH 53/80] docs(dp-placement): record Step 9 validation and extend the policy's documented scope to PP AGENTS.md now states that the vLLM DP placement policy supports any pipeline depth, and that a lane publishes its load at admission while its pipeline has room and otherwise with its next completion, as vLLM does. Task records carry the P2-P5 results: unit and real-event-loop tests, the PP=1 unchanged-behavior check (24 of 24 policy scenarios, 71 of 71 fidelity cases and 16 of 16 examples identical, no suite regressions), the implementation self-review, and two pre-existing defects found during P5: W9-04, a MoE attn_dp=4 online deadlock from a stale first-layer placeholder, and W9-05, requests lost mid-decode under KV pressure. The W9-04 prototype is archived as a patch and is not applied. --- AGENTS.md | 3 +- .../design.md | 32 +- .../issues.md | 100 +++ .../plan.md | 23 + .../progress.md | 23 +- .../review.md | 25 + .../step9_p5/c2_pp1_policy_matrix.py | 350 ++++++++ .../step9_p5/compare_examples.py | 16 + .../step9_p5/deadlock_sweep.py | 47 ++ .../step9_p5/deadlock_trace.py | 67 ++ .../evidence/c2_after_vs_prototype.json | 746 +++++++++++++++++ .../step9_p5/evidence/c2_before_after.json | 756 ++++++++++++++++++ .../evidence/fidelity_comparison.json | 103 +++ .../evidence/integration_compare.json | 26 + .../step9_p5/evidence/sweep_after_lor.txt | 24 + .../step9_p5/evidence/sweep_after_random.txt | 24 + .../step9_p5/evidence/sweep_after_rr.txt | 24 + .../step9_p5/evidence/sweep_main_lor.txt | 24 + .../step9_p5/evidence/sweep_main_random.txt | 24 + .../step9_p5/evidence/sweep_main_rr.txt | 24 + .../step9_p5/evidence/sweep_proto_lor.txt | 24 + .../step9_p5/evidence/sweep_proto_random.txt | 24 + .../step9_p5/evidence/sweep_proto_rr.txt | 24 + .../step9_p5/evidence/unit_compare.json | 62 ++ .../step9_p5/evidence/w9_04_trace.txt | 23 + .../step9_p5/rr_probe_any.py | 18 + .../step9_p5/run_examples.sh | 35 + .../step9_p5/run_fidelity.sh | 17 + .../step9_p5/w9_04_prototype.patch | 17 + .../summary.md | 4 + ...st_report_2026-09-22_w9_pp_dp_placement.md | 163 +++- .../validation.md | 24 +- 32 files changed, 2883 insertions(+), 13 deletions(-) create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/c2_pp1_policy_matrix.py create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/compare_examples.py create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/deadlock_sweep.py create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/deadlock_trace.py create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/c2_after_vs_prototype.json create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/c2_before_after.json create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/fidelity_comparison.json create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/integration_compare.json create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/sweep_after_lor.txt create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/sweep_after_random.txt create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/sweep_after_rr.txt create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/sweep_main_lor.txt create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/sweep_main_random.txt create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/sweep_main_rr.txt create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/sweep_proto_lor.txt create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/sweep_proto_random.txt create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/sweep_proto_rr.txt create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/unit_compare.json create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/evidence/w9_04_trace.txt create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/rr_probe_any.py create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/run_examples.sh create mode 100755 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/run_fidelity.sh create mode 100644 task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/w9_04_prototype.patch diff --git a/AGENTS.md b/AGENTS.md index f9eaaa52..550a90ce 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,6 +12,7 @@ | 2026-09-05 | Added Python module size and plain ML-system naming guidance for refactors. | | 2026-09-06 | Added cleanup-first and split-analysis requirements for critical modules above 2,000 lines. | | 2026-09-22 | Completed the cluster-scheduler implementation list and recorded the opt-in vLLM DP placement policy and its supported scope. | +| 2026-09-23 | Extended the vLLM DP placement policy's supported scope to pipeline parallelism. | - Current public branch supports `co-location`, sequential PDD / `pd-disaggregation`, and sequential PD-AF / `pd-af-disaggregation`. - The public co-location, PDD, and PD-AF examples explicitly select `--cc_backend_config_type analytical` for one-click smoke runs using the built-in analytical model. @@ -617,7 +618,7 @@ The scheduling logic is split across four distinct layers to mirror real-world s - `RandomClusterScheduler`: Random assignment. - `StickyRoundRobinClusterScheduler`: Round-robin over targets, pinned per session so a session's later requests return to the same target. - `StickyLORClusterScheduler`: Least Outstanding Requests with the same per-session pinning. - - `VllmLoadBalancingClusterScheduler`: Models vLLM V1's internal DP selection, choosing the lane with the lowest `waiting * 4 + running` score from a load snapshot the frontend observes with a delay. Opt-in and deliberately narrow: one `co-location` replica, the `vllm_v1` replica scheduler, one pipeline stage, and either a MoE model or `attn_dp=1`. The constructor rejects everything else. No placement or timing equivalence with a real vLLM deployment is claimed. + - `VllmLoadBalancingClusterScheduler`: Models vLLM V1's internal DP selection, choosing the lane with the lowest `waiting * 4 + running` score from a load snapshot the frontend observes with a delay. Opt-in and deliberately narrow: one `co-location` replica, the `vllm_v1` replica scheduler, and either a MoE model or `attn_dp=1`, at any pipeline depth. As in vLLM, a lane publishes its load when it admits a batch while its pipeline still has room, and otherwise with its next completion. The constructor rejects everything else. No placement or timing equivalence with a real vLLM deployment is claimed. 3. **Replica Scheduler** (`ReplicaSchedulerRegistry`): - **Role**: Operates at the level of a single `Replica` (GPU node/instance). diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md index e3962cc6..e0f376df 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/design.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/design.md @@ -7,6 +7,7 @@ scope decisions and the pre-measurement expectation for that package. | Date | Change | | --- | --- | +| 2026-09-23 | W9 as implemented: section "Schedule-time reports under pipeline parallelism" added (P2–P5 outcome, measured against the expectation above); W4 guard row annotated. | | 2026-09-23 | D9-2 decided by the user: the group-anchored rule. | | 2026-09-23 | Added "Design checkpoint D9-2: the key from the fourth shape": reference lockstep facts, seven-shape P1(b) scores, the proposed group-anchored key rule with its invariant argument and residuals, W9-03 pointer. Proposal only; awaits the user's decision. | | 2026-09-22 | Added the design checkpoint section: D9-1 payload settled from the P1 oracle; D9-2 key left open because the `ForwardSyncState` candidate fails invariant I5 at PP>1 and the I1/I5 trade-off is only observable on the shape blocked by W9-01. | @@ -234,7 +235,7 @@ pair to a single scalar. | Decision | Choice | Why | | --- | --- | --- | -| Topology guard | `MONOLITHIC` + one Replica + `PP1` + `vllm_v1` + (**MoE or `attn_dp == 1`**) | The first four are the candidate's. The fifth is decision D1's "reject unsupported configurations explicitly", and the dense multi-lane row above is the measurement behind it. | +| Topology guard | `MONOLITHIC` + one Replica + `PP1` + `vllm_v1` + (**MoE or `attn_dp == 1`**) | The first four are the candidate's. The fifth is decision D1's "reject unsupported configurations explicitly", and the dense multi-lane row above is the measurement behind it. W9 removed the `PP1` clause on 2026-09-23; see "Schedule-time reports under pipeline parallelism". | | Second step counter | **no** | D1 forbids broadening W4 with a new counter. W3's identity is reused where it holds. | | Runtime order assertion | **no** | The reference warns and applies the counts. W4 mirrors the warning. Key equality per shared forward is a test invariant, not a runtime abort. | | `waiting + 1` reservation | keep | One modeled frontend, so `client_count == 1`. The one-frontend restriction is stated in the class docstring. | @@ -725,3 +726,32 @@ lane's admission counter. The policy admits a dense model only at The analytical backend runs MoE `attn_dp=2, PP=3`. W9-02's rejection is the collective-sim topology rule. The P3/P4 matrix can therefore restore the multi-lane PP3 row that I5 needs (C1 amendment, part of this proposal). + +## Schedule-time reports under pipeline parallelism (as implemented, 2026-09-23) + +Commits `2ffe78d` (source and unit tests) and `bacdbb4` (integration tests). + +| Part | As implemented | +| --- | --- | +| Hook | `BaseClusterScheduler.on_replica_batch_scheduled(time, replica_id, replica_local_id, batch)`, inert. `BaseReplicaScheduler` calls it once per admitted batch in the MONOLITHIC/PREFILL admission loop, after `_num_running_batches += 1`. | +| When a report is made | At admission, if `num_running_batches < num_pipeline_stages`. Otherwise the admission's key is held and reported with the lane's next completion, which is the reference iteration that fills the pipeline and applies its oldest output. A completion with nothing held reports under the lane's next key. | +| Key | Group-anchored: `max(stage0.joinable_forward_group_id, last_admitted_key + 1)`, the D9-2 rule. `StageExecutionContext.joinable_forward_group_id` is the one new accessor. `ForwardSyncState` is no longer read by the policy. | +| Guard | The PP1 clause is removed. MONOLITHIC, one Replica, `vllm_v1`, and MoE or `attn_dp=1` remain. | +| PP=1 | Every admission is held, so reports stay completion-only. Key values change (MoE `3, 3, 7, …` became `0, 0, 1, …`), but every pairwise comparison is unchanged. | + +Against "Fidelity expectation, stated before measuring": + +- PP=1 `vllm_load_balancing`: 24 of 24 scenarios identical (artifacts, report + stream, key order, selections). The expectation held. +- Other cluster schedulers: 71 of 71 fidelity cases and 16 of 16 examples + identical. The expectation held. +- PP=2 and PP=3: runs conserve work. On the discriminating scenario the policy + sends the probe to lane 0 and the completion-reporting control sends it to + lane 1. The expectation held. +- Ground truth: not measured. G3–G5 are blocked on GPU authorization. + +The residuals in "What the rule does not reproduce" stand as written. P5 also +found W9-04, a deadlock in the shared-forward placeholder rule that predates +this work (`issues.md`). It is independent of the key: `round_robin` stalls on +the same cells. + diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md index a631a0d1..6f8872b0 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/issues.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | W9-04 (placeholder/join deadlock at `attn_dp=4`, root cause and prototype) and W9-05 (requests lost under KV pressure, also on `main`) recorded from Step 9 P5. | | 2026-09-23 | W9-01 remaining step 3 done (P1(b) complete, D9-2 proposed); W9-02 narrowed to the collective-sim backend; W9-03 recorded (reference DP lockstep under PP, observation). | | 2026-09-23 | W9-01: merged forward (`dd9b8d9`); composition check passes on `03d5f24`. | | 2026-09-23 | W9-01: PR 36 ran its pre-merge untrack (P6, `4d08c5d`); the copies here are now the only published records of that task. | @@ -230,3 +231,102 @@ It is not a report-key defect. The proposed D9-2 key follows Frontier's own grouping, and its residual 1 in `design.md` is the part of this difference that reaches the report stream. + +## W9-04 A lane with a first-layer placeholder can join the forward and deadlock it + +Status: open, root cause established, prototype fix measured in a scratch +tree. Not caused by Step 9: the pre-P2 tree `d1a2a06` and `bacdbb4` stop at the +same state. Decision pending with the user. +Found: 2026-09-23, Step 9 package P5, in the C2 PP=1 policy matrix. + +### Symptom + +MoE co-location, `attn_dp=4, moe_ep=4, PP=1`, `vllm_v1`, 24 Poisson requests +of 8–96 tokens: the event queue drains at 0.1201 s with no request complete +and `RuntimeError: Sequential simulation ended with non-empty scheduler state`. +Lanes 0–2 hold stage 0; lane 3 has a queued batch and is not busy. + +Reachability sweep (`step9_p5/deadlock_sweep.py`; MoE `attn_dp` ∈ {2, 4}, +Poisson qps ∈ {50, 100, 200, 400}, seeds {42, 7, 123}; 24 cells per row): + +| Tree | Cluster scheduler | Stuck cells | +| --- | --- | --- | +| this branch `bacdbb4` | `round_robin` | 5 (all `attn_dp=4`, qps ≥ 200) | +| this branch | `lor` | 1 (`attn_dp=4`, qps 400, seed 42) | +| this branch | `random` | 0 | +| `origin/main` `4ab1964` | `round_robin` | 0 (every request is placed on lane 0: the W2 defect) | +| `origin/main` | `random` | 0 | +| `origin/main` | `lor` | 24 (a different failure: stuck with 2–12 requests done in every cell, `attn_dp=2` included; not diagnosed) | +| prototype below | `round_robin`, `lor`, `random` | 0 of 72 | + +`attn_dp=2` never stalled on this branch. `vllm_load_balancing` stalls the +same way as `round_robin` on the same cells. + +### Mechanism (trace `step9_p5/evidence/w9_04_trace.txt`) + +1. t=12.10 ms: lane 0 reaches layer 0 `pre_moe` of forward group 0. Lane 1 is + bound to group 0 and still in attention. Lanes 2 and 3 have empty, idle + stages, so `_can_supply_idle_lane` (`sync_entry.py:9`) gives each an idle + placeholder in that room. +2. The same instant: request 2 reaches lane 2. The group is not sealed, so + `try_acquire` admits lane 2's batch and `bind_forward_group` binds it to + group 0. +3. t=13.71 ms: lane 1 arrives. The room holds four entries (two real, two + placeholders) and dispatches the layer-0 EP wave. The group is sealed. +4. t=20.38 ms: lane 2's real batch reaches layer 0. That room is closed, so + `resolve_step` opens a new step for layer 0. Lanes 0 and 1 are busy in + group 0 and are not given placeholders. +5. t=27.71 ms: lanes 0 and 1 reach layer 1 and wait for lane 2, which waits + for them at layer 0. Lane 3's new batch cannot join the sealed group. + +The join rule and the placeholder rule disagree. A lane may join a forward +until its first EP wave dispatches, and a real batch replaces the lane's +placeholder only if it reaches the room first. A join after the placeholder +but before dispatch, whose attention outlasts the last peer's arrival, loses +that race. + +### Options + +1. **Drop a stale placeholder (prototype, `step9_p5/w9_04_prototype.patch`, + 8 lines in `enter_layer_sync`).** An idle entry whose lane's stage is busy + belongs to a lane that has since joined this forward with real work, which + will enter the room. It is removed before the room is counted, so the + room waits for the real batch. This is the replacement the code already + performs when the real batch arrives first, applied to the other order. + Measured in `trees/proto`: 72 of 72 sweep cells drain. The 22 C2 cases that + drained before are identical: `request_metrics.csv`, `system_metrics.json`, + the report stream and every selection. The two stalled cases now finish + (24/24, and 23/24 with W9-05). Not yet run: the 71-case fidelity matrix, + the stage-admission groups and the suites. +2. **Keep a lane's placeholder binding (reference behavior).** A lane that has + been given a placeholder does not join that forward; its batch waits for + the next one. This is what the pinned vLLM does: an idle engine is already + inside its dummy forward (W9-03). It changes the timing of every run + where a late join currently succeeds, so it is a fidelity change that + needs its own measurement and approval. +3. Record only and defer. + +Recommendation: option 1 in this PR, because this PR's W2 is what makes the +multi-lane Poisson path reachable under `round_robin`. Option 2 belongs with +W9-03 and the G4 lockstep measurement. + +## W9-05 Requests disappear mid-decode under KV pressure + +Status: open, not diagnosed. Present on `origin/main` `4ab1964`. Outside +Step 9. +Found: 2026-09-23, Step 9 package P5, in the C2 PP=1 policy matrix. + +`vllm_v1` with `num_blocks=12, block_size=16`, 24 Poisson requests of 8–96 +tokens at qps 200. MoE `attn_dp=2` and dense `attn_dp=1` both reproduce it, +under `vllm_load_balancing` and under `round_robin`, on this branch and on +`origin/main`. The run ends normally with 20 of 24 requests complete and +exit code 0. Requests 10, 16, 17 and 21 decode a few tokens (for example +request 10: 3 of 14), then leave every queue. No log line mentions +preemption, and the drain check passes although `is_empty` also counts the +preempted queue, so the requests are held by no queue it reads. Their +`request_metrics.csv` rows have empty latency fields. + +Next step when scheduled: follow one request through +`_try_allocate_with_preemption` and +`_rollback_current_iteration_preempted_requests` +(`vllm_v1_engine_replica_scheduler.py:640-660`). diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md index 5d6b3896..9f9959e9 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/plan.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | §18.16 added: P2–P5 results against their acceptance rows, and the two pre-existing defects found in P5 (W9-04, W9-05). Status line under §18 updated. | | 2026-09-23 | D9-2 decided by the user: group-anchored key (§18.15, `requirements.md`); C1 PP3 row amended; P2 started. | | 2026-09-23 | §18.15 added: P1(b) completed on seven shapes; D9-2 proposal (group-anchored key) and the C1 PP3 amendment await the user's decision. §18.13 blocker marked resolved. | | 2026-09-23 | §18.14 results: K1–K4 pass on `03d5f24`; K2 amended after measuring for online cells (one cell's batches differ after an earlier admission). | @@ -855,6 +856,8 @@ Under [R1]: `frontier/profiling/moe/moe_vllm_kernel.py`, `frontier/entities/stag **Status 2026-09-22:** the user answered every §18.7 decision the same day (verbatim in `requirements.md`). Later the same day an external review of PR34/PR35 (`.local-draft/Frontier_PR34_PR35_Current_Code_and_PP_Extension_Review_2026-09-22.md`, findings P9-01..P9-06) corrected this plan; the corrections are applied in place below and collected with their evidence in §18.11. A second review at the user's direction (2026-09-22, quality gates for core-module changes) is recorded in §18.12 and amended D9-1, D9-2, P1 and §18.10 in place. No Step 9 source edit has been made; execution starts at the first node of the §18.5 graph once the user confirms the start. Research followed the `codebase-design` skill (§18.10); the ground-truth comparison follows `frontier-calibration` v2 as written (§18.9). +**Status 2026-09-23:** P1–P5 are complete on the CPU at `bacdbb4` and P6 records are being published (§18.16). G3–G5 stay blocked on GPU authorization. + ### 18.1 Goal and acceptance criteria `VllmLoadBalancingClusterScheduler` accepts valid `num_pipeline_stages > 1` configurations and reproduces vLLM 0.10.2's per-iteration DP request-count publication under the batch-queue stepping path that PP>1 selects — one observable engine scheduling iteration and its frontend-visible load, not a counter made monotonic after the fact — verified on a controlled or demonstrably matched iteration history against a real `vllm serve --data-parallel-size 2 --pipeline-parallel-size 2` deployment. @@ -1233,3 +1236,23 @@ amendment (`requirements.md`, "[Decision] 2026-09-23 — D9-2 report key"). P2–P5 proceed on the CPU. G3 and G4 need GPU runs whose authorization is still BLOCKED in the case manifest, so G3–G5 wait for a separate go. +### 18.16 P2–P5 results (2026-09-23) + +Commits: P2/P3 `2ffe78d`, P4 `bacdbb4`. Evidence rows are in `validation.md` +Step 9 and the W9 report §4–§6; the self-review is in `review.md` +("Step 9 implementation self-review 2026-09-23"). + +| Package | Acceptance (§18.5) | Result | +| --- | --- | --- | +| P2 | Existing unit tests pass except the inverted guard case | 1 failed (the `pipeline_parallel` guard case), 112 passed | +| P3 | New tests fail before P2 and pass after | 132 passed; 19 of 19 new or changed cases fail on the pre-P2 tree | +| P4 | Pass; each control fails for its stated reason | 9 + 3 passed; the completion-reporting control places the probe on lane 1, the policy on lane 0 | +| P5 | C2 | 24 of 24 PP=1 policy scenarios identical; fidelity 71 of 71; examples 16 of 16; suites 0 regressions | + +Found in P5, both pre-existing (`issues.md`): + +- **W9-04**: a MoE `attn_dp=4` online deadlock caused by a stale first-layer + placeholder. A scratch prototype drains 72 of 72 sweep cells; it is not + applied and awaits the user's decision. +- **W9-05**: `vllm_v1` loses requests mid-decode under KV pressure, also on + `origin/main`. It is not yet diagnosed. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md index 0fb80f69..4ebc5451 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | Step 9 P4 committed (`bacdbb4`) and P5 completed: C2 holds (24 of 24 PP=1 policy scenarios, 71 of 71 fidelity cases, 16 of 16 examples identical; 0 suite regressions). P5 found W9-04 (placeholder/join deadlock at `attn_dp=4`) and W9-05 (requests lost under KV pressure, also on `main`); W9-04 fix decision pending with the user. | | 2026-09-23 | D9-2 decided (group-anchored key); P2 implemented and P3 unit tests added (132 targeted tests pass; the 19 new or changed cases fail on the pre-P2 tree). | | 2026-09-23 | Step 9 P1(b) completed on seven shapes; D9-2 proposal recorded (design.md, plan §18.15); W9-03 observation; awaits the user's D9-2 decision. | | 2026-09-23 | W9-01 merge-forward: `origin/main` merged (`dd9b8d9`); composition check K1–K4 pass on `03d5f24` (drain-reader fix); Step 9 P1(b) resumed. | @@ -34,9 +35,9 @@ | Correctness branch | `fix/issue26-correctness-pr` (worktree `/data/ycfeng/Frontier/.worktrees/issue26-correctness-pr`) | | Base at creation | `refactor/oversized-module-split` @ `41dabfb9d5ef3b51cdf3009d486450515d9a8d2d` (itself on `origin/main` `1f694f7`) | | Prerequisite | MET. All four modules this PR edits are under the 2,000-line gate. The split's final record is 71 of 71 fidelity cases identical with no predictor cache differences, taken with the corrected gate; see the refactor task's Checkpoint B report. | -| Current step | Step 8 closed; external review corrections A–E applied 2026-09-22 (`test_report_2026-09-22_review_corrections.md`). Step 9 is planned (`plan.md` §18, corrected per the review) and **not started**. | +| Current step | Step 9 in progress: P1–P5 completed (P4 `bacdbb4`; P5 evidence in `test_report_2026-09-22_w9_pp_dp_placement.md` §6); P6 (documents, push, PR body) next. G3–G5 blocked on GPU authorization. | | Publication | PUSHED_VERIFIED: `f7c31e4` (C35-01 source + tests), `ca1b9b6` (FP8 `block_shape`, optional-torch skip, W6 report, profiling guide), `57ffa5b` (records, Step 9 plan corrections); remote head `57ffa5b` confirmed; PR34 correction `2310417` merged in as `0d025f8`. Draft PR 35 body PATCHed 2026-09-22T12:12Z through `gh api` and read back; still draft, MERGEABLE, base `refactor/oversized-module-split`. | -| Next action | **User decision.** Start signal for Step 9, at the first node of the `plan.md` §18.5 graph (plan amended by the §18.12 second review). The FP8 native rerun is done and PASS. PR 35 remains draft and nothing was merged. Carried forward in `future.md`: re-point the collective-sim gitlink at `main` once companion PR 1 merges, and repair the `tests/debug/` pointers that 10 baseline unit failures share. Retargeting PR 35's base to `main` waits on PR 34. | +| Next action | P6 records and publication. User decision on W9-04 (`issues.md`: option 1 stale-placeholder removal in this PR, recommended; option 2 reference placeholder binding; option 3 defer) and on W9-05 (defer recommended). Removal of the two P5 detached worktrees `.worktrees/p5-fidelity-{before,after}` needs approval. | ## Step status @@ -52,7 +53,7 @@ | 7 | Optional zero-payload backend | PASS. Companion fix published as `fwyc0573/frontier-htsim` `eb7bc4f` with draft PR 1; Frontier gitlink moved from `b8518af`; no Frontier source change | Companion 9 passed, negative control 6 of 9 fail on pristine sources. Frontier 4 passed, negative control 3 of 4 fail at the old gitlink. Clean checkout resolves `eb7bc4f` from the published remote, builds, and passes. Suite back to the 84-failure baseline with 3782 passing after narrowing three governance scans to Frontier-owned sources | PUSHED_VERIFIED | — | | 8 | Combined regression, PR hand-off | PASS | unit 84 failed / 3782 passed with a `FAILED` set identical to the `origin/main` baseline; integration 15 passed / 22 skipped / 5 errors, the errors environmental and identical on the base; 16 of 16 architecture examples pass; 4 of 4 `PP=2` cases pass; cold and warm predictor-cache runs byte-identical | PUSHED_VERIFIED (records + PR 35 body carrying the Step 8 results, the record links and the implementation commits) | REVIEWED (external review 2026-09-22; corrections below) | | 8+ | External review corrections A–E | PASS | unit 84 failed / 3789 passed / 50 skipped / 10 errors with the `FAILED` set identical to the baseline (+7 passes are the new tests, +1 skip and −1 error are the optional-torch module); mixed-forward unit 26 passed; real-loop hybrid-layer case 2 passed with the negative control failing on the pre-fix source; arithmetic 9 passed under torch | PUSHED_VERIFIED | NOT_REVIEWED | -| 9 | PP>1 support for `vllm_load_balancing` | PLANNED — plan corrected per the external review (`plan.md` §18.11); **not started** | n/a | PUSHED_VERIFIED (records only) | awaiting the user's start signal | +| 9 | PP>1 support for `vllm_load_balancing` | IN PROGRESS — P1–P5 completed; P6 next; G3–G5 blocked (GPU) | unit and integration PASS; C2 PASS (24/24 policy scenarios, 71/71 fidelity, 16/16 examples identical; 0 suite regressions) | LOCAL_ONLY after `9b5d7be` | D9-2 decided by the user; W9-04 decision pending | ## Chronological updates @@ -226,7 +227,9 @@ User start signal: "开始执行step9", with the quality gates repeated (readabi | P1(b) Frontier boundary probe | completed 2026-09-23 (see "Step 9 P1(b) and D9-2" below) | Three shapes probed (`attn_dp=2 PP=1`, `attn_dp=1 PP=2`, `attn_dp=1 PP=3`), tables in `plan.md` §18.13. The fourth shape, MoE `attn_dp=2, moe_ep=2, PP=2`, drains the event queue with requests unfinished — pre-existing defect W9-01 in `issues.md`. | | Design checkpoint (D9-1, D9-2) | settled 2026-09-23 | D9-1's payload was settled from the P1 oracle. D9-2 is the group-anchored key, exact on all seven P1(b) shapes (`design.md` "Design checkpoint D9-2"); the user chose it on 2026-09-23. The first candidate, `ForwardSyncState._next_step_id_by_replica`, failed I5. | | P2 implementation, P3 unit tests | completed 2026-09-23 | See "Step 9 P2 and P3" below. | -| P4, P5, P6 | pending | CPU only. | +| P4 integration | completed 2026-09-23 (`bacdbb4`) | See "Step 9 P4 and P5" below. | +| P5 fidelity and regression | completed 2026-09-23 | See "Step 9 P4 and P5" below. | +| P6 documents and publication | in progress | CPU only. | | G3–G5 | blocked | GPU authorization is `BLOCKED` in the case manifest. | W9-01 is not caused by this PR: `stage_execution_context.py`, `replica_stage_schduler.py` and `stage_contexts.py` are byte-identical to `main`. It is unobserved because every Simulator-level test with `attn_dp > 1` uses `num_pipeline_stages = 1` and no shipped example sets `attn_dp > 1`. Scope decision requested from the user; recommendation is to fix it as a separate correctness item rather than inside this feature branch. @@ -271,6 +274,18 @@ W9-02: `attn_dp=2, moe_ep=2, PP=3` is rejected at construction (6 devices agains | P3 tests | `tests/unit/test_vllm_dp_load_balancer.py`: guard case inverted into PP2/PP3 construct cases (8 shapes) plus dense multi-lane PP2 and uneven-partition rejections; `_ScriptedReplica` drives scripted lane readings and real stage-0 forward groups; cases for PP1 (no schedule-time report), PP2 cold fill in both lane orders (no partial latch), PP3 two admission-only iterations, a full pipeline (MoE and dense), a completion plus the admission it makes room for (one key, no intermediate latch), drain to zero and new work, and the real admission loop at PP2; both seams in the inert and unknown-lane tests. `tests/unit/test_shared_forward_group_admission.py`: `joinable_forward_group_id` across bind, seal and release. | Expected reports written from the reference iteration, before running. Each new case must fail on the pre-P2 tree. | `pytest -q -p no:cacheprovider tests/unit/test_vllm_dp_load_balancer.py tests/integration/test_vllm_dp_placement_runtime.py tests/unit/test_dp_placement_reference_loop.py tests/unit/test_stage_execution_context.py tests/unit/test_shared_forward_group_admission.py`: 132 passed, 7.07 s. The same tests on `git archive HEAD` (pre-P2) plus the new test files: 19 failed, 72 passed; the 19 are exactly the new or changed cases. | | Arrival-order check | Replay of the group-anchored keys from `key_scores.json` in report order, counting keys smaller than the last applied one | Recorded so the balancer's warning is not mistaken for a key defect later. | 1 report in 7 shapes (dp2 PP3 burst, t=0: lane 0 key 0 after lane 1 key 1); the reference has the same race. Noted in `design.md`. | +### Step 9 P4 and P5 (2026-09-23) + +| Step | Change / command | Reason and expectation | Result | +| --- | --- | --- | --- | +| P4 tests | `tests/integration/test_vllm_dp_placement_runtime.py` rewritten around seam records: 5 PP shapes plus the plan §18.6 discriminating case against a completion-reporting control; `tests/integration/test_monolithic_mixed_forward_runtime.py` gains a PP2 `vllm_load_balancing` run | Each report must map to one reference iteration kind under its forward's key; the probe must be placed differently only because of what was published | 9 + 3 passed; report kinds per case and the probe snapshots in the W9 report §5. Committed `bacdbb4`. | +| P5 C2, policy | `step9_p5/c2_pp1_policy_matrix.py`: 24 PP=1 scenarios on `git archive` exports of `d1a2a06` (pre-P2 source) and `bacdbb4` | Artifacts, report stream (without key), key order, and selections identical; no added admission-only report | 24 of 24 identical. Two cases stall on both sides (W9-04) and two lose 4 requests on both sides (W9-05), with identical diagnostics and artifacts. | +| P5 C2, other schedulers | `step9_p5/run_fidelity.sh` on clean detached worktrees `.worktrees/p5-fidelity-{before,after}`; `step9_p5/run_examples.sh` + `compare_examples.py` | 71 of 71 and 16 of 16 identical | 71 of 71 identical, 0 provenance findings; 16 of 16 examples pass on both trees and are identical. | +| P5 suites | `composition_run_suites.sh` on the clean `bacdbb4` worktree, compared by test id against the K4 JUnit of `03d5f24` | 0 regressions, 0 new failures | unit 84 failed / 3829 passed / 51 skipped / 10 errors; integration 5 errors / 26 passed / 22 skipped; 0 regressions, 0 new failures, 0 skip changes. The added skip is `test_collective_sim_zero_payload` (submodule not initialized in the detached worktree); 4 passed in this worktree. | +| W9-04 investigation | `step9_p5/deadlock_trace.py`, `deadlock_sweep.py`; prototype `step9_p5/w9_04_prototype.patch` in a scratch export only | Establish the cause before proposing a change | Root cause and options in `issues.md` W9-04. Prototype: 72 of 72 sweep cells drain (was 66); 22 of 22 previously drained C2 cases identical. Not applied to the branch. | + +Evidence copies: `step9_p5/evidence/`. Raw runs: `/data/ycfeng/tmp/issue26-correctness-pr/step9_p5`. + ### G1 ground-truth instrumentation (2026-09-22, completed) Runs in parallel with P1 in the work graph and does not depend on W9-01. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md index bee5d3cd..f82e314d 100644 --- a/task_memory/task_2026-09-21_issue26_correctness_pr/review.md +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/review.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | Step 9 implementation self-review of `d1a2a06..bacdbb4` against the user's quality gates: findings S9-01..S9-08, no source change required; W9-04 and W9-05 recorded as pre-existing. | | 2026-09-22 | Second Step 9 plan review at the user's direction (quality gates for core-module changes): findings R9-01..R9-08 recorded with dispositions; plan §18.12, design.md. | | 2026-09-21 | Created with the pinned source snapshot. | | 2026-09-21 | Step 1 complete: candidate and vLLM audits landed, dispositions recorded, two decision checkpoints raised. | @@ -451,3 +452,27 @@ Reviewer: this session, against `c231322`, at the user's direction ("确保当 | R9-08 | reference precision | DP engines not iteration-lockstep (all-reduce every 32 steps) | §18.2 row amended; G5 first-cause label | No source change results from this review; Step 9 execution remains unstarted pending the user's start signal. + +## Step 9 implementation self-review 2026-09-23 + +Reviewer: this session. Reviewed range: `d1a2a06..bacdbb4` (P2 `2ffe78d`, P4 `bacdbb4`). Gates: the user's core-module rule ("任何引入的修改和实现都应该是高价值的 ... 禁止hard-coding，禁止临时补丁，禁止过度防御，禁止冗余性设计和实现，禁止使用ai味命名函数和变量") and the AGENTS.md development gates. Inspected: the four changed `frontier/` files in full diff, the admission loop at `base_replica_scheduler.py:1050-1075`, the completion hook call at `global_batch_end_event.py:180-185`, `base_replica_scheduler.py:51` (stage count), and the test diffs. Evidence of behavior: `validation.md` Step 9 and the W9 report §4–§6. + +`frontier/` change: 4 files, +130/−25 lines. `VllmLoadBalancingClusterScheduler` is 158 lines; the largest touched module, `base_cluster_scheduler.py`, is 1,946 lines, under the 2,000-line gate. + +| Id | Gate | Finding | Disposition | +| --- | --- | --- | --- | +| S9-01 | value | The change gives the opt-in policy the PP>1 support the user asked for, and adds one fidelity gain: a schedule-time report while the pipeline has room, which the reference engine publishes and the pre-P2 policy could not express. At PP=1 every admission is held, so C2 shows no behavior change (24 of 24). | Accepted | +| S9-02 | redundancy / superseded path | The completion key no longer reads `ForwardSyncState.get_step_id`; the import and the per-layer step-id key are deleted in the same change. No second key path remains. | Accepted | +| S9-03 | reuse before inventing | The key reads the stage-0 forward group that `StageExecutionContext` already binds. The new property `joinable_forward_group_id` exposes the existing `_forward_group_id` / `_forward_group_sealed` / `_next_forward_group_id` state; it adds no state. | Accepted | +| S9-04 | correctness of the single held slot | One `_held_key` per lane suffices. An admission is held only when it fills the pipeline (`running == PP`); the loop admits only while `running < PP` (`base_replica_scheduler.py:1052`); and `GlobalBatchEndEvent` calls `on_batch_end` (decrement) and then the completion hook in the same handler, before any later `ReplicaScheduleEvent` can admit again. So a held key is always consumed before the next held admission on that lane. The unit case "a completion plus the admission it makes room for" pins that order. | Accepted, no guard added | +| S9-05 | over-defense | No new guard, fallback or `hasattr` reach-up. The `_lane_index` type check is the pre-existing one, extracted so both hooks share it. The inert base hook returns `None` and costs one call per MONOLITHIC/PREFILL admission for the other schedulers; the fidelity matrix is 71 of 71 identical. | Accepted | +| S9-06 | hard-coding | No literal enters the path. The pipeline depth comes from `replica_config.num_pipeline_stages`, the same source as the lane's `_num_stages` (`base_replica_scheduler.py:51`). | Accepted | +| S9-07 | naming | `on_replica_batch_scheduled`, `joinable_forward_group_id`, `_held_key`, `_last_admitted_key`, `_next_report_key`: each names the scheduling concept it holds and follows the neighboring `on_replica_batch_end` and forward-group vocabulary. | Accepted | +| S9-08 | test surface | The tests stay out of `frontier/` (R9-07). Each new or changed unit case fails on the pre-P2 tree (19 of 19). The integration test checks each report against the reference iteration kind, not against Frontier's own output, and its discriminating case separates the policy from a completion-reporting control. | Accepted | + +Found during validation, both outside this change: + +- W9-04, a MoE `attn_dp=4` online deadlock from a stale first-layer placeholder. It is reachable under `round_robin` on this branch and its base, so it is pre-existing. The prototype fix is not applied and awaits the user's decision (`issues.md`). +- W9-05, requests lost mid-decode under KV pressure in `vllm_v1`. It is also present on `origin/main` and is not diagnosed (`issues.md`). + +Result: no source change required by this review. G3–G5 remain blocked on GPU authorization. diff --git a/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/c2_pp1_policy_matrix.py b/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/c2_pp1_policy_matrix.py new file mode 100644 index 00000000..91259b5d --- /dev/null +++ b/task_memory/task_2026-09-21_issue26_correctness_pr/step9_p5/c2_pp1_policy_matrix.py @@ -0,0 +1,350 @@ +"""Step 9 P5 / C2: PP=1 `vllm_load_balancing` scenarios, before vs after P2. + + python c2_pp1_policy_matrix.py run