Fix DP request placement, shared forwards, and MoE routing identity

[fwyc0573]

Related to #26.

Scope

A small set of correctness fixes extracted from bug/ttft-check after a source review, rebuilt against current main rather than merged wholesale.

Latency calibration and any Frontier-versus-vLLM end-to-end latency comparison are out of scope. No accuracy claim is made by this PR. Issue #26 stays open. One scheduler-level comparison against vLLM is authorized as a later step (Step 9, DP placement under PP>1); it compares placement decisions, not latency, and it has not started.

Base

Stacked on #34, which brings the four modules this PR edits under the 2,000-line gate without changing behavior. The base here is refactor/oversized-module-split and will be retargeted to main once #34 merges, so the diff shown is only the behavior change.

The reviewed parent changes were brought in by merge, not rebase, so no commit on this branch was discarded and the published review anchors stay valid. Current relationship: head fix/issue26-correctness-pr, base refactor/oversized-module-split at 2310417, merged in as 0d025f8 after the base corrected its fidelity gate's cache-eligibility rule (review finding C34-01).

Progress

#	Fix	State
W2	Round-robin DP placement keeps rotating across scheduling calls	landed, measured, and re-measured after review
W3	A shared monolithic forward completes once for mixed prefill and decode source lanes	landed; the deadlock reproduced and fixed in the real event loop; a dense-layer decode-credit gap in mixed batches found by the 2026-09-22 external review and fixed
W4	Opt-in vLLM-style DP request placement, bounded and off by default	landed; measured to place differently from round-robin under the same load
W5	Routing implementation identity kept separate from expert-load distribution	closed without a source change; the collision is unreachable
W6	Legacy fused-MoE profiling performs the real gated expert computation	landed; native GPU runs pass 8 of 8 on an H800 twice: seven reference comparisons at rtol=0, atol=0 plus one FP8 structural check, the second run with the corrected block_shape wiring; FP8 numerics not established
W7	Optional collective-sim zero-payload handling	landed companion-side; published as a branch with a draft PR, and the gitlink moved onto it

W2, and how a behavior fix is measured

_schedule_batch_mode drew the replica index from a counter that persists across scheduling calls but drew the DP lane from the index within the current call, so lane assignment restarted at lane zero every time the scheduler was entered. main already applies the correct expression for the unified decode role, so this is an internal-consistency repair rather than a new policy.

The fidelity matrix that gates #34 asserts nothing changes, which is the wrong question for a behavior fix. It was used differently here: the expected-to-move set was stated before measuring, so the result could falsify it.

Expected to move: the three online DP cases. Expected not to move: the two offline DP cases, where the whole request stream is admitted in one call and the two expressions already agree, and the other 66.

Measured: 71 of 71 compared, 68 identical, and the mismatch set was exactly the three predicted cases. Nothing else moved.

The direction was confirmed too, from the lane occupancy the stage ledger records:

Case	Before	After
two lanes, online	all 45 records on lane 0	44 / 45
four lanes, online	all 45 records on lane 0	43 / 43 / 44 / 45
two lanes over two replicas, online	all 56 records on lane 0	55 / 56
two lanes over two replicas, offline (control)	70 / 70	70 / 70 unchanged
attention DP 2, offline (control)	39 / 39	39 / 39 unchanged

The third row is the clearest evidence of the defect: replicas split evenly while every record sat on lane zero, one rotation working and the other not, in a single run.

Re-measured after the review

The first measurement ran the 6ab521d source against the refactor tip's case table and comparator. The harness revision was disclosed in prose but not recorded as a field, and that harness could report success without having compared anything — which is what review comment R34-01 found and #34 now fixes.

It was therefore repeated with the corrected harness, with source and harness at one revision: baseline 6ef0a3c, candidate ceac2b4, both clean detached checkouts, one harness at ceac2b4, no case filter, clean cache, 71 cases executed and 426 predictor cache files on each side.

Every number above reproduces exactly, and the gate now also reports 0 baseline failures, 0 candidate-only failures, 0 cases with missing evidence, 0 cases with differing definitions, 0 cases not compared, and no provenance findings. The re-measurement changed the provenance of the evidence, not the evidence.

Comparison exit code is 1, and that is correct here: the gate reports inequality, while the acceptance criterion for a behavior fix is the stated expectation.

The candidate measured is ceac2b4; up to the W2 checkpoint the later commits touched only tests/ and task_memory/, and git diff ceac2b4..HEAD -- frontier/ was empty at that checkpoint, so these artifacts come from the W2 production tree. W3, W4, W6 and W7 changed frontier/ afterwards; their matrix runs are reported in their own sections.

A coverage limit worth stating

The matrix validates the monolithic half of W2 only. The prefill role reaches the same placement path, but no shipped recipe can give it more than one lane: a dense model in a disaggregated architecture is rejected outright, and the MoE wrappers require ATTN_TP == MOE_TP * MOE_EP while the runtime requires attn_tp * attn_dp == moe_tp * moe_ep, which have no common solution above one lane. Both routes were tried and both were rejected, so this is measured rather than inferred.

That is a limit of the shipped wrappers, not of the runtime, and the docstring recording it now says so. Prefill placement is covered at the scheduler level, through the public schedule(); covering it through the real event loop needs a fixture that builds a runtime configuration directly with deterministic durations injected at the predictor boundary. That fixture now exists, as W3 acceptance work — see below.

Four DP-placement cases were added to the matrix for this fix, since 66 of the previous 67 ran a single lane and could not see lane placement at all.

The placement tests, after review comment R35-01

The earlier tests compared one run against another. That shows placement does not depend on how the request stream is divided between calls, but it would also hold for a wrong rule applied consistently. And test_replica_identity_contract read source text, which cannot tell a correct expression from an equivalent one it does not recognize.

Three topologies now have their full rotation written out by hand, past its wraparound, for the case where both the replica and the lane advance — for example, replicas [3, 11] with two lanes must produce (3,0) (11,0) (3,1) (11,1) repeating. The tests drive the public schedule() rather than _schedule_batch_mode, so the cluster-type dispatch is exercised rather than assumed, and each runs for both roles that reach batch-mode placement, MONOLITHIC and PREFILL. A guard keeps that role list honest: TRANS falls through the same dispatch, and the list is only complete because TRANS is declared and never constructed anywhere in frontier/.

The source-text check remains as a narrow governance check, with a docstring saying it does not establish behavior and must not be satisfied by rewording a correct expression.

What the negative control showed

Re-running these against the pre-fix method, taken verbatim from 6ab521d^ and installed in memory, fails 12 of 23 and leaves the grouping control and the eight unrelated lane tests passing. The failures are more informative than the count: in all three topologies the old code produces the expected sequence exactly when the whole stream arrives in one call, and collapses onto lane 0 only when requests arrive one at a time.

So the hand-derived expectation agrees with the rotation the code already performed for a single call. W2 did not introduce a placement policy; it made incremental arrival reach the placement that batch arrival already had. That is the strongest available statement that this is a bug fix, and it is why only the cases that enter the scheduler many times moved.

W3, and a defect the fidelity matrix cannot reach

A monolithic Replica runs prefill and decode on the same attention-DP lanes, so one forward step can hold a prefill batch on one lane and a pure-decode batch on another. Those two phases were given disjoint waiting rooms and disjoint open-step binding tables. Each room waits for attn_dp members; neither reaches it; and the idle-lane filler cannot rescue either one, because each sibling stage is busy holding the other phase. The forward never dispatches.

This needs more than one attention-DP lane, which is why no shipped recipe has ever hit it — and also why the 71-case matrix cannot prove anything about it. Same constraint as the W2 coverage limit above: the public MoE wrappers require ATTN_TP == MOE_TP * MOE_EP while the runtime requires attn_tp * attn_dp == moe_tp * moe_ep, and those have no common solution above one lane.

So the primary evidence is the direct-construction fixture that R35-02 asked for. With attn_tp=1, attn_dp=2, moe_tp=1, moe_ep=2 on a monolithic MoE Replica, four requests of unequal length arriving together under chunked prefill, and the real Simulator event loop: on the pre-fix source the run drains its event queue with a non-empty scheduler state. That is the deadlock reproduced end to end, not argued from the source. On this branch the same configuration reaches 24 cohorts, 4 of them genuinely mixed-phase, and completes 4 of 4 requests.

The fix is one lifecycle, not just a shared room

The plan forbids an intermediate state where the rooms are shared but completion is still phase-specific, so this lands as one commit:

• MONOLITHIC allocates one waiting room, bound to the forward name and to both phase names, and resolves its lanes in one "forward" step-id namespace.
• The two near-duplicate per-layer sync entries become one enter_layer_sync(..., mode), mirroring the schedule_layer_wave(mode=...) that already existed.
• A new forward_collective.py completes a cohort once: it advances the decode-phase requests once, restores the full-stage owners once, then runs each live source through the phase helper it already had, so every lane predicts its continuation from its own batch rather than from one sampled batch.
• A request may no longer occupy two non-idle source lanes of one forward, checked where the group is formed.
• A dense layer inside a MoE model labels each source by its own phase and keeps the prefill component ledger to prefill sources.

One audit finding was narrowed by reading the source: _next_step_id_by_replica is keyed by replica alone, so step-id allocation was already Replica-scoped and monotonic across kinds. Only the binding table and the waiting room were partitioned, and only those had to change.

Event priority, which constrained the design

Events order by (time, event_type, id), so the collective event class is a priority. Two things follow: a cohort shape that already works must keep the event type it has today, and the class must not depend on which lane happened to close the room — that would make a mixed cohort's ordering depend on arrival order.

Both are satisfied by choosing the class from the cohort's contents, by the same rule the entry already uses: prefill if any live source carries prefill tokens, decode otherwise. A pure-prefill and a pure-decode cohort keep exactly the type they have now; only the mixed cohort, which could not complete at all, is new. No EventType value was added.

One guard that delegation would have dropped

Each per-phase helper refuses a legacy aggregate synchronization by checking its batch for the wave's lane timings — but only when it pops the room itself. Delegation hands it direct_batch, which skips that branch. The check now lives in the shared completion, once per source, against the marker its own phase writes. This was found while re-reading the delegation before measuring, and is covered by its own parametrized test.

Controls, because a test that cannot fail proves nothing

The suite was run against four trees. Each carries the final source and the final tests and differs by exactly one edit, except the baseline tree, which carries the pre-fix parent in full. Each fails for its own reason:

Tree	Unit	Runtime failure
the pre-fix source	22 of 23 fail	RuntimeError: Sequential simulation ended with non-empty scheduler state
every source continues on one borrowed batch	12 of 23 fail	ValueError: one attention-DP lane cannot occupy two open sync cohorts: replica=0, stage=0, lane=1, layer=1, sync_stage=pre_moe
the decode helper advances layers again	2 of 23 fail	ValueError: Decode post_moe layer counter cannot advance: request_id=1, completed_layer_count=4, total_layers=4
the per-source helper restores owners again	16 of 23 fail	ValueError: operation_id is already queued or active in this stage context: ('shared_layer', 0, 1, 0, 1, 'attention', 'FULL_STAGE_WORLD')

The pre-fix column deserves one clarification, because "22 of 23 fail" could be read as the suite collapsing for an unrelated reason. It is not: the same-phase pairs pass every assertion about the forward itself — one shared identity, one collective event, two per-source continuations — and fail only at the last line, which inspects a shared room the pre-fix cluster has no structure for. The mixed pairs fail earlier, at an empty collective list. The gap between the two groups is the deadlock, isolated.

Selecting a source tree needed care, incidentally: PYTHONPATH alone does not override an editable install, because its meta-path finder precedes sys.path. An earlier control run silently tested the installed package and reported everything passing. The runner now strips that finder and asserts the resolved module path before collecting.

Coverage is 23 behavior tests — every phase pairing including true mixed batches, both arrival orders, unequal source tokens, idle participation, duplicate ownership, successive layers, a decoding request carried inside a prefill batch, dense-layer transitions inside a MoE model, a missing wave marker, and a disabled metrics store — plus the real-runtime fixture, which additionally asserts token conservation, one completion per request, drained waiting rooms, no live stage ownership, and an identical run with metrics reporting on and off.

The matrix, and what a null result means

Baseline 3d47417 against candidate 65ed8a7, both clean detached checkouts with source_dirty=False and no dirty paths, both driven by one harness revision, no filter, clean cache, 71 executed and 426 predictor cache files per side.

71 of 71 compared, 71 identical, zero mismatches, zero provenance findings, zero predictor cache differences.

design.md recorded that expectation before the run, together with the one change that is reachable at a single lane: the decode-layer credit for an already-decoding request carried inside a prefill-mode batch. It predicted that credit would move nothing, because on the monolithic path completed_layer_count feeds only admission guards and diagnostics — its two arithmetic consumers are PD-AF DECODE_ATTN/M2N — and it predicted that if a MoE co-location case did move, that credit would be the cause. The primary prediction held and the conditional branch was not taken.

Stated plainly, since the number invites the wrong reading: 71 of 71 identical is not evidence that the deadlock is fixed. No matrix case can reach a multi-lane monolithic MoE forward at all. The matrix answers only "did anything else move", and the answer is no.

A second defect, found by external review and fixed here

A dense layer in a MoE model completes per source batch, outside the shared forward completion that credits a routed layer to every decoding request in the cohort. The decode handler credited its own batch, but the prefill handler credited nothing, so a request that had finished its prefill and was carried in a prefill-mode batch (chunked prefill mixes them) missed every dense layer. On a MoE -> dense -> MoE model its layer count read 1, 1, 2 instead of 1, 2, 3. On a monolithic Replica the undercount is silent; only the PDD decode role asserts the terminal count.

The fix is one helper, advance_decode_layer, holding the validate-then-credit rule once; the shared forward and decode handlers use it, and the dense-layer completion for a prefill-mode source credits that source's decoding members before delegating. A request still prefilling has no decode layer to credit, and a PREFILL-role batch carries none, so the disaggregated roles are unchanged.

Evidence: a dense layer executed for a mixed source (+1 for the decoder, 0 for the prefiller, pure-prefill control credits nothing); the MoE -> dense -> MoE sequence; and a real-loop hybrid-layer variant (moe_layers_enum="0,2,3") of the W3 runtime fixture that reads credits before the token rollout resets them: 4 mixed-source dense completions, 10 decode tokens credited, every one peaking at 4 of 4 layers. On the pre-fix source the same run peaks four of them at 3 of 4. Commit f7c31e4; report test_report_2026-09-22_review_corrections.md.

Scope kept deliberately narrow

The candidate's decode final-metrics hunk calls a helper that main deleted. Rather than rewrite main's decode timing ownership inside a deadlock fix, it is excluded: introducing a second decode component ledger would move every reachable MoE decode case for a reason unrelated to the defect. The exclusion is recorded as a decision with its rationale in design.md, not left as an oversight.

W4, and a policy that has to prove it is not round-robin

Frontier routed requests to a Replica's attention-DP lanes by round-robin. No vLLM deployment does that. A vLLM V1 frontend picks an engine by scoring waiting * 4 + running over counts it only learns when the coordinator publishes a snapshot, so placement follows a delayed view of load, and the delay itself changes where a request lands.

VllmDPLoadBalancer models the two halves of that mechanism separately, because vLLM separates them:

• The frontend's selection (core_client.py:1139-1153): the waiting * 4 + running score, the scan with a strict < so the lowest-index minimum wins, and the local increment of the chosen engine's waiting estimate, which is what spreads a burst of requests across lanes between two snapshots.
• The coordinator's publication schedule (coordinator.py:195-211): 100 ms while counts are changing, 5000 ms while idle, a 50 ms collection wait before the first snapshot, and the latch that publishes the previous step's counts before the current ones.

Every constant carries its citation in the source. Out-of-order step reports warn and are still applied, as the reference does; nothing here aborts a run. The object creates no events, so a drained simulation still drains.

Off by default, and bounded in the constructor

It is reachable only through --cluster_scheduler_config_type vllm_load_balancing. round_robin remains the default and nothing changes when the policy is not selected. Supported scope is enforced, not documented: one co-location Replica, one pipeline stage, the vllm_v1 replica scheduler, and a report key whose ordering actually holds.

That last guard is the interesting one, and it was settled by measurement rather than by argument. The report key is W3's shared forward identity. A probe at the report boundary showed:

Shape	Keys observed	Verdict
MoE, attn_dp=2	strictly increasing, paired when both lanes are live	usable
dense, attn_dp=2, staggered online arrivals	interleaved — lane 1 step 0 arrives after lane 0 step 2	not usable
either, attn_dp=1	trivially ordered	usable

A MoE Replica resolves one shared forward identity across its lanes, which is exactly what W3 delivered. A dense Replica has no per-forward collective across lanes, so each lane keeps its own creation counter. The constructor therefore rejects dense above one lane and says why. No second step identity was invented to paper over it.

Two seams, and no mutable time bridge

Placement now depends on when it happens, so the time has to reach the policy honestly. BaseClusterScheduler.schedule_at(time) defaults to schedule() and ClusterScheduleEvent calls it; the policy overrides it, and its own schedule() raises rather than routing from a stale snapshot. on_replica_batch_end is an inert default hook called after replica_scheduler.on_batch_end, so a policy reading lane populations there sees the post-step state. get_request_load() delegates to the existing decision-log waiting accessor, so a load balancer and the decision log cannot disagree about what "waiting" means.

The runtime test that would catch a no-op

Each case runs one configuration twice — under the policy and under round_robin — in one child process, with identical durations from the dummy predictor. The discriminating case spreads six arrivals over a second and gives the first request a 40-token decode:

	policy	round-robin
placements	[0, 1, 1, 0, 1, 1]	[0, 1, 0, 1, 0, 1]

Same arrivals, same durations, same lane capacity — and the policy puts strictly fewer requests on the lane still draining the long request. Round-robin cannot, because it cannot see load.

The same runs also record that routing times equal the ClusterScheduleEvent times at five distinct instants, that the report keys are non-decreasing with every equal-key pair carrying two distinct lanes, that every report matches the post-step load while strictly fewer match the pre-step load (6 of 10, 38 of 44, 3 of 7), and that no new event type appears.

Controls, five trees and five distinct failure subsets

The one edit	Fails
ClusterScheduleEvent calls schedule() again	all 3 integration cases
the hook moves above on_batch_end	2 integration; reports_after_the_lane_released_the_batch becomes 0 == 10
WAITING_SCORE_WEIGHT = 1	exactly the weight and boundary tests
select stops reserving the chosen lane	4 unit + 2 integration
the dense-lane guard is deleted	exactly the dense_multi_lane construction case

The unmodified control tree passes 64 of 64, so the harness is sound inside a control tree. One real defect surfaced while running the focused regression set and was fixed rather than explained away: the unit file builds both dense and MoE shapes, which trips the process-global IS_MOE latch when it runs after another MoE configuration, so it now resets the simulation globals around each test.

The matrix, and what a null result means here

71 of 71 cases identical, matching the expectation recorded in design.md before the run. Baseline cdfcdf5 against candidate 10dd474, both clean detached checkouts, one harness revision, clean cache, 426 predictor cache files each, zero provenance findings.

Stated plainly: no matrix case selects the new policy. The matrix answers one question — did anything else move — and all three edits reachable from those paths are inert there. The policy's own evidence is the 61 unit tests, the three real-runtime cases and the controls above.

No placement or timing equivalence with a real vLLM deployment is claimed. IPC latency, more than one frontend, elastic scaling and the coordinator's warm-start phase are deliberately absent, and the report key is not a vLLM step counter — it advances per layer, and only its ordering and equality are used, which is all the reference coordinator uses its (wave, step) pair for.

W5, which is not in this PR, and why

The plan's fifth work package would have separated the MoE routing load distribution from the routing implementation used for timing prediction. It is closed without a source change, and the reasoning is worth stating because the audit originally recorded it as a reachable defect.

The collision needs two clusters in one run that resolve different routing implementations. They cannot. moe_routing_distribution_type is declared once, on ReplicaConfig; ClusterConfig declares no {prefill,decode,decode_ffn}_replica_config_moe_routing_distribution_type field, so get_field_value("moe_routing_distribution_type") (frontier/config/cluster_role_config.py:58-63) always takes its miss and returns the global value, and the generated CLI exposes exactly one flag for it. Verified on this branch and on main. The audit had read the get_field_value lookup as evidence of a declared override; that was wrong, and review.md now carries the correction beside the original row rather than in place of it.

So the fix becomes live code only after first adding the per-role override that makes the collision possible — and then a registry routing axis to defend against it. The current contract is kept instead: one global distribution, one derived routing path per run.

What the module decides is narrow: which moe.csv rows train moe_gating_routing_topk, that is, whether that one operator's predicted cost comes from the fused-topk kernel or from the profiler's uniform round-robin routing path (frontier/profiling/moe/moe_impl.py:75-103, :195-225). It never touches expert load, grouped GEMM, EP synchronization or scheduling.

Its magnitude was measured rather than assumed, on the three checked-in datasets that carry both paths. Matched-feature moe_gating_routing_topk medians, standard / uniform:

Dataset	standard_fused_topk	uniform_topk	Difference as a share of summed per-layer operator medians
a800/qwen3-a3b-30b-moe (259 matched rows)	0.073 ms	0.189 ms	3.1% median, up to 24% at the smallest token counts
h800/Phi-tiny-MoE-instruct	0.050 ms	0.082 ms	7.3-7.8%
h800/step-moe-noquant-small	0.041 ms	0.072 ms	4.2-5.4%

That is the cost of the existing mapping selecting the wrong kernel for a scenario, which is why the mapping stays. It is not a gain attributable to the reverted override: with the field unset every configuration resolved exactly as before, so its fidelity effect was zero by construction.

The drafted implementation — the global field, three per-role overrides, four CLI flags, a single-owner resolver, a routing-aware training signature and family gate, and a (model_name, identity, routing_runtime_path) registry key with legacy-conflict rejection — was reverted before any commit and archived as w5_reverted_moe_routing_runtime_path.patch under the task directory.

W6, and a profiling measurement that was missing two kernels

_run_fused_moe_iteration is the profiling path Frontier takes whenever vLLM exposes its low-level fused-MoE API. It ran the first expert GEMM, then handed intermediate_cache1[:, :E] — the gate half of the gate | up projection, unactivated — straight to the second GEMM, and never reduced the per-expert outputs. Two operations were absent: the gated activation and the local top-k reduction.

Reachability was checked before writing any code

The W5 revert set a standard: establish that a defect is reachable from a released configuration, and measure its size, before adding anything. Both checks were run here first.

All five names the low-level branch imports resolve in the pinned reference vLLM v0.10.2, and environment_profiling.yml pins vllm>=0.10,<0.11, so the documented profiling environment takes this path. Worth recording plainly: no environment on the development host reproduces it, because both local Torch environments carry newer vLLM and select the functional fused_experts branch, which was always correct.

The two missing kernels are memory-bound elementwise work, so their cost was estimated as bytes / (peak HBM x 0.80) and compared against the measured moe_grouped_gemm medians in the checked-in datasets:

Dataset	At its largest profiled token count	Over all rows, median / max
a800/qwen3-a3b-30b-moe	4096 tokens: 0.934 ms measured, 0.185 ms missing → 16.5%	6.8% / 26.3%
h800/Qwen3-30B-A3B-tiny	64 tokens: 0.130 ms, 0.0018 ms → 1.4%	0.2% / 1.4%
h800/step-moe-noquant-small	64 tokens: 0.337 ms, 0.0036 ms → 1.1%	0.1% / 2.2%

The error scales with token count: negligible at decode batch sizes, material at prefill and chunked-prefill sizes. Well above the 0.5% bar, and the repair adds no configuration surface at all, so it was implemented.

The repair, and two deliberate departures from the recorded plan

torch.ops._C.silu_and_mul now writes into a dedicated activation buffer, that buffer feeds the second GEMM and the FP8 quantizer, and moe_sum reduces into a preallocated output. Both new buffers are allocated once at the call site, outside the timed step. Operand for operand this is vLLM's own fused_experts_impl.

No gated/non-gated branch was added. profile_fused_moe_kernel only ever materializes w1 with 2 * E rows, so a conditional would be unreachable code.

Two adaptations recorded during the audit were not followed, and the reasons are in review.md rather than left implicit:

• The _custom_ops import went inside the low-level try, not outside it. fused_moe.py imports _custom_ops at its own top, so it cannot fail where the low-level API succeeds and cannot perturb the two-branch detection. Placing it there additionally means a build lacking it selects the functional path instead of running an incomplete computation.
• The existing SiluAndMul wrapper was not reused. Its forward calls torch.empty on every invocation, which would put an allocation inside the timed region and change what moe_grouped_gemm measures.

What the operator now means

moe_grouped_gemm is the complete local expert computation: first GEMM, gated activation, optional activation quantization, second GEMM with the routing weights, and the local top-k reduction. That reduction is a per-token sum, not a collective, so no DP/TP/EP communication cost was added or removed.

This converges the two backends rather than splitting them. vLLM's fused_experts returns reduced hidden states, so the functional path always included both steps; the repair removes a scope disagreement that existed on main. MOE_FAMILY has no operator for the reduction, so counting it here counts it exactly once, with no fifth operator, no new column and no new trained model.

Recorded because it differs: the Frontier-instrumented reference vLLM places moe_sum outside its own record_function("moe_grouped_gemm") scope. That finer split is diagnostics; Frontier's operator of the same name means the whole local expert computation.

Artifact identity: documented, not encoded

Nothing in a profiled row says which version produced it. resolve_grouped_gemm_backend returns vllm_fused for both the low-level and the functional vLLM path, and profiling_patch_tag turns out to hold three historical free-text values in one CSV with nothing in the source writing it. Nothing in frontier/ reads the backend column; one test asserts the MXFP4 label.

The maintainer's decision was to leave the metadata alone and state the limit instead. docs/profiling/README.md now records what the operator measures, how large the pre-repair gap was, and that a row cannot be checked for completeness from its own columns — so the remedy is to re-profile, not to infer.

CPU validation, and what it deliberately does not prove

tests/unit/test_moe_fused_expert_arithmetic.py, 7 tests, all passing. It replaces the native calls with plain-Torch references, so it validates the composition, not native numerics, and its docstring says so. It covers the full path against a written-out gated-SwiGLU reference, the call order GEMM1 → silu_and_mul → GEMM2 → moe_sum with routing weights on the correct GEMM, the reduction's shapes, the FP8 quantizer's input, workspace reuse across profiling steps, and the allocation-site dimensions of all four buffers under TP=2.

One of those tests exists only to make the others falsifiable: it computes the old slice-only arithmetic and asserts the repaired result differs, so the reference-equality test cannot pass against the unrepaired path.

Regression was measured against a detached worktree at the pre-change HEAD over the same four existing files: 1 failed / 136 passed before, 1 failed / 143 passed after. The single failure is identical on both sides and needs an environment with Torch but without vLLM, which no local environment provides. The default-environment suite is unchanged at 84 failed / 3778 passed.

No fidelity matrix, deliberately. The changed module requires Torch and the simulator environment has none, so the simulator cannot import it; the matrix consumes checked-in CSVs rather than fresh profiling. This repair changes what a future profiling run measures. It changes nothing about a simulation run from existing data.

Native parity, because a mock cannot settle this

tests/integration/test_moe_fused_expert_numerical_parity.py drives the repaired iteration with the same buffer shapes, kernel config and block alignment that profile_fused_moe_kernel uses, then compares the output tensor against vLLM's own fused_experts at rtol=0, atol=0. Both sides run in one process on one GPU over identical activations, weights, routing weights, routing ids and expert map.

Case	What it settles
Qwen3-A3B-30B shapes read from the checked-in model config, 4096 and 4097 tokens, EP ranks 0 and 1	The production-shaped case at the token range where the omission cost the most; 4097 exercises the padded tail; two ranks cover global-to-local expert id remapping without an EP cluster
257 tokens, 16 experts, top-k 2 and 4, popularity-weighted routing	A different valid top-k and uneven occupancy, with two local experts asserted to receive no tokens at all
Repeated invocation with different inputs	Workspace reuse cannot leak a stale result
FP8	The real kernels accept the gated activation and return finite output; a structural check, not a reference comparison

The module skips unless CUDA is present and the build reports the low-level API, which is the only configuration that selects the repaired path.

It runs on an H800 worker under the official vllm/vllm-openai:v0.10.2 image, because no local environment selects that path. Result: 8 passed in 13.70 s — seven reference comparisons at rtol=0, atol=0 and one FP8 structural check (gpu-h800-0110, torch 2.8.0+cu128, vLLM 0.10.2, VLLM_API_VERSION=0.10.x). The repaired profiling path reproduces vLLM's fused_experts output bit for bit on the production shapes, on the uneven-occupancy boundary case, and across repeated invocations.

FP8 numerics remain unproven and are recorded as such rather than claimed: the FP8 case checks that the real kernels accept the gated activation and return finite output, not that the quantized arithmetic matches. Frontier quantizes with its own helpers, so a bit-exact FP8 comparison would first require matching those schemes.

The 2026-09-22 external review also found that the FP8 case passed block_dims but not block_shape, so the kernels, as run, read the block-quantized scales through their per-tensor path rather than the production invocation profile_fused_moe_kernel uses. The test now forwards block_shape, and two CPU tests pin that both GEMM invocations receive it (ca1b9b6). The corrected FP8 case was re-run on an H800 under codesign (exp-0922-202645-561899, gpu-h800-0095, same image and launcher): 8 passed in 14.27 s, exit 0, with block_shape=[128, 64] reaching both GEMMs; it is still a structural check, so FP8 numerics remain unproven. Recorded in 9df18d1. The same review corrected a scope claim: the low-level path aligns blocks before the timed step while vLLM's functional entry aligns inside fused_experts, so the two backends time different envelopes, which the profiling guide now states.

W7, an empty collective that the backend refused

The specification-time uncertainty resolved in one direction first: the candidate's submodule commit e564935d is unpublished, not inaccessible. fwyc0573/frontier-htsim is public and readable, its only branch main is exactly the gitlink this branch pinned, that commit returns HTTP 422, and no local object store contains it. So the fix had to be written, not fetched.

Three defects in the published runner, each confirmed by running it rather than by reading it:

• An explicit zero payload is rejected as a missing field. tensor_bytes = 0 and a deleted field produce the identical missing required fields: ['tensor_bytes'].
• A negative payload is not rejected; -1 passes validation and reaches flow generation.
• An explicit command-line zero loses to a positive value in the scenario file.

This is reachable from Frontier. _validate_data_size accepts zero, and the EP all-to-all payload is embedding_dim * 2 * routed_tokens, which is zero for a lane with no routed tokens in a step; predict_reduce_scatter floor-divides by the device count and reaches zero the same way. Frontier cannot fix it alone, because a zero-byte collective still costs the intra-server synchronization latency — short-circuiting to 0.0 here would change the backend's semantics rather than accept the input.

The repair, and where it lives

Entirely in the companion repository: fwyc0573/frontier-htsim#1, branch fix/zero-payload-input-handling, commit eb7bc4f, also draft.

tensor_bytes now merges through the helper that treats only None and "" as unset, instead of the one that reads 0 as missing on both sides. The required-field check became a table carrying, per field, whether zero is a legal value — zero stays "missing" for collective_type, domain_dims, topology, nodes, gpus_per_server and tp, where the argparse defaults genuinely use it to mean absent. A negative payload is rejected in the runner and again in Scenario.validate(), so a serialized scenario cannot smuggle one past the schema. Nothing in flow generation, topology modelling or latency arithmetic changed.

Frontier's side of this package is the gitlink and one test. No Frontier source file changes — which matches the candidate branch, whose own collective_sim_cc_backend.py is byte-identical to this one's.

Two negative controls, because a gitlink bump is easy to mis-verify

Suite	With the fix	Against the pre-fix sources
Companion tests/test_zero_payload_input.py, 9 tests	9 passed	6 failed, each reporting missing required fields: ['tensor_bytes']
Frontier tests/unit/test_collective_sim_zero_payload.py, 4 tests	4 passed	3 failed, same error, with the gitlink moved back to b8518af

The Frontier tests use the canonical TP=4 x DP=2, EP=8 pod from AGENTS.md priced with the analytic NVLink model. An empty all-to-all and an empty reduce-scatter each keep the payload-independent 7 x 0.5 us = 0.0035 ms synchronization term; a 1 MiB all-to-all costs 0.0060486 ms; a negative payload still raises from Frontier's own guard. The module skips when the optional submodule is not initialized and built, which is the default.

A fresh clone confirms the gitlink resolves from the published remote rather than from anything local: the module skips, git submodule update --init checks out eb7bc4f from GitHub, make -j builds, and the four tests pass.

A governance scan the bump exposed

Initializing the submodule took tests/unit from 84 failures to 85. Diffing the FAILED lists named one: test_raw_model_profile_resolution_callsites_are_allowlisted ast.parses every file under frontier/, and the submodule adds 38 vendored files, one of which raises IndentationError under Python 3. Two sibling scans walked the same tree and tolerated it while silently measuring vendored code as Frontier's own.

tests/frontier_sources.iter_frontier_sources() now yields the 413 Frontier-owned files and skips the vendored subtree; all three scans use it. The suite returns to 84 failed / 3782 passed, with an empty diff against the baseline FAILED list in both directions and the four extra passes accounted for by the new module.

This was latent, not caused by the fix: it would have hit anyone who followed the AGENTS.md instruction to initialize the submodule.

One thing to carry into the merge

.gitmodules still records branch = main, which is right for the state after the companion PR merges. git submodule update --init — the documented command — uses the recorded gitlink and lands on eb7bc4f. git submodule update --remote would follow main and drop the fix. Re-point the gitlink at main once the companion PR merges.

The combined regression, and what it does and does not prove

The suites were run together on the integrated branch rather than package by package, on d881357, 32 commits ahead of the base. origin/main was re-fetched first and is still 1f694f7, an ancestor of this head, so no integration merge was needed and none was made.

Check	Result
pytest tests/unit -q --continue-on-collection-errors	84 failed, 3782 passed, 49 skipped, 11 errors
pytest tests/integration -q --continue-on-collection-errors	15 passed, 22 skipped, 5 errors
All 16 release-supported architecture example scripts	16 passed, 0 failed
Four PP=2 cases	4 passed
Trained-predictor path, cold then warm	both pass, byte-identical metrics

The gate for the unit suite is the FAILED set, not the count. Diffed against the recorded origin/main baseline list it is empty in both directions: nothing that passes on the base fails here, and no baseline failure was silently repaired. The 11 collection errors are modules that import torch or matplotlib, which the simulator environment does not carry; they error identically on the base.

On the integration side the one added skip is this branch's own W6 parity module, which skips without a GPU. All 5 errors are test_pdaf_reference_lifecycle_observer.py reporting the same absent pinned PD-AF Reference checkout; they are environmental and identical on the base.

Pipeline coverage, because the new policy is PP1-only

VllmLoadBalancingClusterScheduler refuses anything but one pipeline stage, so it cannot reach the multi-stage path by itself. Lifting that guard is planned as Step 9 (PP>1 placement checked against vLLM's engine-iteration state at the scheduler level, with a CPU reference-loop oracle); the plan was corrected against the 2026-09-22 external review and execution has not started. The cluster-scheduling, stage-dispatch, and metrics code it shares with everything else does run there, so four PP=2 runs cover it: co-location dense offline and online at TP=2, co-location MoE at Attn_TP=4, MoE_TP=2, MoE_EP=2, and sequential PDD dense with both roles at PP=2. Each script echoes its resolved topology and the logs confirm the intended values.

The predictor cache, measured cold on purpose

The checked-in CSV smokes run against a repository cache/ that is already populated, so on their own they only ever measure the cache-hit path. The same simulation was run twice against an empty scratch cache through --metrics_config_cache_dir, leaving the repository cache untouched. The cold run took 26.9 s and wrote 63 predictor artifacts; the warm run took 2.1 s and wrote none; their request_metrics.csv outputs are byte-identical. The persisted-cache path therefore reproduces the freshly trained path exactly.

What this does not prove

CPU only. No native profiling suite was run here, and no vLLM serving or TTFT comparison was performed — it is out of scope for this PR and is not needed to accept it. The PD-AF Reference-checkout tests could not run on this host. Apart from the two CSV smokes, the example runs use dummy execution time, so they validate structure, lifecycle, and conservation rather than latency accuracy.

One pre-existing defect found and deliberately not repaired

AGENTS.md under "Tests" names comm_backend_tests/ and debug/, and tells a reader to start with two scripts under tests/debug/e2e-level/monolith_mode/scripts/. That tree exists neither on this branch nor on main. The same missing tree is why 10 of the 84 baseline unit failures are in test_colocation_release_review_contracts.py, which resolves paths under it, and a docstring at frontier/scheduler/replica_scheduler/vllm_v1_engine_replica_scheduler.py:16 still points into it.

This is one pre-existing defect class from the release scrub that removed tests/debug/, unrelated to Issue 26. Repairing it properly means deciding whether the co-location review contracts should assert against shipped scripts or against a restored tree — a decision about the published test surface, not a documentation tweak. Making the doc read correctly while the contract test still fails on the same paths would hide the gap. It is reported in future.md and left alone; the PP=2 coverage was taken through the example scripts instead.

What the audit established

Every defect was confirmed against main at pinned revisions, not assumed from the candidate's commit messages. The vLLM reference was pinned and shown to be a direct descendant of upstream v0.10.2, with the files defining DP placement and count publication byte-identical to the tag.

One audit finding was later corrected against the source: forward_sync_state does not partition the step-id namespace by synchronization kind. Allocation is already Replica-scoped and monotonic; what is partitioned is the open-step binding table. The W3 defect is unchanged but the required change is narrower. The correction is recorded in review.md rather than quietly applied.

Review material

Tracked under task_memory/task_2026-09-21_issue26_correctness_pr/:

File	Contents
plan.md	The execution specification, with an amendments table
requirements.md	The original requests and every explicit decision
progress.md	Per-step status and the chronological record
review.md	Dispositions, the remediation table, corrections, the maintainer decisions, and the final diff review with its method stated
validation.md	Every measurement with its provenance and limits
future.md	The two deferred items
test_report_2026-09-22_review_corrections.md	Verification of the 2026-09-22 external review corrections (C35-01, C35-03, C35-04) and the records-only findings; review.md carries the finding-by-finding disposition
summary.md	The completion archive: deliverable paths, observed results, and what was not validated

Per-work-package reports sit alongside them, one per fix, plus the three pinned-source audit reports. Logs are not pasted into this description; the reports carry the commands and the numbers.

The implementation commits, as opposed to the record commits:

Fix	Commit
W2	6ab521d fix(scheduler): keep round-robin DP rotation across scheduling calls, with ceac2b4 strengthening its tests after review
W3	65ed8a7 fix(scheduler): give a monolithic Replica one shared forward across phases, with f7c31e4 crediting decoding requests at a dense layer inside a mixed batch (review finding C35-01)
W4	10dd474 feat(scheduler): add an opt-in vLLM-style DP request placement policy
W5	none; closed without a source change in de2bee8
W6	7269bac fix(profiling): complete the legacy fused-MoE expert computation, 697f219 its native parity test, 79f599a the identity limits in the profiling guide, ca1b9b6 the FP8 block_shape wiring and the optional-torch skip (C35-02..04)
W7	1b95187 fix(cc_backend): accept an empty collective through the collective-sim backend, with beded3c narrowing the governance scans the gitlink bump exposed
Docs	d881357 completes the cluster-scheduler list in AGENTS.md

Status

Draft. Do not merge before review.

Technically complete, independently of GitHub's draft status: every work package is closed, the combined regression is recorded, and the final diff review is written up with its method stated — it was a self-review, not an independent one. W5 is closed without a source change. W6 landed with its native run passing 8 of 8 on an H800: seven reference comparisons at rtol=0, atol=0 plus one FP8 structural check; the FP8 wiring correction from the 2026-09-22 external review was re-run on an H800 and passed (8 of 8), still as a structural check. The external review's other findings against this PR (C35-01..05) are fixed and dispositioned in review.md; its Step 9 plan corrections (P9-01..06) are applied to the plan only, a second plan review against the maintainer's core-module quality gates is recorded in plan.md §18.12 (9df18d1), and Step 9 has not started. W7 landed with its companion draft PR open at fwyc0573/frontier-htsim#1.

Two items are carried forward in future.md: re-point the collective-sim gitlink at main once the companion PR merges, and the pre-existing tests/debug/ pointer defect described above.

Merge order: the companion PR first, then re-point this branch's gitlink at its main, then this PR. Issue #26 stays open.