fix(scheduler): stage admission deadlock with attention-DP lanes under pipeline parallelism - #36
Conversation
…n deadlock A MoE replica with more than one attention-DP lane and more than one pipeline stage drains its event queue with requests unfinished. Stage admission mints a ticket per arriving batch and admits only the strict FIFO head; at PP>1 a lane holds several queued tickets while consuming one, so the head can belong to a lane that is busy inside the sync room, and the room in turn waits for the lane the FIFO refuses. Dense completes but serializes its lanes. PP=1 and capacity-1 contexts are unaffected. The records diagnose this from source on main 1f694f7, compare four options, and recommend ordering only exclusive operations: a full-stage ticket waits only for an EP wave queued ahead of it. The lane-aware alternative was rejected because the DES wakes sibling lanes at release, not at a peer's acquisition. No source change in this commit. The task records are tracked through the same narrow .gitignore exception the Issue 26 branches use, so they can be reviewed on the remote.
Each of the ten review findings was re-checked against main 1f694f7 and its disposition recorded in review.md. The records are corrected; nothing was executed and no source file changed. - design.md: cite the MONOLITHIC/PREFILL admission loop; state the drain as a queued-ticket arrangement rather than a shape; note that queued EP waves exist only on DECODE_FFN; remove the admitted ticket with remove(ticket); keep option A rejected on design grounds with its stall trace marked unverified; restate the capacity-1 and PP=1 expectations as caller-level conditions; narrow the queue bound; add the mixed-phase scope boundary; withdraw the unmeasured "lanes serialized" label. - plan.md: C1 targets confirmed deadlock witnesses from a prefill-only, phase-controlled group; C3 uses the stage-batch ledger with half-open overlap duration; C4 takes the reviewer's wording; P0 lists its artifacts and outcome classes; P2 covers both sides of the EP boundary, a DECODE_FFN dense-group control and a second admission round; P3 has separate unchanged, liveness and timing paths; the case list is published on the analytical backend and keeps attn_dp=2, PP=3 (decisions D-6, D-7). - requirements.md, progress.md, review_prompt.md: record the request, the state, and the round-2 reviewer prompt.
…er PP Runs the case list of the stage-admission-ordering plan (section 4) one child process per case: synthetic MoE/dense shapes, the release recipes, and the vLLM-aligned shapes. Each case records its inputs, run provenance and one outcome artifact (metrics hashes, a live drain-state report, or the error). The drain classifier checks the admission-deadlock signature from the live stage contexts, lane queues and sync rooms. The compare command applies the unchanged / repaired-liveness / timing paths, using the stage-ledger lane-overlap metric.
JSON turns integer dict keys into strings, so a state_report.json read back from disk could not be indexed by stage id the way the in-process report is. A list indexed by stage id has the same shape before and after the round trip.
…lelism vllm_burst_driver.py runs inside the vllm-openai v0.10.2 image: it builds the instrumented overlay from the vLLM-BS checkout, accepting it only when the files that differ from the image are exactly the fork's own changes, then drives DP=2, PP=2 bursts of prefill-only requests pinned to rank i mod 2. run_vllm_worker.sh is the 4-GPU worker entry point for the MoE and dense scenarios. compare_lanes.py turns the vLLM per-forward traces and the Frontier G7 stage ledgers into the same lane metrics (completion, per-lane sequences, stage-0 pairing, co-start and co-execution) and writes the workflow-gap table for plan criterion C7.
StageExecutionContext admitted a ticket only from the head of its ready FIFO. Under pipeline parallelism an attention-DP lane that is still busy can queue its next batch at a shared stage, and that ticket then refuses the idle lane's queued batch. With every lane waiting on the head, the sequential run ends with work left (admission deadlock), and dense runs serialize the lanes instead of overlapping them. A full-stage ticket may now be admitted ahead of earlier queued full-stage tickets, but never ahead of an EP wave queued before it. EP waves keep the strict head rule, so they still wait for every operation queued earlier, and the forward-group seal is unchanged. Tests: contract tests for the bypass, both sides of the EP boundary and capacity 1; a DECODE_FFN control that keeps dense groups in counter order around a queued EP wave; a two-lane drain through promotion and restore; and simulator cases that complete the MoE admission-deadlock witnesses and start both dense lanes in the first forward. On the base revision the bypass, capacity-1, drain and simulator tests fail as recorded in the task's negative controls.
The ground-truth checkout passes a fifth renormalize argument to _moe_C::topk_softmax, while both the checkout's own csrc and the v0.10.2 image declare the four-argument op, so the MoE scenario failed in profile_run. The overlay step now takes an optional unified diff, applied after the fork-change check accepts the overlay, and records its SHA-256, the files it touched, and whether each now equals the image's copy. Hunks are applied as exact text replacements because the worker image need not carry patch or git.
Adds the test report, the calibration case for the vLLM comparison (two GPU runs, the recorded four-argument topk_softmax overlay patch, and the workflow-gap analysis), and the selected evidence: base negative controls, G2 identity comparisons, the path-T explanation and the Step 9 probe. Progress, requirements (R-7) and plan are updated. Two plan stop conditions are open for the user: the C3 witness rule at attn_dp=4, and V5 co-execution on the dense vLLM shape.
Contention witnesses now pass on a strictly larger co-execution fraction (multi_lane_busy_time / busy_time). At attn_dp=4 the fix admits all lanes together and shortens the busy period, so the absolute overlap time fell while the overlap became complete. V5 of the vLLM comparison gates the MoE shape only. vLLM's dense DP ranks meet once per forward and vary in duration per rank (co-execution 0.54-0.93 across rounds), which the dummy predictor does not model; the dense value is reported as INFORMATIONAL and the status counts only MISMATCH rows.
Records the user's decision to judge contention witnesses by the co-execution fraction and to gate V5 on the MoE shape only (plan D-9, requirements R-8), the reruns of both comparisons under those rules (no STOP, no MISMATCH), the updated test report and calibration case, and the completion summary.
The test report cites evidence/base_negative_controls.log, but the repository-wide *.log ignore rule kept it out of the tree.
fwyc0573
left a comment
fwyc0573
left a comment
There was a problem hiding this comment.
Review round 2 at ecff89a against merge-base 1f694f7. Review only; no fixes applied in this round.
I found no production-path break in the core rule. The only production caller, replica_stage_schduler.py:328, checks owns() before try_acquire. The three touched unit files pass on the branch (180 passed).
Findings, most important first:
| Id | Area | Topic |
|---|---|---|
| R2-01 | stage_execution_context.py |
try_acquire on an active full-stage ticket now raises ValueError where base returned False (reproduced) |
| R2-02 | compare_lanes.py |
The D-9 rationale for ungating dense V5 is not fully supported by the committed decomposition |
| R2-03 | stage_admission_matrix.py |
No before/after case covers PDD or online with attn_dp>1, PP>1 |
| R2-04, R2-05 | compare_lanes.py |
Placement "ok" ignores unseen requests; V1/V4 fold the base negative control into the vLLM verdict |
| R2-06 | .gitignore |
The task-directory exception would publish records into main on merge |
| R2-07, R2-08 | stage_admission_matrix.py |
No child timeout; the shared work/ directory breaks concurrent sets |
| R2-09, R2-10 | vllm_burst_driver.py |
Overlay file-list order mismatch; fragile apply_patch parser |
| R2-11 | stage_execution_context.py |
Double scope branch and double FIFO pass (simplification) |
| R2-12, R2-13 | contract / design | Capacity-1 ordering contract change; the FIFO no longer orders shared-lane contexts |
| R2-14, R2-15 | task evidence scripts | Hard-coded worktree and scratch paths in the probes that #35 will rerun |
| break | ||
| if queued.scope == EP_WAVE: | ||
| return False | ||
| self._ready_fifo.remove(ticket) |
There was a problem hiding this comment.
R2-01 · correctness · confirmed
If a full-stage ticket is already active, it is no longer in _ready_fifo. The scan then runs off the end and self._ready_fifo.remove(ticket) raises ValueError: StageAdmissionTicket(...) is not in deque. On base, the same call returned False.
_validate_ticket only rejects tickets that are neither queued nor active. So the claim "the scan always ends at the ticket or at an earlier wave" misses the active case.
Reproduced on this branch:
c = StageExecutionContext(replica_id=0, stage_id=0, ep_size=1, full_stage_capacity=2)
t = c.enqueue_full_stage(operation_id="a")
c.try_acquire(t) # True
c.try_acquire(t) # base: False; branch: ValueError ... is not in dequeThe only production caller, replica_stage_schduler.py:328, checks owns() first, so today only direct callers and tests hit this. Still, a public predicate that used to answer "no" now crashes with a message that does not say the ticket is already active.
Suggestion: either make the loop's fall-through explicit (for ... else: return False), or state the "queued, not active" precondition in the docstring and raise a named error for an active ticket.
| # Dense DP ranks meet once per forward and then vary in duration per rank, | ||
| # which the dummy predictor does not model; their co-execution is reported, | ||
| # not gated (plan §4.7 V5, D-9). MoE ranks stay aligned by EP collectives. | ||
| CO_EXECUTION_GATED = {"moe": True, "dense": False} |
There was a problem hiding this comment.
R2-02 · validation evidence · confirmed
D-9 makes dense V5 informational on the grounds that vLLM's dense spread "comes from per-rank duration variance ..., not from admission" (this comment and the PR body). The committed decomposition does not fully support that.
In analysis/co_execution_decomposition_sa-pp-20260923b.json, start offsets exceed end offsets in 3 of the 6 dense rounds. The script calls start offsets "where admission could act".
| round | start (ms) | end (ms) |
|---|---|---|
| dense/n8/r1 | 1.669 | 0.260 |
| dense/n16/r0 | 1.017 | 0.653 |
| dense/n16/r1 | 3.596 | 0.599 |
Equalizing durations does not close the gap either. M5_equal_durations is 0.832, 0.740, 0.667, 0.911, 0.716 and 0.898, never near Frontier's 1.0.
The test report notes that forward_start_ts is taken before the per-forward DP metadata exchange. That may explain the start offsets, but the rationale here and in the PR body states duration variance as the whole cause.
The script also has a bug. decompose_co_execution.py says "union minus overlap equals |Δstart| + |Δend|", which holds only for overlapping pairs; a disjoint pair is overcounted.
Suggestion: either gate dense V5 with a bound derived from vLLM's own spread, or restate the rationale as "two sources", with the start-offset part attributed by evidence, for example by measuring start after the DP exchange.
| else: | ||
| raise ValueError(f"unknown arrival process {case.arrival!r}") | ||
| return SimulationConfig( | ||
| simulation_mode="offline", sys_arch="co-location", |
There was a problem hiding this comment.
R2-03 · coverage gap · plausible
Every before/after case is offline co-location: simulation_mode="offline", sys_arch="co-location", and build_state_report reads only the MONOLITHIC scheduler.
The rule also applies to PDD PREFILL and DECODE contexts, where stage_contexts.py sets capacity to replica_dp_size. It applies to online runs too. The G1 recipes that do cover PDD and online all use PP=1, so they fall in the unchanged set.
No case therefore exercises the changed path on PDD or online with attn_dp > 1 and PP > 1. For example, a change in PREFILL-to-DECODE handoff order would not be caught. I have not checked every guard on whether PDD accepts that shape today.
AGENTS.md (Development Gates) asks for a matrix covering "dense/MoE, offline/online, varied request lengths and counts, varied QPS, feature toggles, and model configs".
Suggestion: add at least PDD attn_dp=2, PP=2 cells and online co-location dp2-pp2 cells, with the same conservation and deadlock classification. If PDD rejects the shape, record the rejection as the result.
| if record["kind"] == "engine_iteration": | ||
| for request_id in record["scheduled_new_req_ids"]: | ||
| scheduled_by[request_id].add(record["engine"]) | ||
| misplaced = sorted(rid for rid, rank in pinned.items() if scheduled_by.get(rid, {rank}) != {rank}) |
There was a problem hiding this comment.
R2-04 · validation tooling · confirmed
vllm_placement treats a request with no engine_iteration record as correctly placed (scheduled_by.get(rid, {rank})). ok ignores unseen, so missing or empty placement logs still give vllm_placement_ok: true.
run_vllm_worker.sh itself notes that engine cores write placement records only at interpreter exit. That exit step is skipped if VLLM_WORKER_MULTIPROC_METHOD=spawn is dropped or timeout 1500 kills an engine; dp_placement_157.jsonl already has 0 rows. In either case every request becomes unseen and the status still says PASS.
Suggestion: "ok": not misplaced and not unseen.
| "frontier_after_outcome": new["outcome"], | ||
| "frontier_after_placement_ok": new.get("placement_ok"), | ||
| "vllm": vllm_metrics} | ||
| base_control = (base["outcome"] == ADMISSION_DEADLOCK) if model == "moe" else (base["outcome"] == SUCCESS) |
There was a problem hiding this comment.
R2-05 · validation tooling · confirmed
The V1 MATCH condition includes the base negative control: base == admission_deadlock for MoE, base == SUCCESS for dense. The dense V4 condition does the same (base_m4 >= CO_START_BOUND).
Consequence: once this lands, rerunning C7 with --before set to a fixed main makes every MoE V1 row and every dense V4 row MISMATCH, owned by stage_execution_context.py. That happens even when Frontier-after matches vLLM exactly. The planned #35 merge-forward check is one such rerun. The tool passes only when the base still has the bug.
Suggestion: report the negative control as its own row or status, separate from the vLLM MATCH/MISMATCH verdict.
| if not self._ready_fifo or self._ready_fifo[0] != ticket: | ||
| return False | ||
| self._ready_fifo.popleft() | ||
| if ticket.scope == EP_WAVE: |
There was a problem hiding this comment.
R2-11 · simplification · confirmed
The scope is now tested twice: if ticket.scope == EP_WAVE at line 333 for the gates, then again here for the FIFO test. After the loop has already found the ticket, deque.remove(ticket) scans the FIFO a second time, so every full-stage admission makes two passes. For an EP wave, remove always hits the head, where popleft() would do.
Suggestion, for readability rather than cost: fold the FIFO test into the existing scope branches and delete by the index found in the scan (enumerate, then del self._ready_fifo[index]). That gives one branch per scope and one pass. Combined with R2-01, the full-stage branch then reads "scan to the ticket; refuse on an earlier wave; refuse if not found".
| assert context.queued_tickets == () | ||
|
|
||
|
|
||
| def test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket() -> None: |
There was a problem hiding this comment.
R2-12 · contract documentation · plausible
This test pins down a contract change at capacity 1, which is the DECODE_ATTN/DECODE_FFN default. On base, enqueue(full0); enqueue(full1); try_acquire(full1) returned False; now it succeeds.
The PR body says capacity-1 (PD-AF) contexts are unaffected by the defect, which holds. But the fix removes the context's insertion-order guarantee among full-stage tickets there as well. Ordering then rests only on each stage scheduler's own heap.
PD-AF outputs stay byte-identical, which is consistent with each such context having a single full-stage producer today. A second full-stage producer on a PD-AF stage would silently lose cross-producer ordering.
Suggestion: state this in the PR body and in design.md as a contract change ("full-stage order is no longer enforced by the context at any capacity"), not only as "unaffected".
| if ticket.scope == EP_WAVE: | ||
| if not self._ready_fifo or self._ready_fifo[0] != ticket: | ||
| return False | ||
| else: |
There was a problem hiding this comment.
R2-13 · design discussion
Design note, not a defect. enqueue_ep_wave has one caller, the DECODE_FFN M2N path (design.md). So on MONOLITHIC, PREFILL and DECODE contexts the full-stage branch always admits, and the context FIFO no longer orders anything there.
The class docstring already states the relaxed order. What remains is that admission_seq and queued_tickets still describe an ordered queue on those contexts, while admission order there now lives implicitly in ReplicaStageScheduler's heap and _is_busy. The FIFO survives on those contexts only for cancel and stale-ticket bookkeeping. The fix switches ordering off by ticket scope rather than removing the cause, which is busy lanes holding queued tickets.
design.md rejects option C (minting at the admission attempt) because of the stale-drop logic. A variant worth recording is to keep an ordered queue only for EP-wave (DECODE_FFN) work and admit lane work from the lane's head under the capacity, seal and EP checks. If it stays out of scope, a sentence on the queued_tickets property saying it carries no admission order on shared-lane contexts would help the next reader.
| from pathlib import Path | ||
|
|
||
| ROOT = "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering" | ||
| sys.path.insert(0, ROOT) |
There was a problem hiding this comment.
R2-14 · reproducibility · confirmed
ROOT = "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering" is placed first on sys.path, so probe_completion.py imports frontier from that worktree whichever checkout runs it.
C6 is to be rerun against #35 after merge-forward. Run from the #35 worktree, from probe_main import build_config executes sys.path.insert(0, ROOT) before from frontier.simulator import Simulator. While this worktree exists, the probe would simulate this tree and report its 6/6 as the merged result.
Suggestion: resolve the repository root from the running checkout, for example Path(__file__).resolve().parents[N], or rely on PYTHONPATH.
| from pathlib import Path | ||
| from tests.comparison.stage_admission_pp import compare_lanes | ||
|
|
||
| S = Path(sys.argv[1]); BASE = Path('/data/ycfeng/tmp/stage_admission_ordering/base') |
There was a problem hiding this comment.
R2-15 · reproducibility · confirmed
BASE = Path('/data/ycfeng/tmp/stage_admission_ordering/base') hard-codes the author's scratch path, while compare_lanes resolves the base set through matrix_root() / resolve_scratch_root(). With a different FRONTIER_TMP_ROOT, or on another machine, shutil.copy(BASE / cid / 'case.json', ...) raises FileNotFoundError.
Relatedly, synthetic_check.py and the probe_main.py/probe_completion.py pair are reusable checks that #35 is expected to rerun, yet they live under task_memory/ rather than tests/.
Suggestion: use matrix_root(), and move the scripts #35 will reuse under tests/comparison/stage_admission_pp/.
Fifteen findings on ecff89a, each re-checked against source and posted to PR 36 as inline comments. Fixes are deferred by the owner.
Round-2 review R2-01: after dac4e69, try_acquire on a full-stage ticket that is already active ran off the FIFO scan and remove() raised ValueError; the base rule returned False. Each scope now has one branch. An EP wave leaves the FIFO by popleft. A full-stage ticket is found in one pass and deleted by position, and a ticket that is not queued is refused (R2-11). The class docstring now states that full-stage order comes from the lane stage schedulers and that admission_seq records enqueue order only (R2-13). A unit test re-acquires an active ticket and checks the context is unchanged.
Round-2 review R2-03, R2-07 and R2-08. - Cases carry sys_arch, simulation_mode, a Poisson rate and recipe environment overrides. New groups: G8 PDD offline, G9 PDD online, G10 co-location online, G11 PD-AF recipes with PREFILL_PP=2. Dense PDD requires attn_dp=1 and PD-AF DECODE_ATTN requires attn_dp=1, so those cells are unchanged-path controls. On main, MONOLITHIC and PREFILL place incremental online arrivals on lane 0, so G9 and G10 add burst cells that reach several lanes. - The drain state report and the deadlock signature read every cluster type, keyed by cluster. - Each child runs in its own session under --case-timeout (default 600 s); a timed-out session is killed and recorded as other_failure. - run holds an exclusive lock on the matrix root, because every set shares work/<case_id> for byte-identical outputs.
Round-2 review R2-02, R2-04, R2-05, R2-09, R2-10, R2-14 and R2-15. - compare_lanes: V1 and V4 compare vLLM with the after revision only. The base negative controls are their own rows, N1 (MoE base admission_deadlock) and N4 (dense base co-start >= 0.5), reported as HOLDS or LOST, with negative_control_holds in the status. Placement is ok only when no request is misplaced or unseen. The D-9 comment names both sources of dense non-overlap. - vllm_burst_driver: overlay acceptance compares file sets. apply_patch reads an empty hunk line as a trimmed context line, rejects any other unknown hunk line, and keeps every section of a file that appears more than once. - tests/unit/test_stage_admission_pp_tools.py replaces the task-local synthetic_check.py and needs no scratch data. On the ecff89a tools 7 of its 9 tests fail. - decompose_co_execution.py states its identity for overlapping pairs only, counts disjoint and unpaired forwards, checks the pairing against M3, and adds M5 with both starts of a pair set to the later one (derived from where vLLM's DP all-reduce sits). Reran on runs a and b: no disjoint pair; existing fields unchanged. - explain_t_path.py takes the after set as an argument. - probe_main.py no longer puts a hard-coded worktree on sys.path.
Plan section 7, the D-9 (b) rationale restated with both sources of dense non-overlap, test report section 8, dispositions of R2-01..R2-15, the capacity-1 contract and FIFO notes in design.md, the calibration rerun with separate negative-control rows, and the round-2 evidence (path-T explanation, G2 comparisons, tool tests on the old tools). R2-06 is recorded as pre-merge step P6, not executed.
Restore the repository-wide task_memory/ ignore rule from main and untrack task_memory/task_2026-09-22_stage_admission_ordering, so that no task record reaches main (review finding R2-06, plan step P6). The parent task on fix/issue26-correctness-pr keeps the archive copies of the summary and the test report. The matrix module docstring now points at build_cases instead of the untracked plan file.
1 participant
Summary
Pre-existing defect on
main(1f694f7): a MoE replica withattn_dp > 1andnum_pipeline_stages > 1drains its event queue with requests unfinished. Dense completes, but its lanes start one forward apart instead of together. This holds for co-location and for PDD (PREFILLand unifiedDECODE), offline and online.PP = 1is unaffected. Capacity-1 contexts change at the context API (see the contract change below), but every capacity-1 caller in the matrix, PD-AF included, is byte-identical.Root cause:
StageExecutionContext.try_acquireadmitted only the strict_ready_fifohead, and tickets are minted at batch arrival. AtPP > 1a lane holds several queued tickets and consumes them one at a time. The FIFO head can therefore belong to a lane that is busy in the sync room, while the room waits for the lane the FIFO refuses:_can_supply_idle_laneandtry_acquiredisagree about the same lane.Fix (plan D-1, option B)
One method in
frontier/scheduler/replica_stage_scheduler/stage_execution_context.py(+35/−15 againstmain, docstrings included):main(round-2 fix R2-01).Capacity, seal and EP-active checks are unchanged. There is no new field, flag, fallback or wake-up.
Contract change (R2-12). An idle capacity-1 context used to admit full-stage tickets in enqueue order. It now admits whichever one its lane stage scheduler presents, unless an EP wave is queued ahead. Order among full-stage tickets comes from the callers, and
admission_seqrecords enqueue order only; the class docstring says so (R2-13). On the callers this changes nothing measurable: the 10 PD-AF release recipes and the 4 PD-AF recipes withPREFILL_PP=2are byte-identical, offline and online. OnMONOLITHIC,PREFILLand unifiedDECODEcontexts the FIFO only ever holds full-stage tickets, becauseenqueue_ep_wavehas one caller, theDECODE_FFNM2N path.Commits
dac4e69test_stage_execution_context.py,test_shared_forward_group_admission.py,test_mixed_layer_decode_ffn_scheduling.py) and integration (tests/integration/test_stage_admission_pipeline_lanes.py)a054d87,5ade853tests/e2e/stage_admission_matrix.py799ccb4,a1b9819tests/comparison/stage_admission_pp/, with support for a recorded overlay patchaeeca931661bf1try_acquirerefactor (R2-01, R2-11), docstring (R2-13), unit testa8e8d8ae35242fdf7868e,fc34341,4bcd616,ecff89a,1218ba6,7a7c22e4d08c5d.gitignorerestored tomain, task directory untracked. The diff againstmainhas notask_memoryfile.Validation
Full report:
test_report_2026-09-23_stage_admission_ordering.md(§8 for round 2), archived with #35's task records because this branch no longer tracks its own. The Frontier side uses the dummy predictor, so no latency parity is claimed (D-8).PP=1cell, G5, the dense PDD cells, and the 4 PD-AFPREFILL_PP=2recipes.peak_lanes ≤ attn_dp). All 4 contention witnesses raise their co-execution fraction strictly, to 1.0 (from 0.818, 0.375, 0.846 and 0.846).tests/unitandtests/integration: no base-passed test regresses, no new failure or error, skips and collection errors unchanged. The 19 new tests pass.sa-pp-20260923b): 56 rows, 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL, and 4 negative-control rows that hold.The P2 tests were also run against the base rule as negative controls: 7 fail and 2 pass, as the plan expects.
C3 stage-0 detail for the differing cases:
Round-2 groups:
attn_dp ∈ {2,4}×PP ∈ {1,2,3}, densedp1-pp2PP > 1burst cellsdp{2,4}-pp{1,2,3}, Poisson 20/s and burst;dp2-pp2at 5/s and 80/sPREFILL_PP=2Dense PDD requires
attn_dp = 1, and PD-AFDECODE_ATTNrequiresattn_dp = 1, so those cells are unchanged-path controls. Onmain,MONOLITHICandPREFILLplace online Poisson arrivals, one per scheduling call, on lane 0 (#35's W2, fixed there). The burst cells deliver all requests att=0in online mode and carry the online multi-lane coverage here.C7 per check, 3 rounds per cell:
admission_deadlockadmission_deadlockThe negative controls are separate rows since round 2 (R2-05). A base without the defect now reports them as LOST without turning any vLLM row into MISMATCH.
Decisions taken during execution
1109c4f16makes the Pythontopk_softmaxpass a fifthrenormalizeargument, but the fork's C++ binding and the v0.10.2 image declare four. MoE run a failed on this mismatch. Run b applies the four-argument form as a recorded overlay patch,inputs/groundtruth_overlay.patch, SHA-2568d476789…3a9c81. Numerics are unchanged, because Python renormalizes after the call. The vLLM-BS checkout is not modified.multi_lane_busy_time / busy_time. Absolute overlap cannot rise when the fix compresses the busy period, which happens atattn_dp = 4.forward_start_ts. The non-overlap therefore has two sources. The rank that arrives first records its wait as busy time (start offsets). Per-rank durations vary (end offsets, CV 0.10–0.29). Neither is admission: both ranks enter the same forward, and V1–V4 match in every round. With both starts of each pair set to the later one, dense M5 is 0.66–0.98 and MoE 0.988–0.994. This value is derived from where the all-reduce sits, not measured, because the traces have no post-exchange timestamp. The dummy predictor models neither source.PP > 1controls.Open
4d08c5d. The.gitignoreexception is gone and the task directory is untracked, somain's tree gains notask_memoryfile. Earlier commits on this branch still add the directory; a squash merge keeps it out ofmain's history as well.attn_dp=2, moe_ep=2, PP=2) already completes 6/6 here, where the base drains.PP=3withattn_dp=2stays rejected by the node-size rule on the default backends (W9-02). That is outside this fix.Found while executing Step 9 of #35 (W9-01).