From 62d25b94d9b858e70c47262d0fbcf31091a306b5 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 00:31:44 +0800 Subject: [PATCH 01/19] docs(stage-admission): plan and design for the pipeline-lane admission deadlock A MoE replica with more than one attention-DP lane and more than one pipeline stage drains its event queue with requests unfinished. Stage admission mints a ticket per arriving batch and admits only the strict FIFO head; at PP>1 a lane holds several queued tickets while consuming one, so the head can belong to a lane that is busy inside the sync room, and the room in turn waits for the lane the FIFO refuses. Dense completes but serializes its lanes. PP=1 and capacity-1 contexts are unaffected. The records diagnose this from source on main 1f694f7, compare four options, and recommend ordering only exclusive operations: a full-stage ticket waits only for an EP wave queued ahead of it. The lane-aware alternative was rejected because the DES wakes sibling lanes at release, not at a peer's acquisition. No source change in this commit. The task records are tracked through the same narrow .gitignore exception the Issue 26 branches use, so they can be reviewed on the remote. --- .gitignore | 4 +- .../design.md | 210 ++++++++++++++++++ .../plan.md | 104 +++++++++ .../progress.md | 35 +++ .../requirements.md | 53 +++++ 5 files changed, 405 insertions(+), 1 deletion(-) create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/design.md create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/plan.md create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/progress.md create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/requirements.md diff --git a/.gitignore b/.gitignore index 0381dbf2..cffd05ad 100644 --- a/.gitignore +++ b/.gitignore @@ -168,7 +168,9 @@ cache settings.json # Task memory and repair receipts are local-only; do not vendor them. -task_memory/ +# The one exception below is reviewed on its branch, as PR 34/35 do. +task_memory/* +!task_memory/task_2026-09-22_stage_admission_ordering/ repairs/ # Local linked worktrees used for isolated feature implementation. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/design.md b/task_memory/task_2026-09-22_stage_admission_ordering/design.md new file mode 100644 index 00000000..db41e905 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/design.md @@ -0,0 +1,210 @@ +# Stage admission ordering under pipeline parallelism — Design + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-22 | Created: defect restated from source on `origin/main` `1f694f7`, what the FIFO guarantees today, four options, recommended rule with its invariants, fidelity expectation. For review before implementation. | + +All line references are to `origin/main` at `1f694f7`, checked out in +`/data/ycfeng/Frontier/.worktrees/stage-admission-ordering`. + +## The defect, restated from source + +One `StageExecutionContext` owns each physical `(replica, stage)`. All +attention-DP lanes of that stage share it. Each lane has its own +`ReplicaStageScheduler` with its own batch heap and its own `_is_busy` flag, so +**a lane consumes at most one ticket at a time**. + +Admission is a two-step handshake: + +1. `ReplicaStageScheduler.add_batch` (`replica_stage_schduler.py:145-162`) mints + a `StageAdmissionTicket` at batch **arrival** through + `StageExecutionContext.enqueue_full_stage`, which appends it to the shared + `_ready_fifo` (`stage_execution_context.py:188`). +2. `pop_batch_if_not_busy` (`:286-359`) takes the lane's own heap head and asks + the context to `try_acquire` its ticket. `try_acquire` (`:322-343`) refuses + when an EP wave is active, when active full-stage owners already fill + `full_stage_capacity`, when the forward group is sealed, and finally when the + ticket **is not the FIFO head**: + + ```python + if not self._ready_fifo or self._ready_fifo[0] != ticket: + return False + ``` + +`full_stage_capacity` is `attn_dp` for `MONOLITHIC`, `PREFILL` and `DECODE` +(`stage_contexts.py`), so up to one ticket per lane may be active at once: that +is how the lanes of one forward co-own the stage. + +`BaseReplicaScheduler.on_schedule` admits up to `num_pipeline_stages` batches +per lane in one round (`base_replica_scheduler.py:893-901`). At +`num_pipeline_stages > 1` a lane therefore holds **several queued tickets** while +being able to consume only one. The FIFO head can then be a ticket whose own +lane is busy, and every other lane is refused although capacity is free. + +### Observed state at the drain (MoE `attn_dp=2, moe_ep=2, PP=2`, 4 requests) + +Both lanes admitted two batches each. Lane 1 scheduled first. + +| Where | State | +| --- | --- | +| Stage `(0,0)` context | `capacity=2 sealed=False group=0`; active `{seq0: batch 0 (lane 1)}`; FIFO `[seq1: batch 1 (lane 1), seq2: batch 2 (lane 0), seq3: batch 3 (lane 0)]` | +| Lane 1 stage 0 | `busy=True`, heap `[(batch 1, global_id 3, seq1)]` | +| Lane 0 stage 0 | `busy=False`, heap `[(batch 2, global_id 0, seq2), (batch 3, global_id 2, seq3)]` | +| Prefill sync room step 0, layer 0, `pre_moe` | `lanes_present=[1]`, waiting for lane 0 | +| Event queue | empty | + +The wait is circular: + +- Lane 1 holds `seq0`, has bound forward group 0 and sits in the sync room + until lane 0 joins. +- Lane 0 presents `seq2`; the FIFO head is `seq1`, which belongs to lane 1, so + `try_acquire` refuses. +- Lane 1 cannot consume `seq1` because it is busy. +- The room does not stand in an idle batch for lane 0: `_can_supply_idle_lane` + (`sync_entry.py:8-14`) returns `False` when the lane has queued work and the + group is unsealed, precisely because such a lane is expected to join. + +The last point is the crispest statement of the defect: **the sync room's +"this lane can still join" predicate and the context's `try_acquire` disagree +about the same lane.** The room is right about the model; the context's FIFO +position test is the part that has no counterpart in the system being +simulated. + +Two orderings also disagree with each other. The lane heap orders by +`global_id = counter * lane_count + lane_id` (`batch_ids.py:19`), which puts +lane 0's batch 2 (`global_id 0`) ahead of lane 1's batch 0 (`global_id 1`), +while the ticket FIFO orders by arrival, which puts lane 1 first. Within one +lane the two agree; across lanes they need not. + +### Why dense completes and MoE does not, and why PP=1 is untouched + +- Dense never calls `bind_forward_group` (the call at + `replica_stage_schduler.py:347-356` is MoE-only), so it never seals and has no + sync room. A refused lane is simply re-woken at the next release + (`batch_stage_end_event.py`, `stage_wakeup.py`), so dense finishes, but the + lanes run **one after the other**: while lane 1 works through both of its + batches, lane 0's slot stays empty. Measured: dense `attn_dp∈{2,4}, PP=2` + completes 6/6 and 8/8; the overlap loss is the softer form of the same defect. +- At `num_pipeline_stages = 1` a lane never holds more than one ticket, tickets + are minted just before the lane attempts to use them, and every refusal + condition other than FIFO position is lane-independent. Measured: MoE + `attn_dp∈{2,4}, PP=1` completes 6/6, 12/12; dense likewise. + +| Shape (origin/main, fresh process each) | Requests | Result | +| --- | --- | --- | +| MoE `attn_dp=2, moe_ep=2, PP=2` | 3 | completes | +| MoE `attn_dp=2, moe_ep=2, PP=2` | 4, 6 | **drained** | +| MoE `attn_dp=4, moe_ep=4, PP=2` | 8 | **drained** | +| MoE `attn_dp=2, moe_ep=2, PP=1` | 6, 12 | completes | +| MoE `attn_dp=4, moe_ep=4, PP=1` | 8, 12 | completes | +| MoE `attn_dp=1, PP=2` / `PP=3` | 6 | completes | +| Dense `attn_dp=2, PP=2`, `attn_dp=4, PP=2` | 6, 8 | completes (lanes serialized) | +| Dense `attn_dp=2, PP=1`, `attn_dp=4, PP=1` | 6, 8 | completes | + +The threshold at 4 requests is where both lanes first hold more than one batch. + +## What the FIFO position test guarantees today + +Read from the unit tests that pin it: + +| Test | Guarantee | +| --- | --- | +| `test_admission_fifo_cannot_skip_an_earlier_ready_wave` | Two queued EP waves are admitted in queue order. | +| `test_ep_wave_owns_stage_before_dense_can_start` | A full-stage ticket queued after an EP wave waits for it. | +| `test_started_group_blocks_new_lane_through_ep_restore_and_partial_release` | A lane arriving after a group started waits until every owner releases (seal), not FIFO. | +| `test_next_group_queue_does_not_block_current_group_idle_participation` | A lane whose queued work is refused by the **seal** is stood in as idle. | +| Comment at `replica_stage_schduler.py:301-303` | The lane must admit the same heap head it inspected (a bypass fix unrelated to cross-lane order). | + +None of them require that two full-stage tickets from **different lanes** be +admitted in arrival order. That ordering is the one piece with no stated +purpose, and it is the one that fails. + +## Options + +| Option | Rule | Verdict | +| --- | --- | --- | +| A. Skip busy owners | Ticket carries its lane; an earlier queued full-stage ticket blocks admission only while its lane holds no active ticket. Provably a no-op wherever a lane holds at most one ticket. | **Rejected.** It leaves a lane waiting for a *peer's acquisition*, and the DES wakes lanes only at *release* (`build_stage_wakeup_events` runs from `BatchStageEndEvent`). Traced on the drain scenario: after the first cohort releases, lane 0 presents `seq3`, lane 1's `seq1` is queued ahead with no active owner, lane 0 is refused, lane 1 then acquires and enters the room, and nothing retries lane 0. It would need a second wake path on acquisition, which is new machinery for a state the model does not have. | +| B. Order only exclusive operations | A full-stage ticket is refused only when an **EP wave** is queued ahead of it; earlier full-stage tickets never block it. EP waves keep the strict head rule. Capacity, seal and EP-active checks unchanged. | **Recommended.** No new field, no interface change, one predicate. Every remaining refusal (capacity, seal, EP active, EP wave ahead) is cleared by a release, which already wakes siblings, so no new wait state exists. The room predicate and the context now agree. | +| C. Mint the ticket at the admission attempt instead of arrival | A busy lane never holds a queued ticket. | Rejected. Changes ordering semantics for every path and breaks the stale-drop logic, which relies on the ticket attached at arrival (`_discard_stale_ticket`, `_drop_queued_lanes_for_ticket`, sibling tickets in `DECODE_FFN`). | +| D. Stand in an idle lane when a lane is blocked by admission order | Change `_can_supply_idle_lane`. | Rejected. Lane 0 has real work for this forward; modelling it as absent skips that work into a later forward. An error-suppressing fallback in the sense of the working gates. | + +## Recommended rule + +In `StageExecutionContext.try_acquire`, replace the head test for full-stage +tickets with: + +> A full-stage ticket may be admitted when no EP wave is queued ahead of it. +> An EP wave may be admitted only as the FIFO head. + +Sketch (final wording at implementation; the existing scope, capacity and seal +checks above it are unchanged): + +```python +if ticket.scope == EP_WAVE: + if not self._ready_fifo or self._ready_fifo[0] != ticket: + return False +elif any(queued.scope == EP_WAVE for queued in self._ready_fifo + if queued.admission_seq < ticket.admission_seq): + return False +self._ready_fifo.remove(ticket) +``` + +The FIFO stays one deque so that an EP wave still sees every full-stage ticket +ahead of it. The scan is bounded by `lanes × num_pipeline_stages` tickets. + +Files touched: `stage_execution_context.py` (rule and the two docstrings that +describe admission as "FIFO-head"), no other source file. `_can_supply_idle_lane` +is left as is; it becomes consistent rather than changed. + +### Invariants after the change + +1. At most one active full-stage ticket per lane per stage (unchanged; from + `_is_busy`). +2. Within one lane, batches enter a stage in heap order (unchanged; the lane + presents only its heap head). +3. Exclusive operations (EP waves) are admitted in queue order and never + overtaken by full-stage work queued behind them (unchanged; pinned by the + two EP tests). +4. A lane with queued work, free capacity and an unsealed group can be admitted + at its next attempt (new; this is the property the sync room already + assumes). +5. Every refusal is cleared by a release event, which wakes idle non-empty + sibling lanes (unchanged mechanism, now sufficient). + +### Where the rule is a no-op by construction + +- `full_stage_capacity = 1` contexts (`DECODE_ATTN`, `DECODE_FFN`): with one + owner slot, a second full-stage ticket is refused by capacity whenever the + first is active; when nothing is active, letting a later full-stage ticket + pass an earlier one is the only new behaviour, and it can arise only if the + later ticket's lane attempts first while both are queued. In `DECODE_FFN` + the shared groups carry one ticket for all sibling lanes, so two distinct + full-stage tickets queued at once means two successive groups, which are + produced and attempted in order. Verified by byte comparison in the plan, not + assumed. +- Dense `attn_dp > 1, PP = 1` and MoE `attn_dp > 1, PP = 1`: one ticket per lane, + minted immediately before the attempt; the only refusals are lane-independent. + Verified by byte comparison. + +## Fidelity expectation, stated before measuring + +| Scenario class | Expected after the change | +| --- | --- | +| Every `num_pipeline_stages = 1` scenario (co-location, PDD, PD-AF examples; Step 8 regression set) | Byte-identical `request_metrics.csv` and `system_metrics.json`. | +| PD-AF `DECODE_ATTN` / `DECODE_FFN` (capacity 1) | Byte-identical. | +| MoE `attn_dp > 1`, `PP > 1` | From drain to completion with request and token conservation. | +| Dense `attn_dp > 1`, `PP > 1` | Completes before and after; lane stage-busy intervals overlap after the change where they were serialized before, so makespan and per-request latencies **change**. This is the same defect's softer symptom and is proposed as an accepted fidelity fix (decision D-2 in the plan). | +| `attn_dp = 1` any PP | Byte-identical (one lane, FIFO order equals heap order). | + +Any difference outside the two "changes" rows is a defect in this change and +stops the work. + +## What this is not + +- Not a change to `full_stage_capacity`, the seal, the EP wave protocol, the + sync rooms, or the wake-up helper. +- Not a new flag or configuration field. +- Not the Step 9 report key (D9-2 in the parent plan); that design resumes once + this lands and the `attn_dp=2, PP=2` shape runs. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/plan.md b/task_memory/task_2026-09-22_stage_admission_ordering/plan.md new file mode 100644 index 00000000..f2c786b3 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/plan.md @@ -0,0 +1,104 @@ +# Stage admission ordering under pipeline parallelism — Plan + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-23 | D-1..D-5 adopted by the user; D-3 executed now (push + draft PR for remote review); D-5 adjusted so the records travel with the branch. Work packages P0–P4 unblocked. | +| 2026-09-22 | Created for user review. Scope, acceptance criteria, work packages P0–P4 with dependencies, verification matrix, decisions D-1..D-4. No source change yet. | + +Diagnosis, options and the recommended rule are in `design.md`. This file is the +executable plan. + +## 1. Scope + +Fix the pre-existing stage admission deadlock (W9-01 in the parent task) on its +own branch: + +- Branch `fix/stage-admission-ordering`, base `origin/main` `1f694f7`, worktree + `/data/ycfeng/Frontier/.worktrees/stage-admission-ordering`. +- Source change confined to + `frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`: + the full-stage admission predicate in `try_acquire` and the two docstrings + that describe admission as "FIFO-head". +- Out of scope: `full_stage_capacity`, the forward-group seal, EP wave + protocol, sync rooms, wake-up helper, any configuration field, and the Step 9 + report key. + +## 2. Acceptance criteria + +| Id | Criterion | Settled by | +| --- | --- | --- | +| C1 | MoE `attn_dp∈{2,4}`, `moe_ep = attn_dp`, `PP∈{2,3}` (layer count divisible by PP), offline, 4/6/8/12 requests: every request completes; request count, prefill tokens and decode tokens conserved; no "non-empty scheduler state" drain. | P2(c) integration test; P3 matrix. | +| C2 | Every `PP = 1` scenario and every PD-AF scenario in the matrix produces byte-identical `request_metrics.csv` and `system_metrics.json` before and after. | P0 baseline vs P3 rerun, SHA-256 per file. | +| C3 | Dense `attn_dp∈{2,4}`, `PP∈{2,3}`: completes before and after; the number of simulated intervals during which two lanes of one stage are busy at once is greater after than before; every metric difference is attributable to that overlap. | P3, with the lane-overlap count from `metrics_ground_truth.jsonl` or the batch-stage trace. | +| C4 | All existing unit tests pass without modification. If any existing test must change, stop and report; that is a design signal, not a test fix. | P2. | +| C5 | The predicate is one readable condition; module and method docstrings state the ordering contract as implemented; no flag, no config field, no `getattr` fallbacks. | Review of the diff against the quality gates. | +| C6 | The Step 9 boundary probe on MoE `attn_dp=2, moe_ep=2, PP=2` runs to completion on this branch (informational: it unblocks the parent task's design checkpoint). | P3, scratch probe rerun. | + +## 3. Work packages + +```text +P0 baseline capture (no source change) + -> P1 rule change + -> {P2 tests, P3 fidelity rerun and comparison} + -> P4 records, commit, push and draft PR (D-3) +``` + +| Package | Content | Acceptance | +| --- | --- | --- | +| P0 Baseline | On `1f694f7`, run the verification matrix of §4 and store outputs under `/data/ycfeng/tmp/stage_admission_ordering/baseline//` with a `sha256sums.txt` per scenario. Store the drain evidence already collected (`design.md` tables) and the reproduction scripts (`repro_main.py`, `drain_state.py`, `drain_lanes.py`, currently in the session scratchpad) under `tests/integration/stage_admission/` as the seed of P2(c). | Every scenario has a hash file; the four drain shapes are recorded as `DRAINED`. | +| P1 Rule | Implement the recommended rule from `design.md`: full-stage tickets are refused only by an EP wave queued ahead; EP waves keep the strict head rule. Update the `StageExecutionContext` class docstring ("A complete operation first enters the ready FIFO, then the owner admits it atomically") and the `try_acquire` docstring ("Acquire the FIFO-head ticket if this stage is currently idle") to describe what is now true. | Diff touches one source file; `python -m pytest tests/unit/test_stage_execution_context.py tests/unit/test_shared_forward_group_admission.py -q` passes unchanged. | +| P2 Tests | (a) Context-level, in `tests/unit/test_stage_execution_context.py`: a full-stage ticket queued behind another lane's queued full-stage ticket is admitted while capacity remains; a full-stage ticket queued behind an EP wave is refused; the two existing EP-order tests stay as they are. (b) Scheduler-level, in `tests/unit/test_shared_forward_group_admission.py` using its `make_stage`/`make_batch` helpers: rebuild the drain state (lane 1 active with a second ticket queued, lane 0 with two queued) and assert lane 0's `pop_batch_if_not_busy` returns its head and binds the same forward group. (c) Simulator-level, `tests/integration/test_stage_admission_pipeline_lanes.py`: one child process per shape (`IS_MOE` is process-global), MoE `attn_dp=2, moe_ep=2, PP=2` at 4 and 6 requests and `attn_dp=4, moe_ep=4, PP=2` at 8, asserting completion and conservation; dense `attn_dp=2, PP=2` asserting completion and lane overlap > 0. Expected values written from the scenario, not copied from a run. | (a) and (b) fail on `1f694f7` and pass after P1; (c) drains on `1f694f7` and passes after P1. | +| P3 Fidelity | Rerun the §4 matrix on the P1 revision into `.../after//`; `diff` the hash files; for every differing scenario, confirm it is in the C3 class and record the overlap count before/after and the makespan delta. Rerun the Step 9 boundary probe for C6. | C1, C2, C3, C6 tables in `test_report_2026-09-22_stage_admission_ordering.md`. | +| P4 Records | Test report, `progress.md`, `summary.md`; commit P1+P2 as one code commit and records as one docs commit; then, under D-3, push the branch and open a draft PR against `main` whose body carries the C1–C3 tables and the W9-01 cross-reference. Note in the parent task (`issues.md` W9-01) the branch and commit. | Pushed and verified, or explicitly left local if D-3 is not granted. | + +## 4. Verification matrix + +Environment: `/data/ycfeng/envs/frontier-py310/bin/python`, `PYTHONPATH` = the +worktree, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`. Each Simulator run in a +fresh process. + +| Group | Scenarios | Expected | +| --- | --- | --- | +| G1 release examples | The 30 `examples/architecture/{co-location,pdd,pd-af-disagg}/{offline,online}/*.sh` recipes (all `PP=1`), metrics dir redirected per scenario | Byte-identical (C2) | +| G2 Step 8 regression set | `pytest tests/unit -q --continue-on-collection-errors` and `pytest tests/integration -q --continue-on-collection-errors`; compare the FAILED **set** to the recorded `1f694f7` baseline (84 entries) | Same set (C4, C2) | +| G3 lanes × stages, MoE | `attn_dp∈{2,4}`, `moe_ep=attn_dp`, `PP∈{1,2,3}`, 6-layer tiny model (all three PP values divide 6), offline, requests ∈ {4, 6, 8, 12} | `PP=1` byte-identical; `PP∈{2,3}` drain → complete (C1) | +| G4 lanes × stages, dense | `attn_dp∈{2,4}`, `PP∈{1,2,3}`, same model with `is_moe=False`, requests ∈ {6, 8} | `PP=1` byte-identical; `PP∈{2,3}` overlap increases (C3) | +| G5 single lane | `attn_dp=1`, `PP∈{1,2,3}`, MoE and dense | Byte-identical | +| G6 PD-AF capacity-1 | The 10 PD-AF recipes from G1 plus `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`, `test_decode_ep_wave_materialization.py`, `test_prefill_ep_wave_materialization.py` | Byte-identical; tests pass (C2, C4) | + +Scenario count: 30 + 2 suites + 24 (G3) + 12 (G4) + 6 (G5) = 72 Simulator runs +plus the two pytest suites, satisfying the AGENTS.md gate of at least 50 +concrete settings. + +Note on G3 `PP=3` with `attn_dp=2`: the collective-sim topology check rejects +`attn_dp=2, moe_ep=2, PP=3` (6 devices against node size 4). Those cells use +`attn_dp=4, moe_ep=4, PP=3` (12 devices) or are marked "rejected at +construction" and excluded; either way the rejection itself is byte-identical +before and after. + +## 5. Decisions (adopted by the user on 2026-09-23: "采纳你d1-d5的推荐决策") + +| Id | Question | Recommendation | Outcome | +| --- | --- | --- | --- | +| D-1 | Adopt option B from `design.md` (full-stage tickets are ordered only behind EP waves) rather than option A (lane-aware skip) or C/D. | B. A needs a wake path on acquisition that the DES does not have; C and D are rejected on the working gates. | Adopted. | +| D-2 | Accept that dense `attn_dp>1, PP>1` timelines change (lanes overlap instead of serializing). | Accept as a fidelity fix; the serialization is the same defect. | Adopted. | +| D-3 | Authorize pushing `fix/stage-admission-ordering` and opening a draft PR against `main`. | Grant at P4; until then everything stays local. | Adopted and brought forward: the user reviews on the remote, so the branch is pushed and a draft PR opened with the plan itself (2026-09-23). Code commits follow per package. | +| D-4 | Baseline for byte comparison is `origin/main` `1f694f7`. PR 35 will merge this branch later instead of carrying the fix itself. | Confirm. | Adopted. | +| D-5 | `task_memory/` is ignored by `.gitignore` on `main` (line 171), so these records are local to the worktree unless force-added. Keep them local and archive the outcome in the parent task, or track them on this branch as PR 35 does? | Keep local; copy `summary.md` and the test report into the parent task at P4. | Adopted with one adjustment required by D-3: remote review needs the records on the branch, so `.gitignore` gets the same narrow exception PR 34/35 use (`task_memory/*` plus `!task_memory/task_2026-09-22_stage_admission_ordering/`). The copy into the parent task at P4 stands. | + +## 6. Dependencies and risks + +- The reproduction and boundary scripts live in the session scratchpad + (`w10/repro_main.py`, `w10/drain_state.py`, `w10/drain_lanes.py`); P0 moves + them under `tests/`. +- Risk: a `DECODE_FFN` path that queues two distinct full-stage tickets from + different sibling lanes at the same time would see admission order change. + `design.md` argues this does not occur; G6 measures it. If G6 differs, stop + and report before adjusting anything. +- Risk: the C3 overlap count needs a per-lane busy-interval source. If + `metrics_ground_truth.jsonl` lacks stage-level lane intervals, P3 derives + them from the batch-stage trace; adding metrics is out of scope. +- The parent task's Step 9 resumes only after this branch is merged into `main` + and merged forward into `fix/issue26-correctness-pr`. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md new file mode 100644 index 00000000..8eb1aa8d --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md @@ -0,0 +1,35 @@ +# Stage admission ordering under pipeline parallelism — Progress + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-23 | Decisions adopted; records pushed for remote review. | +| 2026-09-22 | Created. Worktree and branch created; defect reproduced on `origin/main`; plan and design written for review. No source change. | + +## State + +| Item | State | Evidence | +| --- | --- | --- | +| Worktree `.worktrees/stage-admission-ordering` on `fix/stage-admission-ordering` @ `1f694f7` | completed | `git worktree list` | +| Reproduction on `origin/main`, 15 shapes | completed | `design.md` shape table; logs under `/data/ycfeng/tmp/w10_repro/case_*.log` | +| Drain state dump (FIFO, active owners, lane heaps, sync room) | completed | `design.md` "Observed state at the drain" | +| Root-cause diagnosis and option analysis | completed | `design.md` | +| Plan for review | completed, awaiting user | `plan.md` | +| Records published for remote review (`.gitignore` exception, docs commit, push, draft PR) | completed 2026-09-23 | commit and PR recorded below | +| P0–P4 | pending | unblocked by R-4; P0 next | + +## Commands run (2026-09-22) + +```bash +git -C /data/ycfeng/Frontier worktree add -b fix/stage-admission-ordering \ + /data/ycfeng/Frontier/.worktrees/stage-admission-ordering origin/main + +# fresh process per shape; scripts in the session scratchpad w10/ +python repro_main.py {moe|dense} +python drain_state.py 4 # context FIFO / active owners +python drain_lanes.py # lane heaps and sync room +``` + +Interpreter `/data/ycfeng/envs/frontier-py310/bin/python`, `PYTHONPATH` = the +worktree, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md new file mode 100644 index 00000000..fcfa7ff7 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md @@ -0,0 +1,53 @@ +# Stage admission ordering under pipeline parallelism — Requirements + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-23 | R-4: D-1..D-5 adopted; push and draft PR authorized. | +| 2026-09-22 | Created from the Step 9 finding W9-01 in `task_2026-09-21_issue26_correctness_pr`; recorded the user's scope decision and the request for a reviewable plan. | + +## Origin + +Found on 2026-09-22 while probing Frontier boundaries for Step 9 of the Issue 26 +correctness PR (package P1(b)). Recorded there as W9-01 +(`task_memory/task_2026-09-21_issue26_correctness_pr/issues.md`). The defect +predates both stacked PRs; the three files involved are byte-identical to +`origin/main` at `1f694f7`. + +## Requests + +`[Original Request]` (2026-09-22, after the W9-01 report with three options and +a recommendation to fix it as a separate correctness item): + +> 采纳你的推荐，继续 + +`[Original Request]` (2026-09-22, interrupting the first source read): + +> 先给出"共享 admission 排序问题"的修复计划,将具体的计划落地到文档，由我审阅 + +Quality gates the user repeated for every core-module change in this line of +work, carried over verbatim: + +> 对frontier 核心模块的代码的修改和实现上，确保可读性和可维护，任何引入的修改和实现都应该是高价值的（要么对fidelity有收益，要么与模拟功能直接相关，不可替代），禁止hard-coding，禁止临时补丁，禁止过度防御，禁止冗余性设计和实现，禁止使用ai味命名函数和变量。 + +## Decisions + +| Id | Decision | Source | +| --- | --- | --- | +| R-1 | The defect is fixed as a separate correctness item, not inside the Issue 26 feature branch. Step 9's PP>1 packages stay paused until it lands. | user, 2026-09-22 | +| R-2 | Branch `fix/stage-admission-ordering` from `origin/main` `1f694f7`, worktree `/data/ycfeng/Frontier/.worktrees/stage-admission-ordering`. | agent, under R-1 | +| R-3 | No source change before the user reviews `plan.md` and `design.md`. | user, 2026-09-22 | +| R-4 | Plan decisions D-1..D-5 adopted as recommended. Push the branch and open a draft PR so the review happens on the remote; the reviewer resumes from a prepared prompt. | user, 2026-09-23 | + +## Constraints carried from the parent task + +- `rm` is authorized; `mv`, destructive overwrites, history rewrites, force + pushes and branch or worktree deletion are not. +- Pushing this branch and opening a draft pull request were authorized on + 2026-09-23 (R-4). Merge, marking ready, force-push and history rewrites remain + unauthorized. +- No `Co-Authored-By: Claude` on commits or pull requests. +- Temporary files under `/data/ycfeng/tmp`; the simulator interpreter is + `/data/ycfeng/envs/frontier-py310/bin/python`. +- Never `cd` into the original repository root; use `git -C` and absolute paths. From a6ec6a6ccb995994281b6206eef0440e629c4280 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 00:32:55 +0800 Subject: [PATCH 02/19] docs(stage-admission): record PR 36 and the reviewer resume prompt --- .../progress.md | 9 ++++++++ .../review_prompt.md | 23 +++++++++++++++++++ 2 files changed, 32 insertions(+) create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md index 8eb1aa8d..cc4d2639 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md @@ -33,3 +33,12 @@ python drain_lanes.py # lane heaps and sync room Interpreter `/data/ycfeng/envs/frontier-py310/bin/python`, `PYTHONPATH` = the worktree, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`. + +## Publication (2026-09-23) + +| Item | Value | +| --- | --- | +| Branch | `fix/stage-admission-ordering` on `NetX-lab/Frontier`, base `main` `1f694f7` | +| Records commit | `62d25b9` (plan, design, requirements, progress, `.gitignore` exception) | +| Draft PR | https://github.com/NetX-lab/Frontier/pull/36 | +| Reviewer resume prompt | `review_prompt.md` in this directory | diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md b/task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md new file mode 100644 index 00000000..eb8df287 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md @@ -0,0 +1,23 @@ +# Resume prompt for the reviewing agent + +Copy everything below the line into the review agent's first message. + +--- + +You are reviewing draft PR https://github.com/NetX-lab/Frontier/pull/36 on `NetX-lab/Frontier`, branch `fix/stage-admission-ordering`, base `main` at `1f694f7`. Start with the repository's `AGENTS.md`, then read, in this order, under `task_memory/task_2026-09-22_stage_admission_ordering/`: `requirements.md`, `design.md`, `plan.md`, `progress.md`. + +Context. Frontier is a discrete-event LLM inference simulator. One `StageExecutionContext` (`frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`) owns each physical `(replica, stage)` and is shared by that stage's attention-DP lanes, each of which has its own `ReplicaStageScheduler` (`replica_stage_schduler.py`) with its own batch heap and `_is_busy` flag. `add_batch` mints a `StageAdmissionTicket` at batch arrival into the shared `_ready_fifo`; `pop_batch_if_not_busy` later asks `try_acquire` for the lane's heap head, which is refused unless the ticket is the strict FIFO head (after the EP-active, capacity and forward-group-seal checks). `full_stage_capacity` equals `attn_dp` for `MONOLITHIC`, `PREFILL` and `DECODE`, so the lanes of one forward co-own the stage; MoE lanes then meet in a sync room (`frontier/scheduler/utils/sync_entry.py`), which stands in an idle batch for a missing lane only when that lane has no queued work or the group is sealed (`_can_supply_idle_lane`). + +The defect, reproduced on `main`: with `num_pipeline_stages > 1`, `BaseReplicaScheduler.on_schedule` admits up to `num_pipeline_stages` batches per lane per round, so a lane holds several queued tickets while consuming one. The FIFO head can then be a ticket whose lane is busy inside the sync room; the other lane is refused although capacity is free; the room does not stand it in because it has work and the group is open. MoE `attn_dp∈{2,4}, PP=2` drains with requests unfinished; dense completes but serializes its lanes; every `PP=1` shape and every capacity-1 context completes. `design.md` has the drain-state table (FIFO, active owners, lane heaps, sync room). + +The plan is at the review-before-code stage: the PR has records only, no source change yet. The recommended rule (option B, adopted by the owner as D-1) changes one predicate in `try_acquire`: a full-stage ticket is refused only when an EP wave is queued ahead of it; EP waves keep the strict head rule; capacity, seal and EP-active checks are unchanged. Option A (lane-aware skip of tickets whose lane already holds an active ticket) was rejected because the DES wakes sibling lanes only at release (`frontier/scheduler/utils/stage_wakeup.py`, called from `BatchStageEndEvent`), not at a peer's acquisition, so A leaves a stranded-lane state after the first cohort releases; `design.md` traces that sequence. Decisions D-2 (dense `attn_dp>1, PP>1` timelines may change because lanes now overlap), D-4 (byte-comparison baseline is `main` `1f694f7`) and D-5 (records tracked on the branch through a narrow `.gitignore` exception) are also adopted. + +What to review, in priority order: + +1. The diagnosis in `design.md`: does the source support the circular wait exactly as stated? Check `try_acquire`, `pop_batch_if_not_busy`, `_can_supply_idle_lane`, the admission loop in `base_replica_scheduler.py`, and the wake-up path. +2. The recommended rule: is there any code path where two full-stage tickets from different lanes must stay in arrival order? Look at `DECODE_FFN` (capacity 1, shared sibling tickets, `DenseFFNBatchGroup` in `round_robin_cluster_scheduler.py`) and at `tests/unit/test_stage_execution_context.py`, `tests/unit/test_shared_forward_group_admission.py`, `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`. The plan asserts all existing tests pass unchanged (C4); say whether you agree from reading them. +3. The rejection of option A: confirm or refute the stranded-lane trace using the event classes under `frontier/events/`. +4. The fidelity expectation and the 72-scenario matrix in `plan.md` §4: are the "byte-identical" classes correctly bounded, and is the dense overlap change (C3) measurable with existing outputs? +5. Fit with the owner's core-module gates: readability, no hard-coding, no temporary patches, no over-defensive branches, no redundant mechanisms, plain domain names. + +Report findings as a numbered list with a source anchor (`path:line`) and a verdict per item (agree / disagree / needs evidence), then a one-paragraph recommendation on whether P1 may start as planned. Do not change source or push; the owner decides. From 15ea2e09ed4079b9db5c68d841ecc4f2817c99c0 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 01:33:55 +0800 Subject: [PATCH 03/19] docs(stage-admission): apply the PR 36 round-1 plan review Each of the ten review findings was re-checked against main 1f694f7 and its disposition recorded in review.md. The records are corrected; nothing was executed and no source file changed. - design.md: cite the MONOLITHIC/PREFILL admission loop; state the drain as a queued-ticket arrangement rather than a shape; note that queued EP waves exist only on DECODE_FFN; remove the admitted ticket with remove(ticket); keep option A rejected on design grounds with its stall trace marked unverified; restate the capacity-1 and PP=1 expectations as caller-level conditions; narrow the queue bound; add the mixed-phase scope boundary; withdraw the unmeasured "lanes serialized" label. - plan.md: C1 targets confirmed deadlock witnesses from a prefill-only, phase-controlled group; C3 uses the stage-batch ledger with half-open overlap duration; C4 takes the reviewer's wording; P0 lists its artifacts and outcome classes; P2 covers both sides of the EP boundary, a DECODE_FFN dense-group control and a second admission round; P3 has separate unchanged, liveness and timing paths; the case list is published on the analytical backend and keeps attn_dp=2, PP=3 (decisions D-6, D-7). - requirements.md, progress.md, review_prompt.md: record the request, the state, and the round-2 reviewer prompt. --- .../design.md | 218 +++++++++++++----- .../plan.md | 214 +++++++++++++---- .../progress.md | 9 +- .../requirements.md | 7 + .../review.md | 48 ++++ .../review_prompt.md | 33 ++- 6 files changed, 403 insertions(+), 126 deletions(-) create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/review.md diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/design.md b/task_memory/task_2026-09-22_stage_admission_ordering/design.md index db41e905..9c0ef993 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/design.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/design.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | Applied the round-1 plan review (`review.md`). Changes: the admission-loop anchor now points to the `MONOLITHIC`/`PREFILL` path; the drain condition is stated as a queued-ticket arrangement, not a shape; the shape table is marked author-reported until P0; added where queued EP waves exist; `remove(ticket)` made explicit; option A's stall trace labelled an unverified hypothesis; the capacity-1 section rewritten as a caller-level condition; the queue bound narrowed; added the mixed-phase scope boundary; the dense "lanes serialized" label withdrawn as unmeasured and replaced by the admission sequence read from source. | | 2026-09-22 | Created: defect restated from source on `origin/main` `1f694f7`, what the FIFO guarantees today, four options, recommended rule with its invariants, fidelity expectation. For review before implementation. | All line references are to `origin/main` at `1f694f7`, checked out in @@ -22,29 +23,45 @@ Admission is a two-step handshake: a `StageAdmissionTicket` at batch **arrival** through `StageExecutionContext.enqueue_full_stage`, which appends it to the shared `_ready_fifo` (`stage_execution_context.py:188`). -2. `pop_batch_if_not_busy` (`:286-359`) takes the lane's own heap head and asks - the context to `try_acquire` its ticket. `try_acquire` (`:322-343`) refuses - when an EP wave is active, when active full-stage owners already fill - `full_stage_capacity`, when the forward group is sealed, and finally when the - ticket **is not the FIFO head**: +2. `pop_batch_if_not_busy` (`:286-359`) returns at once when its lane is busy; + otherwise it takes the lane's own heap head and, unless the context already + `owns` that ticket, asks `try_acquire` (`:329`). `try_acquire` (`:322-343`) + refuses when an EP wave is active, when active full-stage owners already + fill `full_stage_capacity`, when the forward group is sealed, and finally + when the ticket **is not the FIFO head**: ```python if not self._ready_fifo or self._ready_fifo[0] != ticket: return False + self._ready_fifo.popleft() ``` `full_stage_capacity` is `attn_dp` for `MONOLITHIC`, `PREFILL` and `DECODE` -(`stage_contexts.py`), so up to one ticket per lane may be active at once: that -is how the lanes of one forward co-own the stage. +and 1 otherwise (`stage_contexts.py:59-81`), so up to one ticket per lane may +be active at once: that is how the lanes of one forward co-own the stage. -`BaseReplicaScheduler.on_schedule` admits up to `num_pipeline_stages` batches -per lane in one round (`base_replica_scheduler.py:893-901`). At -`num_pipeline_stages > 1` a lane therefore holds **several queued tickets** while -being able to consume only one. The FIFO head can then be a ticket whose own -lane is busy, and every other lane is refused although capacity is free. +`BaseReplicaScheduler.on_schedule` admits batches while +`num_running_batches < num_stages`. The co-location reproduction runs the +`MONOLITHIC`/`PREFILL` branch (`base_replica_scheduler.py:1037-1054`, loop at +`:1039`); the unified `DECODE` branch (`:893`) has the same bound. At +`num_pipeline_stages > 1` a lane can therefore hold **several queued tickets** +at one stage while consuming one. + +The drain needs a specific arrangement, not merely `attn_dp > 1` and `PP > 1`: + +- a lane's queued ticket is at the FIFO head while that lane is busy with an + active ticket of an open (unsealed) forward group, and +- another lane with queued work presents a ticket behind it and is refused, + while the busy lane waits in a sync room for that refused lane. + +With too little queued work the same shape completes (the 3-request row +below). ### Observed state at the drain (MoE `attn_dp=2, moe_ep=2, PP=2`, 4 requests) +Author-run on 2026-09-22 with the session scripts. P0 republishes it from the +published case inputs (`plan.md` §4, group R0) and stores the state report. + Both lanes admitted two batches each. Lane 1 scheduled first. | Where | State | @@ -78,19 +95,27 @@ lane 0's batch 2 (`global_id 0`) ahead of lane 1's batch 0 (`global_id 1`), while the ticket FIFO orders by arrival, which puts lane 1 first. Within one lane the two agree; across lanes they need not. -### Why dense completes and MoE does not, and why PP=1 is untouched +### Why dense completes and MoE does not, and why PP=1 is not expected to drain - Dense never calls `bind_forward_group` (the call at `replica_stage_schduler.py:347-356` is MoE-only), so it never seals and has no - sync room. A refused lane is simply re-woken at the next release - (`batch_stage_end_event.py`, `stage_wakeup.py`), so dense finishes, but the - lanes run **one after the other**: while lane 1 works through both of its - batches, lane 0's slot stays empty. Measured: dense `attn_dp∈{2,4}, PP=2` - completes 6/6 and 8/8; the overlap loss is the softer form of the same defect. -- At `num_pipeline_stages = 1` a lane never holds more than one ticket, tickets - are minted just before the lane attempts to use them, and every refusal - condition other than FIFO position is lane-independent. Measured: MoE - `attn_dp∈{2,4}, PP=1` completes 6/6, 12/12; dense likewise. + sync room. A refused lane is re-woken at the next release + (`batch_stage_end_event.py:139-158`, `stage_wakeup.py:8-43`), so dense + finishes, but with lost overlap. From source, with every request at `t=0`: + the lane that schedules first mints two tickets before the other lane mints + any; the other lane is then refused while the first lane's first batch holds + stage 0, although capacity is free, and is admitted only at that release. + The first draft called this "lanes serialized". That was not measured: the + earlier runs checked completion only. P0 measures the loss with the `plan.md` §4.5 + ledger metric. The loss is the softer form of the same defect. +- At `num_pipeline_stages = 1` a lane admits its next batch only after the + previous one leaves the only stage, so it never holds a queued ticket while + busy, and the first bullet of the drain arrangement cannot form. + +Author-reported shapes (2026-09-22; default `astra_sim_analytical` backend, +Poisson `qps=1e6`, prefill 16 / decode 3; logs +`/data/ycfeng/tmp/w10_repro/case_*.log`). They remain author-reported +evidence until P0 reruns them as group R0 from the published inputs. | Shape (origin/main, fresh process each) | Requests | Result | | --- | --- | --- | @@ -100,10 +125,12 @@ lane the two agree; across lanes they need not. | MoE `attn_dp=2, moe_ep=2, PP=1` | 6, 12 | completes | | MoE `attn_dp=4, moe_ep=4, PP=1` | 8, 12 | completes | | MoE `attn_dp=1, PP=2` / `PP=3` | 6 | completes | -| Dense `attn_dp=2, PP=2`, `attn_dp=4, PP=2` | 6, 8 | completes (lanes serialized) | -| Dense `attn_dp=2, PP=1`, `attn_dp=4, PP=1` | 6, 8 | completes | +| Dense `attn_dp=2, PP=2`, `attn_dp=4, PP=2` | 6, 8 | completes (overlap not measured) | +| Dense `attn_dp=2, PP=1`, `attn_dp=4, PP=1`, `attn_dp=1, PP=2` | 6, 8, 6 | completes | +| MoE `attn_dp=2, moe_ep=2, PP=3` | 6 | rejected at construction by the Replica-pod node-size rule (6 devices against node size 4; parent task W9-02) | -The threshold at 4 requests is where both lanes first hold more than one batch. +In this table the drain first appears at 4 requests, where both lanes first +hold more than one batch. ## What the FIFO position test guarantees today @@ -119,14 +146,28 @@ Read from the unit tests that pin it: None of them require that two full-stage tickets from **different lanes** be admitted in arrival order. That ordering is the one piece with no stated -purpose, and it is the one that fails. +purpose, and it is the one that fails. This is a reading of the tests, not a +test result; C4 in the plan settles it. + +### Where queued EP waves exist + +`enqueue_ep_wave` has one caller, the `DECODE_FFN` M2N group path +(`round_robin_cluster_scheduler.py:1052-1057`). On `MONOLITHIC`, `PREFILL` and +`DECODE` contexts, `EP_WAVE` appears only as an active-scope transition of +owners already admitted (`transition_active_scope`, +`replace_full_stage_owners_with_ep_wave`, +`replace_ep_wave_with_full_stage_owners`, driven by +`forward_step_admission.py`); it never enters the FIFO there. So the FIFO of a +shared-lane context holds only full-stage tickets, and a mixed FIFO of +full-stage tickets and EP waves exists only on `DECODE_FFN` contexts, whose +capacity is 1. ## Options | Option | Rule | Verdict | | --- | --- | --- | -| A. Skip busy owners | Ticket carries its lane; an earlier queued full-stage ticket blocks admission only while its lane holds no active ticket. Provably a no-op wherever a lane holds at most one ticket. | **Rejected.** It leaves a lane waiting for a *peer's acquisition*, and the DES wakes lanes only at *release* (`build_stage_wakeup_events` runs from `BatchStageEndEvent`). Traced on the drain scenario: after the first cohort releases, lane 0 presents `seq3`, lane 1's `seq1` is queued ahead with no active owner, lane 0 is refused, lane 1 then acquires and enters the room, and nothing retries lane 0. It would need a second wake path on acquisition, which is new machinery for a state the model does not have. | -| B. Order only exclusive operations | A full-stage ticket is refused only when an **EP wave** is queued ahead of it; earlier full-stage tickets never block it. EP waves keep the strict head rule. Capacity, seal and EP-active checks unchanged. | **Recommended.** No new field, no interface change, one predicate. Every remaining refusal (capacity, seal, EP active, EP wave ahead) is cleared by a release, which already wakes siblings, so no new wait state exists. The room predicate and the context now agree. | +| A. Skip busy owners | The ticket carries its lane; an earlier queued full-stage ticket blocks admission only while its lane holds no active ticket. | **Rejected on design grounds.** It adds lane identity to tickets and makes one lane's admission depend on a peer lane's *acquisition*. Acquisition emits no retry; only `BatchStageEndEvent` wakes siblings (`batch_stage_end_event.py:148-158`), so A would need a new wake path on acquisition. The first draft also sketched a specific second-cohort stall. That trace is an **unverified hypothesis**. It did not account for the releasing lane's own retry, which is emitted before sibling retries (`:139-146`). It did not account for retries from several same-time releases, and the reviewer notes that prefill participants can release at one shared predicted time. And the DES orders equal-time events by `(time, id, event_type)` (`base_event.py:63-64`, `simulator.py:1268`), not by `BaseEvent.__lt__` (`:66-70`), which compares type before id. B does not depend on that trace, so it is not pursued. | +| B. Order only exclusive operations | A full-stage ticket is refused only when an **EP wave** is queued ahead of it; earlier full-stage tickets never block it. EP waves keep the strict head rule. Capacity, seal and EP-active checks unchanged. | **Recommended (adopted as D-1).** No new field, no interface change, one predicate. Every remaining refusal (capacity, seal, EP active, EP wave ahead) is cleared by a release, which already wakes siblings, so no new wait state exists. The room predicate and the context now agree. | | C. Mint the ticket at the admission attempt instead of arrival | A busy lane never holds a queued ticket. | Rejected. Changes ordering semantics for every path and breaks the stale-drop logic, which relies on the ticket attached at arrival (`_discard_stale_ticket`, `_drop_queued_lanes_for_ticket`, sibling tickets in `DECODE_FFN`). | | D. Stand in an idle lane when a lane is blocked by admission order | Change `_can_supply_idle_lane`. | Rejected. Lane 0 has real work for this forward; modelling it as absent skips that work into a later forward. An error-suppressing fallback in the sense of the working gates. | @@ -139,20 +180,39 @@ tickets with: > An EP wave may be admitted only as the FIFO head. Sketch (final wording at implementation; the existing scope, capacity and seal -checks above it are unchanged): +checks above it are unchanged, and the EP-wave line is today's line): ```python if ticket.scope == EP_WAVE: if not self._ready_fifo or self._ready_fifo[0] != ticket: return False -elif any(queued.scope == EP_WAVE for queued in self._ready_fifo - if queued.admission_seq < ticket.admission_seq): - return False +else: + for queued in self._ready_fifo: + if queued == ticket: + break + if queued.scope == EP_WAVE: + return False self._ready_fifo.remove(ticket) ``` -The FIFO stays one deque so that an EP wave still sees every full-stage ticket -ahead of it. The scan is bounded by `lanes × num_pipeline_stages` tickets. +- The admitted ticket is removed with `remove(ticket)`, not `popleft()`: once a + non-head ticket can be admitted, `popleft()` would dequeue a different + ticket. `cancel` already removes a queued ticket the same way + (`stage_execution_context.py:456`). +- The single caller reaches `try_acquire` only with a queued ticket: it checks + `owns` first (`replica_stage_schduler.py:328`), and `_validate_ticket` + rejects a ticket that is neither queued nor active. No branch is added for + other states. +- The FIFO stays one deque so that an EP wave still sees every ticket ahead of + it. +- Queue length: on the shared-lane contexts this change targets, the FIFO holds + at most `attn_dp × num_pipeline_stages` tickets, because each lane runs at + most `num_pipeline_stages` batches (`base_replica_scheduler.py:893,1039`). + `DECODE_FFN` contexts are fed by M2N groups and have no such bound; the scan + there is linear in the queue, as `cancel`'s `remove` already is. No index or + second queue is added. +- On shared-lane contexts no EP wave is ever queued (previous section), so the + loop only walks to the ticket; its EP clause acts on `DECODE_FFN`. Files touched: `stage_execution_context.py` (rule and the two docstrings that describe admission as "FIFO-head"), no other source file. `_can_supply_idle_lane` @@ -165,46 +225,82 @@ is left as is; it becomes consistent rather than changed. 2. Within one lane, batches enter a stage in heap order (unchanged; the lane presents only its heap head). 3. Exclusive operations (EP waves) are admitted in queue order and never - overtaken by full-stage work queued behind them (unchanged; pinned by the - two EP tests). -4. A lane with queued work, free capacity and an unsealed group can be admitted - at its next attempt (new; this is the property the sync room already + overtaken by full-stage work queued behind them; an EP wave still waits for + every earlier queued ticket and every active owner (unchanged; pinned by the + two EP tests and extended by P2(a)). +4. A lane with queued work is admitted at its next attempt when capacity is + free, the group is unsealed, no EP wave is active and no EP wave is queued + ahead of its ticket (new; this is the property the sync room already assumes). 5. Every refusal is cleared by a release event, which wakes idle non-empty sibling lanes (unchanged mechanism, now sufficient). -### Where the rule is a no-op by construction - -- `full_stage_capacity = 1` contexts (`DECODE_ATTN`, `DECODE_FFN`): with one - owner slot, a second full-stage ticket is refused by capacity whenever the - first is active; when nothing is active, letting a later full-stage ticket - pass an earlier one is the only new behaviour, and it can arise only if the - later ticket's lane attempts first while both are queued. In `DECODE_FFN` - the shared groups carry one ticket for all sibling lanes, so two distinct - full-stage tickets queued at once means two successive groups, which are - produced and attempted in order. Verified by byte comparison in the plan, not - assumed. -- Dense `attn_dp > 1, PP = 1` and MoE `attn_dp > 1, PP = 1`: one ticket per lane, - minted immediately before the attempt; the only refusals are lane-independent. - Verified by byte comparison. +Invariant 4 removes the admission-order refusal. It is not a whole-run +liveness proof; see the scope boundary below. + +### Where behaviour is expected to stay unchanged, and why + +The rule is not a no-op at the context API. With an idle capacity-1 context +and FIFO `[full0, full1]`, `try_acquire(full1)` is refused today and admitted +under B. Capacity prevents two simultaneous owners but does not preserve +arrival order when nothing is active. Unchanged behaviour is therefore a claim +about the callers, under this condition: + +> For a scheduler that presents only its heap head, B and today's rule make the +> same decision whenever no full-stage ticket of **another** scheduler is +> queued ahead of the presented ticket, and each scheduler's heap order agrees +> with FIFO order among its own full-stage tickets. Then only EP waves can be +> ahead of the presented ticket, and both rules refuse exactly when something +> is ahead. + +| Context | Why the condition is expected to hold | Evidence planned | +| --- | --- | --- | +| `DECODE_FFN` (capacity 1) | Every `DenseFFNBatchGroup` gets `global_id = _batch_group_creation_counter` (`round_robin_cluster_scheduler.py:1097,1118`) and one full-stage ticket (`:1138`), and is queued on the one full-stage scheduler of its replica (`:1100`), so its heap and FIFO both follow the group counter. EP child batches hold no full-stage ticket; the group shares one `EP_WAVE` ticket (`:1052-1057`). Shared EP sibling tickets are not multiple full-stage owners. | P2(a′) control with two successive dense FFN groups and a neighbouring EP group through the real full-stage scheduler; G6 byte comparison. | +| `DECODE_ATTN` (capacity 1) | `attn_dp=1` with `replica_local_id=None` (AGENTS.md): one scheduler per stage, so no other scheduler's ticket can be ahead. | G6 byte comparison. | +| Shared-lane contexts, `PP = 1` | A lane never holds a queued ticket while busy. A cross-lane inversion needs one release to wake two or more idle siblings whose tickets are queued in the opposite order to the wake order: wake-ups follow lane-key order (`stage_wakeup.py:30-32`), and today's rule refuses the first sibling woken. The releasing lane has no queued ticket at `PP=1` and is excluded, so this needs `attn_dp ≥ 3`. | G1, G3 and G4 `PP=1` byte comparison. `attn_dp=2` is expected unchanged. `attn_dp=4` is expected, not guaranteed, unchanged, and a difference stops the work for diagnosis (plan P3). | +| `attn_dp = 1`, any PP | One lane, so FIFO order equals heap order. | G5 byte comparison. | + +No capacity-1 or `PP=1` special case is added: no supported caller has been +shown to need arbitrary cross-lane full-stage FIFO order. An unexpected +difference in any of these classes stops the work and is reported. + +## Scope boundary: mixed-phase forwards + +This branch is based on `main`, where prefill and decode source lanes still +enter separate synchronization paths. The shared forward across mixed prefill +and decode lanes is PR 35 W3 (`65ed8a7`), not on `main`. Fixing admission does +not fix that. A shape that deadlocked at admission may, once admitted, reach a +mixed-phase cohort and fail another way. Such a failure is recorded and +diagnosed separately; it is not repaired by widening this one-file change. + +Consequences for verification: + +- The C1 witnesses are phase-controlled. All requests arrive at `t=0` with + equal prompt lengths, and the primary group is prefill-only + (`decode_tokens=1`). A `MONOLITHIC` request of that shape completes at the + prefill boundary, which grants its one decode token + (`request.py:1286-1293,1379-1384`), so no decode batch forms. +- Composition with PR 35 is validated in the parent task after this branch is + merged forward, before Step 9 is declared unblocked. ## Fidelity expectation, stated before measuring -| Scenario class | Expected after the change | -| --- | --- | -| Every `num_pipeline_stages = 1` scenario (co-location, PDD, PD-AF examples; Step 8 regression set) | Byte-identical `request_metrics.csv` and `system_metrics.json`. | -| PD-AF `DECODE_ATTN` / `DECODE_FFN` (capacity 1) | Byte-identical. | -| MoE `attn_dp > 1`, `PP > 1` | From drain to completion with request and token conservation. | -| Dense `attn_dp > 1`, `PP > 1` | Completes before and after; lane stage-busy intervals overlap after the change where they were serialized before, so makespan and per-request latencies **change**. This is the same defect's softer symptom and is proposed as an accepted fidelity fix (decision D-2 in the plan). | -| `attn_dp = 1` any PP | Byte-identical (one lane, FIFO order equals heap order). | +| Scenario class | Acceptance path (plan §4.2) | Expected after the change | +| --- | --- | --- | +| Declared `PP = 1` scenarios (release examples, synthetic `PP=1` cells) | U | Byte-identical metrics files. `attn_dp=4` cells carry the caveat in the table above. | +| PD-AF `DECODE_ATTN` / `DECODE_FFN` (capacity 1) in the declared recipes | U | Byte-identical, for the caller-level reason above. | +| `attn_dp = 1`, any PP | U | Byte-identical. | +| MoE `attn_dp > 1`, `PP > 1`, cells P0 classifies as admission deadlock | L | Completes, with request and token conservation. | +| MoE `attn_dp > 1`, `PP > 1`, cells that complete on base | T | Byte-identical, or a difference explained with the stage ledger. | +| Dense `attn_dp > 1`, `PP > 1` | T | Completes before and after. A lane refused only by FIFO position is admitted at once, so lane overlap increases and makespan and per-request latencies may **change**. This is the same defect's softer symptom and was accepted as a fidelity fix (D-2). | -Any difference outside the two "changes" rows is a defect in this change and -stops the work. +Any outcome outside its row stops the work. ## What this is not - Not a change to `full_stage_capacity`, the seal, the EP wave protocol, the sync rooms, or the wake-up helper. - Not a new flag or configuration field. +- Not a fix for mixed-phase forwards on `main` (PR 35 W3). - Not the Step 9 report key (D9-2 in the parent plan); that design resumes once this lands and the `attn_dp=2, PP=2` shape runs. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/plan.md b/task_memory/task_2026-09-22_stage_admission_ordering/plan.md index f2c786b3..cfcc4070 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/plan.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/plan.md @@ -4,11 +4,12 @@ | Date | Change | | --- | --- | +| 2026-09-23 | Applied the round-1 plan review (`review.md`). Changes: C1 now targets confirmed admission-deadlock witnesses from a phase-controlled group; C3 uses the stage ledger and overlap duration; C4 takes the reviewer's wording; P0 lists its artifacts and outcome classes; P2 covers both sides of the EP boundary, the `DECODE_FFN` dense-group control, a second admission round and per-fixture base expectations, and the dense fixture asserts a same-start condition that discriminates on the base; P3 has three acceptance paths. The matrix now publishes a concrete case list on the analytical backend and keeps `attn_dp=2, PP=3`. Added D-6 and D-7. Not executed. | | 2026-09-23 | D-1..D-5 adopted by the user; D-3 executed now (push + draft PR for remote review); D-5 adjusted so the records travel with the branch. Work packages P0–P4 unblocked. | | 2026-09-22 | Created for user review. Scope, acceptance criteria, work packages P0–P4 with dependencies, verification matrix, decisions D-1..D-4. No source change yet. | -Diagnosis, options and the recommended rule are in `design.md`. This file is the -executable plan. +Diagnosis, options and the recommended rule are in `design.md`. Review +dispositions are in `review.md`. This file is the executable plan. ## 1. Scope @@ -19,86 +20,197 @@ own branch: `/data/ycfeng/Frontier/.worktrees/stage-admission-ordering`. - Source change confined to `frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`: - the full-stage admission predicate in `try_acquire` and the two docstrings - that describe admission as "FIFO-head". + the full-stage admission predicate in `try_acquire`, removal of the admitted + ticket with `remove(ticket)`, and the two docstrings that describe admission + as "FIFO-head". - Out of scope: `full_stage_capacity`, the forward-group seal, EP wave - protocol, sync rooms, wake-up helper, any configuration field, and the Step 9 - report key. + protocol, sync rooms, wake-up helper, any configuration field, the Step 9 + report key, and mixed-phase forward failures on `main` (PR 35 W3; see + `design.md` "Scope boundary"). ## 2. Acceptance criteria | Id | Criterion | Settled by | | --- | --- | --- | -| C1 | MoE `attn_dp∈{2,4}`, `moe_ep = attn_dp`, `PP∈{2,3}` (layer count divisible by PP), offline, 4/6/8/12 requests: every request completes; request count, prefill tokens and decode tokens conserved; no "non-empty scheduler state" drain. | P2(c) integration test; P3 matrix. | -| C2 | Every `PP = 1` scenario and every PD-AF scenario in the matrix produces byte-identical `request_metrics.csv` and `system_metrics.json` before and after. | P0 baseline vs P3 rerun, SHA-256 per file. | -| C3 | Dense `attn_dp∈{2,4}`, `PP∈{2,3}`: completes before and after; the number of simulated intervals during which two lanes of one stage are busy at once is greater after than before; every metric difference is attributable to that overlap. | P3, with the lane-overlap count from `metrics_ground_truth.jsonl` or the batch-stage trace. | -| C4 | All existing unit tests pass without modification. If any existing test must change, stop and report; that is a design signal, not a test fix. | P2. | -| C5 | The predicate is one readable condition; module and method docstrings state the ordering contract as implemented; no flag, no config field, no `getattr` fallbacks. | Review of the diff against the quality gates. | -| C6 | The Step 9 boundary probe on MoE `attn_dp=2, moe_ep=2, PP=2` runs to completion on this branch (informational: it unblocks the parent task's design checkpoint). | P3, scratch probe rerun. | +| C1 | Repaired liveness (path L). Every G3a case that P0 classifies as `admission_deadlock` completes after P1, with request count, prefill tokens and decode tokens conserved. G3a is the phase-controlled, prefill-only group. P0 must find at least one such case for each of `(attn_dp, PP)` ∈ {2, 4} × {2, 3}; if a pair has none, stop and report before P1, because the case list does not exercise the defect there. | P0 classification; P2(c); P3 path L. | +| C2 | Unchanged controls (path U). Every run-to-run-stable metrics file is byte-identical before and after for G1 (all 30 release recipes, including the 10 PD-AF recipes), every `PP = 1` cell of G3a, G3b and G4, and every G5 cell. | P0 vs P3 `sha256sums.txt`. | +| C3 | Timing change (path T). Base-successful cases with `attn_dp > 1` and `PP > 1` (all G4 `PP > 1` cells, and G3a/G3b cells P0 classifies as `success`) are either byte-identical, or their difference is explained with the stage-ledger metric of §4.5 plus batch membership and component durations. The designated contention witnesses (§4.2, marked W) show a strictly larger `multi_lane_busy_time` after P1. In every case: no lane overlaps itself, and `peak_lanes ≤ attn_dp`. | P3, §4.5 metric from `frontier_stage_batch_ledger.jsonl`. | +| C4 | Existing passing tests must remain passing without assertion changes. Existing failures, collection errors, and skips must be compared against a fresh run of the exact base revision in the same environment. Any new failure or required change to an ordering assertion stops implementation for review. | G2, §4.6. | +| C5 | The predicate is one readable condition, and the module and method docstrings state the ordering contract as implemented. The change adds no flag, config field, `getattr` fallback, lane field on tickets, acquisition wake-up, PP-specific branch, second queue or capacity-1 special case. | Review of the diff against the quality gates. | +| C6 | Informational. The Step 9 boundary probe on MoE `attn_dp=2, moe_ep=2, PP=2` runs to completion on this branch, or its remaining failure is classified. Composition with PR 35 (W3 mixed-phase forward) is validated in the parent task after merge-forward, before Step 9 is declared unblocked. | P3 probe rerun; parent-task follow-up (§6). | ## 3. Work packages ```text -P0 baseline capture (no source change) +P0 evidence and baseline (no source change) -> P1 rule change - -> {P2 tests, P3 fidelity rerun and comparison} - -> P4 records, commit, push and draft PR (D-3) + -> {P2 tests, P3 rerun and comparison} + -> P4 records, commit, push ``` | Package | Content | Acceptance | | --- | --- | --- | -| P0 Baseline | On `1f694f7`, run the verification matrix of §4 and store outputs under `/data/ycfeng/tmp/stage_admission_ordering/baseline//` with a `sha256sums.txt` per scenario. Store the drain evidence already collected (`design.md` tables) and the reproduction scripts (`repro_main.py`, `drain_state.py`, `drain_lanes.py`, currently in the session scratchpad) under `tests/integration/stage_admission/` as the seed of P2(c). | Every scenario has a hash file; the four drain shapes are recorded as `DRAINED`. | -| P1 Rule | Implement the recommended rule from `design.md`: full-stage tickets are refused only by an EP wave queued ahead; EP waves keep the strict head rule. Update the `StageExecutionContext` class docstring ("A complete operation first enters the ready FIFO, then the owner admits it atomically") and the `try_acquire` docstring ("Acquire the FIFO-head ticket if this stage is currently idle") to describe what is now true. | Diff touches one source file; `python -m pytest tests/unit/test_stage_execution_context.py tests/unit/test_shared_forward_group_admission.py -q` passes unchanged. | -| P2 Tests | (a) Context-level, in `tests/unit/test_stage_execution_context.py`: a full-stage ticket queued behind another lane's queued full-stage ticket is admitted while capacity remains; a full-stage ticket queued behind an EP wave is refused; the two existing EP-order tests stay as they are. (b) Scheduler-level, in `tests/unit/test_shared_forward_group_admission.py` using its `make_stage`/`make_batch` helpers: rebuild the drain state (lane 1 active with a second ticket queued, lane 0 with two queued) and assert lane 0's `pop_batch_if_not_busy` returns its head and binds the same forward group. (c) Simulator-level, `tests/integration/test_stage_admission_pipeline_lanes.py`: one child process per shape (`IS_MOE` is process-global), MoE `attn_dp=2, moe_ep=2, PP=2` at 4 and 6 requests and `attn_dp=4, moe_ep=4, PP=2` at 8, asserting completion and conservation; dense `attn_dp=2, PP=2` asserting completion and lane overlap > 0. Expected values written from the scenario, not copied from a run. | (a) and (b) fail on `1f694f7` and pass after P1; (c) drains on `1f694f7` and passes after P1. | -| P3 Fidelity | Rerun the §4 matrix on the P1 revision into `.../after//`; `diff` the hash files; for every differing scenario, confirm it is in the C3 class and record the overlap count before/after and the makespan delta. Rerun the Step 9 boundary probe for C6. | C1, C2, C3, C6 tables in `test_report_2026-09-22_stage_admission_ordering.md`. | -| P4 Records | Test report, `progress.md`, `summary.md`; commit P1+P2 as one code commit and records as one docs commit; then, under D-3, push the branch and open a draft PR against `main` whose body carries the C1–C3 tables and the W9-01 cross-reference. Note in the parent task (`issues.md` W9-01) the branch and commit. | Pushed and verified, or explicitly left local if D-3 is not granted. | +| P0 Evidence and baseline | (1) Move the reproduction into `tests/e2e/stage_admission_matrix.py`, following the `tests/e2e/moe_ep_non_dummy_matrix.py` precedent. The module holds: the §4.1 fixture builder; the §4.2 case table; a runner with one child process per case, because `IS_MOE` is process-global; the outcome classifier and state-report writer of §4.3, which replace the session scripts `drain_state.py`/`drain_lanes.py`; and the §4.5 ledger metric. Outputs go to `resolve_scratch_root()/stage_admission_ordering/base//` (`tests/scratch_root.py`). (2) Confirm that the branch source equals `1f694f7` (`git diff --stat 1f694f7 -- . ':!task_memory' ':!.gitignore'` is empty). (3) Run R0, G1, G3a, G3b, G4, G5 and the G2 suites. (4) Write the §4.3 artifacts for every case. (5) Rerun two success cases (one G1 recipe, one G4 `PP=2` cell); an unstable file is named and excluded from C2, with the reason. (6) Record the classification table in the test report. | Every case has `case.json`, `run.json` and its class artifact. R0 reproduces the author-reported table, or each difference is explained. No `other_failure`. C1's per-pair witness condition holds. Every successful `attn_dp>1` case has `ATTN_DP_LANE` ledger rows for each of its lanes, otherwise §4.5 cannot be computed and P0 stops. | +| P1 Rule | Implement the `design.md` rule. Full-stage tickets are refused only by an EP wave queued ahead; EP waves keep the strict head rule; the admitted ticket leaves the FIFO by `remove(ticket)`. Update the `StageExecutionContext` class docstring ("A complete operation first enters the ready FIFO, then the owner admits it atomically") and the `try_acquire` docstring ("Acquire the FIFO-head ticket if this stage is currently idle") to state the implemented contract. Queued full-stage work may pass other full-stage work but not an earlier queued EP wave. Queued EP waves keep FIFO admission. Active layer-to-layer scope transitions remain a separate mechanism. | The diff touches one source file. `python -m pytest tests/unit/test_stage_execution_context.py tests/unit/test_shared_forward_group_admission.py -q` passes with no assertion change. | +| P2 Tests | **(a)** Contract tests in `tests/unit/test_stage_execution_context.py`. *Bypass* and *EP boundary* use a capacity-2 context (`ep_size=2`) with FIFO `full0, full1, wave0, full2`. *Bypass*: `full1` acquires before `full0`, and afterwards `queued_tickets == (full0, wave0, full2)`; `full2` is then refused although capacity remains, because `wave0` is ahead; `full0` acquires. *EP boundary* (acquisitions in head order, so it also runs on the base): `full0` and `full1` acquire; after `full1` releases, `full2` is still refused, because `wave0` is ahead; `wave0` is refused while `full0` is active and acquires once it releases; `full2` is refused while `wave0` is active and acquires after `wave0` releases. *Capacity 1*: on an idle context with FIFO `[full0, full1]`, `try_acquire(full1)` succeeds, pinning the API-level change stated in `design.md`. The two existing EP-order tests stay unchanged. **(a′)** A `DECODE_FFN` control in `tests/unit/test_mixed_layer_decode_ffn_scheduling.py` with its mixed-layer fixture. It materializes two successive `DenseFFNBatchGroup`s and a neighbouring EP group on one target replica and stage, through `_schedule_dense_ffn_from_m2n_group` and the real full-stage `ReplicaStageScheduler`. It asserts that FIFO order and heap order both follow the group counter, that the dense groups are admitted in counter order, and that neither dense group crosses an EP wave queued ahead of it. **(b)** A scheduler-level test in `tests/unit/test_shared_forward_group_admission.py` using its `make_stage`/`make_batch` helpers, parametrized over which lane enqueues first. It rebuilds the drain state: the first lane is active with a second ticket queued, and the other lane has two queued tickets. It asserts that the other lane's `pop_batch_if_not_busy` returns its heap head and binds the same forward group. It then continues through promotion to an EP wave and restoration to full-stage owners (`replace_full_stage_owners_with_ep_wave`, `replace_ep_wave_with_full_stage_owners`), release of both owners and `on_stage_end` of both lanes, and it asserts that both lanes admit their next queued batch into a later forward group, leaving the FIFO empty. **(c)** Simulator-level tests in `tests/integration/test_stage_admission_pipeline_lanes.py`, importing the §4.1 builder from `tests.e2e.stage_admission_matrix`, one child process per case. MoE witnesses `G3a-moe-dp2-pp2-n4` and `G3a-moe-dp4-pp2-n8` assert completion and conservation. The dense fixture `G4-dense-dp2-pp2-n8` asserts completion, and that the first stage-0 ledger rows of both lanes start at the same simulated time, because every request arrives at `t=0` and capacity admits both lanes into the first forward. A bare `multi_lane_busy_time > 0` would not discriminate: from source, the base already overlaps the lanes after the first release. Expected values are written from the scenario, not copied from a run. | Expected on `1f694f7`: (a) *bypass* fails at its first assertion, *capacity 1* fails, and *EP boundary* passes; (a′) passes; (b) fails at the other lane's first admission; (c) each MoE witness fails through the documented `admission_deadlock` signature, and the dense fixture completes but fails only its same-start assertion, because the second lane's first row starts at the first lane's first stage-0 end. After P1 all of them pass. The base failures are recorded as negative controls. | +| P3 Rerun | Rerun every P0 case on the P1 revision into `.../after//`, then apply the acceptance path of each case (§4.2). **U**: hashes identical. **L**: the case completes with conservation; there is no base metrics hash to compare. **T**: hashes identical, or the difference is explained by the §4.5 metric (before and after), batch membership and component durations; W cases must show a strict increase. Stop and report, adjusting nothing, on any of these: a U difference (including `attn_dp=4, PP=1`); an L case that fails in any other way, such as a mixed-phase failure in G3b; a T difference that the ledger does not explain; a class change outside these paths; a failure of the self-overlap or `peak_lanes ≤ attn_dp` checks. Rerun the Step 9 boundary probe for C6. | C1–C4 and C6 tables in `test_report__stage_admission_ordering.md`. | +| P4 Records | Test report, `progress.md`, `summary.md`. Commit P0's harness, P1 and P2 as code commits (harness separately from the rule, so the rule commit stays one file plus its tests), and the records as a docs commit. Push the branch and update the draft PR body with the C1–C3 tables. Note in the parent task (`issues.md` W9-01) the branch and commits. | Pushed and verified. | ## 4. Verification matrix -Environment: `/data/ycfeng/envs/frontier-py310/bin/python`, `PYTHONPATH` = the -worktree, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`. Each Simulator run in a -fresh process. +Environment: `/data/ycfeng/envs/frontier-py310/bin/python` (version recorded +in `run.json`), `PYTHONPATH` = the worktree, `WANDB_DISABLED=true`, +`VIDUR_DISABLE_WANDB=1`. Each Simulator run is a fresh process. Before P1 the +source tree is the base: P0 step (2) confirms that it equals `1f694f7`. -| Group | Scenarios | Expected | +### 4.1 Common fixture for the synthetic groups (R0, G3a, G3b, G4, G5) + +| Field | Value | +| --- | --- | +| Model | `num_layers=6` (divisible by PP 1, 2, 3), `num_q_heads=4`, `num_kv_heads=2`, `embedding_dim=256`, `mlp_hidden_dim=64`, `max_position_embeddings=4096`, `use_gated_mlp=True`, `use_bias=False`, `use_qkv_bias=False`, SiLU, RMSNorm, `post_attn_norm=True`, `vocab_size=1024`, `torch_dtype="bfloat16"`. MoE: `is_moe=True`, `num_experts=8`, `num_experts_per_tok=2`. Dense: `is_moe=False`. Injected by monkeypatching `BaseModelConfig.create_from_name`, as `tests/integration/test_pr33_nondummy_acceptance.py:170` does. | +| Replica | `device="a100"`, `network_device="a100_pairwise_nvlink"` (4 devices per node), `attn_tensor_parallel_size=1`, `attn_dp` and `num_pipeline_stages` per case, `memory_margin_fraction=0.1`. MoE: `moe_tensor_parallel_size=1`, `moe_expert_parallel_size=attn_dp`, `total_expert_num=8`, `router_topk=2`. | +| Replica scheduler | `VllmV1SchedulerConfig(num_blocks=128, block_size=16, batch_size_cap=4, max_tokens_in_batch=16, enable_chunked_prefill=True)`. With 16-token prompts each prefill batch carries one request. | +| Cluster scheduler, predictor | `RoundRobinClusterSchedulerConfig()`; `RandomForrestExecutionTimePredictorConfig(enable_dummy_mode=True)`. | +| Simulation | `simulation_mode="offline"`, `sys_arch="co-location"`, `enable_parallel_clusters=False`, `decode_cuda_graph_mode="none"`. | +| CC backend | `ClusterConfig.cc_backend_config = AnalyticalCCBackendConfig()` for G3a, G3b, G4 and G5, the public examples' choice (D-6). `analytical` applies no Replica-pod node-size rule, so `attn_dp=2, PP=3` (6 devices) is constructible; P0 confirms this, and a rejection is classified, not substituted. R0 leaves the default (`astra_sim_analytical`, `config.py:2494`) to reproduce the recorded evidence. | +| Arrivals | G3a, G3b, G4, G5: `StaticRequestIntervalGeneratorConfig()` (every request at `t=0`). R0: `PoissonRequestIntervalGeneratorConfig(qps=1e6)`. | +| Lengths | `FixedRequestLengthGeneratorConfig`. Profile **PF** (prefill-only): `prefill_tokens=16, decode_tokens=1`. Profile **PD**: `prefill_tokens=16, decode_tokens=3`. | +| Metrics | `write_metrics=True` (required: the stage ledger is written by `plot()`, which runs only with `write_metrics`), `store_request_metrics=True`, `store_plots=False`, `enable_chrome_trace=False`, `write_json_trace=False`; `store_frontier_stage_batch_ledger` left at its default `True`; `output_dir` = the case directory. | + +G1 recipes run unchanged with `PYTHON_BIN`, `METRICS_OUTPUT_DIR` and `RUN_ID` +overridden per case (each recipe reads these variables). + +### 4.2 Case list + +Case id: `--dp-pp-n`. "Base +hypothesis" is the expectation before P0: `PP=1` succeeds; at `PP>1`, +`n ≥ 2·attn_dp` (every lane holds two or more batches at stage 0) +deadlocks for MoE; otherwise the case succeeds. **P0's classification, not the +hypothesis, assigns each case its path.** Class `admission_deadlock` → path +L; class `success` at `attn_dp>1, PP>1` → path T; `PP=1` or `attn_dp=1` → +path U; any other class in P0 is handled by §4.3. + +| Group | Cases | Profile | Count | Base hypothesis | Path | +| --- | --- | --- | --- | --- | --- | +| R0 record | The 16 shapes of the `design.md` table: 15 author-run logs plus the MoE `dp2-pp3` rejection | PD, Poisson | 16 | As recorded in `design.md` | Evidence only. After P1: informational, classified the same way. | +| G1 release | The 30 `examples/architecture/{co-location,pdd,pd-af-disagg}/{offline,online}/*.sh` recipes (all `PP=1`) | recipe | 30 | success | U | +| G3a MoE, phase-controlled | `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}` × `n ∈ {4, 8, 12}` | PF | 18 | `PP=1`: success. `PP>1`: deadlock, except `dp4-n4`, which is success (one batch per lane). | U / L / T | +| G3b MoE, standard lengths | `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}` × `n ∈ {4, 8}` | PD | 12 | as G3a | U / L / T. A mixed-phase failure after P1 stops and is reported (scope boundary). | +| G4 dense | `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}` × `n ∈ {4, 8}` | PD | 12 | success; at `PP>1` the second lane's first admission waits for the first lane's first stage-0 release | `PP=1`: U. `PP>1`: T. Contention witnesses **W**: `n=8` at `PP ∈ {2,3}`, `attn_dp ∈ {2,4}` (4 cases). | +| G5 single lane | `attn_dp=1`, `PP ∈ {1,2,3}`, MoE (`moe_ep=1`) and dense, `n=6` | PD | 6 | success | U | +| G6 PD-AF | The 10 PD-AF recipes inside G1, plus `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`, `test_decode_ep_wave_materialization.py` and `test_prefill_ep_wave_materialization.py` | — | (in G1) | success; tests pass | U, C4 | +| G2 suites | `tests/unit` and `tests/integration` | — | 2 | base identities from P0 | C4 | + +Simulator runs: 30 + 18 + 12 + 12 + 6 = 78, plus 16 R0 record runs and the two +pytest suites. This satisfies the AGENTS.md gate of at least 50 concrete +settings. + +### 4.3 Outcome classes and artifacts + +| Class | Definition | +| --- | --- | +| `success` | `Simulator.run()` returns and every request is completed. | +| `admission_deadlock` | The run ends with "Sequential simulation ended with non-empty scheduler state", and the state report read from the live simulator objects shows this signature: on some stage context, active full-stage owners are fewer than capacity, no EP wave is active, and the bound forward group is unsealed; the FIFO head is a full-stage ticket whose lane is busy; another lane is idle, with a non-empty heap whose head ticket is queued behind that head; and a sync room of the bound group lists the busy lane and waits for the idle one. | +| `configuration_rejection` | A `ValueError` from configuration or topology validation, such as the Replica-pod node-size rule, wherever it surfaces. | +| `other_failure` | Anything else, including a drain without the signature. In P0 this stops the work before P1. | + +Every case directory holds: + +- `case.json`: the case id, group, every fixture field of §4.1, and the + resolved `SimulationConfig` as JSON. +- `run.json`: the command line; interpreter path and `python -VV`; a digest of + `pip freeze`; `git rev-parse HEAD`; whether `git status --porcelain` is clean + outside `task_memory/`; start and end wall time; outcome class; exception + type and message. +- One class artifact: + - `success`: `sha256sums.txt` over every file in the metrics directory. + - `admission_deadlock`: `state_report.json`, with per context the capacity, + sealed flag, bound group, active owners, and the FIFO mapped ticket → lane + → batch id → `global_id`; per lane the busy flag and heap; the sync rooms + with the lanes present; and the simulated time. No metrics hash. + - The two failure classes: `error.txt` with the traceback. + +`cases.jsonl` indexes the cases, one line each. Large outputs stay under the +scratch root; the test report keeps the classification table and the metrics. + +### 4.4 Acceptance paths + +| Path | Applies to | Pass condition | | --- | --- | --- | -| G1 release examples | The 30 `examples/architecture/{co-location,pdd,pd-af-disagg}/{offline,online}/*.sh` recipes (all `PP=1`), metrics dir redirected per scenario | Byte-identical (C2) | -| G2 Step 8 regression set | `pytest tests/unit -q --continue-on-collection-errors` and `pytest tests/integration -q --continue-on-collection-errors`; compare the FAILED **set** to the recorded `1f694f7` baseline (84 entries) | Same set (C4, C2) | -| G3 lanes × stages, MoE | `attn_dp∈{2,4}`, `moe_ep=attn_dp`, `PP∈{1,2,3}`, 6-layer tiny model (all three PP values divide 6), offline, requests ∈ {4, 6, 8, 12} | `PP=1` byte-identical; `PP∈{2,3}` drain → complete (C1) | -| G4 lanes × stages, dense | `attn_dp∈{2,4}`, `PP∈{1,2,3}`, same model with `is_moe=False`, requests ∈ {6, 8} | `PP=1` byte-identical; `PP∈{2,3}` overlap increases (C3) | -| G5 single lane | `attn_dp=1`, `PP∈{1,2,3}`, MoE and dense | Byte-identical | -| G6 PD-AF capacity-1 | The 10 PD-AF recipes from G1 plus `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`, `test_decode_ep_wave_materialization.py`, `test_prefill_ep_wave_materialization.py` | Byte-identical; tests pass (C2, C4) | +| U unchanged | G1, G5, every `PP=1` cell, G6 recipes | Identical `sha256sums.txt` (run-to-run-unstable files excluded by P0 with a reason). | +| L repaired liveness | Cases P0 classifies as `admission_deadlock` | `success` after P1; completed requests = generated requests; the sums of prefill and decode tokens over `request_metrics.csv` equal the generated lengths. | +| T timing | Cases P0 classifies as `success` with `attn_dp>1, PP>1` | Identical hashes, or a ledger-explained difference (§4.5). W cases: strictly larger `multi_lane_busy_time`. | + +### 4.5 Lane-overlap metric (C3) -Scenario count: 30 + 2 suites + 24 (G3) + 12 (G4) + 6 (G5) = 72 Simulator runs -plus the two pytest suites, satisfying the AGENTS.md gate of at least 50 -concrete settings. +Source: `frontier_stage_batch_ledger.jsonl` in the case's metrics directory. +Its rows carry `cluster_type`, `replica_id`, `stage_id`, `execution_scope`, +`replica_local_id`, `stage_start_ts` and `stage_end_ts` +(`metrics_store.py:4401-4422`). No production metrics change. + +For each physical stage `(cluster_type, replica_id, stage_id)`, take the rows +with `execution_scope == "ATTN_DP_LANE"` as half-open intervals +`[stage_start_ts, stage_end_ts)`, keyed by `replica_local_id`. + +| Output | Definition | +| --- | --- | +| `multi_lane_busy_time` | Total simulated time during which at least two distinct lanes have an open interval. Touching endpoints overlap for zero time; zero-length rows contribute nothing. | +| `peak_lanes` | The largest number of distinct lanes open at one instant. | +| `makespan` | The largest `stage_end_ts` in the ledger. | +| Checks | No lane's intervals overlap one another. `peak_lanes ≤ attn_dp`. | -Note on G3 `PP=3` with `attn_dp=2`: the collective-sim topology check rejects -`attn_dp=2, moe_ep=2, PP=3` (6 devices against node size 4). Those cells use -`attn_dp=4, moe_ep=4, PP=3` (12 devices) or are marked "rejected at -construction" and excluded; either way the rejection itself is byte-identical -before and after. +A count of overlapping intervals is not used, because it depends on how +intervals are partitioned. For a T difference, the report pairs the metric +before and after with the batch membership (`request_ids` per row) and the +component durations (`execution_time`) of the rows that moved; a changed +aggregate latency alone does not establish the cause. -## 5. Decisions (adopted by the user on 2026-09-23: "采纳你d1-d5的推荐决策") +### 4.6 Test-identity comparison (G2, C4) + +For `tests/unit` and `tests/integration` separately, run +`python -m pytest -q -p no:cacheprovider --continue-on-collection-errors --junitxml=/.xml` +on the base source (P0) and on P1, in the same environment. Compare the node +id → outcome maps (passed, failed, error, skipped) and the collection errors +per module. Pass: every base-passed node id still passes; no node id newly +fails or errors; collection errors and skips are unchanged. A base failure that +now passes is reported, not treated as a stop. No failure count from another +checkpoint is used. + +## 5. Decisions + +Adopted by the user on 2026-09-23 ("采纳你d1-d5的推荐决策"): | Id | Question | Recommendation | Outcome | | --- | --- | --- | --- | -| D-1 | Adopt option B from `design.md` (full-stage tickets are ordered only behind EP waves) rather than option A (lane-aware skip) or C/D. | B. A needs a wake path on acquisition that the DES does not have; C and D are rejected on the working gates. | Adopted. | +| D-1 | Adopt option B from `design.md` (full-stage tickets are ordered only behind EP waves) rather than option A (lane-aware skip) or C/D. | B. A adds lane identity and a dependency on peer acquisition that no wake covers; C and D are rejected on the working gates. | Adopted. | | D-2 | Accept that dense `attn_dp>1, PP>1` timelines change (lanes overlap instead of serializing). | Accept as a fidelity fix; the serialization is the same defect. | Adopted. | | D-3 | Authorize pushing `fix/stage-admission-ordering` and opening a draft PR against `main`. | Grant at P4; until then everything stays local. | Adopted and brought forward: the user reviews on the remote, so the branch is pushed and a draft PR opened with the plan itself (2026-09-23). Code commits follow per package. | | D-4 | Baseline for byte comparison is `origin/main` `1f694f7`. PR 35 will merge this branch later instead of carrying the fix itself. | Confirm. | Adopted. | | D-5 | `task_memory/` is ignored by `.gitignore` on `main` (line 171), so these records are local to the worktree unless force-added. Keep them local and archive the outcome in the parent task, or track them on this branch as PR 35 does? | Keep local; copy `summary.md` and the test report into the parent task at P4. | Adopted with one adjustment required by D-3: remote review needs the records on the branch, so `.gitignore` gets the same narrow exception PR 34/35 use (`task_memory/*` plus `!task_memory/task_2026-09-22_stage_admission_ordering/`). The copy into the parent task at P4 stands. | +Adopted from the round-1 review on 2026-09-23, under the user's instruction +"采纳高价值和必要决策" (`review.md`): + +| Id | Decision | Reason | +| --- | --- | --- | +| D-6 | The synthetic groups select `AnalyticalCCBackendConfig` explicitly and keep `attn_dp=2, PP=3`, instead of substituting `attn_dp=4, PP=3`. | The change is admission-only. The node-size rule belongs to the `collective_sim`/`astra_sim_analytical` backends, and substituting the shape would leave part of C1 untested. | +| D-7 | C1 witnesses come from the phase-controlled prefill-only group G3a. Mixed-phase failures are out of scope: stop, report, diagnose separately. Composition with PR 35 is checked in the parent task. | `main` lacks PR 35 W3; this keeps the admission repair separable from the mixed-phase lifecycle. | + ## 6. Dependencies and risks -- The reproduction and boundary scripts live in the session scratchpad - (`w10/repro_main.py`, `w10/drain_state.py`, `w10/drain_lanes.py`); P0 moves - them under `tests/`. -- Risk: a `DECODE_FFN` path that queues two distinct full-stage tickets from - different sibling lanes at the same time would see admission order change. - `design.md` argues this does not occur; G6 measures it. If G6 differs, stop - and report before adjusting anything. -- Risk: the C3 overlap count needs a per-lane busy-interval source. If - `metrics_ground_truth.jsonl` lacks stage-level lane intervals, P3 derives - them from the batch-stage trace; adding metrics is out of scope. +- The reproduction scripts still live in the session scratchpad + (`w10/probe_main.py`, `w10/repro_main.py`, `w10/drain_state.py`, + `w10/drain_lanes.py`); P0 replaces them with `tests/e2e/stage_admission_matrix.py`. +- Risk: the `PP=1, attn_dp=4` cells could differ through the wake-order + inversion described in `design.md`. That is a stop-and-report condition, + not an automatic acceptance. +- Risk: a `DECODE_FFN` or `DECODE_ATTN` path that queues full-stage tickets from + two schedulers on one context would see admission order change. `design.md` + gives the caller-level reason this is not expected; P2(a′) and G6 measure it. + If either differs, stop and report before adjusting anything. +- Risk: after P1, a G3b case may reach a mixed-phase cohort and fail (scope + boundary). Stop and report; it is not repaired on this branch. - The parent task's Step 9 resumes only after this branch is merged into `main` - and merged forward into `fix/issue26-correctness-pr`. + and merged forward into `fix/issue26-correctness-pr`. The parent task then + reruns G3b on that branch, where W3 is present, as the composition check + (C6). diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md index cc4d2639..604dcc01 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | Round-1 plan review verified against source and applied to `design.md`, `plan.md` and `requirements.md`; `review.md` created; resume prompt updated for round 2. Not executed. | | 2026-09-23 | Decisions adopted; records pushed for remote review. | | 2026-09-22 | Created. Worktree and branch created; defect reproduced on `origin/main`; plan and design written for review. No source change. | @@ -12,12 +13,13 @@ | Item | State | Evidence | | --- | --- | --- | | Worktree `.worktrees/stage-admission-ordering` on `fix/stage-admission-ordering` @ `1f694f7` | completed | `git worktree list` | -| Reproduction on `origin/main`, 15 shapes | completed | `design.md` shape table; logs under `/data/ycfeng/tmp/w10_repro/case_*.log` | +| Reproduction on `origin/main`, 15 author-run shapes | completed; P0 republishes them as group R0 from published inputs | `design.md` shape table; logs under `/data/ycfeng/tmp/w10_repro/case_*.log` | | Drain state dump (FIFO, active owners, lane heaps, sync room) | completed | `design.md` "Observed state at the drain" | | Root-cause diagnosis and option analysis | completed | `design.md` | -| Plan for review | completed, awaiting user | `plan.md` | +| Plan for review | completed; round-1 review applied | `plan.md`, `review.md` | | Records published for remote review (`.gitignore` exception, docs commit, push, draft PR) | completed 2026-09-23 | commit and PR recorded below | -| P0–P4 | pending | unblocked by R-4; P0 next | +| Round-1 plan review (10 findings) verified and applied | completed 2026-09-23 | `review.md`; plan D-6, D-7 | +| P0–P4 | pending | P0 is defined (`plan.md` P0, §4); it waits for the owner's start signal (R-5: "暂不执行") | ## Commands run (2026-09-22) @@ -42,3 +44,4 @@ worktree, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`. | Records commit | `62d25b9` (plan, design, requirements, progress, `.gitignore` exception) | | Draft PR | https://github.com/NetX-lab/Frontier/pull/36 | | Reviewer resume prompt | `review_prompt.md` in this directory | +| Round-1 review applied | the commit that adds `review.md` (`git log -- task_memory/task_2026-09-22_stage_admission_ordering/review.md`) | diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md index fcfa7ff7..375a4341 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-5: round-1 plan review verified and applied to the records; execution deferred. | | 2026-09-23 | R-4: D-1..D-5 adopted; push and draft PR authorized. | | 2026-09-22 | Created from the Step 9 finding W9-01 in `task_2026-09-21_issue26_correctness_pr`; recorded the user's scope decision and the request for a reviewable plan. | @@ -26,6 +27,11 @@ a recommendation to fix it as a separate correctness item): > 先给出"共享 admission 排序问题"的修复计划,将具体的计划落地到文档，由我审阅 +`[Original Request]` (2026-09-23, after the first external review of PR 36 at +`a6ec6a6`; the pasted review is recorded finding by finding in `review.md`): + +> 以下是最新review结果，请你核实每个comments，采纳高价值和必要决策，修复完善docs，暂不执行。 + Quality gates the user repeated for every core-module change in this line of work, carried over verbatim: @@ -39,6 +45,7 @@ work, carried over verbatim: | R-2 | Branch `fix/stage-admission-ordering` from `origin/main` `1f694f7`, worktree `/data/ycfeng/Frontier/.worktrees/stage-admission-ordering`. | agent, under R-1 | | R-3 | No source change before the user reviews `plan.md` and `design.md`. | user, 2026-09-22 | | R-4 | Plan decisions D-1..D-5 adopted as recommended. Push the branch and open a draft PR so the review happens on the remote; the reviewer resumes from a prepared prompt. | user, 2026-09-23 | +| R-5 | Verify every review finding against the source, adopt the high-value and necessary corrections into the records (dispositions in `review.md`, new decisions D-6 and D-7 in `plan.md`), and do not execute: no P0 run, no source change. The docs commit is pushed to the draft PR under R-4. | user, 2026-09-23 | ## Constraints carried from the parent task diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/review.md b/task_memory/task_2026-09-22_stage_admission_ordering/review.md new file mode 100644 index 00000000..aa67897f --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/review.md @@ -0,0 +1,48 @@ +# Stage admission ordering under pipeline parallelism — Review record + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-23 | Created. First external plan review of PR 36 at `a6ec6a6` recorded; each finding re-checked against `1f694f7` source, with a disposition and the place it was applied. | + +## Round 1: plan review of PR 36 at `a6ec6a6` + +| Item | Value | +| --- | --- | +| Component / phase | Plan and design, before P0; no source change on the branch | +| Reviewer | External review agent, started from `review_prompt.md` | +| Inspected by the reviewer | `stage_execution_context.py`, `replica_stage_schduler.py`, `sync_entry.py`, `base_replica_scheduler.py`, `stage_contexts.py`, `stage_wakeup.py`, `batch_stage_end_event.py`, `replica_stage_schedule_event.py`, `base_event.py`, `round_robin_cluster_scheduler.py`, `metrics_store.py`, `batch_stage.py`, the three admission test files, `AGENTS.md`, and the task records | +| Reviewer's recommendation | Conditional GO for option B. P0 may proceed. Before P1: fix C4 and the matrix outcome classes, name the stage ledger for C3, publish the reproduction inputs, and qualify the capacity-1 and option-A claims. | +| Re-check | Every source anchor below re-read on `1f694f7` in this worktree on 2026-09-23. No simulator was run. | +| Owner instruction | "核实每个comments，采纳高价值和必要决策，修复完善docs，暂不执行" (requirements R-5) | + +### Findings and dispositions + +| # | Reviewer verdict | Re-check against source | Disposition | Applied in | +| --- | --- | --- | --- | --- | +| 1 | Agree with the diagnosis; the cited admission loop is the wrong branch | Confirmed. `base_replica_scheduler.py:893` is the unified `DECODE` loop. The co-location reproduction runs the `MONOLITHIC`/`PREFILL` `else` branch at `:1037-1054`, with the loop at `:1039`. Both loops use the same `num_running_batches < num_stages` bound. | Adopted. Anchor corrected. The shape table is labelled author-reported until P0 republishes it from the case inputs. | `design.md` "The defect" and shape table | +| 2 | Agree with B; make `remove(ticket)` explicit; test both sides of the EP boundary | Confirmed. `try_acquire` ends with `popleft()` at `stage_execution_context.py:337`, which would remove the wrong ticket once a non-head ticket can be admitted. `cancel` already uses `self._ready_fifo.remove(ticket)` (`:456`). | Adopted. The sketch removes the admitted ticket with `remove(ticket)`. P2(a) adds a `full0, full1, wave0, full2` contract test. | `design.md` "Recommended rule"; `plan.md` P1, P2(a) | +| 3 | Needs evidence. At capacity 1 the context API does change. The unchanged-behaviour claim belongs to the callers. | Confirmed on four points:
(a) On an idle capacity-1 context with FIFO `[full0, full1]`, the current rule refuses `full1` and B admits it.
(b) `DenseFFNBatchGroup` takes `global_id = _batch_group_creation_counter` (`round_robin_cluster_scheduler.py:1097,1118`). It gets one full-stage ticket (`:1138`) and is queued on the one full-stage scheduler per replica (`:1100`).
(c) EP child batches share one `EP_WAVE` ticket (`:1052-1057`).
(d) New fact: `enqueue_ep_wave` has no other caller. Queued EP waves therefore exist only on `DECODE_FFN` contexts. On `MONOLITHIC`/`PREFILL`/`DECODE`, `EP_WAVE` is only an active-scope transition of owners that were already admitted. | Adopted. The section is retitled and restated as a caller-level condition, with the API-level change stated explicitly. P2(a′) adds the control with two successive dense FFN groups and a neighbouring EP group. No capacity-1 special case is added. | `design.md` "Where behaviour is expected to stay unchanged"; `plan.md` P2(a′) | +| 4 | Needs evidence. Acquisition emits no wake, but the exact stall trace for option A is not established. | Confirmed on four points:
(a) `BatchStageEndEvent` emits the releasing lane's own retry (`batch_stage_end_event.py:139-146`) before its sibling retries (`:148-158`).
(b) `build_stage_wakeup_events` orders siblings by lane key, not by FIFO position (`stage_wakeup.py:30-32`).
(c) The queue key is `_priority_number = (time, id, event_type)` (`base_event.py:63-64`, `simulator.py:1268`). `BaseEvent.__lt__` (`:66-70`) compares type before id.
(d) A refused attempt returns `[]` (`replica_stage_schedule_event.py` "No batch to schedule" branch). | Adopted. A stays rejected on design grounds. The first-draft trace is now labelled an unverified hypothesis and is not pursued, because B does not depend on it. No acquisition wake-up is added. | `design.md` Options table, row A | +| 5 | Disagree with C4 as written | Confirmed. C4 required that all unit tests pass, while G2 expected a baseline failure set of 84 imported from another checkpoint. | Adopted. C4 is replaced with the reviewer's wording, verbatim. G2 now compares node id → outcome per suite against a fresh run of the base source in the same environment. | `plan.md` C4, §4.6 | +| 6 | Needs evidence. The liveness claims are too broad, and a mixed-phase scope boundary is missing. | Confirmed on three points:
(a) `attn_dp=2, PP=2` with 3 requests completes on main (author-run log), so the shape alone does not imply a drain.
(b) The shared forward across mixed prefill and decode source lanes is PR 35 W3 (`65ed8a7`), which is not on main.
(c) New fact: a `MONOLITHIC` request with `decode_tokens=1` completes at the prefill boundary (`request.py:1286-1293,1379-1384`), so it gives a prefill-only witness. | Adopted. The drain condition is restated as a queued-ticket arrangement. Mixed-phase failures become a stop-and-report boundary. C1 witnesses come from a phase-controlled prefill-only group. Composition with PR 35 is checked in the parent task before Step 9 resumes. | `design.md` "The defect", "Scope boundary"; `plan.md` C1, C6, G3a/G3b, §6 | +| 7 | Disagree with the matrix as written | Confirmed on three points:
(a) G3 labelled every `PP>1` cell "drain → complete".
(b) P3 allowed differences only in C3.
(c) DP2/PP3 was replaced by DP4/PP3.
New fact: the node-size rule (`parallel_semantics.py:236-262`) is applied only when materializing `collective_sim` (`cluster.py:203`) and `astra_sim_analytical` (`cluster.py:267`). `analytical` has no such rule, so `attn_dp=2, PP=3` on 6 devices is constructible there. The earlier probe used the default `astra_sim_analytical` (`config.py:2494`), and there it was rejected (parent W9-02). | Adopted. P0 now classifies outcomes into four classes. Acceptance runs on three separate paths: U unchanged, L repaired liveness, T timing. Synthetic cases set `AnalyticalCCBackendConfig` explicitly. A concrete case list is published. | `plan.md` §4.1-§4.4, P3 | +| 8 | Agree; name the ledger and measure overlap duration | Confirmed. `frontier_stage_batch_ledger.jsonl` rows carry `cluster_type`, `replica_id`, `stage_id`, `execution_scope`, `replica_local_id`, `stage_start_ts` and `stage_end_ts` (`metrics_store.py:4401-4422`). `execution_scope` is `ATTN_DP_LANE` for lane rows outside `DECODE_FFN` (`:1510-1521`). Capture defaults to on (`config.py:1202-1205`). The file is written by `plot()`, which runs only with `write_metrics=True` (`metrics_store.py:62-67,2200-2218`). | Adopted, with the reviewer's metric definition. The fixture sets `write_metrics=True`, because the earlier probe's `write_metrics=False` would have produced no ledger. | `plan.md` C3, §4.5 | +| 9 | Needs evidence. P0 must make the evidence reproducible. P2(b) and P2(c) need correcting. | Confirmed. The reproduction scripts exist only in the session scratchpad. P2(b) covered only the first blocked group. P2(c) said "drains on main" for the dense fixture too. | Adopted. P0 now has an artifact list per case, and DRAINED cases get a state report instead of a hash. P2(b) runs through release and restore into the next group, in both lane orders. P2(c) gives per-fixture base expectations. | `plan.md` P0, P2(b), P2(c), §4.3 | +| 10 | Agree; narrow the queue bound | Confirmed. The per-lane bound comes from the admission loops (`:893`, `:1039`). `DECODE_FFN` queues are fed by M2N groups and have no such bound. | Adopted. The bound is restated for the shared-lane contexts only. | `design.md` "Recommended rule" | + +### Items not adopted, and why + +- **A finite event trace for option A (finding 4, optional).** Not produced. It would require implementing A in order to reject it, and the reviewer states that B does not depend on it. The trace stays labelled unverified. +- **None of the required corrections was declined.** + +### New facts found during the re-check (not in the review) + +1. Queued EP waves exist only on `DECODE_FFN` (see finding 3(d)). On the shared-lane contexts where the defect lives, B's "EP wave queued ahead" clause never fires, and B reduces to "admit any queued full-stage ticket within capacity and seal". +2. Sibling wake-ups follow lane-key order (`stage_wakeup.py:30-32`), not FIFO order. At `PP=1` with `attn_dp ≥ 3`, one release can wake two idle siblings whose tickets are queued in the opposite order. The current rule refuses the first sibling woken. So `PP=1` cells with `attn_dp=4` are expected, not guaranteed, to stay unchanged. A difference there stops the work for diagnosis (plan P3). It is not accepted automatically. +3. The ledger needs `write_metrics=True` (see finding 8). + +### Status after round 1 + +Docs corrected. P0 has not started, per the owner's "暂不执行". The next step is the owner's decision to start P0. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md b/task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md index eb8df287..e2b954b1 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md @@ -1,23 +1,34 @@ # Resume prompt for the reviewing agent +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-23 | Round 2: the prompt now asks the reviewer to verify the round-1 dispositions in `review.md` and the corrected plan. | +| 2026-09-23 | Created for round 1. | + Copy everything below the line into the review agent's first message. --- -You are reviewing draft PR https://github.com/NetX-lab/Frontier/pull/36 on `NetX-lab/Frontier`, branch `fix/stage-admission-ordering`, base `main` at `1f694f7`. Start with the repository's `AGENTS.md`, then read, in this order, under `task_memory/task_2026-09-22_stage_admission_ordering/`: `requirements.md`, `design.md`, `plan.md`, `progress.md`. +You are continuing the review of draft PR https://github.com/NetX-lab/Frontier/pull/36 on `NetX-lab/Frontier`, branch `fix/stage-admission-ordering`, base `main` at `1f694f7`. This is round 2. In round 1 you reviewed the plan at `a6ec6a6` and gave a conditional GO for option B, with ten findings. The owner had every finding verified against the source and the records corrected. Nothing was executed: no P0 run, no source change. -Context. Frontier is a discrete-event LLM inference simulator. One `StageExecutionContext` (`frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`) owns each physical `(replica, stage)` and is shared by that stage's attention-DP lanes, each of which has its own `ReplicaStageScheduler` (`replica_stage_schduler.py`) with its own batch heap and `_is_busy` flag. `add_batch` mints a `StageAdmissionTicket` at batch arrival into the shared `_ready_fifo`; `pop_batch_if_not_busy` later asks `try_acquire` for the lane's heap head, which is refused unless the ticket is the strict FIFO head (after the EP-active, capacity and forward-group-seal checks). `full_stage_capacity` equals `attn_dp` for `MONOLITHIC`, `PREFILL` and `DECODE`, so the lanes of one forward co-own the stage; MoE lanes then meet in a sync room (`frontier/scheduler/utils/sync_entry.py`), which stands in an idle batch for a missing lane only when that lane has no queued work or the group is sealed (`_can_supply_idle_lane`). +Start with the repository's `AGENTS.md`. Then read, under `task_memory/task_2026-09-22_stage_admission_ordering/`, in this order: `review.md` (your findings, the source re-check, the disposition of each, and three new facts found during the re-check), `design.md`, `plan.md`, `requirements.md` (R-5), `progress.md`. -The defect, reproduced on `main`: with `num_pipeline_stages > 1`, `BaseReplicaScheduler.on_schedule` admits up to `num_pipeline_stages` batches per lane per round, so a lane holds several queued tickets while consuming one. The FIFO head can then be a ticket whose lane is busy inside the sync room; the other lane is refused although capacity is free; the room does not stand it in because it has work and the group is open. MoE `attn_dp∈{2,4}, PP=2` drains with requests unfinished; dense completes but serializes its lanes; every `PP=1` shape and every capacity-1 context completes. `design.md` has the drain-state table (FIFO, active owners, lane heaps, sync room). +Background, briefly. One `StageExecutionContext` (`frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`) owns each physical `(replica, stage)` and is shared by that stage's attention-DP lanes. Today `try_acquire` admits only the strict FIFO head. At `PP > 1` a lane holds several queued tickets while it consumes one, so a busy lane's queued ticket at the head can block an idle lane that its own sync room is waiting for. Option B, adopted as D-1: a full-stage ticket is refused only by an EP wave queued ahead of it; EP waves keep the strict head rule; the admitted ticket leaves the FIFO by `remove(ticket)`. -The plan is at the review-before-code stage: the PR has records only, no source change yet. The recommended rule (option B, adopted by the owner as D-1) changes one predicate in `try_acquire`: a full-stage ticket is refused only when an EP wave is queued ahead of it; EP waves keep the strict head rule; capacity, seal and EP-active checks are unchanged. Option A (lane-aware skip of tickets whose lane already holds an active ticket) was rejected because the DES wakes sibling lanes only at release (`frontier/scheduler/utils/stage_wakeup.py`, called from `BatchStageEndEvent`), not at a peer's acquisition, so A leaves a stranded-lane state after the first cohort releases; `design.md` traces that sequence. Decisions D-2 (dense `attn_dp>1, PP>1` timelines may change because lanes now overlap), D-4 (byte-comparison baseline is `main` `1f694f7`) and D-5 (records tracked on the branch through a narrow `.gitignore` exception) are also adopted. +What to check, in priority order: -What to review, in priority order: +1. For each of the ten findings in `review.md`: is the disposition faithful to what you asked, and is it applied where the table says? Flag anything weakened, misread or missing. +2. The three new facts in `review.md`. Check each against the source: + - Queued EP waves exist only on `DECODE_FFN`, because `enqueue_ep_wave`'s sole caller is `round_robin_cluster_scheduler.py:1052`. + - Sibling wake-ups follow lane-key order (`stage_wakeup.py:30-32`), which makes `PP=1, attn_dp=4` an expected-unchanged class rather than a guaranteed one. + - The stage ledger is written only with `write_metrics=True`. -1. The diagnosis in `design.md`: does the source support the circular wait exactly as stated? Check `try_acquire`, `pop_batch_if_not_busy`, `_can_supply_idle_lane`, the admission loop in `base_replica_scheduler.py`, and the wake-up path. -2. The recommended rule: is there any code path where two full-stage tickets from different lanes must stay in arrival order? Look at `DECODE_FFN` (capacity 1, shared sibling tickets, `DenseFFNBatchGroup` in `round_robin_cluster_scheduler.py`) and at `tests/unit/test_stage_execution_context.py`, `tests/unit/test_shared_forward_group_admission.py`, `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`. The plan asserts all existing tests pass unchanged (C4); say whether you agree from reading them. -3. The rejection of option A: confirm or refute the stranded-lane trace using the event classes under `frontier/events/`. -4. The fidelity expectation and the 72-scenario matrix in `plan.md` §4: are the "byte-identical" classes correctly bounded, and is the dense overlap change (C3) measurable with existing outputs? -5. Fit with the owner's core-module gates: readability, no hard-coding, no temporary patches, no over-defensive branches, no redundant mechanisms, plain domain names. + Also check one correction made during the re-check: the first draft's dense "lanes serialized" label is withdrawn, because from source the base already overlaps lanes after the first release. P2(c)'s dense assertion was changed to a same-start condition for that reason. +3. `design.md` "Where behaviour is expected to stay unchanged". Is the caller-level condition correct and sufficient for `DECODE_FFN` and `DECODE_ATTN`, and is it stated with the right strength? +4. `plan.md` §4: fixture, case list, outcome classes and signature, acceptance paths U/L/T, the ledger metric, and the test-identity comparison. Can P0 run from this text alone? Are the base hypotheses and paths consistent with C1–C4? Is any stop condition missing? +5. P2(a), (a′), (b) and (c). Does each test fail on the base for the stated reason and pass after P1? Is any of them redundant with an existing test? +6. Fit with the owner's core-module gates: readability, no hard-coding, no temporary patches, no over-defensive branches, no redundant mechanisms, plain domain names. This covers the planned harness `tests/e2e/stage_admission_matrix.py` as well as the one-file rule change. -Report findings as a numbered list with a source anchor (`path:line`) and a verdict per item (agree / disagree / needs evidence), then a one-paragraph recommendation on whether P1 may start as planned. Do not change source or push; the owner decides. +Report findings as a numbered list with a source or record anchor (`path:line`) and a verdict per item (agree / disagree / needs evidence). End with a one-paragraph recommendation on whether P0 may start as written. Do not change source or push; the owner decides. From a054d87b640368dda3708d1202595e3c123d254b Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 02:05:00 +0800 Subject: [PATCH 04/19] tests: add the stage admission case matrix for attention-DP lanes under PP Runs the case list of the stage-admission-ordering plan (section 4) one child process per case: synthetic MoE/dense shapes, the release recipes, and the vLLM-aligned shapes. Each case records its inputs, run provenance and one outcome artifact (metrics hashes, a live drain-state report, or the error). The drain classifier checks the admission-deadlock signature from the live stage contexts, lane queues and sync rooms. The compare command applies the unchanged / repaired-liveness / timing paths, using the stage-ledger lane-overlap metric. --- tests/e2e/stage_admission_matrix.py | 742 ++++++++++++++++++++++++++++ 1 file changed, 742 insertions(+) create mode 100644 tests/e2e/stage_admission_matrix.py diff --git a/tests/e2e/stage_admission_matrix.py b/tests/e2e/stage_admission_matrix.py new file mode 100644 index 00000000..38bcc8dc --- /dev/null +++ b/tests/e2e/stage_admission_matrix.py @@ -0,0 +1,742 @@ +#!/usr/bin/env python3 +"""Case matrix for stage admission of attention-DP lanes under pipeline parallelism. + +Runs the case list of the stage-admission-ordering plan +(``task_memory/task_2026-09-22_stage_admission_ordering/plan.md`` §4) on the +current source tree and writes, for each case, its inputs (``case.json``), its +run provenance (``run.json``) and one outcome artifact: + +* ``success``: ``sha256sums.txt`` over the copied metrics tree; +* ``admission_deadlock``: ``state_report.json`` read from the live scheduler + objects after the sequential run ends with work left; +* ``configuration_rejection`` / ``other_failure``: ``error.txt``. + +Each case runs in its own child process because ``IS_MOE`` is process-global. +Every child writes its simulator output under ``/work/``, a path +shared by all sets, so that files embedding the output path compare byte for +byte between a set run before a change and one run after it. + +Usage:: + + python -m tests.e2e.stage_admission_matrix run --set base [--group G3a] + python -m tests.e2e.stage_admission_matrix compare --before base --after after +""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import os +import shutil +import subprocess +import sys +import time +import traceback +from collections import defaultdict +from concurrent.futures import ThreadPoolExecutor +from dataclasses import asdict, dataclass +from importlib import metadata as importlib_metadata +from pathlib import Path +from typing import Iterable, Sequence + +from tests.scratch_root import resolve_scratch_root + + +REPO_ROOT = Path(__file__).resolve().parents[2] +MATRIX_DIR_NAME = "stage_admission_ordering" +DRAIN_MESSAGE = "Sequential simulation ended with non-empty scheduler state" +ATTN_DP_LANE = "ATTN_DP_LANE" + +SUCCESS = "success" +ADMISSION_DEADLOCK = "admission_deadlock" +CONFIGURATION_REJECTION = "configuration_rejection" +OTHER_FAILURE = "other_failure" + +# Plan §4.1 synthetic fixture and §4.7 vLLM-aligned fixture. +SYNTHETIC = "synthetic" +VLLM_ALIGNED = "vllm_aligned" +VLLM_ALIGNED_MODELS = {True: "Qwen3-30B-A3B-tiny", False: "Llama-3.2-1B-Instruct"} + +PREFILL_ONLY = (16, 1) +PREFILL_DECODE = (16, 3) +VLLM_ALIGNED_PREFILL_ONLY = (256, 1) + + +@dataclass(frozen=True) +class Case: + case_id: str + group: str + fixture: str = SYNTHETIC + is_moe: bool = False + attn_dp: int = 1 + stages: int = 1 + num_requests: int = 0 + prefill_tokens: int = 0 + decode_tokens: int = 0 + arrival: str = "static" + cc_backend: str = "analytical" + recipe: str | None = None + contention_witness: bool = False + + @property + def path(self) -> str: + """Plan §4.2 path before P0 classification refines L/T.""" + if self.recipe is not None or self.stages == 1 or self.attn_dp == 1: + return "U" + return "LT" + + +def _shape_id(group: str, is_moe: bool, attn_dp: int, stages: int, num_requests: int) -> str: + kind = "moe" if is_moe else "dense" + return f"{group}-{kind}-dp{attn_dp}-pp{stages}-n{num_requests}" + + +def _synthetic(group: str, is_moe: bool, attn_dp: int, stages: int, num_requests: int, + lengths: tuple[int, int], **fields) -> Case: + return Case( + case_id=_shape_id(group, is_moe, attn_dp, stages, num_requests), + group=group, + is_moe=is_moe, + attn_dp=attn_dp, + stages=stages, + num_requests=num_requests, + prefill_tokens=lengths[0], + decode_tokens=lengths[1], + **fields, + ) + + +def _release_recipes() -> list[Case]: + architecture_root = REPO_ROOT / "examples" / "architecture" + recipes = [] + for architecture in ("co-location", "pdd", "pd-af-disagg"): + for mode in ("offline", "online"): + for script in sorted((architecture_root / architecture / mode).glob("*.sh")): + recipes.append( + Case( + case_id=f"G1-{architecture}-{mode}-{script.stem}", + group="G1", + recipe=str(script.relative_to(REPO_ROOT)), + ) + ) + return recipes + + +def build_cases() -> list[Case]: + """Return the plan §4.2 case list in a fixed order.""" + cases: list[Case] = [] + # R0 reproduces the author-reported shapes of design.md with their inputs: + # Poisson arrivals, the default CC backend and prefill 16 / decode 3. + r0_shapes = [ + (True, 2, 2, 3), (True, 2, 2, 4), (True, 2, 2, 6), (True, 4, 2, 8), + (True, 2, 1, 6), (True, 2, 1, 12), (True, 4, 1, 8), (True, 4, 1, 12), + (True, 1, 2, 6), (True, 1, 3, 6), + (False, 2, 2, 6), (False, 4, 2, 8), (False, 2, 1, 6), (False, 4, 1, 8), + (False, 1, 2, 6), (True, 2, 3, 6), + ] + for is_moe, attn_dp, stages, num_requests in r0_shapes: + cases.append( + _synthetic("R0", is_moe, attn_dp, stages, num_requests, PREFILL_DECODE, + arrival="poisson", cc_backend="default") + ) + cases.extend(_release_recipes()) + for attn_dp in (2, 4): + for stages in (1, 2, 3): + for num_requests in (4, 8, 12): + cases.append(_synthetic("G3a", True, attn_dp, stages, num_requests, PREFILL_ONLY)) + for attn_dp in (2, 4): + for stages in (1, 2, 3): + for num_requests in (4, 8): + cases.append(_synthetic("G3b", True, attn_dp, stages, num_requests, PREFILL_DECODE)) + for attn_dp in (2, 4): + for stages in (1, 2, 3): + for num_requests in (4, 8): + cases.append( + _synthetic("G4", False, attn_dp, stages, num_requests, PREFILL_DECODE, + contention_witness=stages > 1 and num_requests == 8) + ) + for is_moe in (True, False): + for stages in (1, 2, 3): + cases.append(_synthetic("G5", is_moe, 1, stages, 6, PREFILL_DECODE)) + for is_moe in (True, False): + for num_requests in (8, 16): + cases.append( + _synthetic("G7", is_moe, 2, 2, num_requests, VLLM_ALIGNED_PREFILL_ONLY, + fixture=VLLM_ALIGNED) + ) + return cases + + +# --------------------------------------------------------------------------- +# Fixture (runs inside the child process) +# --------------------------------------------------------------------------- + + +def _synthetic_model(is_moe: bool): + from frontier.config import BaseModelConfig + from frontier.types import ActivationType, NormType + + model = BaseModelConfig( + num_layers=6, num_q_heads=4, num_kv_heads=2, embedding_dim=256, + mlp_hidden_dim=64, max_position_embeddings=4096, use_gated_mlp=True, + use_bias=False, use_qkv_bias=False, activation=ActivationType.SILU, + norm=NormType.RMS_NORM, post_attn_norm=True, vocab_size=1024, + is_moe=is_moe, num_experts=8 if is_moe else 0, + num_experts_per_tok=2 if is_moe else 0, torch_dtype="bfloat16", + ) + model._model_name = f"stage_admission_{'moe' if is_moe else 'dense'}" + registered = BaseModelConfig.create_from_name.__func__ + BaseModelConfig.create_from_name = classmethod( + lambda cls, name: model if name == model._model_name else registered(cls, name) + ) + return model._model_name + + +def build_config(case: Case, output_dir: Path, cache_dir: Path): + """Build the SimulationConfig of one synthetic or vLLM-aligned case.""" + from frontier.cc_backend.cc_backend_config import AnalyticalCCBackendConfig + from frontier.config import ( + ClusterConfig, FixedRequestLengthGeneratorConfig, MetricsConfig, + PoissonRequestIntervalGeneratorConfig, RandomForrestExecutionTimePredictorConfig, + ReplicaConfig, RoundRobinClusterSchedulerConfig, SimulationConfig, + StaticRequestIntervalGeneratorConfig, SyntheticRequestGeneratorConfig, + VllmV1SchedulerConfig, + ) + + if case.fixture == SYNTHETIC: + model_name = _synthetic_model(case.is_moe) + device, network_device = "a100", "a100_pairwise_nvlink" + scheduler = VllmV1SchedulerConfig( + num_blocks=128, block_size=16, batch_size_cap=4, + max_tokens_in_batch=16, enable_chunked_prefill=True, + ) + elif case.fixture == VLLM_ALIGNED: + model_name = VLLM_ALIGNED_MODELS[case.is_moe] + device, network_device = "h800", "h800_dgx" + scheduler = VllmV1SchedulerConfig( + num_blocks=1024, block_size=16, batch_size_cap=4, + max_tokens_in_batch=case.prefill_tokens, enable_chunked_prefill=True, + ) + else: + raise ValueError(f"unknown fixture {case.fixture!r}") + + moe_fields = ( + dict(moe_tensor_parallel_size=1, moe_expert_parallel_size=case.attn_dp) + if case.is_moe else {} + ) + replica = ReplicaConfig( + model_name=model_name, device=device, network_device=network_device, + num_pipeline_stages=case.stages, attn_tensor_parallel_size=1, + attn_dp=case.attn_dp, memory_margin_fraction=0.1, **moe_fields, + ) + cluster_fields = {} + if case.cc_backend == "analytical": + cluster_fields["cc_backend_config"] = AnalyticalCCBackendConfig() + elif case.cc_backend != "default": + raise ValueError(f"unknown CC backend selector {case.cc_backend!r}") + cluster = ClusterConfig( + replica_config=replica, + replica_scheduler_config=scheduler, + cluster_scheduler_config=RoundRobinClusterSchedulerConfig(), + execution_time_predictor_config=RandomForrestExecutionTimePredictorConfig( + enable_dummy_mode=True + ), + **cluster_fields, + ) + if case.arrival == "static": + interval = StaticRequestIntervalGeneratorConfig() + elif case.arrival == "poisson": + interval = PoissonRequestIntervalGeneratorConfig(qps=1e6) + else: + raise ValueError(f"unknown arrival process {case.arrival!r}") + return SimulationConfig( + simulation_mode="offline", sys_arch="co-location", + enable_parallel_clusters=False, decode_cuda_graph_mode="none", + cluster_config=cluster, + metrics_config=MetricsConfig( + output_dir=str(output_dir), cache_dir=str(cache_dir), + run_id=case.case_id, write_metrics=True, store_request_metrics=True, + store_plots=False, enable_chrome_trace=False, write_json_trace=False, + ), + request_generator_config=SyntheticRequestGeneratorConfig( + num_requests=case.num_requests, + length_generator_config=FixedRequestLengthGeneratorConfig( + prefill_tokens=case.prefill_tokens, decode_tokens=case.decode_tokens, + ), + interval_generator_config=interval, + ), + ) + + +# --------------------------------------------------------------------------- +# Drain state and outcome classification (child process) +# --------------------------------------------------------------------------- + + +def _ticket_view(ticket) -> dict: + return { + "admission_seq": ticket.admission_seq, + "operation_id": str(ticket.operation_id), + "scope": ticket.scope, + } + + +def build_state_report(simulator) -> dict: + """Read stage contexts, lane queues and sync rooms after a drain.""" + from frontier.types import ClusterType + + cluster_scheduler = simulator.scheduler.get_cluster_scheduler(ClusterType.MONOLITHIC) + lanes = {} + queued_owner = {} + for (replica_id, lane_id), replica_scheduler in sorted( + cluster_scheduler._replica_schedulers.items(), key=lambda item: str(item[0]) + ): + stage_views = {} + for stage_id in range(replica_scheduler._num_stages): + stage = replica_scheduler.get_replica_stage_scheduler(stage_id) + heap = [] + for batch in stage.get_queue_batches(): + ticket = batch._stage_admission_ticket + queued_owner[(replica_id, stage_id, ticket.admission_seq)] = lane_id + heap.append({"batch_id": batch.id, "global_id": batch.global_id, + **_ticket_view(ticket)}) + stage_views[stage_id] = {"busy": stage.is_busy, "heap": heap} + lanes[f"{replica_id}/{lane_id}"] = { + "replica_id": replica_id, "lane": lane_id, "stages": stage_views, + } + + contexts = [] + for (replica_id, stage_id), context in sorted(cluster_scheduler._stage_execution_contexts.items()): + contexts.append({ + "replica_id": replica_id, + "stage_id": stage_id, + "capacity": context.full_stage_capacity, + "sealed": context.forward_group_sealed, + "bound_group": context._forward_group_id, + "ep_wave_active": context._active_ep_ticket is not None, + "active_full_stage": sorted( + (_ticket_view(ticket) for ticket in context._active_full_stage_tickets), + key=lambda view: view["admission_seq"], + ), + "fifo": [ + {**_ticket_view(ticket), + "lane": queued_owner.get((replica_id, stage_id, ticket.admission_seq))} + for ticket in context.queued_tickets + ], + }) + + rooms = [] + for room_name in ("_prefill_sync_waiting_room", "_decode_sync_waiting_room"): + by_replica = getattr(cluster_scheduler, room_name) or {} + for replica_id, by_stage in by_replica.items(): + for stage_id, by_step in by_stage.items(): + for step, by_layer in by_step.items(): + for layer, by_sync in by_layer.items(): + for sync_stage, room in by_sync.items(): + if not room["batches"]: + continue + rooms.append({ + "room": room_name.strip("_"), + "replica_id": replica_id, "stage_id": stage_id, + "step": step, "layer": layer, "sync_stage": str(sync_stage), + "lanes_present": sorted(room["batches"]), + }) + return { + "simulation_time": simulator._time, + "contexts": contexts, + "lanes": lanes, + "sync_rooms": rooms, + } + + +def has_admission_deadlock_signature(report: dict) -> bool: + """Plan §4.3: a busy lane's queued ticket heads the FIFO while an idle lane + with queued work, needed by that lane's sync room, is refused behind it.""" + lanes = report["lanes"] + for context in report["contexts"]: + if (context["ep_wave_active"] or context["sealed"] or not context["fifo"] + or len(context["active_full_stage"]) >= context["capacity"]): + continue + head = context["fifo"][0] + if head["scope"] != "FULL_STAGE_WORLD" or head["lane"] is None: + continue + replica_id, stage_id = context["replica_id"], context["stage_id"] + head_stage = lanes[f"{replica_id}/{head['lane']}"]["stages"][stage_id] + if not head_stage["busy"]: + continue + for lane in lanes.values(): + if lane["replica_id"] != replica_id or lane["lane"] == head["lane"]: + continue + stage = lane["stages"][stage_id] + if stage["busy"] or not stage["heap"]: + continue + if stage["heap"][0]["admission_seq"] <= head["admission_seq"]: + continue + for room in report["sync_rooms"]: + if (room["replica_id"] == replica_id and room["stage_id"] == stage_id + and head["lane"] in room["lanes_present"] + and lane["lane"] not in room["lanes_present"]): + return True + return False + + +def _run_simulator_case(case: Case, work_dir: Path, case_dir: Path) -> dict: + output_root = work_dir / "metrics" + try: + config = build_config(case, output_root, work_dir / "cache") + from frontier.simulator import Simulator + + simulator = Simulator(config) + except ValueError as exc: + (case_dir / "error.txt").write_text(traceback.format_exc()) + return {"outcome": CONFIGURATION_REJECTION, "exception": repr(exc)} + resolved_config = json.loads( + (Path(config.metrics_config.output_dir) / "config.json").read_text() + ) + (case_dir / "resolved_config.json").write_text(json.dumps(resolved_config, indent=1, sort_keys=True)) + try: + simulator.run() + except RuntimeError as exc: + if not str(exc).startswith(DRAIN_MESSAGE): + (case_dir / "error.txt").write_text(traceback.format_exc()) + return {"outcome": OTHER_FAILURE, "exception": repr(exc)[:2000]} + report = build_state_report(simulator) + (case_dir / "state_report.json").write_text(json.dumps(report, indent=1, sort_keys=True)) + outcome = ADMISSION_DEADLOCK if has_admission_deadlock_signature(report) else OTHER_FAILURE + return {"outcome": outcome, "exception": DRAIN_MESSAGE, + "simulation_time": report["simulation_time"]} + except Exception as exc: # classified, never converted into success + (case_dir / "error.txt").write_text(traceback.format_exc()) + return {"outcome": OTHER_FAILURE, "exception": repr(exc)[:2000]} + requests = list(simulator._all_requests) + completed = sum(1 for request in requests if request.completed) + if completed != len(requests): + (case_dir / "error.txt").write_text( + f"run returned with {completed} of {len(requests)} requests completed\n" + ) + return {"outcome": OTHER_FAILURE, "exception": "incomplete requests"} + return {"outcome": SUCCESS, "completed_requests": completed, + "metrics_run_dir": str(Path(config.metrics_config.output_dir).relative_to(work_dir))} + + +def _run_recipe_case(case: Case, work_dir: Path, case_dir: Path) -> dict: + env = dict(os.environ) + env.update({ + "PYTHON_BIN": sys.executable, + "METRICS_OUTPUT_DIR": str(work_dir / "metrics"), + "RUN_ID": case.case_id, + }) + result = subprocess.run( + ["bash", str(REPO_ROOT / case.recipe)], cwd=REPO_ROOT, env=env, + capture_output=True, text=True, + ) + (case_dir / "stdout.log").write_text(result.stdout[-200_000:] + result.stderr[-200_000:]) + if result.returncode != 0: + (case_dir / "error.txt").write_text(result.stderr[-50_000:]) + return {"outcome": OTHER_FAILURE, "exception": f"exit code {result.returncode}"} + return {"outcome": SUCCESS} + + +def run_case_in_child(case: Case, root: Path, set_name: str) -> None: + """Child entry point: run one case, then publish its artifacts.""" + case_dir = root / set_name / case.case_id + work_dir = root / "work" / case.case_id + for directory in (case_dir, work_dir): + if directory.exists(): + shutil.rmtree(directory) + directory.mkdir(parents=True) + started = time.time() + if case.recipe is None: + result = _run_simulator_case(case, work_dir, case_dir) + else: + result = _run_recipe_case(case, work_dir, case_dir) + result["wall_start"] = started + result["wall_end"] = time.time() + if result["outcome"] == SUCCESS: + shutil.copytree(work_dir / "metrics", case_dir / "metrics") + (case_dir / "sha256sums.txt").write_text(sha256_lines(case_dir / "metrics")) + shutil.rmtree(work_dir) + (case_dir / "outcome.json").write_text(json.dumps(result, indent=1, sort_keys=True)) + + +def sha256_lines(directory: Path) -> str: + lines = [] + for path in sorted(p for p in directory.rglob("*") if p.is_file()): + digest = hashlib.sha256(path.read_bytes()).hexdigest() + lines.append(f"{digest} {path.relative_to(directory)}") + return "\n".join(lines) + "\n" + + +# --------------------------------------------------------------------------- +# Lane-overlap metric (plan §4.5) +# --------------------------------------------------------------------------- + + +def interval_overlap(intervals: Iterable[tuple[float, float, object]]) -> dict: + """Half-open ``[start, end)`` intervals keyed by lane. + + Returns the time with at least one open lane, the time with at least two + distinct open lanes, the peak number of distinct open lanes, the latest + end, and whether any lane overlaps itself. + """ + intervals = list(intervals) + events = [] + for start, end, lane in intervals: + if end > start: + events.append((start, 1, lane)) + events.append((end, -1, lane)) + # Ends sort before starts at the same instant: touching intervals do not overlap. + events.sort(key=lambda event: (event[0], event[1])) + open_by_lane: dict[object, int] = defaultdict(int) + busy_time = multi_lane_time = 0.0 + peak_lanes = 0 + self_overlap = False + previous_time = None + for event_time, delta, lane in events: + if previous_time is not None: + open_lanes = sum(1 for count in open_by_lane.values() if count > 0) + if open_lanes >= 1: + busy_time += event_time - previous_time + if open_lanes >= 2: + multi_lane_time += event_time - previous_time + open_by_lane[lane] += delta + if open_by_lane[lane] > 1: + self_overlap = True + peak_lanes = max(peak_lanes, sum(1 for count in open_by_lane.values() if count > 0)) + previous_time = event_time + return { + "busy_time": busy_time, + "multi_lane_busy_time": multi_lane_time, + "peak_lanes": peak_lanes, + "makespan": max((end for _, end, _ in intervals), default=0.0), + "self_overlap": self_overlap, + } + + +def read_ledger(metrics_dir: Path) -> list[dict]: + paths = sorted(metrics_dir.rglob("frontier_stage_batch_ledger.jsonl")) + if len(paths) != 1: + raise ValueError(f"expected one stage ledger under {metrics_dir}, found {len(paths)}") + return [json.loads(line) for line in paths[0].read_text().splitlines() if line.strip()] + + +def lane_intervals(rows: Sequence[dict]) -> dict[tuple, list[tuple[float, float, int]]]: + """``ATTN_DP_LANE`` ledger intervals per physical stage.""" + by_stage: dict[tuple, list[tuple[float, float, int]]] = defaultdict(list) + for row in rows: + if row["execution_scope"] != ATTN_DP_LANE: + continue + key = (row["cluster_type"], row["replica_id"], row["stage_id"]) + by_stage[key].append((row["stage_start_ts"], row["stage_end_ts"], row["replica_local_id"])) + return dict(by_stage) + + +def ledger_lane_metric(metrics_dir: Path) -> dict: + """Plan §4.5 metric per physical stage, keyed ``cluster/replica/stage``.""" + stages = {} + for (cluster_type, replica_id, stage_id), intervals in sorted(lane_intervals(read_ledger(metrics_dir)).items()): + metric = interval_overlap(intervals) + metric["lanes"] = sorted({lane for _, _, lane in intervals}) + stages[f"{cluster_type}/{replica_id}/{stage_id}"] = metric + return stages + + +# --------------------------------------------------------------------------- +# Parent: set runner and provenance +# --------------------------------------------------------------------------- + + +def _git(*args: str) -> str: + return subprocess.run(["git", "-C", str(REPO_ROOT), *args], check=True, + capture_output=True, text=True).stdout.strip() + + +def set_provenance() -> dict: + distributions = sorted( + f"{dist.metadata['Name']}=={dist.version}" for dist in importlib_metadata.distributions() + ) + status = _git("status", "--porcelain", "--", ".", ":!task_memory") + return { + "interpreter": sys.executable, + "python_vv": subprocess.run([sys.executable, "-VV"], check=True, + capture_output=True, text=True).stdout.strip(), + "distributions_sha256": hashlib.sha256("\n".join(distributions).encode()).hexdigest(), + "git_head": _git("rev-parse", "HEAD"), + "clean_outside_task_memory": status == "", + "status_outside_task_memory": status, + } + + +def matrix_root() -> Path: + return resolve_scratch_root() / MATRIX_DIR_NAME + + +def _run_one(case: Case, root: Path, set_name: str, provenance: dict) -> dict: + command = [sys.executable, "-m", "tests.e2e.stage_admission_matrix", "child", + "--set", set_name, "--case", case.case_id] + env = dict(os.environ, PYTHONPATH=str(REPO_ROOT), WANDB_DISABLED="true", + VIDUR_DISABLE_WANDB="1") + result = subprocess.run(command, cwd=REPO_ROOT, env=env, capture_output=True, text=True) + case_dir = root / set_name / case.case_id + case_dir.mkdir(parents=True, exist_ok=True) + outcome_path = case_dir / "outcome.json" + if result.returncode != 0 or not outcome_path.exists(): + (case_dir / "error.txt").write_text(result.stdout[-50_000:] + result.stderr[-50_000:]) + outcome = {"outcome": OTHER_FAILURE, "exception": f"child exit code {result.returncode}"} + else: + outcome = json.loads(outcome_path.read_text()) + (case_dir / "case.json").write_text(json.dumps(asdict(case), indent=1, sort_keys=True)) + run = {"command": command, **provenance, **outcome} + (case_dir / "run.json").write_text(json.dumps(run, indent=1, sort_keys=True)) + return {"case_id": case.case_id, "group": case.group, "outcome": outcome["outcome"], + "exception": outcome.get("exception")} + + +def run_set(set_name: str, cases: Sequence[Case], jobs: int) -> list[dict]: + root = matrix_root() + (root / set_name).mkdir(parents=True, exist_ok=True) + provenance = set_provenance() + with ThreadPoolExecutor(max_workers=jobs) as pool: + rows = list(pool.map(lambda case: _run_one(case, root, set_name, provenance), cases)) + if set_provenance()["git_head"] != provenance["git_head"]: + raise RuntimeError("git HEAD changed while the set was running") + index = root / set_name / "cases.jsonl" + existing = {} + if index.exists(): + existing = {row["case_id"]: row for row in map(json.loads, index.read_text().splitlines())} + existing.update({row["case_id"]: row for row in rows}) + order = [case.case_id for case in build_cases()] + index.write_text("".join(json.dumps(existing[case_id]) + "\n" + for case_id in order if case_id in existing)) + return rows + + +# --------------------------------------------------------------------------- +# Parent: before/after comparison (plan §4.4) +# --------------------------------------------------------------------------- + + +def _case_state(root: Path, set_name: str, case_id: str) -> dict: + case_dir = root / set_name / case_id + run = json.loads((case_dir / "run.json").read_text()) + state = {"outcome": run["outcome"]} + if run["outcome"] == SUCCESS: + state["sha256sums"] = (case_dir / "sha256sums.txt").read_text() + if (case_dir / "metrics").exists(): + state["metrics_dir"] = case_dir / "metrics" + return state + + +def _conservation(case: Case, metrics_dir: Path) -> dict: + paths = sorted(metrics_dir.rglob("request_metrics.csv")) + if len(paths) != 1: + return {"ok": False, "reason": f"{len(paths)} request_metrics.csv files"} + with paths[0].open() as handle: + rows = list(csv.DictReader(handle)) + prefill = sum(int(float(row["request_num_prefill_tokens"])) for row in rows) + decode = sum(int(float(row["request_num_decode_tokens"])) for row in rows) + expected = (case.num_requests, case.num_requests * case.prefill_tokens, + case.num_requests * case.decode_tokens) + observed = (len(rows), prefill, decode) + return {"ok": observed == expected, "observed": observed, "expected": expected} + + +def _differing_files(before: str, after: str) -> list[str]: + def parse(text): + return dict(reversed(line.split(" ", 1)) for line in text.splitlines() if line) + left, right = parse(before), parse(after) + return sorted(name for name in set(left) | set(right) if left.get(name) != right.get(name)) + + +def compare_sets(before: str, after: str) -> list[dict]: + root = matrix_root() + rows = [] + for case in build_cases(): + if not (root / before / case.case_id / "run.json").exists(): + continue + base = _case_state(root, before, case.case_id) + new = _case_state(root, after, case.case_id) + row = {"case_id": case.case_id, "group": case.group, + "before": base["outcome"], "after": new["outcome"]} + if case.group == "R0": + row.update(path="R0", verdict="informational") + elif case.path == "U": + row["path"] = "U" + identical = base["outcome"] == new["outcome"] == SUCCESS and base["sha256sums"] == new["sha256sums"] + row["verdict"] = "PASS" if identical else "STOP" + if base["outcome"] == new["outcome"] == SUCCESS and not identical: + row["differing_files"] = _differing_files(base["sha256sums"], new["sha256sums"]) + elif base["outcome"] == ADMISSION_DEADLOCK: + row["path"] = "L" + if new["outcome"] == SUCCESS: + row["conservation"] = _conservation(case, new["metrics_dir"]) + row["verdict"] = "PASS" if row["conservation"]["ok"] else "STOP" + else: + row["verdict"] = "STOP" + elif base["outcome"] == SUCCESS: + row["path"] = "T" + if new["outcome"] != SUCCESS: + row["verdict"] = "STOP" + else: + before_metric = ledger_lane_metric(base["metrics_dir"]) + after_metric = ledger_lane_metric(new["metrics_dir"]) + row["lane_metric_before"] = before_metric + row["lane_metric_after"] = after_metric + checks_ok = all( + not stage["self_overlap"] and stage["peak_lanes"] <= case.attn_dp + for stage in after_metric.values() + ) + identical = base["sha256sums"] == new["sha256sums"] + if not identical: + row["differing_files"] = _differing_files(base["sha256sums"], new["sha256sums"]) + if case.contention_witness: + total = lambda metric: sum(stage["multi_lane_busy_time"] for stage in metric.values()) + row["witness_increase"] = total(after_metric) > total(before_metric) + checks_ok = checks_ok and row["witness_increase"] + row["verdict"] = ("PASS" if identical and checks_ok + else "EXPLAIN" if checks_ok else "STOP") + else: + row.update(path="class", verdict="STOP") + rows.append(row) + return rows + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0]) + commands = parser.add_subparsers(dest="command", required=True) + run_parser = commands.add_parser("run", help="run cases into one named set") + run_parser.add_argument("--set", required=True) + run_parser.add_argument("--group", action="append", default=[]) + run_parser.add_argument("--case", action="append", default=[]) + run_parser.add_argument("--jobs", type=int, default=8) + child_parser = commands.add_parser("child", help=argparse.SUPPRESS) + child_parser.add_argument("--set", required=True) + child_parser.add_argument("--case", required=True) + compare_parser = commands.add_parser("compare", help="apply the plan §4.4 paths") + compare_parser.add_argument("--before", required=True) + compare_parser.add_argument("--after", required=True) + compare_parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args(argv) + + cases_by_id = {case.case_id: case for case in build_cases()} + if args.command == "child": + run_case_in_child(cases_by_id[args.case], matrix_root(), args.set) + return 0 + if args.command == "compare": + rows = compare_sets(args.before, args.after) + args.output.write_text(json.dumps(rows, indent=1, sort_keys=True, default=str)) + for row in rows: + print(f"{row['case_id']:<48} {row['path']:<5} {row['before']:<24} {row['after']:<24} {row['verdict']}") + return 0 + selected = [case for case in cases_by_id.values() + if (not args.group or case.group in args.group) + and (not args.case or case.case_id in args.case)] + for row in run_set(args.set, selected, args.jobs): + print(f"{row['case_id']:<48} {row['outcome']}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From 5ade853e72d440849f565e090126b2109aba178f Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 02:21:18 +0800 Subject: [PATCH 05/19] tests: keep per-stage lane views as a list in the admission state report JSON turns integer dict keys into strings, so a state_report.json read back from disk could not be indexed by stage id the way the in-process report is. A list indexed by stage id has the same shape before and after the round trip. --- tests/e2e/stage_admission_matrix.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/e2e/stage_admission_matrix.py b/tests/e2e/stage_admission_matrix.py index 38bcc8dc..e08a63cc 100644 --- a/tests/e2e/stage_admission_matrix.py +++ b/tests/e2e/stage_admission_matrix.py @@ -293,7 +293,7 @@ def build_state_report(simulator) -> dict: for (replica_id, lane_id), replica_scheduler in sorted( cluster_scheduler._replica_schedulers.items(), key=lambda item: str(item[0]) ): - stage_views = {} + stage_views = [] for stage_id in range(replica_scheduler._num_stages): stage = replica_scheduler.get_replica_stage_scheduler(stage_id) heap = [] @@ -302,7 +302,7 @@ def build_state_report(simulator) -> dict: queued_owner[(replica_id, stage_id, ticket.admission_seq)] = lane_id heap.append({"batch_id": batch.id, "global_id": batch.global_id, **_ticket_view(ticket)}) - stage_views[stage_id] = {"busy": stage.is_busy, "heap": heap} + stage_views.append({"busy": stage.is_busy, "heap": heap}) lanes[f"{replica_id}/{lane_id}"] = { "replica_id": replica_id, "lane": lane_id, "stages": stage_views, } From 799ccb4bd16aa0ce78e46975a74c714a337536ea Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 02:21:18 +0800 Subject: [PATCH 06/19] tests: compare DP-lane stage admission with vLLM under pipeline parallelism vllm_burst_driver.py runs inside the vllm-openai v0.10.2 image: it builds the instrumented overlay from the vLLM-BS checkout, accepting it only when the files that differ from the image are exactly the fork's own changes, then drives DP=2, PP=2 bursts of prefill-only requests pinned to rank i mod 2. run_vllm_worker.sh is the 4-GPU worker entry point for the MoE and dense scenarios. compare_lanes.py turns the vLLM per-forward traces and the Frontier G7 stage ledgers into the same lane metrics (completion, per-lane sequences, stage-0 pairing, co-start and co-execution) and writes the workflow-gap table for plan criterion C7. --- .../stage_admission_pp/compare_lanes.py | 249 ++++++++++++++++++ .../stage_admission_pp/run_vllm_worker.sh | 102 +++++++ .../stage_admission_pp/vllm_burst_driver.py | 221 ++++++++++++++++ 3 files changed, 572 insertions(+) create mode 100644 tests/comparison/stage_admission_pp/compare_lanes.py create mode 100644 tests/comparison/stage_admission_pp/run_vllm_worker.sh create mode 100644 tests/comparison/stage_admission_pp/vllm_burst_driver.py diff --git a/tests/comparison/stage_admission_pp/compare_lanes.py b/tests/comparison/stage_admission_pp/compare_lanes.py new file mode 100644 index 00000000..035eb68a --- /dev/null +++ b/tests/comparison/stage_admission_pp/compare_lanes.py @@ -0,0 +1,249 @@ +#!/usr/bin/env python3 +"""Compare DP-lane stage admission between vLLM traces and Frontier ledgers. + +Reads the vLLM ground truth written by ``run_vllm_worker.sh`` and the G7 cases +of ``tests.e2e.stage_admission_matrix`` for a set run before the change and one +run after it, computes the per-forward lane metrics of the stage-admission plan +(§4.7, M1–M5) on both sides with the same definitions, and writes the +workflow-gap table, summary and status for the calibration case. + +A vLLM forward row is one ``pp_boundary`` record: its lane is the DP rank the +driver pinned its requests to, and its stage is ``pp_rank``. Stage-0 +intervals end at ``send_start_ts``, taken after the post-forward synchronize; +last-stage intervals end at the record's wall-clock ``timestamp`` converted to +the monotonic clock with the offset the driver sampled around the round. A +Frontier forward row is one ``ATTN_DP_LANE`` ledger row. +""" + +from __future__ import annotations + +import argparse +import csv +import json +import statistics +from collections import defaultdict +from pathlib import Path + +from tests.e2e.stage_admission_matrix import ( + ADMISSION_DEADLOCK, + ATTN_DP_LANE, + SUCCESS, + interval_overlap, + matrix_root, + read_ledger, +) + +MODELS = ("moe", "dense") +BURSTS = (8, 16) +CO_START_BOUND = 0.5 +CO_EXECUTION_BOUND = 0.10 +FRONTIER_OWNER = "frontier/scheduler/replica_stage_scheduler/stage_execution_context.py" + + +def _forward(lane: int, stage: int, start: float, end: float, indices) -> dict: + return {"lane": lane, "stage": stage, "start": start, "end": end, + "indices": tuple(sorted(indices))} + + +def vllm_forwards(scenario_dir: Path) -> dict[tuple[int, int], dict]: + """Formal vLLM forwards keyed by ``(burst size, round)``.""" + requests = {row["request_id"]: row for row in map(json.loads, (scenario_dir / "requests.jsonl").read_text().splitlines())} + summary = json.loads((scenario_dir / "summary.json").read_text()) + offsets = { + entry["label"]: (entry["wall_minus_monotonic_before"] + entry["wall_minus_monotonic_after"]) / 2 + for entry in summary["rounds"] + } + runs: dict[tuple[int, int], dict] = {} + for line in (scenario_dir / "pp_boundary.jsonl").read_text().splitlines(): + record = json.loads(line) + members = [requests[request_id] for request_id in record["request_ids"]] + labels = {member["burst"] for member in members} + ranks = {member["rank"] for member in members} + if len(labels) != 1 or len(ranks) != 1: + raise ValueError(f"forward mixes bursts or ranks: {record['request_ids']}") + label = labels.pop() + if label == "warmup": + continue + if record["is_last_rank"]: + end = record["timestamp"] - offsets[label] + else: + end = record["send_start_ts"] + member = members[0] + run = runs.setdefault( + (int(label.split("-")[0][1:]), member["round"]), + {"forwards": [], "requests": [row for row in requests.values() if row["burst"] == label]}, + ) + run["forwards"].append(_forward(ranks.pop(), record["pp_rank"], record["forward_start_ts"], end, + (m["index"] for m in members))) + for run in runs.values(): + rows = run.pop("requests") + run["submitted"] = len(rows) + run["completed"] = sum(1 for row in rows if row["num_output_tokens"] == 1) + return runs + + +def vllm_placement(scenario_dir: Path) -> dict: + """Check from engine iterations that each formal request ran on its pinned rank.""" + pinned = {row["request_id"]: row["rank"] for row in map(json.loads, (scenario_dir / "requests.jsonl").read_text().splitlines())} + scheduled_by = defaultdict(set) + for path in sorted((scenario_dir / "dp_placement").glob("*.jsonl")): + for record in map(json.loads, path.read_text().splitlines()): + if record["kind"] == "engine_iteration": + for request_id in record["scheduled_new_req_ids"]: + scheduled_by[request_id].add(record["engine"]) + misplaced = sorted(rid for rid, rank in pinned.items() if scheduled_by.get(rid, {rank}) != {rank}) + unseen = sorted(rid for rid in pinned if rid not in scheduled_by) + return {"requests": len(pinned), "misplaced": misplaced, "unseen": unseen, "ok": not misplaced} + + +def frontier_run(set_dir: Path, case_id: str) -> dict: + case_dir = set_dir / case_id + run = json.loads((case_dir / "run.json").read_text()) + case = json.loads((case_dir / "case.json").read_text()) + result = {"outcome": run["outcome"], "submitted": case["num_requests"], "forwards": []} + if run["outcome"] != SUCCESS: + result["completed"] = None + return result + result["completed"] = case["num_requests"] + for row in read_ledger(case_dir / "metrics"): + if row["execution_scope"] != ATTN_DP_LANE: + continue + result["forwards"].append(_forward(row["replica_local_id"], row["stage_id"], row["stage_start_ts"], + row["stage_end_ts"], (int(rid) for rid in row["request_ids"]))) + lanes_match_index = all(index % 2 == forward["lane"] for forward in result["forwards"] + for index in forward["indices"]) + result["placement_ok"] = lanes_match_index + return result + + +def lane_metrics(forwards: list[dict]) -> dict: + """M2–M5 of plan §4.7 from one run's forwards.""" + by_lane_stage = defaultdict(list) + for forward in forwards: + by_lane_stage[(forward["lane"], forward["stage"])].append(forward) + sequences = { + f"lane{lane}/stage{stage}": [list(f["indices"]) for f in sorted(rows, key=lambda f: f["start"])] + for (lane, stage), rows in sorted(by_lane_stage.items()) + } + metrics = {"M2_sequences": sequences} + for stage in sorted({forward["stage"] for forward in forwards}): + lane0 = sorted(by_lane_stage[(0, stage)], key=lambda f: f["start"]) + lane1 = sorted(by_lane_stage[(1, stage)], key=lambda f: f["start"]) + pairing = [] + for forward in lane0: + overlaps = [(min(forward["end"], other["end"]) - max(forward["start"], other["start"]), other) + for other in lane1] + overlap, partner = max(overlaps, key=lambda item: item[0], default=(0.0, None)) + pairing.append([list(forward["indices"]), list(partner["indices"]) if partner and overlap > 0 else None]) + durations = [f["end"] - f["start"] for f in lane0 + lane1] + skew = abs(lane0[0]["start"] - lane1[0]["start"]) / statistics.median(durations) if lane0 and lane1 else None + overlap = interval_overlap([(f["start"], f["end"], f["lane"]) for f in lane0 + lane1]) + metrics[f"stage{stage}"] = { + "M3_pairing": pairing, + "M4_co_start": skew, + "M5_co_execution": overlap["multi_lane_busy_time"] / overlap["busy_time"] if overlap["busy_time"] else None, + "median_forward_duration": statistics.median(durations) if durations else None, + "self_overlap": overlap["self_overlap"], + } + return metrics + + +def _row(check, model, burst, round_index, metric, groundtruth, after, base, status, note=""): + return {"check": check, "model": model, "burst": burst, "round": round_index, "metric": metric, + "groundtruth": json.dumps(groundtruth), "frontier_after": json.dumps(after), + "frontier_base": json.dumps(base), "status": status, + "frontier_owner": FRONTIER_OWNER if status == "MISMATCH" else "", "note": note} + + +def compare(vllm_run: Path, frontier_root: Path, before: str, after: str) -> tuple[list[dict], dict]: + rows, details = [], {"placement": {}, "runs": {}} + for model in MODELS: + scenario_dir = vllm_run / "runs" / model + vllm_runs = vllm_forwards(scenario_dir) + details["placement"][model] = vllm_placement(scenario_dir) + for burst in BURSTS: + case_id = f"G7-{model}-dp2-pp2-n{burst}" + base = frontier_run(frontier_root / before, case_id) + new = frontier_run(frontier_root / after, case_id) + base_metrics = lane_metrics(base["forwards"]) if base["outcome"] == SUCCESS else None + new_metrics = lane_metrics(new["forwards"]) if new["outcome"] == SUCCESS else None + rounds = sorted(r for (b, r) in vllm_runs if b == burst) + vllm_metrics = {r: lane_metrics(vllm_runs[(burst, r)]["forwards"]) for r in rounds} + details["runs"][case_id] = {"frontier_base": base_metrics, "frontier_after": new_metrics, + "frontier_base_outcome": base["outcome"], + "frontier_after_outcome": new["outcome"], + "frontier_after_placement_ok": new.get("placement_ok"), + "vllm": vllm_metrics} + base_control = (base["outcome"] == ADMISSION_DEADLOCK) if model == "moe" else (base["outcome"] == SUCCESS) + for r in rounds: + run = vllm_runs[(burst, r)] + completed = run["completed"] == run["submitted"] == burst + status = "MATCH" if completed and new["outcome"] == SUCCESS and base_control else "MISMATCH" + rows.append(_row("V1", model, burst, r, "M1 completion", + f"{run['completed']}/{run['submitted']}", new["outcome"], base["outcome"], status)) + gt = vllm_metrics[r] + after_m2 = new_metrics["M2_sequences"] if new_metrics else None + rows.append(_row("V2", model, burst, r, "M2 lane sequences", gt["M2_sequences"], after_m2, + base_metrics["M2_sequences"] if base_metrics else None, + "MATCH" if after_m2 == gt["M2_sequences"] else "MISMATCH")) + after_m3 = new_metrics["stage0"]["M3_pairing"] if new_metrics else None + rows.append(_row("V3", model, burst, r, "M3 stage-0 pairing", gt["stage0"]["M3_pairing"], after_m3, + base_metrics["stage0"]["M3_pairing"] if base_metrics else None, + "MATCH" if after_m3 == gt["stage0"]["M3_pairing"] else "MISMATCH")) + after_m4 = new_metrics["stage0"]["M4_co_start"] if new_metrics else None + base_m4 = base_metrics["stage0"]["M4_co_start"] if base_metrics else None + m4_ok = (gt["stage0"]["M4_co_start"] < CO_START_BOUND and after_m4 is not None + and after_m4 < CO_START_BOUND + and (model == "moe" or (base_m4 is not None and base_m4 >= CO_START_BOUND))) + rows.append(_row("V4", model, burst, r, "M4 stage-0 co-start", gt["stage0"]["M4_co_start"], + after_m4, base_m4, "MATCH" if m4_ok else "MISMATCH")) + gt_m5 = statistics.mean(vllm_metrics[r]["stage0"]["M5_co_execution"] for r in rounds) + after_m5 = new_metrics["stage0"]["M5_co_execution"] if new_metrics else None + base_m5 = base_metrics["stage0"]["M5_co_execution"] if base_metrics else None + m5_ok = after_m5 is not None and abs(after_m5 - gt_m5) <= CO_EXECUTION_BOUND + if model == "dense": + m5_ok = m5_ok and base_m5 is not None and abs(base_m5 - gt_m5) > abs(after_m5 - gt_m5) + rows.append(_row("V5", model, burst, "mean", "M5 stage-0 co-execution", gt_m5, after_m5, base_m5, + "MATCH" if m5_ok else "MISMATCH")) + return rows, details + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0]) + parser.add_argument("--vllm-run", type=Path, required=True, help="evidence directory of one worker run") + parser.add_argument("--before", default="base") + parser.add_argument("--after", default="after") + parser.add_argument("--output", type=Path, required=True, help="case analysis directory") + args = parser.parse_args(argv) + + rows, details = compare(args.vllm_run, matrix_root(), args.before, args.after) + args.output.mkdir(parents=True, exist_ok=True) + with (args.output / "workflow_gap_table.csv").open("w", newline="") as handle: + writer = csv.DictWriter(handle, fieldnames=list(rows[0])) + writer.writeheader() + writer.writerows(rows) + (args.output / "lane_metrics.json").write_text(json.dumps(details, indent=1, sort_keys=True)) + mismatches = [row for row in rows if row["status"] != "MATCH"] + placement_ok = all(p["ok"] for p in details["placement"].values()) + placement_unseen = sum(len(p["unseen"]) for p in details["placement"].values()) + status = { + "analysis_state": "COMPLETE", + "status": "PASS" if not mismatches and placement_ok else "FAIL", + "correction_state": "not_applicable", + "rows": len(rows), + "mismatches": len(mismatches), + "vllm_placement_ok": placement_ok, + "vllm_placement_unseen_requests": placement_unseen, + "next_action": ("record C7 in the test report" if not mismatches and placement_ok + else "report each MISMATCH row with its cause before P4; adjust nothing"), + } + (args.output / "workflow_gap_status.json").write_text(json.dumps(status, indent=1)) + for row in rows: + print(f"{row['check']} {row['model']:<5} n{row['burst']:<3} r{row['round']!s:<5} {row['status']:<9} " + f"gt={row['groundtruth'][:40]} after={row['frontier_after'][:40]} base={row['frontier_base'][:40]}") + print(json.dumps(status)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/comparison/stage_admission_pp/run_vllm_worker.sh b/tests/comparison/stage_admission_pp/run_vllm_worker.sh new file mode 100644 index 00000000..0b5896ea --- /dev/null +++ b/tests/comparison/stage_admission_pp/run_vllm_worker.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +# Worker entry point for the stage-admission vLLM ground truth (4 GPUs, +# vllm/vllm-openai:v0.10.2). Builds the instrumented overlay, runs the MoE and +# dense bursts, and publishes the traces to the cloud-volume archive and to the +# calibration case directory on the mounted workspace. +# +# Required environment: +# RUN_TAG identifier of this run +# FRONTIER_TREE Frontier worktree on the mounted workspace +# GROUNDTRUTH vLLM-BS checkout on the mounted workspace +# CASE_DIR calibration case directory on the mounted workspace; reads +# inputs/, writes runs/vllm-instrumented// +# ARCHIVE_DIR cloud-volume directory for this run +set -euo pipefail +set +x +: "${RUN_TAG:?}" "${FRONTIER_TREE:?}" "${GROUNDTRUTH:?}" "${CASE_DIR:?}" "${ARCHIVE_DIR:?}" +EVIDENCE_DIR="$CASE_DIR/runs/vllm-instrumented/$RUN_TAG" + +for d in /usr/local/nvidia/lib64 /usr/local/nvidia/lib /usr/lib/x86_64-linux-gnu; do + if [ -e "$d/libcuda.so.1" ]; then + export LD_LIBRARY_PATH="$d${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" + break + fi +done + +PY=python3 +SCRIPT_DIR="$FRONTIER_TREE/tests/comparison/stage_admission_pp" +WORK=/tmp/stage_admission_pp/$RUN_TAG +mkdir -p "$WORK/runs" +export VLLM_CACHE_ROOT="$WORK/vllm_cache" HF_HOME="$WORK/hf" HF_HUB_OFFLINE=1 \ + TRANSFORMERS_OFFLINE=1 VLLM_NO_USAGE_STATS=1 DO_NOT_TRACK=1 + +publish() { + local target="$1" + mkdir -p "$target" + for item in runs overlay_report.json worker_env.json vllm_import.txt; do + if [ -e "$WORK/$item" ]; then cp -r "$WORK/$item" "$target/"; fi + done + echo "status=$status" > "$target/COMPLETE" +} +# The worker writes the mounted workspace as root; hand the evidence back to +# the owner of the case directory. +publish_evidence() { + publish "$EVIDENCE_DIR" + chown -R "$(stat -c %u:%g "$CASE_DIR")" "$EVIDENCE_DIR" +} +status=0 +"$PY" - <<'PY' | tee "$WORK/worker_env.json" +import json, platform, torch +print(json.dumps({ + "python": platform.python_version(), + "torch": torch.__version__, + "cuda_available": torch.cuda.is_available(), + "device_count": torch.cuda.device_count(), + "devices": [torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())], +})) +PY + +SITE_VLLM=$("$PY" -c 'import importlib.util, os; print(os.path.dirname(importlib.util.find_spec("vllm").origin))') +"$PY" "$SCRIPT_DIR/vllm_burst_driver.py" overlay \ + --site-vllm "$SITE_VLLM" --checkout "$GROUNDTRUTH" --destination "$WORK/overlay" \ + --expected-changes "$CASE_DIR/inputs/fork_changed_files.txt" \ + --report "$WORK/overlay_report.json" || status=3 +if [ "$status" -ne 0 ]; then + publish "$ARCHIVE_DIR"; publish_evidence + echo "WORKER_STATUS=$status overlay rejected" + exit "$status" +fi +export PYTHONPATH="$WORK/overlay" +"$PY" -c 'import vllm, vllm.v1.frontier_trace as t; print("VLLM_IMPORT", vllm.__version__, vllm.__file__, t.__file__)' \ + | tee "$WORK/vllm_import.txt" + +run_scenario() { + local name="$1"; shift + local out="$WORK/runs/$name" + mkdir -p "$out" + # Engine cores write their placement records at interpreter exit, which a + # forked multiprocessing child skips; spawned children run it. + if VLLM_FRONTIER_INSTRUMENTATION=1 VLLM_WORKER_MULTIPROC_METHOD=spawn \ + VLLM_FRONTIER_PP_BOUNDARY_LOG_PATH="$out/pp_boundary.jsonl" \ + VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR="$out/dp_placement" \ + timeout 1500 "$PY" "$SCRIPT_DIR/vllm_burst_driver.py" run --output-dir "$out" "$@" \ + > "$out/driver.log" 2>&1; then + echo "SCENARIO_PASS $name" + else + echo "SCENARIO_FAIL $name exit=$?" + tail -n 60 "$out/driver.log" + status=1 + fi +} +run_scenario moe --model-config "$FRONTIER_TREE/data/config/models/Qwen3-30B-A3B-tiny.json" --enable-expert-parallel +run_scenario dense --model-config "$FRONTIER_TREE/data/config/models/Llama-3.2-1B-Instruct.json" + +publish "$ARCHIVE_DIR" +publish_evidence + +for name in moe dense; do + grep -h "DRIVER_DONE" "$WORK/runs/$name/driver.log" | sed "s/^/$name /" || true + wc -l "$WORK/runs/$name/pp_boundary.jsonl" 2>/dev/null || true +done +echo "WORKER_STATUS=$status RUN_TAG=$RUN_TAG" +exit "$status" diff --git a/tests/comparison/stage_admission_pp/vllm_burst_driver.py b/tests/comparison/stage_admission_pp/vllm_burst_driver.py new file mode 100644 index 00000000..a82db805 --- /dev/null +++ b/tests/comparison/stage_admission_pp/vllm_burst_driver.py @@ -0,0 +1,221 @@ +#!/usr/bin/env python3 +"""vLLM ground truth for stage admission of DP ranks under pipeline parallelism. + +Runs inside the ``vllm/vllm-openai:v0.10.2`` image on one 4-GPU worker. Two +subcommands: + +``overlay`` + Build the instrumented vLLM package: copy the image's installed ``vllm`` + package (which carries the compiled extensions) and copy every + ``vllm/**/*.py`` of the ground-truth checkout over it. The overlay is + accepted only when the files where the image and the checkout differ are + exactly the checkout's own changes over its upstream base, listed in + ``--expected-changes``. + +``run`` + Start one ``AsyncLLM`` with DP=2, PP=2, TP=1 (EP for the MoE model), run + warmup requests, then bursts of prefill-only requests pinned to rank + ``i mod 2``. Every request of a burst is added before any output is + awaited. The engines are idle between rounds. Per-forward traces come + from the checkout's own instrumentation; this script records each + request's rank, submit and finish times, and the wall/monotonic clock + offset around each round. + +The script imports vLLM only inside ``run`` so that ``overlay`` never loads +the package it is building. +""" + +from __future__ import annotations + +import argparse +import asyncio +import filecmp +import json +import os +import random +import shutil +import sys +import time +from pathlib import Path + + +def build_overlay(site_vllm: Path, checkout: Path, destination: Path, expected_changes: Path) -> dict: + target = destination / "vllm" + if target.exists(): + shutil.rmtree(target) + shutil.copytree(site_vllm, target, symlinks=True) + differing = [] + for source in sorted((checkout / "vllm").rglob("*.py")): + relative = source.relative_to(checkout) + installed = site_vllm.parent / relative + if not installed.exists() or not filecmp.cmp(source, installed, shallow=False): + differing.append(str(relative)) + copy_target = destination / relative + copy_target.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, copy_target) + expected = sorted( + line.strip() for line in expected_changes.read_text().splitlines() + if line.strip().endswith(".py") + ) + return { + "site_vllm": str(site_vllm), + "checkout": str(checkout), + "overlay": str(target), + "differing_py_files": differing, + "expected_py_changes": expected, + "unexpected": sorted(set(differing) - set(expected)), + "missing": sorted(set(expected) - set(differing)), + "accepted": differing == expected, + } + + +def write_model_dir(model_config: Path, model_dir: Path) -> dict: + config = json.loads(model_config.read_text()) + model_dir.mkdir(parents=True, exist_ok=True) + (model_dir / "config.json").write_text(json.dumps(config, indent=1)) + return config + + +def prompt_token_ids(request_id: str, length: int, vocab_size: int) -> list[int]: + generator = random.Random(request_id) + return [generator.randrange(100, vocab_size - 100) for _ in range(length)] + + +async def run_bursts(args: argparse.Namespace) -> dict: + from vllm import SamplingParams + from vllm.engine.arg_utils import AsyncEngineArgs + from vllm.inputs import TokensPrompt + from vllm.sampling_params import RequestOutputKind + from vllm.v1.engine.async_llm import AsyncLLM + + output_dir = Path(args.output_dir) + model_config = write_model_dir(Path(args.model_config), output_dir / "model") + engine_args = AsyncEngineArgs( + model=str(output_dir / "model"), + load_format="dummy", + skip_tokenizer_init=True, + dtype="bfloat16", + tensor_parallel_size=1, + pipeline_parallel_size=args.pipeline_parallel_size, + data_parallel_size=args.data_parallel_size, + enable_expert_parallel=args.enable_expert_parallel, + enforce_eager=True, + enable_prefix_caching=False, + enable_chunked_prefill=True, + max_num_batched_tokens=args.prompt_tokens, + max_num_seqs=args.max_num_seqs, + block_size=16, + max_model_len=args.max_model_len, + gpu_memory_utilization=args.gpu_memory_utilization, + seed=0, + disable_log_stats=True, + ) + engine = AsyncLLM.from_engine_args(engine_args) + sampling = SamplingParams( + max_tokens=1, ignore_eos=True, temperature=0.0, detokenize=False, + output_kind=RequestOutputKind.FINAL_ONLY, + ) + vocab_size = int(model_config["vocab_size"]) + records: list[dict] = [] + + async def wait_until_idle() -> float: + started = time.monotonic() + while engine.engine_core.dp_engines_running(): + if time.monotonic() - started > args.idle_timeout_s: + raise RuntimeError("DP engines did not pause between rounds") + await asyncio.sleep(0.05) + await asyncio.sleep(args.idle_gap_s) + return time.monotonic() - started + + async def burst(label: str, round_index: int, num_requests: int) -> dict: + offset_before = time.time() - time.monotonic() + queues = [] + for index in range(num_requests): + request_id = f"{label}-q{index}" + rank = index % args.data_parallel_size + prompt = TokensPrompt( + prompt_token_ids=prompt_token_ids(request_id, args.prompt_tokens, vocab_size) + ) + submitted = time.monotonic() + queue = await engine.add_request(request_id, prompt, sampling, data_parallel_rank=rank) + queues.append((request_id, index, rank, submitted, queue)) + for request_id, index, rank, submitted, queue in queues: + output = await queue.get() + while not output.finished: + output = await queue.get() + records.append({ + "request_id": request_id, "burst": label, "round": round_index, + "index": index, "rank": rank, "submit_monotonic": submitted, + "finish_monotonic": time.monotonic(), + "num_prompt_tokens": len(output.prompt_token_ids), + "num_output_tokens": len(output.outputs[0].token_ids), + "finish_reason": output.outputs[0].finish_reason, + }) + offset_after = time.time() - time.monotonic() + return {"label": label, "round": round_index, "num_requests": num_requests, + "wall_minus_monotonic_before": offset_before, + "wall_minus_monotonic_after": offset_after, + "idle_wait_s": await wait_until_idle()} + + rounds = [] + try: + rounds.append(await burst("warmup", 0, args.warmups)) + for num_requests in args.bursts: + for round_index in range(args.rounds): + rounds.append(await burst(f"b{num_requests}-r{round_index}", round_index, num_requests)) + cache_config = engine.vllm_config.cache_config + summary = { + "model_config": args.model_config, + "num_gpu_blocks": cache_config.num_gpu_blocks, + "block_size": cache_config.block_size, + "engine_args": {key: value for key, value in vars(engine_args).items() + if isinstance(value, (bool, int, float, str, type(None)))}, + "rounds": rounds, + } + finally: + engine.shutdown() + (output_dir / "requests.jsonl").write_text("".join(json.dumps(row) + "\n" for row in records)) + return summary + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0]) + commands = parser.add_subparsers(dest="command", required=True) + overlay = commands.add_parser("overlay") + overlay.add_argument("--site-vllm", type=Path, required=True) + overlay.add_argument("--checkout", type=Path, required=True) + overlay.add_argument("--destination", type=Path, required=True) + overlay.add_argument("--expected-changes", type=Path, required=True) + overlay.add_argument("--report", type=Path, required=True) + run = commands.add_parser("run") + run.add_argument("--model-config", required=True) + run.add_argument("--output-dir", required=True) + run.add_argument("--enable-expert-parallel", action="store_true") + run.add_argument("--data-parallel-size", type=int, default=2) + run.add_argument("--pipeline-parallel-size", type=int, default=2) + run.add_argument("--prompt-tokens", type=int, default=256) + run.add_argument("--max-num-seqs", type=int, default=4) + run.add_argument("--max-model-len", type=int, default=512) + run.add_argument("--gpu-memory-utilization", type=float, default=0.5) + run.add_argument("--bursts", type=int, nargs="+", default=[8, 16]) + run.add_argument("--rounds", type=int, default=3) + run.add_argument("--warmups", type=int, default=4) + run.add_argument("--idle-gap-s", type=float, default=1.0) + run.add_argument("--idle-timeout-s", type=float, default=60.0) + args = parser.parse_args(argv) + + if args.command == "overlay": + report = build_overlay(args.site_vllm, args.checkout, args.destination, args.expected_changes) + args.report.write_text(json.dumps(report, indent=1)) + print(json.dumps({key: report[key] for key in ("accepted", "unexpected", "missing")})) + return 0 if report["accepted"] else 3 + Path(args.output_dir).mkdir(parents=True, exist_ok=True) + summary = asyncio.run(run_bursts(args)) + Path(args.output_dir, "summary.json").write_text(json.dumps(summary, indent=1)) + print("DRIVER_DONE", json.dumps({"rounds": len(summary["rounds"]), + "num_gpu_blocks": summary["num_gpu_blocks"]})) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From dac4e693610880b52a92d6775638a86e5697c73e Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 02:33:56 +0800 Subject: [PATCH 07/19] fix(scheduler): order full-stage admission only behind queued EP waves StageExecutionContext admitted a ticket only from the head of its ready FIFO. Under pipeline parallelism an attention-DP lane that is still busy can queue its next batch at a shared stage, and that ticket then refuses the idle lane's queued batch. With every lane waiting on the head, the sequential run ends with work left (admission deadlock), and dense runs serialize the lanes instead of overlapping them. A full-stage ticket may now be admitted ahead of earlier queued full-stage tickets, but never ahead of an EP wave queued before it. EP waves keep the strict head rule, so they still wait for every operation queued earlier, and the forward-group seal is unchanged. Tests: contract tests for the bypass, both sides of the EP boundary and capacity 1; a DECODE_FFN control that keeps dense groups in counter order around a queued EP wave; a two-lane drain through promotion and restore; and simulator cases that complete the MoE admission-deadlock witnesses and start both dense lanes in the first forward. On the base revision the bypass, capacity-1, drain and simulator tests fail as recorded in the task's negative controls. --- .../stage_execution_context.py | 28 ++++++-- .../test_stage_admission_pipeline_lanes.py | 65 +++++++++++++++++ .../test_mixed_layer_decode_ffn_scheduling.py | 69 +++++++++++++++++++ .../test_shared_forward_group_admission.py | 37 ++++++++++ tests/unit/test_stage_execution_context.py | 51 ++++++++++++++ 5 files changed, 243 insertions(+), 7 deletions(-) create mode 100644 tests/integration/test_stage_admission_pipeline_lanes.py diff --git a/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py b/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py index e92a4edc..1523586f 100644 --- a/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py +++ b/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py @@ -35,9 +35,12 @@ class StageExecutionContext: The context is intentionally independent of event timing and child lane queues. A complete operation first enters the ready FIFO, then the owner - admits it atomically. EP child schedulers may start only after their - wave's ticket has been acquired, and the ticket remains active through the - wave-level combine/cleanup boundary. + admits it atomically. An EP wave is admitted only from the FIFO head, so + it waits for every operation queued before it. A full-stage operation may + be admitted ahead of earlier queued full-stage operations, but never ahead + of an EP wave queued before it. EP child schedulers may start only after + their wave's ticket has been acquired, and the ticket remains active + through the wave-level combine/cleanup boundary. """ def __init__( @@ -320,7 +323,11 @@ def _validate_ticket(self, ticket: StageAdmissionTicket) -> None: ) def try_acquire(self, ticket: StageAdmissionTicket) -> bool: - """Acquire the FIFO-head ticket if this stage is currently idle.""" + """Acquire ``ticket`` if the stage can admit it now. + + An EP wave must be the FIFO head. A full-stage ticket must have no EP + wave queued ahead of it. + """ self._validate_ticket(ticket) if ticket.scope == EP_WAVE: @@ -332,9 +339,16 @@ def try_acquire(self, ticket: StageAdmissionTicket) -> bool: return False elif self._forward_group_sealed: return False - if not self._ready_fifo or self._ready_fifo[0] != ticket: - return False - self._ready_fifo.popleft() + if ticket.scope == EP_WAVE: + if not self._ready_fifo or self._ready_fifo[0] != ticket: + return False + else: + for queued in self._ready_fifo: + if queued == ticket: + break + if queued.scope == EP_WAVE: + return False + self._ready_fifo.remove(ticket) if ticket.scope == EP_WAVE: self._active_ep_ticket = ticket else: diff --git a/tests/integration/test_stage_admission_pipeline_lanes.py b/tests/integration/test_stage_admission_pipeline_lanes.py new file mode 100644 index 00000000..11970c18 --- /dev/null +++ b/tests/integration/test_stage_admission_pipeline_lanes.py @@ -0,0 +1,65 @@ +"""Simulator regression for attention-DP lanes sharing pipeline stages.""" + +import csv +import json +import os +from pathlib import Path +import subprocess +import sys + +import pytest + +from tests.e2e.stage_admission_matrix import ATTN_DP_LANE, SUCCESS, build_cases, read_ledger, run_case_in_child + +REPO_ROOT = Path(__file__).resolve().parents[2] +SET_NAME = "test" + + +def run_case(root, case_id): + """Run one matrix case in its own process, because ``IS_MOE`` is process-global.""" + result = subprocess.run( + [sys.executable, str(Path(__file__).resolve()), str(root), case_id], + cwd=REPO_ROOT, + env={**os.environ, "PYTHONPATH": str(REPO_ROOT), "WANDB_DISABLED": "true", + "VIDUR_DISABLE_WANDB": "1", "OMP_NUM_THREADS": "1", "OPENBLAS_NUM_THREADS": "1"}, + text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=300, + ) + (root / "run.log").write_text(result.stdout) + assert result.returncode == 0, result.stdout[-15000:] + case_dir = root / SET_NAME / case_id + outcome = json.loads((case_dir / "outcome.json").read_text()) + assert outcome["outcome"] == SUCCESS, outcome + return case_dir / "metrics" + + +@pytest.mark.parametrize("case_id, expected", [ + # (requests, prefill tokens, decode tokens): 16 prompt tokens and one output token each. + ("G3a-moe-dp2-pp2-n4", (4, 64, 4)), + ("G3a-moe-dp4-pp2-n8", (8, 128, 8)), +]) +def test_moe_lanes_complete_every_request(tmp_path, case_id, expected): + metrics_dir = run_case(tmp_path, case_id) + with next(metrics_dir.rglob("request_metrics.csv")).open() as handle: + rows = list(csv.DictReader(handle)) + observed = ( + len(rows), + sum(int(float(row["request_num_prefill_tokens"])) for row in rows), + sum(int(float(row["request_num_decode_tokens"])) for row in rows), + ) + assert observed == expected + + +def test_dense_lanes_start_in_the_same_first_forward(tmp_path): + """Every request arrives at t=0 and the stage has room for both lanes.""" + rows = read_ledger(run_case(tmp_path, "G4-dense-dp2-pp2-n8")) + first_start = {} + for row in sorted(rows, key=lambda row: row["stage_start_ts"]): + if row["execution_scope"] == ATTN_DP_LANE and row["stage_id"] == 0: + first_start.setdefault(row["replica_local_id"], row["stage_start_ts"]) + assert sorted(first_start) == [0, 1] + assert first_start[0] == first_start[1] + + +if __name__ == "__main__": + cases = {case.case_id: case for case in build_cases()} + run_case_in_child(cases[sys.argv[2]], Path(sys.argv[1]), SET_NAME) diff --git a/tests/unit/test_mixed_layer_decode_ffn_scheduling.py b/tests/unit/test_mixed_layer_decode_ffn_scheduling.py index 7f3db6fe..9c959682 100644 --- a/tests/unit/test_mixed_layer_decode_ffn_scheduling.py +++ b/tests/unit/test_mixed_layer_decode_ffn_scheduling.py @@ -799,6 +799,75 @@ def test_decode_ffn_wave_materialization_attaches_one_parent_ticket( assert context.queued_tickets == (tickets[0],) +def test_decode_ffn_dense_groups_keep_counter_order_around_a_queued_ep_wave( + mixed_model_config, +) -> None: + """Dense groups enter in group-counter order and never pass an EP wave queued ahead.""" + + scheduler, _, _, lane_sinks = _atomicity_scheduler( + mixed_model_config, + layer_id=4, + ep_size=2, + ) + full_stage_sink = _QueuedBatchSink() + scheduler._full_stage_replica_schedulers = {0: full_stage_sink} + scheduler.get_full_stage_replica_scheduler = Mock(return_value=full_stage_sink) + scheduler._m2n_ready_groups = deque( + [ + [(_source_batch(layer_id=layer_id), _transfer_info(layer_id=layer_id))] + for layer_id in (3, 4, 3) + ] + ) + context = scheduler.get_stage_execution_context(0, 2) + for _ in range(3): + scheduler.schedule_ffn_with_m2n_immediate() + + first_dense, second_dense = full_stage_sink._m2n_immediate_batch_queue + lane_batches = [lane_sinks[ep_id]._m2n_immediate_batch_queue[0] for ep_id in (0, 1)] + wave = lane_batches[0]._stage_admission_ticket + assert [first_dense.global_id, lane_batches[0].global_id, second_dense.global_id] == [0, 1, 2] + assert context.queued_tickets == ( + first_dense._stage_admission_ticket, + wave, + second_dense._stage_admission_ticket, + ) + + def stage_scheduler(replica_local_id): + return ReplicaStageScheduler( + replica_id=0, + stage_id=2, + is_last_stage=True, + is_moe=True, + execution_time_predictor=object(), + cluster_type=ClusterType.DECODE_FFN, + replica_local_id=replica_local_id, + stage_execution_context=context, + ) + + full_stage = stage_scheduler(None) + ep_lanes = [stage_scheduler(ep_id) for ep_id in (0, 1)] + full_stage.add_batch(first_dense) + full_stage.add_batch(second_dense) + for lane, batch in zip(ep_lanes, lane_batches): + lane.add_batch(batch) + assert full_stage.get_queue_batches() == [first_dense, second_dense] + + assert full_stage.pop_batch_if_not_busy() is first_dense + full_stage.on_stage_end() + context.release(first_dense._stage_admission_ticket) + assert full_stage.pop_batch_if_not_busy() is None + + assert ep_lanes[0].pop_batch_if_not_busy() is lane_batches[0] + assert ep_lanes[1].pop_batch_if_not_busy() is lane_batches[1] + assert full_stage.pop_batch_if_not_busy() is None + for lane in ep_lanes: + lane.on_stage_end() + context.release(wave) + + assert full_stage.pop_batch_if_not_busy() is second_dense + assert context.queued_tickets == () + + def _atomicity_snapshot( scheduler, source_batch, queue_sinks, *, include_entity_ids: bool = True ): diff --git a/tests/unit/test_shared_forward_group_admission.py b/tests/unit/test_shared_forward_group_admission.py index 7b5284f8..5d97ebc7 100644 --- a/tests/unit/test_shared_forward_group_admission.py +++ b/tests/unit/test_shared_forward_group_admission.py @@ -51,6 +51,43 @@ def test_admitted_lanes_share_identity_after_unequal_batch_histories(cluster_typ assert [batch.global_id for batch in batches] == [2, 1] +@pytest.mark.parametrize("first_lane", [0, 1]) +def test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket(first_lane): + """Under PP a busy lane can hold the FIFO head; the other lane must still join.""" + + other_lane = 1 - first_lane + context = StageExecutionContext(replica_id=0, stage_id=0, ep_size=2, full_stage_capacity=2) + stages = [make_stage(context, lane, ClusterType.MONOLITHIC) for lane in range(2)] + first_now, first_next = make_batch(first_lane, 0), make_batch(first_lane, 1) + # Distinct provisional ids, so that sharing a bound group is observable. + other_now, other_next = make_batch(other_lane, 5), make_batch(other_lane, 6) + stages[first_lane].add_batch(first_now) + assert stages[first_lane].pop_batch_if_not_busy() is first_now + stages[first_lane].add_batch(first_next) + stages[other_lane].add_batch(other_now) + stages[other_lane].add_batch(other_next) + assert context.queued_tickets[0] == first_next._stage_admission_ticket + + assert stages[other_lane].pop_batch_if_not_busy() is other_now + assert other_now._forward_cohort_provisional_id == first_now._forward_cohort_provisional_id + + wave = context.replace_full_stage_owners_with_ep_wave( + (first_now._stage_admission_ticket, other_now._stage_admission_ticket), + operation_id="wave", participant_ep_ids=(0, 1), + ) + owners = context.replace_ep_wave_with_full_stage_owners(wave, operation_ids=("restored0", "restored1")) + for owner in owners: + context.release(owner) + for stage in stages: + stage.on_stage_end() + + assert stages[first_lane].pop_batch_if_not_busy() is first_next + assert stages[other_lane].pop_batch_if_not_busy() is other_next + assert first_next._forward_cohort_provisional_id == other_next._forward_cohort_provisional_id + assert first_next._forward_cohort_provisional_id > first_now._forward_cohort_provisional_id + assert context.queued_tickets == () + + def test_started_group_blocks_new_lane_through_ep_restore_and_partial_release(): context = StageExecutionContext(replica_id=0, stage_id=0, ep_size=2, full_stage_capacity=4) first = context.enqueue_full_stage(operation_id="first") diff --git a/tests/unit/test_stage_execution_context.py b/tests/unit/test_stage_execution_context.py index 4eb05de0..6777cdb5 100644 --- a/tests/unit/test_stage_execution_context.py +++ b/tests/unit/test_stage_execution_context.py @@ -91,6 +91,57 @@ def test_admission_fifo_cannot_skip_an_earlier_ready_wave() -> None: context.release(second) +def _context_with_wave_between_full_stage_tickets(): + context = StageExecutionContext( + replica_id=0, + stage_id=0, + ep_size=2, + full_stage_capacity=2, + ) + full0 = context.enqueue_full_stage(operation_id="full0") + full1 = context.enqueue_full_stage(operation_id="full1") + wave0 = context.enqueue_ep_wave(operation_id="wave0", participant_ep_ids=(0, 1)) + full2 = context.enqueue_full_stage(operation_id="full2") + return context, full0, full1, wave0, full2 + + +def test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave() -> None: + context, full0, full1, wave0, full2 = _context_with_wave_between_full_stage_tickets() + + assert context.try_acquire(full1) is True + assert context.queued_tickets == (full0, wave0, full2) + # Capacity remains, but wave0 is queued ahead of full2. + assert context.try_acquire(full2) is False + assert context.try_acquire(full0) is True + + +def test_queued_ep_wave_orders_full_stage_work_on_both_sides() -> None: + context, full0, full1, wave0, full2 = _context_with_wave_between_full_stage_tickets() + + assert context.try_acquire(full0) is True + assert context.try_acquire(full1) is True + context.release(full1) + assert context.try_acquire(full2) is False + assert context.try_acquire(wave0) is False + context.release(full0) + assert context.try_acquire(wave0) is True + assert context.try_acquire(full2) is False + context.release(wave0) + assert context.try_acquire(full2) is True + context.release(full2) + assert context.is_idle + assert context.queued_tickets == () + + +def test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket() -> None: + context = StageExecutionContext(replica_id=0, stage_id=0, ep_size=1) + full0 = context.enqueue_full_stage(operation_id="full0") + full1 = context.enqueue_full_stage(operation_id="full1") + + assert context.try_acquire(full1) is True + assert context.queued_tickets == (full0,) + + def test_release_requires_the_active_operation_ticket() -> None: context = StageExecutionContext(replica_id=0, stage_id=0, ep_size=1) wave = context.enqueue_ep_wave(operation_id=30, participant_ep_ids=(0,)) From a1b98194c9bc4ff3a28bc6440449a46120482299 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 02:41:36 +0800 Subject: [PATCH 08/19] tests: apply a recorded patch to the accepted vLLM overlay The ground-truth checkout passes a fifth renormalize argument to _moe_C::topk_softmax, while both the checkout's own csrc and the v0.10.2 image declare the four-argument op, so the MoE scenario failed in profile_run. The overlay step now takes an optional unified diff, applied after the fork-change check accepts the overlay, and records its SHA-256, the files it touched, and whether each now equals the image's copy. Hunks are applied as exact text replacements because the worker image need not carry patch or git. --- .../stage_admission_pp/run_vllm_worker.sh | 4 +- .../stage_admission_pp/vllm_burst_driver.py | 57 ++++++++++++++++++- 2 files changed, 59 insertions(+), 2 deletions(-) diff --git a/tests/comparison/stage_admission_pp/run_vllm_worker.sh b/tests/comparison/stage_admission_pp/run_vllm_worker.sh index 0b5896ea..e7dea8ff 100644 --- a/tests/comparison/stage_admission_pp/run_vllm_worker.sh +++ b/tests/comparison/stage_admission_pp/run_vllm_worker.sh @@ -11,6 +11,8 @@ # CASE_DIR calibration case directory on the mounted workspace; reads # inputs/, writes runs/vllm-instrumented// # ARCHIVE_DIR cloud-volume directory for this run +# Optional: +# OVERLAY_PATCH recorded unified diff applied to the accepted overlay set -euo pipefail set +x : "${RUN_TAG:?}" "${FRONTIER_TREE:?}" "${GROUNDTRUTH:?}" "${CASE_DIR:?}" "${ARCHIVE_DIR:?}" @@ -60,7 +62,7 @@ SITE_VLLM=$("$PY" -c 'import importlib.util, os; print(os.path.dirname(importlib "$PY" "$SCRIPT_DIR/vllm_burst_driver.py" overlay \ --site-vllm "$SITE_VLLM" --checkout "$GROUNDTRUTH" --destination "$WORK/overlay" \ --expected-changes "$CASE_DIR/inputs/fork_changed_files.txt" \ - --report "$WORK/overlay_report.json" || status=3 + --report "$WORK/overlay_report.json" ${OVERLAY_PATCH:+--patch "$OVERLAY_PATCH"} || status=3 if [ "$status" -ne 0 ]; then publish "$ARCHIVE_DIR"; publish_evidence echo "WORKER_STATUS=$status overlay rejected" diff --git a/tests/comparison/stage_admission_pp/vllm_burst_driver.py b/tests/comparison/stage_admission_pp/vllm_burst_driver.py index a82db805..70ab06cf 100644 --- a/tests/comparison/stage_admission_pp/vllm_burst_driver.py +++ b/tests/comparison/stage_admission_pp/vllm_burst_driver.py @@ -10,7 +10,8 @@ ``vllm/**/*.py`` of the ground-truth checkout over it. The overlay is accepted only when the files where the image and the checkout differ are exactly the checkout's own changes over its upstream base, listed in - ``--expected-changes``. + ``--expected-changes``. An accepted overlay may then take one recorded + ``--patch`` (a unified diff), whose SHA-256 and files enter the report. ``run`` Start one ``AsyncLLM`` with DP=2, PP=2, TP=1 (EP for the MoE model), run @@ -30,9 +31,11 @@ import argparse import asyncio import filecmp +import hashlib import json import os import random +import re import shutil import sys import time @@ -69,6 +72,46 @@ def build_overlay(site_vllm: Path, checkout: Path, destination: Path, expected_c } +def apply_patch(patch: Path, root: Path) -> list[str]: + """Apply the unified diff ``patch`` to files under ``root``. + + The worker image need not carry ``patch`` or ``git``, so hunks are applied + here as text replacements; each hunk must match its file exactly once. + """ + lines = patch.read_text().splitlines(keepends=True) + hunks: dict[str, list[tuple[str, str]]] = {} + index = 0 + while index < len(lines): + line = lines[index] + index += 1 + if line.startswith("+++ "): + target = line[4:].strip().removeprefix("b/") + hunks[target] = [] + elif line.startswith("@@ "): + header = re.match(r"@@ -\d+(?:,(\d+))? \+\d+(?:,(\d+))? @@", line) + old_count, new_count = (int(count or 1) for count in header.groups()) + old, new = [], [] + while old_count or new_count: + tag, text = lines[index][0], lines[index][1:] + index += 1 + if tag in " -": + old.append(text) + old_count -= 1 + if tag in " +": + new.append(text) + new_count -= 1 + hunks[target].append(("".join(old), "".join(new))) + for relative, edits in hunks.items(): + path = root / relative + text = path.read_text() + for old, new in edits: + if text.count(old) != 1: + raise ValueError(f"{patch}: a hunk does not match {relative} exactly once") + text = text.replace(old, new) + path.write_text(text) + return sorted(hunks) + + def write_model_dir(model_config: Path, model_dir: Path) -> dict: config = json.loads(model_config.read_text()) model_dir.mkdir(parents=True, exist_ok=True) @@ -187,6 +230,7 @@ def main(argv: list[str] | None = None) -> int: overlay.add_argument("--destination", type=Path, required=True) overlay.add_argument("--expected-changes", type=Path, required=True) overlay.add_argument("--report", type=Path, required=True) + overlay.add_argument("--patch", type=Path) run = commands.add_parser("run") run.add_argument("--model-config", required=True) run.add_argument("--output-dir", required=True) @@ -206,6 +250,17 @@ def main(argv: list[str] | None = None) -> int: if args.command == "overlay": report = build_overlay(args.site_vllm, args.checkout, args.destination, args.expected_changes) + if report["accepted"] and args.patch is not None: + files = apply_patch(args.patch, args.destination) + report["patch"] = { + "path": str(args.patch), + "sha256": hashlib.sha256(args.patch.read_bytes()).hexdigest(), + "files": files, + "equal_to_image_after_patch": { + name: filecmp.cmp(args.destination / name, args.site_vllm.parent / name, shallow=False) + for name in files + }, + } args.report.write_text(json.dumps(report, indent=1)) print(json.dumps({key: report[key] for key in ("accepted", "unexpected", "missing")})) return 0 if report["accepted"] else 3 From df7868e4587d4ee74185db11eef56daac11dcf2b Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 02:48:28 +0800 Subject: [PATCH 09/19] docs(stage-admission): record P0-P3 and the vLLM comparison Adds the test report, the calibration case for the vLLM comparison (two GPU runs, the recorded four-argument topk_softmax overlay patch, and the workflow-gap analysis), and the selected evidence: base negative controls, G2 identity comparisons, the path-T explanation and the Step 9 probe. Progress, requirements (R-7) and plan are updated. Two plan stop conditions are open for the user: the C3 witness rule at attn_dp=4, and V5 co-execution on the dense vLLM shape. --- ...ecution_decomposition_sa-pp-20260923a.json | 56 + ...ecution_decomposition_sa-pp-20260923b.json | 110 + .../analysis/lane_metrics.json | 3568 +++++++++++++++++ .../analysis/synthetic_check.py | 55 + .../analysis/workflow_gap_status.json | 10 + .../analysis/workflow_gap_summary.md | 51 + .../analysis/workflow_gap_table.csv | 53 + .../stage_admission_case_001/case_init.md | 20 + .../inputs/fork_changed_files.txt | 61 + .../inputs/groundtruth_local_commit.diff | 412 ++ .../inputs/groundtruth_overlay.patch | 24 + .../stage_admission_case_001/manifest.yaml | 148 + .../sa-pp-20260923a/COMPLETE | 1 + .../sa-pp-20260923a/overlay_report.json | 130 + .../sa-pp-20260923a/replica_log.txt | 73 + .../dense/dp_placement/dp_placement_588.jsonl | 22 + .../dense/dp_placement/dp_placement_663.jsonl | 65 + .../dense/dp_placement/dp_placement_664.jsonl | 65 + .../runs/dense/model/config.json | 95 + .../runs/dense/pp_boundary.jsonl | 152 + .../sa-pp-20260923a/runs/dense/requests.jsonl | 76 + .../sa-pp-20260923a/runs/dense/summary.json | 186 + .../runs/moe/model/config.json | 95 + .../sa-pp-20260923a/vllm_import.txt | 1 + .../sa-pp-20260923a/worker_env.json | 1 + .../sa-pp-20260923b/COMPLETE | 1 + .../sa-pp-20260923b/overlay_report.json | 142 + .../sa-pp-20260923b/replica_log.txt | 14 + .../dense/dp_placement/dp_placement_844.jsonl | 22 + .../dense/dp_placement/dp_placement_919.jsonl | 65 + .../dense/dp_placement/dp_placement_920.jsonl | 65 + .../runs/dense/model/config.json | 95 + .../runs/dense/pp_boundary.jsonl | 152 + .../sa-pp-20260923b/runs/dense/requests.jsonl | 76 + .../sa-pp-20260923b/runs/dense/summary.json | 186 + .../moe/dp_placement/dp_placement_157.jsonl | 22 + .../moe/dp_placement/dp_placement_232.jsonl | 65 + .../moe/dp_placement/dp_placement_233.jsonl | 65 + .../runs/moe/model/config.json | 95 + .../runs/moe/pp_boundary.jsonl | 152 + .../sa-pp-20260923b/runs/moe/requests.jsonl | 76 + .../sa-pp-20260923b/runs/moe/summary.json | 186 + .../sa-pp-20260923b/vllm_import.txt | 1 + .../sa-pp-20260923b/worker_env.json | 1 + .../evidence/decompose_co_execution.py | 46 + .../evidence/explain_t_path.py | 68 + .../evidence/g2_integration_compare.json | 12 + .../evidence/g2_unit_compare.json | 15 + .../evidence/p3_t_path_explanation.json | 436 ++ .../step9_probe/after_dense_dp1_pp2.json | 10 + .../step9_probe/after_moe_dp2_pp1.json | 10 + .../step9_probe/after_moe_dp2_pp2.json | 10 + .../step9_probe/after_moe_dp2_pp3.json | 9 + .../step9_probe/base_moe_dp2_pp2.json | 9 + .../evidence/step9_probe/probe_completion.py | 33 + .../evidence/step9_probe/probe_main.py | 179 + .../plan.md | 134 +- .../progress.md | 34 +- .../requirements.md | 20 + ...ort_2026-09-23_stage_admission_ordering.md | 260 ++ 60 files changed, 8292 insertions(+), 4 deletions(-) create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/lane_metrics.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/synthetic_check.py create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/case_init.md create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/fork_changed_files.txt create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_local_commit.diff create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/COMPLETE create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/overlay_report.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/replica_log.txt create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_588.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_663.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_664.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/model/config.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/pp_boundary.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/requests.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/summary.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/moe/model/config.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/vllm_import.txt create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/worker_env.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/COMPLETE create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/overlay_report.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/replica_log.txt create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_844.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_919.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_920.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/model/config.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/pp_boundary.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/requests.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/summary.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_157.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_232.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_233.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/model/config.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/pp_boundary.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/requests.jsonl create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/summary.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/vllm_import.txt create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/worker_env.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_integration_compare.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_unit_compare.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/p3_t_path_explanation.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_dense_dp1_pp2.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp1.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp2.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp3.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/base_moe_dp2_pp2.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_completion.py create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json new file mode 100644 index 00000000..fda094a3 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json @@ -0,0 +1,56 @@ +{ + "dense/n8/r0": { + "pairs": 4, + "M5_observed": 0.6421, + "M5_equal_durations": 0.7101, + "non_overlap_ms_from_start_offsets": 2.2, + "non_overlap_ms_from_end_offsets": 3.808, + "stage0_duration_ms_median": 3.435, + "stage0_duration_ms_cv": 0.116 + }, + "dense/n8/r1": { + "pairs": 4, + "M5_observed": 0.7393, + "M5_equal_durations": 0.7521, + "non_overlap_ms_from_start_offsets": 1.805, + "non_overlap_ms_from_end_offsets": 2.014, + "stage0_duration_ms_median": 3.222, + "stage0_duration_ms_cv": 0.183 + }, + "dense/n8/r2": { + "pairs": 4, + "M5_observed": 0.7604, + "M5_equal_durations": 0.9323, + "non_overlap_ms_from_start_offsets": 0.395, + "non_overlap_ms_from_end_offsets": 3.071, + "stage0_duration_ms_median": 2.651, + "stage0_duration_ms_cv": 0.261 + }, + "dense/n16/r0": { + "pairs": 8, + "M5_observed": 0.5372, + "M5_equal_durations": 0.5704, + "non_overlap_ms_from_start_offsets": 12.44, + "non_overlap_ms_from_end_offsets": 11.9, + "stage0_duration_ms_median": 4.561, + "stage0_duration_ms_cv": 0.16 + }, + "dense/n16/r1": { + "pairs": 8, + "M5_observed": 0.752, + "M5_equal_durations": 0.6557, + "non_overlap_ms_from_start_offsets": 5.729, + "non_overlap_ms_from_end_offsets": 1.232, + "stage0_duration_ms_median": 3.032, + "stage0_duration_ms_cv": 0.162 + }, + "dense/n16/r2": { + "pairs": 8, + "M5_observed": 0.7668, + "M5_equal_durations": 0.6756, + "non_overlap_ms_from_start_offsets": 5.235, + "non_overlap_ms_from_end_offsets": 1.092, + "stage0_duration_ms_median": 2.738, + "stage0_duration_ms_cv": 0.178 + } +} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json new file mode 100644 index 00000000..04089bd8 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json @@ -0,0 +1,110 @@ +{ + "moe/n8/r0": { + "pairs": 4, + "M5_observed": 0.9769, + "M5_equal_durations": 0.9713, + "non_overlap_ms_from_start_offsets": 0.321, + "non_overlap_ms_from_end_offsets": 0.194, + "stage0_duration_ms_median": 5.312, + "stage0_duration_ms_cv": 0.066 + }, + "moe/n8/r1": { + "pairs": 4, + "M5_observed": 0.9735, + "M5_equal_durations": 0.9648, + "non_overlap_ms_from_start_offsets": 0.402, + "non_overlap_ms_from_end_offsets": 0.197, + "stage0_duration_ms_median": 5.335, + "stage0_duration_ms_cv": 0.091 + }, + "moe/n8/r2": { + "pairs": 4, + "M5_observed": 0.9774, + "M5_equal_durations": 0.9676, + "non_overlap_ms_from_start_offsets": 0.536, + "non_overlap_ms_from_end_offsets": 0.204, + "stage0_duration_ms_median": 8.45, + "stage0_duration_ms_cv": 0.088 + }, + "moe/n16/r0": { + "pairs": 8, + "M5_observed": 0.9782, + "M5_equal_durations": 0.9686, + "non_overlap_ms_from_start_offsets": 0.698, + "non_overlap_ms_from_end_offsets": 0.26, + "stage0_duration_ms_median": 5.294, + "stage0_duration_ms_cv": 0.067 + }, + "moe/n16/r1": { + "pairs": 8, + "M5_observed": 0.9365, + "M5_equal_durations": 0.9029, + "non_overlap_ms_from_start_offsets": 3.274, + "non_overlap_ms_from_end_offsets": 0.688, + "stage0_duration_ms_median": 7.424, + "stage0_duration_ms_cv": 0.05 + }, + "moe/n16/r2": { + "pairs": 8, + "M5_observed": 0.9283, + "M5_equal_durations": 0.8972, + "non_overlap_ms_from_start_offsets": 3.827, + "non_overlap_ms_from_end_offsets": 0.682, + "stage0_duration_ms_median": 7.483, + "stage0_duration_ms_cv": 0.055 + }, + "dense/n8/r0": { + "pairs": 4, + "M5_observed": 0.6569, + "M5_equal_durations": 0.832, + "non_overlap_ms_from_start_offsets": 1.487, + "non_overlap_ms_from_end_offsets": 4.198, + "stage0_duration_ms_median": 3.052, + "stage0_duration_ms_cv": 0.291 + }, + "dense/n8/r1": { + "pairs": 4, + "M5_observed": 0.8508, + "M5_equal_durations": 0.7401, + "non_overlap_ms_from_start_offsets": 1.669, + "non_overlap_ms_from_end_offsets": 0.26, + "stage0_duration_ms_median": 2.967, + "stage0_duration_ms_cv": 0.128 + }, + "dense/n8/r2": { + "pairs": 4, + "M5_observed": 0.6089, + "M5_equal_durations": 0.6668, + "non_overlap_ms_from_start_offsets": 2.627, + "non_overlap_ms_from_end_offsets": 4.399, + "stage0_duration_ms_median": 3.752, + "stage0_duration_ms_cv": 0.19 + }, + "dense/n16/r0": { + "pairs": 8, + "M5_observed": 0.9259, + "M5_equal_durations": 0.9111, + "non_overlap_ms_from_start_offsets": 1.017, + "non_overlap_ms_from_end_offsets": 0.653, + "stage0_duration_ms_median": 2.616, + "stage0_duration_ms_cv": 0.103 + }, + "dense/n16/r1": { + "pairs": 8, + "M5_observed": 0.8332, + "M5_equal_durations": 0.7156, + "non_overlap_ms_from_start_offsets": 3.596, + "non_overlap_ms_from_end_offsets": 0.599, + "stage0_duration_ms_median": 2.615, + "stage0_duration_ms_cv": 0.182 + }, + "dense/n16/r2": { + "pairs": 8, + "M5_observed": 0.8371, + "M5_equal_durations": 0.8982, + "non_overlap_ms_from_start_offsets": 1.324, + "non_overlap_ms_from_end_offsets": 2.807, + "stage0_duration_ms_median": 2.676, + "stage0_duration_ms_cv": 0.193 + } +} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/lane_metrics.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/lane_metrics.json new file mode 100644 index 00000000..3333d51c --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/lane_metrics.json @@ -0,0 +1,3568 @@ +{ + "placement": { + "dense": { + "misplaced": [], + "ok": true, + "requests": 76, + "unseen": [] + }, + "moe": { + "misplaced": [], + "ok": true, + "requests": 76, + "unseen": [] + } + }, + "runs": { + "G7-dense-dp2-pp2-n16": { + "frontier_after": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.0, + "M5_co_execution": 1.0, + "median_forward_duration": 0.12, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.0, + "M5_co_execution": 1.0, + "median_forward_duration": 0.12, + "self_overlap": false + } + }, + "frontier_after_outcome": "success", + "frontier_after_placement_ok": true, + "frontier_base": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 3 + ] + ], + [ + [ + 2 + ], + [ + 5 + ] + ], + [ + [ + 4 + ], + [ + 7 + ] + ], + [ + [ + 6 + ], + [ + 9 + ] + ], + [ + [ + 8 + ], + [ + 11 + ] + ], + [ + [ + 10 + ], + [ + 13 + ] + ], + [ + [ + 12 + ], + [ + 15 + ] + ], + [ + [ + 14 + ], + null + ] + ], + "M4_co_start": 1.0000000000000002, + "M5_co_execution": 0.7777777777777778, + "median_forward_duration": 0.12, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 3 + ] + ], + [ + [ + 2 + ], + [ + 5 + ] + ], + [ + [ + 4 + ], + [ + 7 + ] + ], + [ + [ + 6 + ], + [ + 9 + ] + ], + [ + [ + 8 + ], + [ + 11 + ] + ], + [ + [ + 10 + ], + [ + 13 + ] + ], + [ + [ + 12 + ], + [ + 15 + ] + ], + [ + [ + 14 + ], + null + ] + ], + "M4_co_start": 1.0000000000000002, + "M5_co_execution": 0.7777777777777778, + "median_forward_duration": 0.12, + "self_overlap": false + } + }, + "frontier_base_outcome": "success", + "vllm": { + "0": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.17091455981041143, + "M5_co_execution": 0.9259308116414676, + "median_forward_duration": 0.00261572003364563, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.015250329608562718, + "M5_co_execution": 0.9556969257536658, + "median_forward_duration": 0.0029625799506902695, + "self_overlap": false + } + }, + "1": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.09309313158893653, + "M5_co_execution": 0.8331530258980444, + "median_forward_duration": 0.002615438774228096, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.08185507172547078, + "M5_co_execution": 0.7170726933331707, + "median_forward_duration": 0.003510111942887306, + "self_overlap": false + } + }, + "2": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.04588904145721386, + "M5_co_execution": 0.8370898411019174, + "median_forward_duration": 0.0026763956993818283, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.11631847952009687, + "M5_co_execution": 0.7891339057001696, + "median_forward_duration": 0.0030781766399741173, + "self_overlap": false + } + } + } + }, + "G7-dense-dp2-pp2-n8": { + "frontier_after": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.0, + "M5_co_execution": 1.0, + "median_forward_duration": 0.12000000000000001, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.0, + "M5_co_execution": 1.0, + "median_forward_duration": 0.12000000000000002, + "self_overlap": false + } + }, + "frontier_after_outcome": "success", + "frontier_after_placement_ok": true, + "frontier_base": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 3 + ] + ], + [ + [ + 2 + ], + [ + 5 + ] + ], + [ + [ + 4 + ], + [ + 7 + ] + ], + [ + [ + 6 + ], + null + ] + ], + "M4_co_start": 1.0, + "M5_co_execution": 0.6, + "median_forward_duration": 0.12000000000000001, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 3 + ] + ], + [ + [ + 2 + ], + [ + 5 + ] + ], + [ + [ + 4 + ], + [ + 7 + ] + ], + [ + [ + 6 + ], + null + ] + ], + "M4_co_start": 0.9999999999999999, + "M5_co_execution": 0.6000000000000001, + "median_forward_duration": 0.12000000000000002, + "self_overlap": false + } + }, + "frontier_base_outcome": "success", + "vllm": { + "0": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.008418151150125628, + "M5_co_execution": 0.6568505845035172, + "median_forward_duration": 0.0030516916885972023, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.04170587227710599, + "M5_co_execution": 0.6089605761452099, + "median_forward_duration": 0.004431265406310558, + "self_overlap": false + } + }, + "1": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.04251974485265244, + "M5_co_execution": 0.8508060484104523, + "median_forward_duration": 0.002966976724565029, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.02229866292638622, + "M5_co_execution": 0.6404995388202928, + "median_forward_duration": 0.004050368443131447, + "self_overlap": false + } + }, + "2": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.24755095323041817, + "M5_co_execution": 0.6088974398480734, + "median_forward_duration": 0.0037516485899686813, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.37008108266775924, + "M5_co_execution": 0.6159455220656315, + "median_forward_duration": 0.0037597408518195152, + "self_overlap": false + } + } + } + }, + "G7-moe-dp2-pp2-n16": { + "frontier_after": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.0, + "M5_co_execution": 1.0, + "median_forward_duration": 0.062000000000000055, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.0, + "M5_co_execution": 1.0, + "median_forward_duration": 0.062000000000000055, + "self_overlap": false + } + }, + "frontier_after_outcome": "success", + "frontier_after_placement_ok": true, + "frontier_base": null, + "frontier_base_outcome": "admission_deadlock", + "vllm": { + "0": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.023619782508997624, + "M5_co_execution": 0.9782382605035794, + "median_forward_duration": 0.005294156260788441, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.01174241455626505, + "M5_co_execution": 0.9788841920364375, + "median_forward_duration": 0.005645160563290119, + "self_overlap": false + } + }, + "1": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.004534493121171677, + "M5_co_execution": 0.9365388912624988, + "median_forward_duration": 0.007423891685903072, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.025260268381604473, + "M5_co_execution": 0.9713419954163164, + "median_forward_duration": 0.005722890608012676, + "self_overlap": false + } + }, + "2": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ], + [ + 8 + ], + [ + 10 + ], + [ + 12 + ], + [ + 14 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ], + [ + 9 + ], + [ + 11 + ], + [ + 13 + ], + [ + 15 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.022678808030265958, + "M5_co_execution": 0.9283263327604807, + "median_forward_duration": 0.0074830856174230576, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ], + [ + [ + 8 + ], + [ + 9 + ] + ], + [ + [ + 10 + ], + [ + 11 + ] + ], + [ + [ + 12 + ], + [ + 13 + ] + ], + [ + [ + 14 + ], + [ + 15 + ] + ] + ], + "M4_co_start": 0.0778888599864643, + "M5_co_execution": 0.9677368487359697, + "median_forward_duration": 0.005574577488005161, + "self_overlap": false + } + } + } + }, + "G7-moe-dp2-pp2-n8": { + "frontier_after": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.0, + "M5_co_execution": 1.0, + "median_forward_duration": 0.062000000000000055, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.0, + "M5_co_execution": 1.0, + "median_forward_duration": 0.062000000000000055, + "self_overlap": false + } + }, + "frontier_after_outcome": "success", + "frontier_after_placement_ok": true, + "frontier_base": null, + "frontier_base_outcome": "admission_deadlock", + "vllm": { + "0": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.00864423422319438, + "M5_co_execution": 0.9769112551681639, + "median_forward_duration": 0.005312402732670307, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.044595115345557684, + "M5_co_execution": 0.9602193830355867, + "median_forward_duration": 0.007738551124930382, + "self_overlap": false + } + }, + "1": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.035494719435994984, + "M5_co_execution": 0.9734773526983487, + "median_forward_duration": 0.005334779620170593, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.029321800955295463, + "M5_co_execution": 0.9702129108716677, + "median_forward_duration": 0.007605820894241333, + "self_overlap": false + } + }, + "2": { + "M2_sequences": { + "lane0/stage0": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane0/stage1": [ + [ + 0 + ], + [ + 2 + ], + [ + 4 + ], + [ + 6 + ] + ], + "lane1/stage0": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ], + "lane1/stage1": [ + [ + 1 + ], + [ + 3 + ], + [ + 5 + ], + [ + 7 + ] + ] + }, + "stage0": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.06341533360159404, + "M5_co_execution": 0.9774133792071, + "median_forward_duration": 0.00845013465732336, + "self_overlap": false + }, + "stage1": { + "M3_pairing": [ + [ + [ + 0 + ], + [ + 1 + ] + ], + [ + [ + 2 + ], + [ + 3 + ] + ], + [ + [ + 4 + ], + [ + 5 + ] + ], + [ + [ + 6 + ], + [ + 7 + ] + ] + ], + "M4_co_start": 0.01704118680493941, + "M5_co_execution": 0.968491350994991, + "median_forward_duration": 0.0076489923521876335, + "self_overlap": false + } + } + } + } + } +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/synthetic_check.py b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/synthetic_check.py new file mode 100644 index 00000000..d26e683c --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/synthetic_check.py @@ -0,0 +1,55 @@ +"""Synthetic inputs for compare_lanes: ideal vLLM pairing, a matching Frontier +'after' set, the real P0 base G7 cases, and one vLLM round shifted by a dummy.""" +import json, shutil, sys +from pathlib import Path +from tests.comparison.stage_admission_pp import compare_lanes + +S = Path(sys.argv[1]); BASE = Path('/data/ycfeng/tmp/stage_admission_ordering/base') +D = 0.12 # forward duration +def ideal(n, shift_round=None, rnd=0): + """Stage 0: lane pair k runs [k*D, (k+1)*D); stage 1 one slot later.""" + fw = [] + for i in range(n): + lane, k = i % 2, i // 2 + s0 = k * D + (D if (shift_round == rnd and lane == 1) else 0.0) + fw.append((lane, 0, s0, s0 + D, i)); fw.append((lane, 1, s0 + D, s0 + 2 * D, i)) + return fw +def write_vllm(model, shift): + d = S / 'vllm' / 'runs' / model; (d / 'dp_placement').mkdir(parents=True) + reqs, pp, rounds, place, t0 = [], [], [], {0: [], 1: []}, 100.0 + for label, rnd, n in [('warmup', 0, 4)] + [(f'b{b}-r{r}', r, b) for b in (8, 16) for r in range(3)]: + off = 1.7e9 + for lane, stage, s, e, i in (ideal(n, shift, rnd) if label != 'warmup' else ideal(n)): + rid = f'{label}-q{i}' + rec = {'request_ids': [rid], 'pp_rank': stage, 'is_last_rank': stage == 1, + 'forward_start_ts': t0 + s, 'send_start_ts': None if stage else t0 + e, + 'timestamp': off + t0 + e} + pp.append(rec) + if stage == 0: place[lane].append({'kind': 'engine_iteration', 'engine': lane, 'scheduled_new_req_ids': [rid]}) + for i in range(n): + reqs.append({'request_id': f'{label}-q{i}', 'burst': label, 'round': rnd, 'index': i, 'rank': i % 2, + 'num_output_tokens': 1}) + rounds.append({'label': label, 'wall_minus_monotonic_before': off, 'wall_minus_monotonic_after': off}) + t0 += 10.0 + (d / 'requests.jsonl').write_text(''.join(json.dumps(r) + '\n' for r in reqs)) + (d / 'pp_boundary.jsonl').write_text(''.join(json.dumps(r) + '\n' for r in pp)) + (d / 'summary.json').write_text(json.dumps({'rounds': rounds})) + for e, rows in place.items(): + (d / 'dp_placement' / f'dp_placement_{e}.jsonl').write_text(''.join(json.dumps(r) + '\n' for r in rows)) +def write_after(model, n): + cid = f'G7-{model}-dp2-pp2-n{n}'; d = S / 'frontier' / 'after' / cid; (d / 'metrics' / 'x').mkdir(parents=True) + (d / 'run.json').write_text(json.dumps({'outcome': 'success'})) + shutil.copy(BASE / cid / 'case.json', d / 'case.json') + rows = [{'execution_scope': 'ATTN_DP_LANE', 'replica_local_id': l, 'stage_id': st, 'stage_start_ts': s, + 'stage_end_ts': e, 'request_ids': [str(i)]} for l, st, s, e, i in ideal(n)] + (d / 'metrics' / 'x' / 'frontier_stage_batch_ledger.jsonl').write_text(''.join(json.dumps(r) + '\n' for r in rows)) + (S / 'frontier' / 'base').mkdir(parents=True, exist_ok=True) + (S / 'frontier' / 'base' / cid).symlink_to(BASE / cid) +write_vllm('moe', shift=None); write_vllm('dense', shift=1) +for m in ('moe', 'dense'): + for n in (8, 16): write_after(m, n) +rows, details = compare_lanes.compare(S / 'vllm', S / 'frontier', 'base', 'after') +bad = [(r['check'], r['model'], r['burst'], r['round']) for r in rows if r['status'] != 'MATCH'] +print('rows', len(rows)); print('mismatch', bad) +print('dense base n8 stage0', {k: v for k, v in details['runs']['G7-dense-dp2-pp2-n8']['frontier_base']['stage0'].items() if k != 'M3_pairing'}) +print('placement', {m: (p['ok'], len(p['unseen'])) for m, p in details['placement'].items()}) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json new file mode 100644 index 00000000..2475be98 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json @@ -0,0 +1,10 @@ +{ + "analysis_state": "COMPLETE", + "status": "FAIL", + "correction_state": "not_applicable", + "rows": 52, + "mismatches": 2, + "vllm_placement_ok": true, + "vllm_placement_unseen_requests": 0, + "next_action": "report each MISMATCH row with its cause before P4; adjust nothing" +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md new file mode 100644 index 00000000..3c6a5768 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md @@ -0,0 +1,51 @@ +# Workflow-gap summary — stage_admission_case_001 + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-23 | Created from run `sa-pp-20260923b` against Frontier sets `base` (`1f694f7` rule) and `after` (`dac4e69`). | + +## Inputs + +| Side | Source | +| --- | --- | +| vLLM | `runs/vllm-instrumented/sa-pp-20260923b/` — vLLM-BS `494b9f327` plus `inputs/groundtruth_overlay.patch` (SHA-256 `8d476789…3a9c81`), DP=2, PP=2, TP=1, MoE with EP, 4×H800 | +| Frontier before | `/data/ycfeng/tmp/stage_admission_ordering/base/G7-*` | +| Frontier after | `/data/ycfeng/tmp/stage_admission_ordering/after/G7-*` | +| Producer | `tests/comparison/stage_admission_pp/compare_lanes.py` → `workflow_gap_table.csv`, `lane_metrics.json`, `workflow_gap_status.json` | + +## Result + +52 rows: 50 `MATCH`, 2 `MISMATCH`. `vllm_placement_ok = true`, no unseen +request. + +| Metric | MoE (n8, n16) | Dense (n8, n16) | +| --- | --- | --- | +| V1 completion | MATCH in 6/6 rounds; base `admission_deadlock` (negative control holds) | MATCH in 6/6 rounds | +| V2 lane sequences | MATCH 6/6 | MATCH 6/6 | +| V3 stage-0 pairing | MATCH 6/6 | MATCH 6/6; base pairs are shifted by one forward | +| V4 first-forward co-start | MATCH 6/6 (vLLM ≤ 0.063, after 0.0) | MATCH 6/6 (vLLM ≤ 0.248, after 0.0, base 1.0: negative control holds) | +| V5 stage-0 co-execution | MATCH: vLLM 0.976 / 0.948, after 1.0 | **MISMATCH**: vLLM 0.706 / 0.865, after 1.0, base 0.600 / 0.778 | + +## The two MISMATCH rows + +- Observed: vLLM's dense co-execution varies from round to round by more than + the 0.10 bound: 0.537–0.926 over the 12 dense rounds of runs a and b. The + n16 means of the two runs differ by 0.18. +- Observed (`co_execution_decomposition_sa-pp-20260923{a,b}.json`): the dense + non-overlap has two sources. + - Per-pair start offsets of up to about 1.5 ms, measured before the + per-forward DP metadata exchange. + - End offsets from per-rank duration variation: CV 0.10–0.29 on forwards of + about 3 ms. + MoE ends stay within 0.2–0.7 ms in total. +- Inference: the gap is a duration-variance property of vLLM's dense ranks, + which meet once per forward. It is not an admission difference, because + admission is measured by V1–V4, and they match in every round. The Frontier + owner is the execution-time model: the dummy predictor gives equal + durations. It is not `stage_execution_context.py`, the default owner label + written into the table. +- Plan §3 P5 requires a stop before P4 with nothing adjusted. Proposed + resolution, pending the user: V5 becomes informational for the dense shape + and stays a gate for MoE. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv new file mode 100644 index 00000000..4396e801 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv @@ -0,0 +1,53 @@ +check,model,burst,round,metric,groundtruth,frontier_after,frontier_base,status,frontier_owner,note +V1,moe,8,0,M1 completion,"""8/8""","""success""","""admission_deadlock""",MATCH,, +V2,moe,8,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",null,MATCH,, +V3,moe,8,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]",null,MATCH,, +V4,moe,8,0,M4 stage-0 co-start,0.00864423422319438,0.0,null,MATCH,, +V1,moe,8,1,M1 completion,"""8/8""","""success""","""admission_deadlock""",MATCH,, +V2,moe,8,1,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",null,MATCH,, +V3,moe,8,1,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]",null,MATCH,, +V4,moe,8,1,M4 stage-0 co-start,0.035494719435994984,0.0,null,MATCH,, +V1,moe,8,2,M1 completion,"""8/8""","""success""","""admission_deadlock""",MATCH,, +V2,moe,8,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",null,MATCH,, +V3,moe,8,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]",null,MATCH,, +V4,moe,8,2,M4 stage-0 co-start,0.06341533360159404,0.0,null,MATCH,, +V5,moe,8,mean,M5 stage-0 co-execution,0.9759339956912042,1.0,null,MATCH,, +V1,moe,16,0,M1 completion,"""16/16""","""success""","""admission_deadlock""",MATCH,, +V2,moe,16,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",null,MATCH,, +V3,moe,16,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]",null,MATCH,, +V4,moe,16,0,M4 stage-0 co-start,0.023619782508997624,0.0,null,MATCH,, +V1,moe,16,1,M1 completion,"""16/16""","""success""","""admission_deadlock""",MATCH,, +V2,moe,16,1,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",null,MATCH,, +V3,moe,16,1,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]",null,MATCH,, +V4,moe,16,1,M4 stage-0 co-start,0.004534493121171677,0.0,null,MATCH,, +V1,moe,16,2,M1 completion,"""16/16""","""success""","""admission_deadlock""",MATCH,, +V2,moe,16,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",null,MATCH,, +V3,moe,16,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]",null,MATCH,, +V4,moe,16,2,M4 stage-0 co-start,0.022678808030265958,0.0,null,MATCH,, +V5,moe,16,mean,M5 stage-0 co-execution,0.947701161508853,1.0,null,MATCH,, +V1,dense,8,0,M1 completion,"""8/8""","""success""","""success""",MATCH,, +V2,dense,8,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",MATCH,, +V3,dense,8,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], null]]",MATCH,, +V4,dense,8,0,M4 stage-0 co-start,0.008418151150125628,0.0,1.0,MATCH,, +V1,dense,8,1,M1 completion,"""8/8""","""success""","""success""",MATCH,, +V2,dense,8,1,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",MATCH,, +V3,dense,8,1,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], null]]",MATCH,, +V4,dense,8,1,M4 stage-0 co-start,0.04251974485265244,0.0,1.0,MATCH,, +V1,dense,8,2,M1 completion,"""8/8""","""success""","""success""",MATCH,, +V2,dense,8,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",MATCH,, +V3,dense,8,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], null]]",MATCH,, +V4,dense,8,2,M4 stage-0 co-start,0.24755095323041817,0.0,1.0,MATCH,, +V5,dense,8,mean,M5 stage-0 co-execution,0.7055180242540143,1.0,0.6,MISMATCH,frontier/scheduler/replica_stage_scheduler/stage_execution_context.py, +V1,dense,16,0,M1 completion,"""16/16""","""success""","""success""",MATCH,, +V2,dense,16,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",MATCH,, +V3,dense,16,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], [9]], [[8], [11]], [[10], [13]], [[12], [15]], [[14], null]]",MATCH,, +V4,dense,16,0,M4 stage-0 co-start,0.17091455981041143,0.0,1.0000000000000002,MATCH,, +V1,dense,16,1,M1 completion,"""16/16""","""success""","""success""",MATCH,, +V2,dense,16,1,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",MATCH,, +V3,dense,16,1,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], [9]], [[8], [11]], [[10], [13]], [[12], [15]], [[14], null]]",MATCH,, +V4,dense,16,1,M4 stage-0 co-start,0.09309313158893653,0.0,1.0000000000000002,MATCH,, +V1,dense,16,2,M1 completion,"""16/16""","""success""","""success""",MATCH,, +V2,dense,16,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",MATCH,, +V3,dense,16,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], [9]], [[8], [11]], [[10], [13]], [[12], [15]], [[14], null]]",MATCH,, +V4,dense,16,2,M4 stage-0 co-start,0.04588904145721386,0.0,1.0000000000000002,MATCH,, +V5,dense,16,mean,M5 stage-0 co-execution,0.8653912262138098,1.0,0.7777777777777778,MISMATCH,frontier/scheduler/replica_stage_scheduler/stage_execution_context.py, diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/case_init.md b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/case_init.md new file mode 100644 index 00000000..3aea24cb --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/case_init.md @@ -0,0 +1,20 @@ +# case_init — stage_admission_case_001 + +Immutable record. Written 2026-09-23, before the first GPU run. + +| Field | Value | +| --- | --- | +| `case_id` | `stage_admission_case_001` | +| `run_generation` | 1 | +| `requesting_user` | `i-fengyicheng` | +| `reviewer_identity` | `i-fengyicheng` | +| `auto_recycle` | `false` | +| Ground-truth checkout | `/data/ycfeng/Frontier/.real-engine/vLLM-BS` | +| Ground-truth branch | `feature/frontier-comparison-instrumentation` | +| Ground-truth commit | `494b9f327036d4493034a9b37ebb343354884e01` | +| Ground-truth remote tip | `ea95f571e20937c7c908c6d59ddd1cd6bf9268f1` (local is one unpushed commit ahead) | +| Tree dirty | `false` | +| Diff artifact | `inputs/groundtruth_local_commit.diff`, SHA-256 `84fc24db0e2411268a93f8be7ca5f8e4e5072ea09d86063cac2cfb98381feb2c` | +| Fork changes over `upstream-v0.10.2` (`01efc7ef7`) | `inputs/fork_changed_files.txt`, SHA-256 `be638438ff661e1178f0c5ab68da7b9d249ebf8e641dcb1f6cd15d128c0abdaa` | +| Weight mode | `dummy`, no real weight download | +| Mode | instrumented only; no clean E2E run (plan D-8) | diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/fork_changed_files.txt b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/fork_changed_files.txt new file mode 100644 index 00000000..893d3620 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/fork_changed_files.txt @@ -0,0 +1,61 @@ +vllm/_C.py +vllm/_custom_ops.py +vllm/_moe_C.py +vllm/attention/layer.py +vllm/benchmarks/throughput.py +vllm/compilation/compiler_interface.py +vllm/config/__init__.py +vllm/distributed/communication_op.py +vllm/distributed/kv_transfer/kv_connector/v1/base.py +vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py +vllm/distributed/kv_transfer/kv_connector/v1/p2p/frontier_kv_transfer_logger.py +vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py +vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py +vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py +vllm/distributed/parallel_state.py +vllm/engine/arg_utils.py +vllm/engine/llm_engine.py +vllm/entrypoints/openai/frontier_request_metrics.py +vllm/entrypoints/openai/serving_chat.py +vllm/entrypoints/openai/serving_completion.py +vllm/entrypoints/openai/serving_engine.py +vllm/envs.py +vllm/model_executor/custom_op.py +vllm/model_executor/layers/fused_moe/configs/specific-README +vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py +vllm/model_executor/layers/fused_moe/deepep_ll_prepare_finalize.py +vllm/model_executor/layers/fused_moe/fused_moe.py +vllm/model_executor/layers/fused_moe/layer.py +vllm/model_executor/layers/fused_moe/pplx_prepare_finalize.py +vllm/model_executor/layers/linear.py +vllm/model_executor/layers/vocab_parallel_embedding.py +vllm/model_executor/models/llama.py +vllm/model_executor/models/phimoe.py +vllm/model_executor/models/qwen3_moe.py +vllm/model_executor/models/qwen3_moe_mtp.py +vllm/model_executor/models/registry.py +vllm/request_generator/__init__.py +vllm/request_generator/config.py +vllm/request_generator/kv_sync.py +vllm/request_generator/plan.md +vllm/request_generator/prompt_generator.py +vllm/request_generator/vllm_request_generator.py +vllm/v1/attention/backends/flash_attn.py +vllm/v1/attention/backends/flashinfer.py +vllm/v1/attention/backends/mla/common.py +vllm/v1/attention/backends/mla/flashinfer_mla.py +vllm/v1/attention/backends/utils.py +vllm/v1/core/sched/scheduler.py +vllm/v1/engine/coordinator.py +vllm/v1/engine/core.py +vllm/v1/engine/core_client.py +vllm/v1/engine/output_processor.py +vllm/v1/engine/processor.py +vllm/v1/frontier_trace.py +vllm/v1/metrics/stats.py +vllm/v1/spec_decode/eagle.py +vllm/v1/utils.py +vllm/v1/worker/gpu_model_runner.py +vllm/v1/worker/gpu_worker.py +vllm/worker/model_runner.py +vllm/worker/worker.py diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_local_commit.diff b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_local_commit.diff new file mode 100644 index 00000000..5d788f9a --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_local_commit.diff @@ -0,0 +1,412 @@ +diff --git a/vllm/v1/engine/coordinator.py b/vllm/v1/engine/coordinator.py +index 596edfdbe..1fd9ad16f 100644 +--- a/vllm/v1/engine/coordinator.py ++++ b/vllm/v1/engine/coordinator.py +@@ -12,6 +12,7 @@ import zmq + from vllm.config import ParallelConfig + from vllm.logger import init_logger + from vllm.utils import get_mp_context, make_zmq_socket, set_process_title ++from vllm.v1 import frontier_trace + from vllm.v1.engine import EngineCoreOutputs, EngineCoreRequestType + from vllm.v1.serial_utils import MsgpackDecoder + from vllm.v1.utils import get_engine_client_zmq_addr, shutdown +@@ -157,6 +158,10 @@ class DPCoordinatorProc: + last_stats_wave = -1 + last_step_counts: Optional[list[list[int]]] = None + ++ # Identifies each set of counts sent to the front ends, so a placement ++ # can be traced back to the engine reports it was computed from. ++ snapshot_id = 0 ++ + with make_zmq_socket( + path=front_publish_address, # IPC + ctx=self.ctx, +@@ -208,12 +213,23 @@ class DPCoordinatorProc: + if last_step_counts is not None: + engine_req_counts_list = last_step_counts + last_step_counts = None ++ counts_source = "latched_previous_step" + else: + engine_req_counts_list = self._get_engine_counts() + stats_changed = False ++ counts_source = "current" ++ ++ snapshot_id += 1 ++ frontier_trace.log_dp_placement_record( ++ "coordinator_publish", ++ snapshot=snapshot_id, ++ counts=engine_req_counts_list, ++ counts_source=counts_source, ++ wave=current_wave, ++ engines_running=engines_running) + + to_publish = (engine_req_counts_list, current_wave, +- engines_running) ++ engines_running, snapshot_id) + publish_front.send(msgspec.msgpack.encode(to_publish)) + last_publish_time = int(time.time() * 1000) + continue +@@ -290,21 +306,35 @@ class DPCoordinatorProc: + stats = self.engines[eng_index].request_counts + stats_step = scheduler_stats.step_counter + stats_wave = scheduler_stats.current_wave ++ disposition = "applied_without_latch" + if (stats_wave > last_stats_wave + or stats_wave == last_stats_wave + and stats_step > last_stats_step): + if stats_changed: + last_step_counts = self._get_engine_counts( + do_copy=True) ++ disposition = "latched_previous_step" ++ else: ++ disposition = "advanced_without_latch" + last_stats_step = stats_step + last_stats_wave = stats_wave + elif stats_wave != last_stats_wave or ( + stats_step != last_stats_step): ++ disposition = "out_of_order" + logger.warning( + "Received stats for out-of-order " + "step (%d, %d) from engine %d (expected " + "> (%d, %d))", stats_wave, stats_step, + eng_index, last_stats_wave, last_stats_step) ++ frontier_trace.log_dp_placement_record( ++ "coordinator_receive", ++ engine=eng_index, ++ wave=stats_wave, ++ step=stats_step, ++ waiting=scheduler_stats.num_waiting_reqs, ++ running=scheduler_stats.num_running_reqs, ++ disposition=disposition, ++ latched_counts=last_step_counts) + stats[0] = scheduler_stats.num_waiting_reqs + stats[1] = scheduler_stats.num_running_reqs + stats_changed = True +@@ -335,7 +365,8 @@ class DPCoordinatorProc: + self._send_start_wave(publish_back, wave, eng_index) + + if wave_state_changed: +- message = (None, current_wave, engines_running) ++ message = (None, current_wave, engines_running, ++ snapshot_id) + publish_front.send(msgspec.msgpack.encode(message)) + + @staticmethod +diff --git a/vllm/v1/engine/core.py b/vllm/v1/engine/core.py +index bfc29fc78..c61236776 100644 +--- a/vllm/v1/engine/core.py ++++ b/vllm/v1/engine/core.py +@@ -45,6 +45,7 @@ from vllm.v1.engine.utils import (EngineHandshakeMetadata, EngineZmqAddresses, + from vllm.v1.executor.abstract import Executor + from vllm.v1.kv_cache_interface import KVCacheConfig + from vllm.v1.metrics.stats import SchedulerStats ++from vllm.v1 import frontier_trace + from vllm.v1.outputs import ModelRunnerOutput + from vllm.v1.request import Request, RequestStatus + from vllm.v1.serial_utils import MsgpackDecoder, MsgpackEncoder +@@ -156,6 +157,12 @@ class EngineCore: + self.batch_queue_size) + self.batch_queue = deque(maxlen=self.batch_queue_size) + ++ # Classification of the most recent iteration, written by the step ++ # methods and consumed by the data-parallel busy loop, which is where ++ # the wave, step counter and published counts are known. Stays None ++ # while Frontier placement logging is off. ++ self.frontier_iteration: Optional[dict[str, Any]] = None ++ + self.request_block_hasher: Optional[Callable[[Request], + list[BlockHash]]] = None + if (self.vllm_config.cache_config.enable_prefix_caching +@@ -305,9 +312,48 @@ class EngineCore: + engine_core_outputs = self.scheduler.update_from_output( + scheduler_output, model_output) # type: ignore + ++ self._record_frontier_iteration(scheduler_output, ++ applied_output=True, ++ queue_occupancy=0) ++ + return (engine_core_outputs, + scheduler_output.total_num_scheduled_tokens > 0) + ++ def _record_frontier_iteration(self, ++ scheduled_output: Optional[SchedulerOutput], ++ *, applied_output: bool, ++ queue_occupancy: int) -> None: ++ """Classify one engine iteration for Frontier placement analysis. ++ ++ Which branch an iteration takes is what decides whether it publishes ++ request counts on its own, so the branch is recorded rather than ++ inferred later from the counts. ++ """ ++ if not frontier_trace.is_dp_placement_logging_enabled(): ++ return ++ ++ if scheduled_output is None: ++ branch = "applied_without_scheduling" ++ elif applied_output: ++ branch = "applied_after_scheduling" ++ else: ++ branch = "scheduled_without_applying" ++ ++ self.frontier_iteration = { ++ "branch": ++ branch, ++ "applied_output": ++ applied_output, ++ "queue_occupancy": ++ queue_occupancy, ++ "scheduled_new_req_ids": ++ [data.req_id for data in scheduled_output.scheduled_new_reqs] ++ if scheduled_output is not None else [], ++ "num_scheduled_tokens": ++ scheduled_output.total_num_scheduled_tokens ++ if scheduled_output is not None else 0, ++ } ++ + def post_step(self, model_executed: bool) -> None: + if self.use_spec_decode and model_executed: + # Take the draft token ids. +@@ -339,17 +385,22 @@ class EngineCore: + assert len(batch_queue) < self.batch_queue_size + + model_executed = False ++ scheduled_output: Optional[SchedulerOutput] = None + if self.scheduler.has_requests(): +- scheduler_output = self.scheduler.schedule() +- future = self.model_executor.execute_model(scheduler_output) ++ scheduled_output = self.scheduler.schedule() ++ future = self.model_executor.execute_model(scheduled_output) + batch_queue.appendleft( +- (future, scheduler_output)) # type: ignore[arg-type] ++ (future, scheduled_output)) # type: ignore[arg-type] + +- model_executed = scheduler_output.total_num_scheduled_tokens > 0 ++ model_executed = scheduled_output.total_num_scheduled_tokens > 0 + if model_executed and len(batch_queue) < self.batch_queue_size \ + and not batch_queue[-1][0].done(): + # Don't block on next worker response unless the queue is full + # or there are no more requests to schedule. ++ self._record_frontier_iteration( ++ scheduled_output, ++ applied_output=False, ++ queue_occupancy=len(batch_queue)) + return None, True + + elif not batch_queue: +@@ -366,6 +417,10 @@ class EngineCore: + engine_core_outputs = self.scheduler.update_from_output( + scheduler_output, model_output) + ++ self._record_frontier_iteration(scheduled_output, ++ applied_output=True, ++ queue_occupancy=len(batch_queue)) ++ + return engine_core_outputs, model_executed + + def shutdown(self): +@@ -1072,9 +1127,10 @@ class DPEngineCoreProc(EngineCoreProc): + else: + super()._handle_client_request(request_type, request) + +- def _maybe_publish_request_counts(self): ++ def _maybe_publish_request_counts(self) -> bool: ++ """Returns whether this iteration published its request counts.""" + if not self.publish_dp_lb_stats: +- return ++ return False + + # Publish our request counts (if they've changed). + counts = self.scheduler.get_request_counts() +@@ -1085,6 +1141,31 @@ class DPEngineCoreProc(EngineCoreProc): + current_wave=self.current_wave) + self.output_queue.put_nowait( + (-1, EngineCoreOutputs(scheduler_stats=stats))) ++ return True ++ return False ++ ++ def _log_frontier_iteration(self, published: bool) -> None: ++ """Emit the iteration the step methods classified. ++ ++ `step_counter` is read before `_has_global_unfinished_reqs` advances ++ it, so it is the same value a published report carried, which makes ++ `(engine, wave, step)` the join for the whole placement chain. ++ """ ++ iteration = self.frontier_iteration ++ if iteration is None: ++ return ++ self.frontier_iteration = None ++ ++ num_running_reqs, num_waiting_reqs = self.scheduler.get_request_counts() ++ frontier_trace.log_dp_placement_record( ++ "engine_iteration", ++ engine=self.dp_rank, ++ wave=self.current_wave, ++ step=self.step_counter, ++ waiting=num_waiting_reqs, ++ running=num_running_reqs, ++ published=published, ++ **iteration) + + def run_busy_loop(self): + """Core busy loop of the EngineCore for data parallel case.""" +@@ -1096,7 +1177,7 @@ class DPEngineCoreProc(EngineCoreProc): + + # 2) Step the engine core. + executed = self._process_engine_step() +- self._maybe_publish_request_counts() ++ self._log_frontier_iteration(self._maybe_publish_request_counts()) + + local_unfinished_reqs = self.scheduler.has_unfinished_requests() + if not executed: +diff --git a/vllm/v1/engine/core_client.py b/vllm/v1/engine/core_client.py +index 605bedaf1..daf8a67ef 100644 +--- a/vllm/v1/engine/core_client.py ++++ b/vllm/v1/engine/core_client.py +@@ -29,6 +29,7 @@ from vllm.v1.engine import (EngineCoreOutputs, EngineCoreRequest, + EngineCoreRequestType, + ReconfigureDistributedRequest, ReconfigureRankType, + UtilityOutput) ++from vllm.v1 import frontier_trace + from vllm.v1.engine.coordinator import DPCoordinator + from vllm.v1.engine.core import EngineCore, EngineCoreProc + from vllm.v1.engine.exceptions import EngineDeadError +@@ -977,6 +978,10 @@ class DPAsyncMPClient(AsyncMPClient): + # List of [waiting, running] pair per engine. + # Used only by DPLBAsyncMPClient subclass. + self.lb_engines: list[list[int]] = [[0, 0] for _ in self.core_engines] ++ # Coordinator snapshot the counts above came from; 0 until the first ++ # one arrives, so a placement made from the initial zeros is visible ++ # as such. ++ self.lb_snapshot: int = 0 + + self.first_req_sock_addr = get_open_zmq_inproc_path() + self.first_req_send_socket = self.resources.first_req_send_socket = ( +@@ -1070,12 +1075,20 @@ class DPAsyncMPClient(AsyncMPClient): + continue + + # Update local load-balancing state. +- counts, wave, running = msgspec.msgpack.decode(buf) ++ counts, wave, running, snapshot = msgspec.msgpack.decode( ++ buf) + self.current_wave = wave + self.engines_running = running + if counts is not None: + sliced_counts = counts[count_slice] + self.lb_engines = sliced_counts ++ self.lb_snapshot = snapshot ++ frontier_trace.log_dp_placement_record( ++ "frontend_snapshot", ++ snapshot=snapshot, ++ counts=sliced_counts, ++ wave=wave, ++ engines_running=running) + logger.debug("Received counts: %s (%s)", sliced_counts, + count_slice) + +@@ -1147,6 +1160,15 @@ class DPLBAsyncMPClient(DPAsyncMPClient): + if score < min_score: + min_score = score + eng_index = idx ++ frontier_trace.log_dp_placement_record( ++ "frontend_route", ++ request_id=request.request_id, ++ engine=eng_index, ++ snapshot=self.lb_snapshot, ++ counts=[list(counts) for counts in current_counts], ++ score=min_score, ++ start_index=self.eng_start_index, ++ reservation=self.client_count) + # Increment local waiting count for better balancing between stats + # updates from the coordinator (which happen every 100ms). + current_counts[eng_index][0] += self.client_count +diff --git a/vllm/v1/frontier_trace.py b/vllm/v1/frontier_trace.py +index d9bddeea5..a727a86b9 100644 +--- a/vllm/v1/frontier_trace.py ++++ b/vllm/v1/frontier_trace.py +@@ -2,14 +2,18 @@ + # SPDX-FileCopyrightText: Copyright contributors to the vLLM project + """Runtime gate for Frontier trace logging.""" + ++import atexit + from contextlib import contextmanager + import json + import os ++import time + from typing import Any, Mapping + + _SKIP_WARMUP = os.environ.get("VLLM_FRONTIER_TRACE_SKIP_WARMUP", "0") == "1" + _TRACE_ACTIVE = not _SKIP_WARMUP + _PP_BOUNDARY_LOG_ENV_VAR = "VLLM_FRONTIER_PP_BOUNDARY_LOG_PATH" ++_DP_PLACEMENT_LOG_DIR_ENV_VAR = "VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR" ++_DP_PLACEMENT_FLUSH_EVERY = 1024 + _PP_BOUNDARY_REQUIRED_FIELDS = ( + "model_name", + "timestamp", +@@ -99,3 +103,67 @@ def disable_for_warmup(): + yield + finally: + _TRACE_ACTIVE = True ++ ++ ++# Data-parallel placement records. Every process that takes part in a placement ++# decision -- each engine core, the coordinator, each API server -- writes its ++# own file, so records never interleave and the reader can tell the roles ++# apart. They are buffered because an engine core writes one per iteration of ++# its busy loop; they reach disk every _DP_PLACEMENT_FLUSH_EVERY records and at ++# normal process exit. ++_dp_placement_records: list[dict[str, Any]] = [] ++_dp_placement_seq = 0 ++ ++ ++def get_dp_placement_log_dir() -> str: ++ return os.environ.get(_DP_PLACEMENT_LOG_DIR_ENV_VAR, "") ++ ++ ++def is_dp_placement_logging_enabled() -> bool: ++ return is_active() and bool(get_dp_placement_log_dir()) ++ ++ ++def log_dp_placement_record(kind: str, **fields: Any) -> None: ++ """Buffer one placement record of the given kind. ++ ++ `seq` orders the records one process wrote and is the tie-break when two ++ of them carry the same correlation id. Timestamps are for reading; the ++ join between processes is always a correlation id. ++ """ ++ if not is_dp_placement_logging_enabled(): ++ return ++ ++ global _dp_placement_seq ++ record: dict[str, Any] = { ++ "kind": kind, ++ "pid": os.getpid(), ++ "seq": _dp_placement_seq, ++ "monotonic": time.monotonic(), ++ } ++ record.update(fields) ++ _dp_placement_records.append(record) ++ _dp_placement_seq += 1 ++ ++ if len(_dp_placement_records) >= _DP_PLACEMENT_FLUSH_EVERY: ++ flush_dp_placement_records() ++ ++ ++def flush_dp_placement_records() -> None: ++ if not _dp_placement_records: ++ return ++ ++ log_dir = get_dp_placement_log_dir() ++ os.makedirs(log_dir, exist_ok=True) ++ log_path = os.path.join(log_dir, f"dp_placement_{os.getpid()}.jsonl") ++ try: ++ with open(log_path, "a", encoding="utf-8") as file: ++ for record in _dp_placement_records: ++ file.write(json.dumps(record) + "\n") ++ except OSError as exc: ++ raise RuntimeError( ++ f"Failed to write Frontier DP placement log file: {log_path}" ++ ) from exc ++ _dp_placement_records.clear() ++ ++ ++atexit.register(flush_dp_placement_records) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch new file mode 100644 index 00000000..ec4805ba --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch @@ -0,0 +1,24 @@ +--- a/vllm/_custom_ops.py ++++ b/vllm/_custom_ops.py +@@ -1505,9 +1505,9 @@ + + def topk_softmax(topk_weights: torch.Tensor, topk_ids: torch.Tensor, + token_expert_indices: torch.Tensor, +- gating_output: torch.Tensor, renormalize: bool) -> None: ++ gating_output: torch.Tensor) -> None: + torch.ops._moe_C.topk_softmax(topk_weights, topk_ids, token_expert_indices, +- gating_output, renormalize) ++ gating_output) + + + def grouped_topk(scores: torch.Tensor, scores_with_bias: torch.Tensor, +--- a/vllm/model_executor/layers/fused_moe/fused_moe.py ++++ b/vllm/model_executor/layers/fused_moe/fused_moe.py +@@ -890,7 +890,6 @@ + topk_indices, + token_expert_indices, + gating_output, +- renormalize, + ) + if renormalize: + topk_weights = topk_weights / topk_weights.sum(dim=-1, keepdim=True) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml new file mode 100644 index 00000000..1b634e51 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml @@ -0,0 +1,148 @@ +# Frontier calibration case: stage admission of attention-DP lanes under PP. +# Structural comparison (plan §4.7, D-8). No E2E latency gate: the Frontier +# side runs the dummy predictor, and the fix changes admission, not durations. + +case_id: stage_admission_case_001 +run_generation: 1 +created_at_utc: "2026-09-23" +requesting_user: i-fengyicheng +reviewer_identity: i-fengyicheng +auto_recycle: false + +purpose: >- + Check that the stage-admission rule of plan P1 makes Frontier's attention-DP + lanes behave like vLLM V1 data-parallel ranks at pipeline_parallel_size 2: + a rank with runnable work is not refused at its stage by another rank's + queued work, both ranks start in the same forward slot, and stage-0 forwards + pair one to one. Criterion C7 of plan.md. + +selected_checkout: + frontier_worktree: /data/ycfeng/Frontier/.worktrees/stage-admission-ordering + frontier_branch: fix/stage-admission-ordering + frontier_before_source: 1f694f7 # P0 set "base"; source identical to origin/main + frontier_after_commit: dac4e69 # P1 rule commit; P3 set "after" + frontier_harness: tests/e2e/stage_admission_matrix.py (group G7) + +groundtruth_checkout_path: /data/ycfeng/Frontier/.real-engine/vLLM-BS +groundtruth_branch: feature/frontier-comparison-instrumentation +groundtruth_ref: refs/heads/feature/frontier-comparison-instrumentation +groundtruth_commit: 494b9f327036d4493034a9b37ebb343354884e01 +groundtruth_tree_dirty: false +groundtruth_remote_url: https://github.com/fwyc0573/vLLM-BS.git +groundtruth_remote_tip: ea95f571e20937c7c908c6d59ddd1cd6bf9268f1 +groundtruth_remote_tip_note: >- + The local commit is one commit ahead of the remote tip (494b9f327, "Trace the + data-parallel placement chain for Frontier calibration"), unpushed; it adds + only the DP placement records. Its diff is inputs/groundtruth_local_commit.diff, + identical to the parent task's dp_pp_case_001/g1_instrumentation.diff. +groundtruth_diff_artifact: inputs/groundtruth_local_commit.diff +groundtruth_diff_sha256: 84fc24db0e2411268a93f8be7ca5f8e4e5072ea09d86063cac2cfb98381feb2c +groundtruth_upstream_base: 01efc7ef7 (tag upstream-v0.10.2) +groundtruth_fork_changes: inputs/fork_changed_files.txt # 61 files under vllm/, 59 .py +groundtruth_dirty_patch_sha256: null +groundtruth_overlay_patch_applied: true # run sa-pp-20260923b; run sa-pp-20260923a had none +groundtruth_overlay_patch: inputs/groundtruth_overlay.patch +groundtruth_overlay_patch_sha256: 8d47678911b7b689bc644ea81bd9d2200ea296d773ee9c07a1c5a9bc9b3a9c81 +groundtruth_overlay_patch_reason: >- + Fork commit 1109c4f16 passes a fifth renormalize argument to + _moe_C::topk_softmax; the fork's own csrc (unchanged from upstream-v0.10.2) + and the v0.10.2 image declare four, so run a failed in the MoE profile_run. + The patch restores the upstream four-argument wrapper (vllm/_custom_ops.py) + and call (fused_moe.py vllm_topk_softmax). Numerics are unchanged: Python + renormalizes after the call. The checkout is not modified. +groundtruth_overlay: >- + The worker copies the image's installed vllm package and copies every + vllm/**/*.py of the checkout over it. It runs only when the files where image + and checkout differ are exactly the .py files of fork_changed_files.txt + (overlay_report.json in the run directory). +groundtruth_weight_mode: dummy +real_weight_download: false + +topology: + vllm: "data_parallel_size=2, pipeline_parallel_size=2, tensor_parallel_size=1; MoE adds enable_expert_parallel" + frontier: "attn_dp=2, num_pipeline_stages=2, attn_tp=1, one Replica; MoE adds moe_tp=1, moe_ep=2" + devices: 4x H800 (vLLM); h800 / h800_dgx (Frontier) + cluster_scheduler: round_robin (request i to lane i mod 2) + replica_scheduler: vllm_v1 + +models: + moe: data/config/models/Qwen3-30B-A3B-tiny.json + dense: data/config/models/Llama-3.2-1B-Instruct.json + +settings: + dtype: bfloat16 + prompt_tokens: 256 + output_tokens: 1 + max_num_batched_tokens: 256 + max_num_seqs: 4 + block_size: 16 + chunked_prefill: true + prefix_caching: false + graph_mode: eager + frontier_num_blocks: 1024 + vllm_gpu_memory_utilization: 0.5 + vllm_num_gpu_blocks: recorded in runs/vllm-instrumented//runs//summary.json + +workload: + request_id_namespace: "-q; burst = warmup | b-r" + request_id_encoding: "AsyncLLM.add_request request_id, unchanged into the engine (n=1)" + frontier_request_ids: "0..n-1, mapped to vLLM b-r-q by index" + lane_assignment: "index mod 2 (vLLM data_parallel_rank; Frontier round robin)" + warmup_request_ids: [warmup-q0, warmup-q1, warmup-q2, warmup-q3] + formal_request_ids: "b8-r{0,1,2}-q{0..7}, b16-r{0,1,2}-q{0..15}" + rounds_per_burst: 3 + idle_between_rounds: "engines paused (dp_engines_running false), then 1 s" + +modes: + groundtruth_instrumented: + env: "VLLM_FRONTIER_INSTRUMENTATION=1, VLLM_FRONTIER_PP_BOUNDARY_LOG_PATH, VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR, VLLM_WORKER_MULTIPROC_METHOD=spawn" + producer: tests/comparison/stage_admission_pp/run_vllm_worker.sh + artifact_path: runs/vllm-instrumented// (archive /mnt/codesign-exp/ycfeng/frontier/stage_admission_pp//) + status: COMPLETE + runs: + - run_tag: sa-pp-20260923a + rjob: exp-0923-022226-151935 + overlay_patch: none + result: "dense complete; MoE failed (_moe_C::topk_softmax 5 vs 4 arguments); job Failed" + - run_tag: sa-pp-20260923b + rjob: exp-0923-024146-345158 + overlay_patch: inputs/groundtruth_overlay.patch + result: "MoE and dense complete; job Succeeded; used by analysis/" + groundtruth_clean: + status: NOT_APPLICABLE # no E2E latency gate (D-8) + simulator_before: + source: /data/ycfeng/tmp/stage_admission_ordering/base/G7-* + status: COMPLETE # MoE n8/n16 admission_deadlock; dense n8/n16 success + simulator_after: + source: /data/ycfeng/tmp/stage_admission_ordering/after/G7-* + status: COMPLETE # all four G7 cases success at dac4e69 + +analysis: + producer: tests/comparison/stage_admission_pp/compare_lanes.py + outputs: [analysis/workflow_gap_table.csv, analysis/lane_metrics.json, analysis/workflow_gap_status.json, analysis/workflow_gap_summary.md] + gates_not_applicable: + e2e_gate: "structural comparison; Frontier uses the dummy predictor (D-8)" + moe_routing_distortion_gate: "no request-level parity claim; admission order does not depend on routing" + +decisions: + - decision_id: R-6 + question: Validate the admission fix against vLLM on a GPU worker? + answer: "按照已有plan执行上述修复（该修复需要和在gpu worker上运行的vllm进行合理的对比验证，确保修改的有效性）" + decided_at_utc: "2026-09-23" + - decision_id: D-8 + question: Comparison scope and mode. + answer: Structural M1-M5 in instrumented mode, prefill-only, existing model configs, no E2E gate (plan.md D-8). + decided_at_utc: "2026-09-23" + - decision_id: gpu-charged-group + question: GPU cluster. + answer: "codesign only (后续的gpu worker集群只允许使用 codesign（暂停对steptron_ci的使用，直至得到我允许）)" + - decision_id: topk-softmax-abi + question: How to run the MoE ground truth after the topk_softmax ABI failure of run a? + answer: "topk_softmax 统一修复为4 个参数的版本" + decided_at_utc: "2026-09-23" + +analysis_result: >- + 50 of 52 rows MATCH. V5 (stage-0 co-execution) MISMATCH on dense n8 and n16; + cause and proposal in analysis/workflow_gap_summary.md. Plan §3 P5 stop: + nothing adjusted, awaiting the user's decision. +status: STOPPED_FOR_DECISION diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/COMPLETE b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/COMPLETE new file mode 100644 index 00000000..f47e4716 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/COMPLETE @@ -0,0 +1 @@ +status=1 diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/overlay_report.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/overlay_report.json new file mode 100644 index 00000000..a41bc82e --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/overlay_report.json @@ -0,0 +1,130 @@ +{ + "site_vllm": "/usr/local/lib/python3.12/dist-packages/vllm", + "checkout": "/data/ycfeng/Frontier/.real-engine/vLLM-BS", + "overlay": "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm", + "differing_py_files": [ + "vllm/_C.py", + "vllm/_custom_ops.py", + "vllm/_moe_C.py", + "vllm/attention/layer.py", + "vllm/benchmarks/throughput.py", + "vllm/compilation/compiler_interface.py", + "vllm/config/__init__.py", + "vllm/distributed/communication_op.py", + "vllm/distributed/kv_transfer/kv_connector/v1/base.py", + "vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/frontier_kv_transfer_logger.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", + "vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py", + "vllm/distributed/parallel_state.py", + "vllm/engine/arg_utils.py", + "vllm/engine/llm_engine.py", + "vllm/entrypoints/openai/frontier_request_metrics.py", + "vllm/entrypoints/openai/serving_chat.py", + "vllm/entrypoints/openai/serving_completion.py", + "vllm/entrypoints/openai/serving_engine.py", + "vllm/envs.py", + "vllm/model_executor/custom_op.py", + "vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", + "vllm/model_executor/layers/fused_moe/deepep_ll_prepare_finalize.py", + "vllm/model_executor/layers/fused_moe/fused_moe.py", + "vllm/model_executor/layers/fused_moe/layer.py", + "vllm/model_executor/layers/fused_moe/pplx_prepare_finalize.py", + "vllm/model_executor/layers/linear.py", + "vllm/model_executor/layers/vocab_parallel_embedding.py", + "vllm/model_executor/models/llama.py", + "vllm/model_executor/models/phimoe.py", + "vllm/model_executor/models/qwen3_moe.py", + "vllm/model_executor/models/qwen3_moe_mtp.py", + "vllm/model_executor/models/registry.py", + "vllm/request_generator/__init__.py", + "vllm/request_generator/config.py", + "vllm/request_generator/kv_sync.py", + "vllm/request_generator/prompt_generator.py", + "vllm/request_generator/vllm_request_generator.py", + "vllm/v1/attention/backends/flash_attn.py", + "vllm/v1/attention/backends/flashinfer.py", + "vllm/v1/attention/backends/mla/common.py", + "vllm/v1/attention/backends/mla/flashinfer_mla.py", + "vllm/v1/attention/backends/utils.py", + "vllm/v1/core/sched/scheduler.py", + "vllm/v1/engine/coordinator.py", + "vllm/v1/engine/core.py", + "vllm/v1/engine/core_client.py", + "vllm/v1/engine/output_processor.py", + "vllm/v1/engine/processor.py", + "vllm/v1/frontier_trace.py", + "vllm/v1/metrics/stats.py", + "vllm/v1/spec_decode/eagle.py", + "vllm/v1/utils.py", + "vllm/v1/worker/gpu_model_runner.py", + "vllm/v1/worker/gpu_worker.py", + "vllm/worker/model_runner.py", + "vllm/worker/worker.py" + ], + "expected_py_changes": [ + "vllm/_C.py", + "vllm/_custom_ops.py", + "vllm/_moe_C.py", + "vllm/attention/layer.py", + "vllm/benchmarks/throughput.py", + "vllm/compilation/compiler_interface.py", + "vllm/config/__init__.py", + "vllm/distributed/communication_op.py", + "vllm/distributed/kv_transfer/kv_connector/v1/base.py", + "vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/frontier_kv_transfer_logger.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", + "vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py", + "vllm/distributed/parallel_state.py", + "vllm/engine/arg_utils.py", + "vllm/engine/llm_engine.py", + "vllm/entrypoints/openai/frontier_request_metrics.py", + "vllm/entrypoints/openai/serving_chat.py", + "vllm/entrypoints/openai/serving_completion.py", + "vllm/entrypoints/openai/serving_engine.py", + "vllm/envs.py", + "vllm/model_executor/custom_op.py", + "vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", + "vllm/model_executor/layers/fused_moe/deepep_ll_prepare_finalize.py", + "vllm/model_executor/layers/fused_moe/fused_moe.py", + "vllm/model_executor/layers/fused_moe/layer.py", + "vllm/model_executor/layers/fused_moe/pplx_prepare_finalize.py", + "vllm/model_executor/layers/linear.py", + "vllm/model_executor/layers/vocab_parallel_embedding.py", + "vllm/model_executor/models/llama.py", + "vllm/model_executor/models/phimoe.py", + "vllm/model_executor/models/qwen3_moe.py", + "vllm/model_executor/models/qwen3_moe_mtp.py", + "vllm/model_executor/models/registry.py", + "vllm/request_generator/__init__.py", + "vllm/request_generator/config.py", + "vllm/request_generator/kv_sync.py", + "vllm/request_generator/prompt_generator.py", + "vllm/request_generator/vllm_request_generator.py", + "vllm/v1/attention/backends/flash_attn.py", + "vllm/v1/attention/backends/flashinfer.py", + "vllm/v1/attention/backends/mla/common.py", + "vllm/v1/attention/backends/mla/flashinfer_mla.py", + "vllm/v1/attention/backends/utils.py", + "vllm/v1/core/sched/scheduler.py", + "vllm/v1/engine/coordinator.py", + "vllm/v1/engine/core.py", + "vllm/v1/engine/core_client.py", + "vllm/v1/engine/output_processor.py", + "vllm/v1/engine/processor.py", + "vllm/v1/frontier_trace.py", + "vllm/v1/metrics/stats.py", + "vllm/v1/spec_decode/eagle.py", + "vllm/v1/utils.py", + "vllm/v1/worker/gpu_model_runner.py", + "vllm/v1/worker/gpu_worker.py", + "vllm/worker/model_runner.py", + "vllm/worker/worker.py" + ], + "unexpected": [], + "missing": [], + "accepted": true +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/replica_log.txt b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/replica_log.txt new file mode 100644 index 00000000..834e9102 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/replica_log.txt @@ -0,0 +1,73 @@ +# Replica log tail of RJob exp-0923-022226-151935 (codesign, H800 x4, creator i-fengyicheng). +# Platform init lines (node addresses, NCCL interface settings) are removed; workload output is verbatim. +{"python": "3.12.11", "torch": "2.8.0+cu128", "cuda_available": true, "device_count": 4, "devices": ["NVIDIA H800", "NVIDIA H800", "NVIDIA H800", "NVIDIA H800"]} +{"accepted": true, "unexpected": [], "missing": []} +/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. + import pynvml # type: ignore[import] +VLLM_IMPORT 0.10.2 /tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/__init__.py /tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/frontier_trace.py +SCENARIO_FAIL moe exit=1 +(EngineCore_DP0 pid=232) return self.collective_rpc("determine_available_memory") +(EngineCore_DP0 pid=232) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore_DP0 pid=232) File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/executor/multiproc_executor.py", line 257, in collective_rpc +(EngineCore_DP0 pid=232) result = result.result() +(EngineCore_DP0 pid=232) ^^^^^^^^^^^^^^^ +(EngineCore_DP0 pid=232) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result +(EngineCore_DP0 pid=232) return self.__get_result() +(EngineCore_DP0 pid=232) ^^^^^^^^^^^^^^^^^^^ +(EngineCore_DP0 pid=232) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result +(EngineCore_DP0 pid=232) raise self._exception +(EngineCore_DP0 pid=232) File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run +(EngineCore_DP0 pid=232) result = self.fn(*self.args, **self.kwargs) +(EngineCore_DP0 pid=232) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore_DP0 pid=232) File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/executor/multiproc_executor.py", line 243, in get_response +(EngineCore_DP0 pid=232) raise RuntimeError( +(EngineCore_DP0 pid=232) RuntimeError: Worker failed with error '_moe_C::topk_softmax() expected at most 4 argument(s) but received 5 argument(s). Declaration: _moe_C::topk_softmax(Tensor($0! -> ) topk_weights, Tensor($1! -> ) topk_indices, Tensor($2! -> ) token_expert_indices, Tensor gating_output) -> ()', please check the stack trace above for the root cause +Traceback (most recent call last): + File "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/tests/comparison/stage_admission_pp/vllm_burst_driver.py", line 221, in + sys.exit(main()) + ^^^^^^ + File "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/tests/comparison/stage_admission_pp/vllm_burst_driver.py", line 213, in main + summary = asyncio.run(run_bursts(args)) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run + return runner.run(main) + ^^^^^^^^^^^^^^^^ + File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run + return self._loop.run_until_complete(task) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete + return future.result() + ^^^^^^^^^^^^^^^ + File "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/tests/comparison/stage_admission_pp/vllm_burst_driver.py", line 113, in run_bursts + engine = AsyncLLM.from_engine_args(engine_args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/async_llm.py", line 240, in from_engine_args + return cls( + ^^^^ + File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/async_llm.py", line 136, in __init__ + self.engine_core = EngineCoreClient.make_async_mp_client( + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client + return DPLBAsyncMPClient(*client_args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/core_client.py", line 1137, in __init__ + super().__init__(vllm_config, executor_class, log_stats, + File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/core_client.py", line 975, in __init__ + super().__init__(vllm_config, executor_class, log_stats, + File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/core_client.py", line 770, in __init__ + super().__init__( + File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/core_client.py", line 449, in __init__ + with launch_core_engines(vllm_config, executor_class, + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ + next(self.gen) + File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/utils.py", line 729, in launch_core_engines + wait_for_engine_startup( + File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/utils.py", line 782, in wait_for_engine_startup + raise RuntimeError("Engine core initialization failed. " +RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} +SCENARIO_PASS dense +dense DRIVER_DONE {"rounds": 7, "num_gpu_blocks": 304854} +152 /tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/pp_boundary.jsonl +WORKER_STATUS=1 RUN_TAG=sa-pp-20260923a +ExitCode= 1 diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_588.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_588.jsonl new file mode 100644 index 00000000..7ff91c38 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_588.jsonl @@ -0,0 +1,22 @@ +{"kind": "frontend_snapshot", "pid": 588, "seq": 0, "monotonic": 9119575.220499707, "snapshot": 2, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 1, "monotonic": 9119576.500151489, "snapshot": 3, "counts": [[1, 1], [1, 1]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 2, "monotonic": 9119576.66924366, "snapshot": 4, "counts": [[0, 1], [0, 1]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 3, "monotonic": 9119576.769698702, "snapshot": 5, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 4, "monotonic": 9119577.8353699, "snapshot": 6, "counts": [[0, 1], [0, 1]], "wave": 1, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 5, "monotonic": 9119577.935569072, "snapshot": 7, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 588, "seq": 6, "monotonic": 9119578.035744542, "snapshot": 8, "counts": [[0, 0], [0, 0]], "wave": 2, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 588, "seq": 7, "monotonic": 9119578.971015744, "snapshot": 9, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 8, "monotonic": 9119579.070546191, "snapshot": 10, "counts": [[0, 1], [0, 1]], "wave": 3, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 588, "seq": 9, "monotonic": 9119579.17074387, "snapshot": 11, "counts": [[0, 0], [0, 0]], "wave": 3, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 588, "seq": 10, "monotonic": 9119580.105585678, "snapshot": 12, "counts": [[0, 1], [0, 1]], "wave": 3, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 11, "monotonic": 9119580.205964977, "snapshot": 13, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 588, "seq": 12, "monotonic": 9119580.306053106, "snapshot": 14, "counts": [[0, 0], [0, 0]], "wave": 4, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 588, "seq": 13, "monotonic": 9119581.244764512, "snapshot": 15, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 14, "monotonic": 9119581.344573118, "snapshot": 16, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 15, "monotonic": 9119581.444366362, "snapshot": 17, "counts": [[0, 0], [0, 0]], "wave": 5, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 588, "seq": 16, "monotonic": 9119582.409289015, "snapshot": 18, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 17, "monotonic": 9119582.509353423, "snapshot": 19, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 18, "monotonic": 9119582.610055115, "snapshot": 20, "counts": [[0, 0], [0, 0]], "wave": 6, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 588, "seq": 19, "monotonic": 9119583.568792408, "snapshot": 21, "counts": [[0, 1], [0, 1]], "wave": 6, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 20, "monotonic": 9119583.669192385, "snapshot": 22, "counts": [[0, 1], [0, 1]], "wave": 6, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 588, "seq": 21, "monotonic": 9119583.768215796, "snapshot": 23, "counts": [[0, 0], [0, 0]], "wave": 7, "engines_running": false} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_663.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_663.jsonl new file mode 100644 index 00000000..6dbe9039 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_663.jsonl @@ -0,0 +1,65 @@ +{"kind": "engine_iteration", "pid": 663, "seq": 0, "monotonic": 9119576.449129984, "engine": 0, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 1, "monotonic": 9119576.663859723, "engine": 0, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 2, "monotonic": 9119576.66974478, "engine": 0, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 3, "monotonic": 9119576.673491668, "engine": 0, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 4, "monotonic": 9119576.675901318, "engine": 0, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 5, "monotonic": 9119577.823941316, "engine": 0, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 6, "monotonic": 9119577.834619695, "engine": 0, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 7, "monotonic": 9119577.838431131, "engine": 0, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 8, "monotonic": 9119577.846998611, "engine": 0, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 9, "monotonic": 9119577.851663468, "engine": 0, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 10, "monotonic": 9119577.856308192, "engine": 0, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 11, "monotonic": 9119577.859561805, "engine": 0, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 12, "monotonic": 9119577.862482356, "engine": 0, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 13, "monotonic": 9119578.960288996, "engine": 0, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 14, "monotonic": 9119578.970791968, "engine": 0, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 15, "monotonic": 9119578.973420218, "engine": 0, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 16, "monotonic": 9119578.982067386, "engine": 0, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 17, "monotonic": 9119578.986149205, "engine": 0, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 18, "monotonic": 9119578.990410643, "engine": 0, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 19, "monotonic": 9119578.992635172, "engine": 0, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 20, "monotonic": 9119578.994783333, "engine": 0, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 21, "monotonic": 9119580.09426199, "engine": 0, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 22, "monotonic": 9119580.104928317, "engine": 0, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 23, "monotonic": 9119580.108950997, "engine": 0, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 24, "monotonic": 9119580.11632106, "engine": 0, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 25, "monotonic": 9119580.122566212, "engine": 0, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 26, "monotonic": 9119580.127797948, "engine": 0, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 27, "monotonic": 9119580.131552352, "engine": 0, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 28, "monotonic": 9119580.135168187, "engine": 0, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 29, "monotonic": 9119581.232589915, "engine": 0, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 30, "monotonic": 9119581.244141446, "engine": 0, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 31, "monotonic": 9119581.24668991, "engine": 0, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 32, "monotonic": 9119581.255638912, "engine": 0, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 33, "monotonic": 9119581.262119388, "engine": 0, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 34, "monotonic": 9119581.268199256, "engine": 0, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q8"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 35, "monotonic": 9119581.2744289, "engine": 0, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q10"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 36, "monotonic": 9119581.280617448, "engine": 0, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q12"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 37, "monotonic": 9119581.286810076, "engine": 0, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q14"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 38, "monotonic": 9119581.29294334, "engine": 0, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 39, "monotonic": 9119581.296367176, "engine": 0, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 40, "monotonic": 9119581.299756235, "engine": 0, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 41, "monotonic": 9119582.398610272, "engine": 0, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 42, "monotonic": 9119582.40874644, "engine": 0, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 43, "monotonic": 9119582.41231597, "engine": 0, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 44, "monotonic": 9119582.420458922, "engine": 0, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 45, "monotonic": 9119582.425835129, "engine": 0, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 46, "monotonic": 9119582.431018058, "engine": 0, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q8"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 47, "monotonic": 9119582.436320836, "engine": 0, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q10"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 48, "monotonic": 9119582.441395594, "engine": 0, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q12"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 49, "monotonic": 9119582.446620272, "engine": 0, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q14"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 50, "monotonic": 9119582.451802012, "engine": 0, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 51, "monotonic": 9119582.45515298, "engine": 0, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 52, "monotonic": 9119582.457941312, "engine": 0, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 53, "monotonic": 9119583.557296367, "engine": 0, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 54, "monotonic": 9119583.568288304, "engine": 0, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 55, "monotonic": 9119583.571709411, "engine": 0, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 56, "monotonic": 9119583.57967443, "engine": 0, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 57, "monotonic": 9119583.584644731, "engine": 0, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 58, "monotonic": 9119583.590087384, "engine": 0, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q8"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 59, "monotonic": 9119583.595089003, "engine": 0, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q10"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 60, "monotonic": 9119583.600473017, "engine": 0, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q12"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 61, "monotonic": 9119583.60545546, "engine": 0, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q14"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 663, "seq": 62, "monotonic": 9119583.610792452, "engine": 0, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 63, "monotonic": 9119583.61387913, "engine": 0, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 663, "seq": 64, "monotonic": 9119583.616643798, "engine": 0, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_664.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_664.jsonl new file mode 100644 index 00000000..bb20bb4f --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_664.jsonl @@ -0,0 +1,65 @@ +{"kind": "engine_iteration", "pid": 664, "seq": 0, "monotonic": 9119576.448517464, "engine": 1, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 1, "monotonic": 9119576.656539064, "engine": 1, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 2, "monotonic": 9119576.668673038, "engine": 1, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 3, "monotonic": 9119576.67321912, "engine": 1, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 4, "monotonic": 9119576.675677937, "engine": 1, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 5, "monotonic": 9119577.824109633, "engine": 1, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 6, "monotonic": 9119577.835726645, "engine": 1, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 7, "monotonic": 9119577.838386636, "engine": 1, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 8, "monotonic": 9119577.847060977, "engine": 1, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 9, "monotonic": 9119577.851408212, "engine": 1, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 10, "monotonic": 9119577.856307589, "engine": 1, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 11, "monotonic": 9119577.858678106, "engine": 1, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 12, "monotonic": 9119577.861769507, "engine": 1, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 13, "monotonic": 9119578.960459523, "engine": 1, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 14, "monotonic": 9119578.970415901, "engine": 1, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 15, "monotonic": 9119578.973142773, "engine": 1, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 16, "monotonic": 9119578.9821197, "engine": 1, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 17, "monotonic": 9119578.986046756, "engine": 1, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 18, "monotonic": 9119578.990198545, "engine": 1, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 19, "monotonic": 9119578.992546445, "engine": 1, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 20, "monotonic": 9119578.99476318, "engine": 1, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 21, "monotonic": 9119580.0946057, "engine": 1, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 22, "monotonic": 9119580.105009504, "engine": 1, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 23, "monotonic": 9119580.107590236, "engine": 1, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 24, "monotonic": 9119580.116323853, "engine": 1, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 25, "monotonic": 9119580.12049956, "engine": 1, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 26, "monotonic": 9119580.126651892, "engine": 1, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 27, "monotonic": 9119580.130101508, "engine": 1, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 28, "monotonic": 9119580.133673932, "engine": 1, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 29, "monotonic": 9119581.232942274, "engine": 1, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 30, "monotonic": 9119581.244137796, "engine": 1, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 31, "monotonic": 9119581.2480332, "engine": 1, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 32, "monotonic": 9119581.257721173, "engine": 1, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 33, "monotonic": 9119581.26389318, "engine": 1, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 34, "monotonic": 9119581.270159176, "engine": 1, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q9"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 35, "monotonic": 9119581.276280183, "engine": 1, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q11"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 36, "monotonic": 9119581.282644592, "engine": 1, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q13"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 37, "monotonic": 9119581.288699504, "engine": 1, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q15"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 38, "monotonic": 9119581.29402939, "engine": 1, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 39, "monotonic": 9119581.297625517, "engine": 1, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 40, "monotonic": 9119581.301283844, "engine": 1, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 41, "monotonic": 9119582.398801848, "engine": 1, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 42, "monotonic": 9119582.409782464, "engine": 1, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 43, "monotonic": 9119582.412839167, "engine": 1, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 44, "monotonic": 9119582.421456927, "engine": 1, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 45, "monotonic": 9119582.42656548, "engine": 1, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 46, "monotonic": 9119582.431824055, "engine": 1, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q9"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 47, "monotonic": 9119582.43702584, "engine": 1, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q11"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 48, "monotonic": 9119582.442215288, "engine": 1, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q13"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 49, "monotonic": 9119582.447442189, "engine": 1, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q15"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 50, "monotonic": 9119582.452716084, "engine": 1, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 51, "monotonic": 9119582.455679407, "engine": 1, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 52, "monotonic": 9119582.45858248, "engine": 1, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 53, "monotonic": 9119583.557630615, "engine": 1, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 54, "monotonic": 9119583.56901594, "engine": 1, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 55, "monotonic": 9119583.57212383, "engine": 1, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 56, "monotonic": 9119583.580448361, "engine": 1, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 57, "monotonic": 9119583.585726876, "engine": 1, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 58, "monotonic": 9119583.590783136, "engine": 1, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q9"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 59, "monotonic": 9119583.596036963, "engine": 1, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q11"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 60, "monotonic": 9119583.601235135, "engine": 1, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q13"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 61, "monotonic": 9119583.606379524, "engine": 1, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q15"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 664, "seq": 62, "monotonic": 9119583.611563925, "engine": 1, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 63, "monotonic": 9119583.614556208, "engine": 1, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 664, "seq": 64, "monotonic": 9119583.617356433, "engine": 1, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/model/config.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/model/config.json new file mode 100644 index 00000000..e9b20534 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/model/config.json @@ -0,0 +1,95 @@ +{ + "vocab_size": 128256, + "max_position_embeddings": 131072, + "hidden_size": 2048, + "intermediate_size": 8192, + "num_hidden_layers": 16, + "num_attention_heads": 32, + "num_key_value_heads": 8, + "hidden_act": "silu", + "initializer_range": 0.02, + "rms_norm_eps": 1e-05, + "pretraining_tp": 1, + "use_cache": true, + "rope_theta": 500000.0, + "rope_scaling": { + "factor": 32.0, + "high_freq_factor": 4.0, + "low_freq_factor": 1.0, + "original_max_position_embeddings": 8192, + "rope_type": "llama3" + }, + "attention_bias": false, + "attention_dropout": 0.0, + "mlp_bias": false, + "head_dim": 64, + "return_dict": true, + "output_hidden_states": false, + "torchscript": false, + "dtype": "bfloat16", + "torch_dtype": "bfloat16", + "pruned_heads": {}, + "tie_word_embeddings": true, + "chunk_size_feed_forward": 0, + "is_encoder_decoder": false, + "is_decoder": false, + "cross_attention_hidden_size": null, + "add_cross_attention": false, + "tie_encoder_decoder": false, + "architectures": [ + "LlamaForCausalLM" + ], + "finetuning_task": null, + "id2label": { + "0": "LABEL_0", + "1": "LABEL_1" + }, + "label2id": { + "LABEL_0": 0, + "LABEL_1": 1 + }, + "task_specific_params": null, + "problem_type": null, + "tokenizer_class": null, + "prefix": null, + "bos_token_id": 128000, + "pad_token_id": null, + "eos_token_id": [ + 128001, + 128008, + 128009 + ], + "sep_token_id": null, + "decoder_start_token_id": null, + "max_length": 20, + "min_length": 0, + "do_sample": false, + "early_stopping": false, + "num_beams": 1, + "temperature": 1.0, + "top_k": 50, + "top_p": 1.0, + "typical_p": 1.0, + "repetition_penalty": 1.0, + "length_penalty": 1.0, + "no_repeat_ngram_size": 0, + "encoder_no_repeat_ngram_size": 0, + "bad_words_ids": null, + "num_return_sequences": 1, + "output_scores": false, + "return_dict_in_generate": false, + "forced_bos_token_id": null, + "forced_eos_token_id": null, + "remove_invalid_values": false, + "exponential_decay_length_penalty": null, + "suppress_tokens": null, + "begin_suppress_tokens": null, + "num_beam_groups": 1, + "diversity_penalty": 0.0, + "_name_or_path": "meta-llama/Llama-3.2-1B-Instruct", + "transformers_version": "4.57.3", + "model_type": "llama", + "tf_legacy_loss": false, + "use_bfloat16": false, + "output_attentions": false +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/pp_boundary.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/pp_boundary.jsonl new file mode 100644 index 00000000..1998efab --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/pp_boundary.jsonl @@ -0,0 +1,152 @@ +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119576.449235303, "preprocess_end_ts": 9119576.451109972, "forward_start_ts": 9119576.451299587, "forward_end_ts": 9119576.460063873, "timestamp": 1790101597.6622696, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119576.46019184, "send_end_ts": 9119576.629189717} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119576.44977724, "preprocess_end_ts": 9119576.451516641, "forward_start_ts": 9119576.451719085, "forward_end_ts": 9119576.46017466, "timestamp": 1790101597.6628628, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119576.460299674, "send_end_ts": 9119576.629782932} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119576.629035799, "preprocess_end_ts": 9119576.630523184, "forward_start_ts": 9119576.630714431, "forward_end_ts": 9119576.639729928, "timestamp": 1790101597.6889172, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119576.448764324, "recv_end_ts": 9119576.628623897, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119576.629958017, "preprocess_end_ts": 9119576.630545435, "forward_start_ts": 9119576.630566584, "forward_end_ts": 9119576.63488724, "timestamp": 1790101597.6897466, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119576.635018112, "send_end_ts": 9119576.65667815} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119576.629221069, "preprocess_end_ts": 9119576.631134983, "forward_start_ts": 9119576.631357692, "forward_end_ts": 9119576.64234354, "timestamp": 1790101597.696209, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119576.449436652, "recv_end_ts": 9119576.62873706, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119576.63039467, "preprocess_end_ts": 9119576.630852029, "forward_start_ts": 9119576.630868305, "forward_end_ts": 9119576.633887624, "timestamp": 1790101597.6971579, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119576.633995306, "send_end_ts": 9119576.664088145} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119576.656937769, "preprocess_end_ts": 9119576.65735066, "forward_start_ts": 9119576.657365354, "forward_end_ts": 9119576.667654537, "timestamp": 1790101597.701157, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119576.65626875, "recv_end_ts": 9119576.656732377, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119576.664351743, "preprocess_end_ts": 9119576.664894195, "forward_start_ts": 9119576.664911393, "forward_end_ts": 9119576.668773573, "timestamp": 1790101597.7023337, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119576.663628172, "recv_end_ts": 9119576.664111456, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.824357212, "preprocess_end_ts": 9119577.82498523, "forward_start_ts": 9119577.825001959, "forward_end_ts": 9119577.828014908, "timestamp": 1790101598.8617194, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.828126712, "send_end_ts": 9119577.828654032} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.824468488, "preprocess_end_ts": 9119577.825073266, "forward_start_ts": 9119577.825089, "forward_end_ts": 9119577.829188433, "timestamp": 1790101598.8629055, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.829313964, "send_end_ts": 9119577.82983897} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.828956548, "preprocess_end_ts": 9119577.829398211, "forward_start_ts": 9119577.829411317, "forward_end_ts": 9119577.833676076, "timestamp": 1790101598.867198, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.824065764, "recv_end_ts": 9119577.828752125, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.830214947, "preprocess_end_ts": 9119577.830757245, "forward_start_ts": 9119577.830772884, "forward_end_ts": 9119577.83463634, "timestamp": 1790101598.868191, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.824199883, "recv_end_ts": 9119577.829978593, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.838627815, "preprocess_end_ts": 9119577.838983875, "forward_start_ts": 9119577.838994114, "forward_end_ts": 9119577.841707371, "timestamp": 1790101598.8752441, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.84179205, "send_end_ts": 9119577.842179088} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.838771084, "preprocess_end_ts": 9119577.83919682, "forward_start_ts": 9119577.839209195, "forward_end_ts": 9119577.842601636, "timestamp": 1790101598.8762252, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.842700256, "send_end_ts": 9119577.843159849} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.842395412, "preprocess_end_ts": 9119577.842754915, "forward_start_ts": 9119577.84276472, "forward_end_ts": 9119577.84616774, "timestamp": 1790101598.879648, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.838463878, "recv_end_ts": 9119577.842247857, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.843345407, "preprocess_end_ts": 9119577.843700437, "forward_start_ts": 9119577.84371037, "forward_end_ts": 9119577.84624232, "timestamp": 1790101598.879721, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.838398213, "recv_end_ts": 9119577.84319912, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.842513002, "preprocess_end_ts": 9119577.842835452, "forward_start_ts": 9119577.842845012, "forward_end_ts": 9119577.84654935, "timestamp": 1790101598.8802576, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.84662892, "send_end_ts": 9119577.847191827} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.843563482, "preprocess_end_ts": 9119577.843991311, "forward_start_ts": 9119577.844012097, "forward_end_ts": 9119577.84732702, "timestamp": 1790101598.8808856, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.847418964, "send_end_ts": 9119577.847820364} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.847960591, "preprocess_end_ts": 9119577.84833606, "forward_start_ts": 9119577.84834514, "forward_end_ts": 9119577.85068873, "timestamp": 1790101598.8841507, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.84689201, "recv_end_ts": 9119577.847834667, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.84745461, "preprocess_end_ts": 9119577.84791852, "forward_start_ts": 9119577.84792845, "forward_end_ts": 9119577.850802308, "timestamp": 1790101598.8842638, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.84683005, "recv_end_ts": 9119577.847290784, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.847578784, "preprocess_end_ts": 9119577.847931173, "forward_start_ts": 9119577.847940467, "forward_end_ts": 9119577.851136187, "timestamp": 1790101598.884929, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.851210548, "send_end_ts": 9119577.851864755} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.84824756, "preprocess_end_ts": 9119577.84866038, "forward_start_ts": 9119577.848671071, "forward_end_ts": 9119577.852039874, "timestamp": 1790101598.885622, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.852133228, "send_end_ts": 9119577.852556845} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.852132866, "preprocess_end_ts": 9119577.852478208, "forward_start_ts": 9119577.85248706, "forward_end_ts": 9119577.855527874, "timestamp": 1790101598.8889954, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.851488229, "recv_end_ts": 9119577.851950396, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.852745475, "preprocess_end_ts": 9119577.853077188, "forward_start_ts": 9119577.853086252, "forward_end_ts": 9119577.855551347, "timestamp": 1790101598.8890097, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.85131053, "recv_end_ts": 9119577.85258692, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.960668135, "preprocess_end_ts": 9119578.96123012, "forward_start_ts": 9119578.961244408, "forward_end_ts": 9119578.964522295, "timestamp": 1790101599.9982448, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.964636814, "send_end_ts": 9119578.96517948} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.96082784, "preprocess_end_ts": 9119578.961398747, "forward_start_ts": 9119578.961414317, "forward_end_ts": 9119578.965519715, "timestamp": 1790101599.9992168, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.965648709, "send_end_ts": 9119578.96615019} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.966426106, "preprocess_end_ts": 9119578.966873564, "forward_start_ts": 9119578.966887629, "forward_end_ts": 9119578.969609275, "timestamp": 1790101600.003122, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.960561955, "recv_end_ts": 9119578.966229323, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.965645164, "preprocess_end_ts": 9119578.966221929, "forward_start_ts": 9119578.966246396, "forward_end_ts": 9119578.969896998, "timestamp": 1790101600.003438, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.960641623, "recv_end_ts": 9119578.965302303, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.973591277, "preprocess_end_ts": 9119578.9739243, "forward_start_ts": 9119578.973934824, "forward_end_ts": 9119578.976901937, "timestamp": 1790101600.0104132, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.976986956, "send_end_ts": 9119578.977347884} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.973924132, "preprocess_end_ts": 9119578.974346958, "forward_start_ts": 9119578.974358551, "forward_end_ts": 9119578.977764774, "timestamp": 1790101600.0114522, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.977866407, "send_end_ts": 9119578.97838593} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.977527997, "preprocess_end_ts": 9119578.97787376, "forward_start_ts": 9119578.977883596, "forward_end_ts": 9119578.981314112, "timestamp": 1790101600.0147886, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.97344376, "recv_end_ts": 9119578.977390269, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.978509312, "preprocess_end_ts": 9119578.978842791, "forward_start_ts": 9119578.978853147, "forward_end_ts": 9119578.981392836, "timestamp": 1790101600.0148735, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.973179871, "recv_end_ts": 9119578.978368731, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.977689132, "preprocess_end_ts": 9119578.978013683, "forward_start_ts": 9119578.978024937, "forward_end_ts": 9119578.981589071, "timestamp": 1790101600.0153449, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.981677303, "send_end_ts": 9119578.982279787} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.978735156, "preprocess_end_ts": 9119578.979071628, "forward_start_ts": 9119578.979081316, "forward_end_ts": 9119578.981540589, "timestamp": 1790101600.0155048, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.981624395, "send_end_ts": 9119578.982438985} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.982645798, "preprocess_end_ts": 9119578.982993113, "forward_start_ts": 9119578.983002448, "forward_end_ts": 9119578.985380031, "timestamp": 1790101600.0188494, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.982039647, "recv_end_ts": 9119578.98251816, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.982495245, "preprocess_end_ts": 9119578.982915264, "forward_start_ts": 9119578.982923692, "forward_end_ts": 9119578.985420316, "timestamp": 1790101600.0188775, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.981969379, "recv_end_ts": 9119578.982336044, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.982795704, "preprocess_end_ts": 9119578.983114818, "forward_start_ts": 9119578.983123522, "forward_end_ts": 9119578.985487437, "timestamp": 1790101600.0194476, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.985561136, "send_end_ts": 9119578.986381425} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.982634112, "preprocess_end_ts": 9119578.982959624, "forward_start_ts": 9119578.982968632, "forward_end_ts": 9119578.985553572, "timestamp": 1790101600.0194964, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.985630652, "send_end_ts": 9119578.98643162} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.986565378, "preprocess_end_ts": 9119578.98689252, "forward_start_ts": 9119578.986901361, "forward_end_ts": 9119578.98951606, "timestamp": 1790101600.022985, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.986011704, "recv_end_ts": 9119578.986432532, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.98668281, "preprocess_end_ts": 9119578.987024937, "forward_start_ts": 9119578.987034166, "forward_end_ts": 9119578.989679078, "timestamp": 1790101600.0231612, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.986048156, "recv_end_ts": 9119578.986526525, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.094688626, "preprocess_end_ts": 9119580.095366264, "forward_start_ts": 9119580.095383598, "forward_end_ts": 9119580.09878654, "timestamp": 1790101601.1325057, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.098893972, "send_end_ts": 9119580.099439714} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.09495793, "preprocess_end_ts": 9119580.09543242, "forward_start_ts": 9119580.095445976, "forward_end_ts": 9119580.099593991, "timestamp": 1790101601.1333818, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.099735184, "send_end_ts": 9119580.100314468} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.099875676, "preprocess_end_ts": 9119580.100399459, "forward_start_ts": 9119580.100419892, "forward_end_ts": 9119580.104041908, "timestamp": 1790101601.1375847, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.094411604, "recv_end_ts": 9119580.099614464, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.100623075, "preprocess_end_ts": 9119580.10111244, "forward_start_ts": 9119580.101126062, "forward_end_ts": 9119580.104044553, "timestamp": 1790101601.1376822, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.09472104, "recv_end_ts": 9119580.100411221, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.109105298, "preprocess_end_ts": 9119580.109411485, "forward_start_ts": 9119580.109421978, "forward_end_ts": 9119580.111969125, "timestamp": 1790101601.1455402, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.112062896, "send_end_ts": 9119580.112475628} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.109155536, "preprocess_end_ts": 9119580.10950957, "forward_start_ts": 9119580.10952064, "forward_end_ts": 9119580.112086548, "timestamp": 1790101601.145642, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.112181759, "send_end_ts": 9119580.11257712} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.112802109, "preprocess_end_ts": 9119580.113141969, "forward_start_ts": 9119580.113151772, "forward_end_ts": 9119580.11560526, "timestamp": 1790101601.1490958, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.107629674, "recv_end_ts": 9119580.1126576, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.11271294, "preprocess_end_ts": 9119580.113064012, "forward_start_ts": 9119580.113074807, "forward_end_ts": 9119580.115638377, "timestamp": 1790101601.149124, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.109004192, "recv_end_ts": 9119580.112562789, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.11292049, "preprocess_end_ts": 9119580.11325336, "forward_start_ts": 9119580.113263048, "forward_end_ts": 9119580.115688888, "timestamp": 1790101601.1496584, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.115762897, "send_end_ts": 9119580.11659234} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.112800252, "preprocess_end_ts": 9119580.113136455, "forward_start_ts": 9119580.113145532, "forward_end_ts": 9119580.115697565, "timestamp": 1790101601.1497052, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.115773346, "send_end_ts": 9119580.116639584} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.116801972, "preprocess_end_ts": 9119580.117155287, "forward_start_ts": 9119580.117163748, "forward_end_ts": 9119580.119808223, "timestamp": 1790101601.1532898, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.116262624, "recv_end_ts": 9119580.116673816, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.11686216, "preprocess_end_ts": 9119580.117208404, "forward_start_ts": 9119580.117217377, "forward_end_ts": 9119580.12174194, "timestamp": 1790101601.1552868, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.11630194, "recv_end_ts": 9119580.116723644, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.116933325, "preprocess_end_ts": 9119580.11725814, "forward_start_ts": 9119580.117267575, "forward_end_ts": 9119580.121900078, "timestamp": 1790101601.1555579, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.121995708, "send_end_ts": 9119580.122491708} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.117043357, "preprocess_end_ts": 9119580.117375923, "forward_start_ts": 9119580.117384301, "forward_end_ts": 9119580.119822728, "timestamp": 1790101601.1559808, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.11989524, "send_end_ts": 9119580.122915024} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.122579215, "preprocess_end_ts": 9119580.122919368, "forward_start_ts": 9119580.12292863, "forward_end_ts": 9119580.125977028, "timestamp": 1790101601.159441, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.12045625, "recv_end_ts": 9119580.122441912, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.123137092, "preprocess_end_ts": 9119580.123574857, "forward_start_ts": 9119580.123585664, "forward_end_ts": 9119580.127115352, "timestamp": 1790101601.16059, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.122521417, "recv_end_ts": 9119580.122969313, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.233396504, "preprocess_end_ts": 9119581.233994463, "forward_start_ts": 9119581.234011868, "forward_end_ts": 9119581.238167107, "timestamp": 1790101602.2719352, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.238298027, "send_end_ts": 9119581.238868516} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.233019358, "preprocess_end_ts": 9119581.233695393, "forward_start_ts": 9119581.233710907, "forward_end_ts": 9119581.238902632, "timestamp": 1790101602.272517, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.239007484, "send_end_ts": 9119581.239451487} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.239113316, "preprocess_end_ts": 9119581.239545468, "forward_start_ts": 9119581.239558164, "forward_end_ts": 9119581.243230795, "timestamp": 1790101602.2767441, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.233050244, "recv_end_ts": 9119581.238915678, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.239849389, "preprocess_end_ts": 9119581.240372729, "forward_start_ts": 9119581.240387648, "forward_end_ts": 9119581.243204897, "timestamp": 1790101602.2767386, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.232745023, "recv_end_ts": 9119581.239610076, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.24686108, "preprocess_end_ts": 9119581.247199334, "forward_start_ts": 9119581.247208878, "forward_end_ts": 9119581.251029195, "timestamp": 1790101602.284532, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.251113068, "send_end_ts": 9119581.25146761} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.24822918, "preprocess_end_ts": 9119581.248552652, "forward_start_ts": 9119581.248562204, "forward_end_ts": 9119581.251013158, "timestamp": 1790101602.284648, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.25109625, "send_end_ts": 9119581.251581362} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.251669185, "preprocess_end_ts": 9119581.252024984, "forward_start_ts": 9119581.252034616, "forward_end_ts": 9119581.254945168, "timestamp": 1790101602.2884212, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.246744012, "recv_end_ts": 9119581.251521748, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.251804125, "preprocess_end_ts": 9119581.252146717, "forward_start_ts": 9119581.252156204, "forward_end_ts": 9119581.254929967, "timestamp": 1790101602.2890077, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.255009815, "send_end_ts": 9119581.255942978} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.251821188, "preprocess_end_ts": 9119581.2523388, "forward_start_ts": 9119581.252351632, "forward_end_ts": 9119581.256925946, "timestamp": 1790101602.2904587, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.248077651, "recv_end_ts": 9119581.251637876, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.251998393, "preprocess_end_ts": 9119581.252419272, "forward_start_ts": 9119581.252429983, "forward_end_ts": 9119581.256907957, "timestamp": 1790101602.2910933, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.257015044, "send_end_ts": 9119581.258026555} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.25614549, "preprocess_end_ts": 9119581.256490285, "forward_start_ts": 9119581.256499529, "forward_end_ts": 9119581.261377309, "timestamp": 1790101602.294866, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.255611897, "recv_end_ts": 9119581.255996894, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.25627696, "preprocess_end_ts": 9119581.256595548, "forward_start_ts": 9119581.256605532, "forward_end_ts": 9119581.261548655, "timestamp": 1790101602.2954452, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.26162432, "send_end_ts": 9119581.2623796} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.258215534, "preprocess_end_ts": 9119581.258696891, "forward_start_ts": 9119581.25870836, "forward_end_ts": 9119581.263100836, "timestamp": 1790101602.2966354, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.257630471, "recv_end_ts": 9119581.25805199, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.258432096, "preprocess_end_ts": 9119581.258829989, "forward_start_ts": 9119581.258840077, "forward_end_ts": 9119581.26315874, "timestamp": 1790101602.297273, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.263248276, "send_end_ts": 9119581.264206264} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.262610223, "preprocess_end_ts": 9119581.262939585, "forward_start_ts": 9119581.262948113, "forward_end_ts": 9119581.267470272, "timestamp": 1790101602.3009524, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.262037065, "recv_end_ts": 9119581.262459122, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.262731668, "preprocess_end_ts": 9119581.263055056, "forward_start_ts": 9119581.26306337, "forward_end_ts": 9119581.267570898, "timestamp": 1790101602.3015566, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.267647356, "send_end_ts": 9119581.268490193} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.264412211, "preprocess_end_ts": 9119581.264882725, "forward_start_ts": 9119581.264894173, "forward_end_ts": 9119581.269408816, "timestamp": 1790101602.3029337, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.2637949, "recv_end_ts": 9119581.264242757, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.2646153, "preprocess_end_ts": 9119581.2650162, "forward_start_ts": 9119581.265026176, "forward_end_ts": 9119581.269391648, "timestamp": 1790101602.3035674, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.26949292, "send_end_ts": 9119581.270500356} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.268705528, "preprocess_end_ts": 9119581.269011945, "forward_start_ts": 9119581.269020028, "forward_end_ts": 9119581.2737324, "timestamp": 1790101602.3072135, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.268147953, "recv_end_ts": 9119581.26856769, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.268824603, "preprocess_end_ts": 9119581.26916552, "forward_start_ts": 9119581.269175928, "forward_end_ts": 9119581.273862423, "timestamp": 1790101602.3078134, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.273937115, "send_end_ts": 9119581.274747202} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.270700352, "preprocess_end_ts": 9119581.271185782, "forward_start_ts": 9119581.271197073, "forward_end_ts": 9119581.275499867, "timestamp": 1790101602.3090343, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.270090705, "recv_end_ts": 9119581.270530147, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.270897005, "preprocess_end_ts": 9119581.271298785, "forward_start_ts": 9119581.271308472, "forward_end_ts": 9119581.275674287, "timestamp": 1790101602.3096607, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.275766958, "send_end_ts": 9119581.276593788} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.274924587, "preprocess_end_ts": 9119581.2752397, "forward_start_ts": 9119581.275248451, "forward_end_ts": 9119581.27991012, "timestamp": 1790101602.3134055, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.274385016, "recv_end_ts": 9119581.274778528, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.2750879, "preprocess_end_ts": 9119581.27542442, "forward_start_ts": 9119581.275435092, "forward_end_ts": 9119581.279930985, "timestamp": 1790101602.3139422, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.280019982, "send_end_ts": 9119581.280877493} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.27679698, "preprocess_end_ts": 9119581.277281515, "forward_start_ts": 9119581.277292844, "forward_end_ts": 9119581.281861791, "timestamp": 1790101602.3153794, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.276213275, "recv_end_ts": 9119581.276628692, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.277001103, "preprocess_end_ts": 9119581.2773985, "forward_start_ts": 9119581.277407784, "forward_end_ts": 9119581.281844368, "timestamp": 1790101602.3159869, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.281945448, "send_end_ts": 9119581.282920448} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.28111526, "preprocess_end_ts": 9119581.28146688, "forward_start_ts": 9119581.281476619, "forward_end_ts": 9119581.286099674, "timestamp": 1790101602.3195717, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.280585507, "recv_end_ts": 9119581.280974768, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.281199312, "preprocess_end_ts": 9119581.281508658, "forward_start_ts": 9119581.281517176, "forward_end_ts": 9119581.286307205, "timestamp": 1790101602.3201282, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.286381297, "send_end_ts": 9119581.287063923} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.283120563, "preprocess_end_ts": 9119581.283600004, "forward_start_ts": 9119581.283611668, "forward_end_ts": 9119581.287884478, "timestamp": 1790101602.3214347, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.282539101, "recv_end_ts": 9119581.282952696, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.283317965, "preprocess_end_ts": 9119581.283716684, "forward_start_ts": 9119581.28372626, "forward_end_ts": 9119581.288230387, "timestamp": 1790101602.3220446, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.288325038, "send_end_ts": 9119581.288978308} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.287315471, "preprocess_end_ts": 9119581.287657458, "forward_start_ts": 9119581.28766616, "forward_end_ts": 9119581.292271994, "timestamp": 1790101602.3257442, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.28679424, "recv_end_ts": 9119581.28717838, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.289188562, "preprocess_end_ts": 9119581.289658416, "forward_start_ts": 9119581.289669557, "forward_end_ts": 9119581.293196363, "timestamp": 1790101602.3267112, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.288626568, "recv_end_ts": 9119581.289021444, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.399505384, "preprocess_end_ts": 9119582.400126928, "forward_start_ts": 9119582.400153896, "forward_end_ts": 9119582.403315112, "timestamp": 1790101603.4370353, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.403440015, "send_end_ts": 9119582.403964752} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.399045372, "preprocess_end_ts": 9119582.399619577, "forward_start_ts": 9119582.399633583, "forward_end_ts": 9119582.40350036, "timestamp": 1790101603.4372048, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.40361334, "send_end_ts": 9119582.404137751} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.404470395, "preprocess_end_ts": 9119582.404925624, "forward_start_ts": 9119582.404938199, "forward_end_ts": 9119582.407814419, "timestamp": 1790101603.4413486, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.398723489, "recv_end_ts": 9119582.4042588, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.404359484, "preprocess_end_ts": 9119582.4049385, "forward_start_ts": 9119582.404952949, "forward_end_ts": 9119582.408852275, "timestamp": 1790101603.442421, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.398926804, "recv_end_ts": 9119582.404123345, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.413005928, "preprocess_end_ts": 9119582.4133225, "forward_start_ts": 9119582.413331948, "forward_end_ts": 9119582.415712643, "timestamp": 1790101603.4492488, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.415795052, "send_end_ts": 9119582.41618442} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.412871776, "preprocess_end_ts": 9119582.413243042, "forward_start_ts": 9119582.413253022, "forward_end_ts": 9119582.416106895, "timestamp": 1790101603.4496534, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.41620316, "send_end_ts": 9119582.416588666} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.4167915, "preprocess_end_ts": 9119582.417173, "forward_start_ts": 9119582.417184316, "forward_end_ts": 9119582.419704515, "timestamp": 1790101603.453184, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.412371628, "recv_end_ts": 9119582.416646589, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.416910045, "preprocess_end_ts": 9119582.417237205, "forward_start_ts": 9119582.417245656, "forward_end_ts": 9119582.41993879, "timestamp": 1790101603.4537365, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.420030707, "send_end_ts": 9119582.42067046} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.416453078, "preprocess_end_ts": 9119582.416901791, "forward_start_ts": 9119582.416913465, "forward_end_ts": 9119582.420616964, "timestamp": 1790101603.454135, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.412896767, "recv_end_ts": 9119582.41628768, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.416508028, "preprocess_end_ts": 9119582.416822912, "forward_start_ts": 9119582.41683184, "forward_end_ts": 9119582.419869848, "timestamp": 1790101603.4547722, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.419945642, "send_end_ts": 9119582.42170742} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.42084617, "preprocess_end_ts": 9119582.421175722, "forward_start_ts": 9119582.421185223, "forward_end_ts": 9119582.424993424, "timestamp": 1790101603.4584696, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.420361483, "recv_end_ts": 9119582.420718808, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.421002967, "preprocess_end_ts": 9119582.421317281, "forward_start_ts": 9119582.421325749, "forward_end_ts": 9119582.42481764, "timestamp": 1790101603.4590611, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.424892709, "send_end_ts": 9119582.425996372} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.421963971, "preprocess_end_ts": 9119582.422395866, "forward_start_ts": 9119582.422406385, "forward_end_ts": 9119582.425856559, "timestamp": 1790101603.459334, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.421367256, "recv_end_ts": 9119582.421805905, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.42202926, "preprocess_end_ts": 9119582.422343152, "forward_start_ts": 9119582.422351869, "forward_end_ts": 9119582.424909133, "timestamp": 1790101603.4599416, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.42499226, "send_end_ts": 9119582.426876452} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.42624215, "preprocess_end_ts": 9119582.426581772, "forward_start_ts": 9119582.426591672, "forward_end_ts": 9119582.430140233, "timestamp": 1790101603.4636054, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.425643357, "recv_end_ts": 9119582.426081654, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.426323969, "preprocess_end_ts": 9119582.426648231, "forward_start_ts": 9119582.426657196, "forward_end_ts": 9119582.43019246, "timestamp": 1790101603.464215, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.430267744, "send_end_ts": 9119582.431150008} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.427180208, "preprocess_end_ts": 9119582.427611424, "forward_start_ts": 9119582.427621692, "forward_end_ts": 9119582.431075092, "timestamp": 1790101603.464572, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.426556783, "recv_end_ts": 9119582.427014092, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.427201988, "preprocess_end_ts": 9119582.427503897, "forward_start_ts": 9119582.4275129, "forward_end_ts": 9119582.429906465, "timestamp": 1790101603.465202, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.429982044, "send_end_ts": 9119582.432137204} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.431276957, "preprocess_end_ts": 9119582.431606408, "forward_start_ts": 9119582.431616047, "forward_end_ts": 9119582.435424551, "timestamp": 1790101603.468891, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.430785645, "recv_end_ts": 9119582.431143759, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.431484496, "preprocess_end_ts": 9119582.431882525, "forward_start_ts": 9119582.431892028, "forward_end_ts": 9119582.435344663, "timestamp": 1790101603.4694836, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.435420744, "send_end_ts": 9119582.436418865} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.432389896, "preprocess_end_ts": 9119582.432831071, "forward_start_ts": 9119582.432842351, "forward_end_ts": 9119582.43632091, "timestamp": 1790101603.469801, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.43179648, "recv_end_ts": 9119582.432225011, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.432463683, "preprocess_end_ts": 9119582.43278008, "forward_start_ts": 9119582.432788637, "forward_end_ts": 9119582.435299978, "timestamp": 1790101603.470434, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.435382178, "send_end_ts": 9119582.437368937} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.436651569, "preprocess_end_ts": 9119582.436987516, "forward_start_ts": 9119582.436997196, "forward_end_ts": 9119582.440632757, "timestamp": 1790101603.474107, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.436072463, "recv_end_ts": 9119582.43649928, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.43674938, "preprocess_end_ts": 9119582.43708408, "forward_start_ts": 9119582.437093578, "forward_end_ts": 9119582.440551614, "timestamp": 1790101603.4746928, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.440625504, "send_end_ts": 9119582.441628233} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.437632522, "preprocess_end_ts": 9119582.438060312, "forward_start_ts": 9119582.438070623, "forward_end_ts": 9119582.441493956, "timestamp": 1790101603.4749963, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.437028263, "recv_end_ts": 9119582.437467912, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.43771562, "preprocess_end_ts": 9119582.438037302, "forward_start_ts": 9119582.438046437, "forward_end_ts": 9119582.440517, "timestamp": 1790101603.4756887, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.440590328, "send_end_ts": 9119582.442623949} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.441851616, "preprocess_end_ts": 9119582.442183517, "forward_start_ts": 9119582.442192648, "forward_end_ts": 9119582.445875488, "timestamp": 1790101603.47935, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.441275446, "recv_end_ts": 9119582.44170566, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.441955976, "preprocess_end_ts": 9119582.442265827, "forward_start_ts": 9119582.442274155, "forward_end_ts": 9119582.445797782, "timestamp": 1790101603.4799547, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.445872858, "send_end_ts": 9119582.446887596} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.442881905, "preprocess_end_ts": 9119582.443313569, "forward_start_ts": 9119582.443323756, "forward_end_ts": 9119582.4466977, "timestamp": 1790101603.480205, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.44221971, "recv_end_ts": 9119582.442719487, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.442944948, "preprocess_end_ts": 9119582.44325064, "forward_start_ts": 9119582.4432588, "forward_end_ts": 9119582.445691545, "timestamp": 1790101603.480905, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.445766605, "send_end_ts": 9119582.447840024} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.447107095, "preprocess_end_ts": 9119582.447438892, "forward_start_ts": 9119582.447447576, "forward_end_ts": 9119582.45109568, "timestamp": 1790101603.4845707, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.4465232, "recv_end_ts": 9119582.446960269, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.448111612, "preprocess_end_ts": 9119582.448540188, "forward_start_ts": 9119582.44855094, "forward_end_ts": 9119582.45196748, "timestamp": 1790101603.4855008, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.447434956, "recv_end_ts": 9119582.447922956, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.558165964, "preprocess_end_ts": 9119583.558815712, "forward_start_ts": 9119583.558831848, "forward_end_ts": 9119583.562208481, "timestamp": 1790101604.5959175, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.56232964, "send_end_ts": 9119583.56285016} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.558093507, "preprocess_end_ts": 9119583.55883775, "forward_start_ts": 9119583.558868157, "forward_end_ts": 9119583.562416296, "timestamp": 1790101604.596084, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.5625461, "send_end_ts": 9119583.563014574} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.563554471, "preprocess_end_ts": 9119583.564202191, "forward_start_ts": 9119583.564220864, "forward_end_ts": 9119583.567211548, "timestamp": 1790101604.600776, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.557605112, "recv_end_ts": 9119583.563252116, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.56323242, "preprocess_end_ts": 9119583.563799536, "forward_start_ts": 9119583.563816488, "forward_end_ts": 9119583.568085903, "timestamp": 1790101604.601647, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.557727855, "recv_end_ts": 9119583.562990518, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.572272748, "preprocess_end_ts": 9119583.572581254, "forward_start_ts": 9119583.572590098, "forward_end_ts": 9119583.57498684, "timestamp": 1790101604.6084995, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.575065097, "send_end_ts": 9119583.575435093} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.571874056, "preprocess_end_ts": 9119583.57219586, "forward_start_ts": 9119583.57220512, "forward_end_ts": 9119583.575020477, "timestamp": 1790101604.6085463, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.575109651, "send_end_ts": 9119583.575481975} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.575726492, "preprocess_end_ts": 9119583.5761047, "forward_start_ts": 9119583.576115075, "forward_end_ts": 9119583.57891527, "timestamp": 1790101604.612406, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.571748208, "recv_end_ts": 9119583.575573377, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.575801192, "preprocess_end_ts": 9119583.576123495, "forward_start_ts": 9119583.576132122, "forward_end_ts": 9119583.578572733, "timestamp": 1790101604.6129484, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.578646213, "send_end_ts": 9119583.579884293} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.575709935, "preprocess_end_ts": 9119583.576208178, "forward_start_ts": 9119583.576221002, "forward_end_ts": 9119583.579639962, "timestamp": 1790101604.6131327, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.572167996, "recv_end_ts": 9119583.575538272, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.575764172, "preprocess_end_ts": 9119583.57608008, "forward_start_ts": 9119583.5760887, "forward_end_ts": 9119583.578583768, "timestamp": 1790101604.61376, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.578659184, "send_end_ts": 9119583.58069524} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.580092108, "preprocess_end_ts": 9119583.580472684, "forward_start_ts": 9119583.580483211, "forward_end_ts": 9119583.583912114, "timestamp": 1790101604.6173956, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.579599932, "recv_end_ts": 9119583.579955412, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.580210296, "preprocess_end_ts": 9119583.580518411, "forward_start_ts": 9119583.580526842, "forward_end_ts": 9119583.584042951, "timestamp": 1790101604.6179266, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.58412996, "send_end_ts": 9119583.584862104} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.580960056, "preprocess_end_ts": 9119583.58138511, "forward_start_ts": 9119583.58139578, "forward_end_ts": 9119583.584819203, "timestamp": 1790101604.618354, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.580370374, "recv_end_ts": 9119583.580805363, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.581018431, "preprocess_end_ts": 9119583.581334531, "forward_start_ts": 9119583.581343347, "forward_end_ts": 9119583.583715232, "timestamp": 1790101604.6189904, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.583787864, "send_end_ts": 9119583.585925717} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.58506961, "preprocess_end_ts": 9119583.585432064, "forward_start_ts": 9119583.58544058, "forward_end_ts": 9119583.58920126, "timestamp": 1790101604.6226776, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.584563052, "recv_end_ts": 9119583.58492705, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.585209263, "preprocess_end_ts": 9119583.585548878, "forward_start_ts": 9119583.585558156, "forward_end_ts": 9119583.58915847, "timestamp": 1790101604.6232069, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.589234628, "send_end_ts": 9119583.590142863} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.586181484, "preprocess_end_ts": 9119583.586610127, "forward_start_ts": 9119583.58662052, "forward_end_ts": 9119583.59006598, "timestamp": 1790101604.623529, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.585576976, "recv_end_ts": 9119583.58602296, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.586244408, "preprocess_end_ts": 9119583.586552983, "forward_start_ts": 9119583.586561283, "forward_end_ts": 9119583.589022089, "timestamp": 1790101604.6242101, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.589095414, "send_end_ts": 9119583.591145668} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.590339264, "preprocess_end_ts": 9119583.590705585, "forward_start_ts": 9119583.590714654, "forward_end_ts": 9119583.594391404, "timestamp": 1790101604.627862, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.589851815, "recv_end_ts": 9119583.59020824, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.590481328, "preprocess_end_ts": 9119583.59083188, "forward_start_ts": 9119583.59084208, "forward_end_ts": 9119583.594294608, "timestamp": 1790101604.6283815, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.59436937, "send_end_ts": 9119583.59531654} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.591388373, "preprocess_end_ts": 9119583.591828829, "forward_start_ts": 9119583.591839086, "forward_end_ts": 9119583.595257632, "timestamp": 1790101604.6287599, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.59074742, "recv_end_ts": 9119583.591225345, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.591467204, "preprocess_end_ts": 9119583.591777526, "forward_start_ts": 9119583.591786014, "forward_end_ts": 9119583.594190637, "timestamp": 1790101604.6294143, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.594262984, "send_end_ts": 9119583.596349882} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.5955398, "preprocess_end_ts": 9119583.5958819, "forward_start_ts": 9119583.595890991, "forward_end_ts": 9119583.599585062, "timestamp": 1790101604.6330583, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.595037963, "recv_end_ts": 9119583.595384393, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.595650123, "preprocess_end_ts": 9119583.5959664, "forward_start_ts": 9119583.595974859, "forward_end_ts": 9119583.599450566, "timestamp": 1790101604.6335962, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.599524356, "send_end_ts": 9119583.600531695} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.596603245, "preprocess_end_ts": 9119583.597033508, "forward_start_ts": 9119583.597043771, "forward_end_ts": 9119583.600491628, "timestamp": 1790101604.633969, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.595993277, "recv_end_ts": 9119583.596438153, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.596669847, "preprocess_end_ts": 9119583.596980136, "forward_start_ts": 9119583.59698864, "forward_end_ts": 9119583.59938621, "timestamp": 1790101604.6346073, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.599459851, "send_end_ts": 9119583.601542953} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.600735752, "preprocess_end_ts": 9119583.601096991, "forward_start_ts": 9119583.601106236, "forward_end_ts": 9119583.60471283, "timestamp": 1790101604.638188, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.60024349, "recv_end_ts": 9119583.600602657, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.600859625, "preprocess_end_ts": 9119583.601178572, "forward_start_ts": 9119583.601187274, "forward_end_ts": 9119583.604693076, "timestamp": 1790101604.6387181, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.6047763, "send_end_ts": 9119583.60565389} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.601779047, "preprocess_end_ts": 9119583.602213588, "forward_start_ts": 9119583.602223707, "forward_end_ts": 9119583.605615718, "timestamp": 1790101604.6391351, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.601194521, "recv_end_ts": 9119583.6016186, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.601859516, "preprocess_end_ts": 9119583.602171978, "forward_start_ts": 9119583.60218029, "forward_end_ts": 9119583.60453762, "timestamp": 1790101604.6397457, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.604610441, "send_end_ts": 9119583.606681332} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.60586302, "preprocess_end_ts": 9119583.606205292, "forward_start_ts": 9119583.606213644, "forward_end_ts": 9119583.609991904, "timestamp": 1790101604.643459, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.605360894, "recv_end_ts": 9119583.605715549, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.606932731, "preprocess_end_ts": 9119583.607365588, "forward_start_ts": 9119583.607375856, "forward_end_ts": 9119583.610797776, "timestamp": 1790101604.6443014, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.606361384, "recv_end_ts": 9119583.606772551, "send_start_ts": null, "send_end_ts": null} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/requests.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/requests.jsonl new file mode 100644 index 00000000..be5aaf77 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/requests.jsonl @@ -0,0 +1,76 @@ +{"request_id": "warmup-q0", "burst": "warmup", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9119570.292903332, "finish_monotonic": 9119576.664363926, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "warmup-q1", "burst": "warmup", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9119570.2957572, "finish_monotonic": 9119576.664373389, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "warmup-q2", "burst": "warmup", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9119570.296333695, "finish_monotonic": 9119576.670321444, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "warmup-q3", "burst": "warmup", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9119570.296722282, "finish_monotonic": 9119576.670327768, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q0", "burst": "b8-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9119577.822749784, "finish_monotonic": 9119577.835280377, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q1", "burst": "b8-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9119577.823230114, "finish_monotonic": 9119577.83628296, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q2", "burst": "b8-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9119577.823472217, "finish_monotonic": 9119577.847771548, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q3", "burst": "b8-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9119577.823677383, "finish_monotonic": 9119577.847779376, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q4", "burst": "b8-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9119577.823869893, "finish_monotonic": 9119577.85206842, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q5", "burst": "b8-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9119577.824067151, "finish_monotonic": 9119577.852074748, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q6", "burst": "b8-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9119577.824303448, "finish_monotonic": 9119577.856742447, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q7", "burst": "b8-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9119577.824581333, "finish_monotonic": 9119577.856747039, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q0", "burst": "b8-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9119578.95893912, "finish_monotonic": 9119578.9712066, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q1", "burst": "b8-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9119578.959609669, "finish_monotonic": 9119578.971213512, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q2", "burst": "b8-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9119578.95999268, "finish_monotonic": 9119578.982687123, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q3", "burst": "b8-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9119578.960343532, "finish_monotonic": 9119578.982695224, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q4", "burst": "b8-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9119578.9606352, "finish_monotonic": 9119578.986493504, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q5", "burst": "b8-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9119578.960871626, "finish_monotonic": 9119578.986498725, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q6", "burst": "b8-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9119578.961075956, "finish_monotonic": 9119578.990688741, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q7", "burst": "b8-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9119578.961258136, "finish_monotonic": 9119578.990693668, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q0", "burst": "b8-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9119580.093087101, "finish_monotonic": 9119580.105471551, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q1", "burst": "b8-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9119580.093768604, "finish_monotonic": 9119580.105479572, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q2", "burst": "b8-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9119580.094143052, "finish_monotonic": 9119580.116958952, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q3", "burst": "b8-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9119580.094440961, "finish_monotonic": 9119580.116967266, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q4", "burst": "b8-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9119580.094725056, "finish_monotonic": 9119580.122895185, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q5", "burst": "b8-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9119580.09502366, "finish_monotonic": 9119580.122903796, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q6", "burst": "b8-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9119580.095331525, "finish_monotonic": 9119580.128065204, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q7", "burst": "b8-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9119580.095670745, "finish_monotonic": 9119580.12807052, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q0", "burst": "b16-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9119581.231372556, "finish_monotonic": 9119581.244669287, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q1", "burst": "b16-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9119581.232019192, "finish_monotonic": 9119581.244676992, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q2", "burst": "b16-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9119581.23240773, "finish_monotonic": 9119581.256066673, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q3", "burst": "b16-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9119581.232733017, "finish_monotonic": 9119581.258137325, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q4", "burst": "b16-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9119581.233046625, "finish_monotonic": 9119581.262572609, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q5", "burst": "b16-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9119581.233357828, "finish_monotonic": 9119581.264275284, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q6", "burst": "b16-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9119581.233661748, "finish_monotonic": 9119581.268603608, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q7", "burst": "b16-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9119581.233952317, "finish_monotonic": 9119581.2705439, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q8", "burst": "b16-r0", "round": 0, "index": 8, "rank": 0, "submit_monotonic": 9119581.234164078, "finish_monotonic": 9119581.274798244, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q9", "burst": "b16-r0", "round": 0, "index": 9, "rank": 1, "submit_monotonic": 9119581.234352224, "finish_monotonic": 9119581.276661308, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q10", "burst": "b16-r0", "round": 0, "index": 10, "rank": 0, "submit_monotonic": 9119581.234614696, "finish_monotonic": 9119581.280968891, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q11", "burst": "b16-r0", "round": 0, "index": 11, "rank": 1, "submit_monotonic": 9119581.234914288, "finish_monotonic": 9119581.283019535, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q12", "burst": "b16-r0", "round": 0, "index": 12, "rank": 0, "submit_monotonic": 9119581.235235328, "finish_monotonic": 9119581.28715656, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q13", "burst": "b16-r0", "round": 0, "index": 13, "rank": 1, "submit_monotonic": 9119581.235567328, "finish_monotonic": 9119581.288995408, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q14", "burst": "b16-r0", "round": 0, "index": 14, "rank": 0, "submit_monotonic": 9119581.235901883, "finish_monotonic": 9119581.293288851, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q15", "burst": "b16-r0", "round": 0, "index": 15, "rank": 1, "submit_monotonic": 9119581.23628505, "finish_monotonic": 9119581.294291746, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q0", "burst": "b16-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9119582.397575928, "finish_monotonic": 9119582.409182351, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q1", "burst": "b16-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9119582.398024382, "finish_monotonic": 9119582.410082297, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q2", "burst": "b16-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9119582.398251675, "finish_monotonic": 9119582.420878284, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q3", "burst": "b16-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9119582.398454148, "finish_monotonic": 9119582.421804003, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q4", "burst": "b16-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9119582.398645584, "finish_monotonic": 9119582.426290512, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q5", "burst": "b16-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9119582.398865044, "finish_monotonic": 9119582.4269211, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q6", "burst": "b16-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9119582.399150457, "finish_monotonic": 9119582.431440856, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q7", "burst": "b16-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9119582.399460008, "finish_monotonic": 9119582.432184016, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q8", "burst": "b16-r1", "round": 1, "index": 8, "rank": 0, "submit_monotonic": 9119582.399763105, "finish_monotonic": 9119582.436708044, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q9", "burst": "b16-r1", "round": 1, "index": 9, "rank": 1, "submit_monotonic": 9119582.400090864, "finish_monotonic": 9119582.437363334, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q10", "burst": "b16-r1", "round": 1, "index": 10, "rank": 0, "submit_monotonic": 9119582.400385031, "finish_monotonic": 9119582.441793997, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q11", "burst": "b16-r1", "round": 1, "index": 11, "rank": 1, "submit_monotonic": 9119582.400697349, "finish_monotonic": 9119582.442543592, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q12", "burst": "b16-r1", "round": 1, "index": 12, "rank": 0, "submit_monotonic": 9119582.40096812, "finish_monotonic": 9119582.446937904, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q13", "burst": "b16-r1", "round": 1, "index": 13, "rank": 1, "submit_monotonic": 9119582.40129884, "finish_monotonic": 9119582.4477446, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q14", "burst": "b16-r1", "round": 1, "index": 14, "rank": 0, "submit_monotonic": 9119582.401608504, "finish_monotonic": 9119582.452116895, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q15", "burst": "b16-r1", "round": 1, "index": 15, "rank": 1, "submit_monotonic": 9119582.401885726, "finish_monotonic": 9119582.452971255, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q0", "burst": "b16-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9119583.556243872, "finish_monotonic": 9119583.568614528, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q1", "burst": "b16-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9119583.556679232, "finish_monotonic": 9119583.569503242, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q2", "burst": "b16-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9119583.556910612, "finish_monotonic": 9119583.580152176, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q3", "burst": "b16-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9119583.557116171, "finish_monotonic": 9119583.58094232, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q4", "burst": "b16-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9119583.557302792, "finish_monotonic": 9119583.585108727, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q5", "burst": "b16-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9119583.55747951, "finish_monotonic": 9119583.58618874, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q6", "burst": "b16-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9119583.55769601, "finish_monotonic": 9119583.590519594, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q7", "burst": "b16-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9119583.557939816, "finish_monotonic": 9119583.591250531, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q8", "burst": "b16-r2", "round": 2, "index": 8, "rank": 0, "submit_monotonic": 9119583.558238704, "finish_monotonic": 9119583.595475748, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q9", "burst": "b16-r2", "round": 2, "index": 9, "rank": 1, "submit_monotonic": 9119583.558546081, "finish_monotonic": 9119583.596459309, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q10", "burst": "b16-r2", "round": 2, "index": 10, "rank": 0, "submit_monotonic": 9119583.55885946, "finish_monotonic": 9119583.600849977, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q11", "burst": "b16-r2", "round": 2, "index": 11, "rank": 1, "submit_monotonic": 9119583.55918352, "finish_monotonic": 9119583.601664387, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q12", "burst": "b16-r2", "round": 2, "index": 12, "rank": 0, "submit_monotonic": 9119583.5594886, "finish_monotonic": 9119583.605823291, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q13", "burst": "b16-r2", "round": 2, "index": 13, "rank": 1, "submit_monotonic": 9119583.559807884, "finish_monotonic": 9119583.606727397, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q14", "burst": "b16-r2", "round": 2, "index": 14, "rank": 0, "submit_monotonic": 9119583.5601061, "finish_monotonic": 9119583.611053113, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q15", "burst": "b16-r2", "round": 2, "index": 15, "rank": 1, "submit_monotonic": 9119583.560411915, "finish_monotonic": 9119583.611848287, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/summary.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/summary.json new file mode 100644 index 00000000..b1c8e0d3 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/summary.json @@ -0,0 +1,186 @@ +{ + "model_config": "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/data/config/models/Llama-3.2-1B-Instruct.json", + "num_gpu_blocks": 304854, + "block_size": 16, + "engine_args": { + "model": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", + "served_model_name": null, + "tokenizer": null, + "hf_config_path": null, + "runner": "auto", + "convert": "auto", + "task": null, + "skip_tokenizer_init": true, + "enable_prompt_embeds": false, + "tokenizer_mode": "auto", + "trust_remote_code": false, + "allowed_local_media_path": "", + "download_dir": null, + "safetensors_load_strategy": "lazy", + "load_format": "dummy", + "config_format": "auto", + "dtype": "bfloat16", + "kv_cache_dtype": "auto", + "seed": 0, + "max_model_len": 512, + "distributed_executor_backend": null, + "pipeline_parallel_size": 2, + "tensor_parallel_size": 1, + "decode_context_parallel_size": 1, + "data_parallel_size": 2, + "data_parallel_rank": null, + "data_parallel_start_rank": null, + "data_parallel_size_local": null, + "data_parallel_address": null, + "data_parallel_rpc_port": null, + "data_parallel_hybrid_lb": false, + "data_parallel_backend": "mp", + "enable_expert_parallel": false, + "enable_eplb": false, + "num_redundant_experts": 0, + "eplb_window_size": 1000, + "eplb_step_interval": 3000, + "eplb_log_balancedness": false, + "max_parallel_loading_workers": null, + "block_size": 16, + "enable_prefix_caching": false, + "prefix_caching_hash_algo": "sha256", + "disable_sliding_window": false, + "disable_cascade_attn": false, + "swap_space": 4, + "cpu_offload_gb": 0, + "gpu_memory_utilization": 0.5, + "kv_cache_memory_bytes": null, + "max_num_batched_tokens": 256, + "max_num_partial_prefills": 1, + "max_long_partial_prefills": 1, + "long_prefill_token_threshold": 0, + "max_num_seqs": 4, + "max_logprobs": 20, + "disable_log_stats": true, + "revision": null, + "code_revision": null, + "rope_theta": null, + "hf_token": null, + "tokenizer_revision": null, + "quantization": null, + "enforce_eager": true, + "max_seq_len_to_capture": 8192, + "disable_custom_all_reduce": false, + "interleave_mm_strings": false, + "mm_processor_kwargs": null, + "disable_mm_preprocessor_cache": false, + "mm_processor_cache_gb": 4, + "mm_encoder_tp_mode": "weights", + "io_processor_plugin": null, + "skip_mm_profiling": false, + "enable_lora": false, + "enable_lora_bias": false, + "max_loras": 1, + "max_lora_rank": 16, + "default_mm_loras": null, + "fully_sharded_loras": false, + "max_cpu_loras": null, + "lora_dtype": "auto", + "lora_extra_vocab_size": 256, + "ray_workers_use_nsight": false, + "num_gpu_blocks_override": null, + "num_lookahead_slots": 0, + "ignore_patterns": null, + "preemption_mode": null, + "scheduler_delay_factor": 0.0, + "enable_chunked_prefill": true, + "disable_chunked_mm_input": false, + "disable_hybrid_kv_cache_manager": false, + "guided_decoding_backend": "auto", + "guided_decoding_disable_fallback": false, + "guided_decoding_disable_any_whitespace": false, + "guided_decoding_disable_additional_properties": false, + "logits_processor_pattern": null, + "speculative_config": null, + "show_hidden_metrics_for_version": null, + "otlp_traces_endpoint": null, + "collect_detailed_traces": null, + "disable_async_output_proc": false, + "scheduling_policy": "fcfs", + "scheduler_cls": "vllm.v1.core.sched.scheduler.Scheduler", + "override_pooler_config": null, + "worker_cls": "auto", + "worker_extension_cls": "", + "kv_transfer_config": null, + "kv_events_config": null, + "generation_config": "auto", + "enable_sleep_mode": false, + "model_impl": "auto", + "override_attention_dtype": null, + "calculate_kv_scales": false, + "mamba_cache_dtype": "auto", + "mamba_ssm_cache_dtype": "auto", + "reasoning_parser": "", + "use_tqdm_on_load": true, + "pt_load_map_location": "cpu", + "enable_multimodal_encoder_data_parallel": false, + "logits_processors": null, + "async_scheduling": false, + "kv_sharing_fast_prefill": false, + "enable_log_requests": false + }, + "rounds": [ + { + "label": "warmup", + "round": 0, + "num_requests": 4, + "wall_minus_monotonic_before": 1780982021.0330582, + "wall_minus_monotonic_after": 1780982021.0330582, + "idle_wait_s": 1.1522713880985975 + }, + { + "label": "b8-r0", + "round": 0, + "num_requests": 8, + "wall_minus_monotonic_before": 1780982021.0330582, + "wall_minus_monotonic_after": 1780982021.033058, + "idle_wait_s": 1.1019583977758884 + }, + { + "label": "b8-r1", + "round": 1, + "num_requests": 8, + "wall_minus_monotonic_before": 1780982021.0330582, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.1021581627428532 + }, + { + "label": "b8-r2", + "round": 2, + "num_requests": 8, + "wall_minus_monotonic_before": 1780982021.0330584, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.1030645035207272 + }, + { + "label": "b16-r0", + "round": 0, + "num_requests": 16, + "wall_minus_monotonic_before": 1780982021.0330584, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.1031487435102463 + }, + { + "label": "b16-r1", + "round": 1, + "num_requests": 16, + "wall_minus_monotonic_before": 1780982021.0330584, + "wall_minus_monotonic_after": 1780982021.0330582, + "idle_wait_s": 1.1031373273581266 + }, + { + "label": "b16-r2", + "round": 2, + "num_requests": 16, + "wall_minus_monotonic_before": 1780982021.0330586, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.1024576723575592 + } + ] +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/moe/model/config.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/moe/model/config.json new file mode 100644 index 00000000..9b897034 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/moe/model/config.json @@ -0,0 +1,95 @@ +{ + "_comment": "Qwen3-30B-A3B-tiny: Modified Qwen3-30B-A3B for MoE profiling testing (reduced layers and experts)", + "vocab_size": 151936, + "max_position_embeddings": 40960, + "hidden_size": 2048, + "intermediate_size": 6144, + "num_hidden_layers": 8, + "num_attention_heads": 32, + "use_sliding_window": false, + "sliding_window": null, + "num_key_value_heads": 4, + "hidden_act": "silu", + "initializer_range": 0.02, + "rms_norm_eps": 1e-06, + "use_cache": true, + "rope_theta": 1000000.0, + "rope_scaling": null, + "attention_bias": false, + "attention_dropout": 0.0, + "decoder_sparse_step": 1, + "moe_intermediate_size": 768, + "num_experts_per_tok": 8, + "num_experts": 16, + "norm_topk_prob": true, + "output_router_logits": false, + "router_aux_loss_coef": 0.001, + "mlp_only_layers": [], + "return_dict": true, + "output_hidden_states": false, + "torchscript": false, + "dtype": "bfloat16", + "pruned_heads": {}, + "tie_word_embeddings": false, + "chunk_size_feed_forward": 0, + "is_encoder_decoder": false, + "is_decoder": false, + "cross_attention_hidden_size": null, + "add_cross_attention": false, + "tie_encoder_decoder": false, + "architectures": [ + "Qwen3MoeForCausalLM" + ], + "finetuning_task": null, + "id2label": { + "0": "LABEL_0", + "1": "LABEL_1" + }, + "label2id": { + "LABEL_0": 0, + "LABEL_1": 1 + }, + "task_specific_params": null, + "problem_type": null, + "tokenizer_class": null, + "prefix": null, + "bos_token_id": 151643, + "pad_token_id": null, + "eos_token_id": 151645, + "sep_token_id": null, + "decoder_start_token_id": null, + "max_length": 20, + "min_length": 0, + "do_sample": false, + "early_stopping": false, + "num_beams": 1, + "temperature": 1.0, + "top_k": 50, + "top_p": 1.0, + "typical_p": 1.0, + "repetition_penalty": 1.0, + "length_penalty": 1.0, + "no_repeat_ngram_size": 0, + "encoder_no_repeat_ngram_size": 0, + "bad_words_ids": null, + "num_return_sequences": 1, + "output_scores": false, + "return_dict_in_generate": false, + "forced_bos_token_id": null, + "forced_eos_token_id": null, + "remove_invalid_values": false, + "exponential_decay_length_penalty": null, + "suppress_tokens": null, + "begin_suppress_tokens": null, + "num_beam_groups": 1, + "diversity_penalty": 0.0, + "_name_or_path": "Qwen/Qwen3-30B-A3B", + "transformers_version": "4.57.3", + "head_dim": 128, + "max_window_layers": 48, + "model_type": "qwen3_moe", + "tf_legacy_loss": false, + "use_bfloat16": false, + "output_attentions": false, + "use_qk_norm": true +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/vllm_import.txt b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/vllm_import.txt new file mode 100644 index 00000000..6aa03d4f --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/vllm_import.txt @@ -0,0 +1 @@ +VLLM_IMPORT 0.10.2 /tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/__init__.py /tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/frontier_trace.py diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/worker_env.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/worker_env.json new file mode 100644 index 00000000..00b7f124 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/worker_env.json @@ -0,0 +1 @@ +{"python": "3.12.11", "torch": "2.8.0+cu128", "cuda_available": true, "device_count": 4, "devices": ["NVIDIA H800", "NVIDIA H800", "NVIDIA H800", "NVIDIA H800"]} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/COMPLETE b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/COMPLETE new file mode 100644 index 00000000..bbfcfc41 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/COMPLETE @@ -0,0 +1 @@ +status=0 diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/overlay_report.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/overlay_report.json new file mode 100644 index 00000000..f3a11b19 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/overlay_report.json @@ -0,0 +1,142 @@ +{ + "site_vllm": "/usr/local/lib/python3.12/dist-packages/vllm", + "checkout": "/data/ycfeng/Frontier/.real-engine/vLLM-BS", + "overlay": "/tmp/stage_admission_pp/sa-pp-20260923b/overlay/vllm", + "differing_py_files": [ + "vllm/_C.py", + "vllm/_custom_ops.py", + "vllm/_moe_C.py", + "vllm/attention/layer.py", + "vllm/benchmarks/throughput.py", + "vllm/compilation/compiler_interface.py", + "vllm/config/__init__.py", + "vllm/distributed/communication_op.py", + "vllm/distributed/kv_transfer/kv_connector/v1/base.py", + "vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/frontier_kv_transfer_logger.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", + "vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py", + "vllm/distributed/parallel_state.py", + "vllm/engine/arg_utils.py", + "vllm/engine/llm_engine.py", + "vllm/entrypoints/openai/frontier_request_metrics.py", + "vllm/entrypoints/openai/serving_chat.py", + "vllm/entrypoints/openai/serving_completion.py", + "vllm/entrypoints/openai/serving_engine.py", + "vllm/envs.py", + "vllm/model_executor/custom_op.py", + "vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", + "vllm/model_executor/layers/fused_moe/deepep_ll_prepare_finalize.py", + "vllm/model_executor/layers/fused_moe/fused_moe.py", + "vllm/model_executor/layers/fused_moe/layer.py", + "vllm/model_executor/layers/fused_moe/pplx_prepare_finalize.py", + "vllm/model_executor/layers/linear.py", + "vllm/model_executor/layers/vocab_parallel_embedding.py", + "vllm/model_executor/models/llama.py", + "vllm/model_executor/models/phimoe.py", + "vllm/model_executor/models/qwen3_moe.py", + "vllm/model_executor/models/qwen3_moe_mtp.py", + "vllm/model_executor/models/registry.py", + "vllm/request_generator/__init__.py", + "vllm/request_generator/config.py", + "vllm/request_generator/kv_sync.py", + "vllm/request_generator/prompt_generator.py", + "vllm/request_generator/vllm_request_generator.py", + "vllm/v1/attention/backends/flash_attn.py", + "vllm/v1/attention/backends/flashinfer.py", + "vllm/v1/attention/backends/mla/common.py", + "vllm/v1/attention/backends/mla/flashinfer_mla.py", + "vllm/v1/attention/backends/utils.py", + "vllm/v1/core/sched/scheduler.py", + "vllm/v1/engine/coordinator.py", + "vllm/v1/engine/core.py", + "vllm/v1/engine/core_client.py", + "vllm/v1/engine/output_processor.py", + "vllm/v1/engine/processor.py", + "vllm/v1/frontier_trace.py", + "vllm/v1/metrics/stats.py", + "vllm/v1/spec_decode/eagle.py", + "vllm/v1/utils.py", + "vllm/v1/worker/gpu_model_runner.py", + "vllm/v1/worker/gpu_worker.py", + "vllm/worker/model_runner.py", + "vllm/worker/worker.py" + ], + "expected_py_changes": [ + "vllm/_C.py", + "vllm/_custom_ops.py", + "vllm/_moe_C.py", + "vllm/attention/layer.py", + "vllm/benchmarks/throughput.py", + "vllm/compilation/compiler_interface.py", + "vllm/config/__init__.py", + "vllm/distributed/communication_op.py", + "vllm/distributed/kv_transfer/kv_connector/v1/base.py", + "vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/frontier_kv_transfer_logger.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", + "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", + "vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py", + "vllm/distributed/parallel_state.py", + "vllm/engine/arg_utils.py", + "vllm/engine/llm_engine.py", + "vllm/entrypoints/openai/frontier_request_metrics.py", + "vllm/entrypoints/openai/serving_chat.py", + "vllm/entrypoints/openai/serving_completion.py", + "vllm/entrypoints/openai/serving_engine.py", + "vllm/envs.py", + "vllm/model_executor/custom_op.py", + "vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", + "vllm/model_executor/layers/fused_moe/deepep_ll_prepare_finalize.py", + "vllm/model_executor/layers/fused_moe/fused_moe.py", + "vllm/model_executor/layers/fused_moe/layer.py", + "vllm/model_executor/layers/fused_moe/pplx_prepare_finalize.py", + "vllm/model_executor/layers/linear.py", + "vllm/model_executor/layers/vocab_parallel_embedding.py", + "vllm/model_executor/models/llama.py", + "vllm/model_executor/models/phimoe.py", + "vllm/model_executor/models/qwen3_moe.py", + "vllm/model_executor/models/qwen3_moe_mtp.py", + "vllm/model_executor/models/registry.py", + "vllm/request_generator/__init__.py", + "vllm/request_generator/config.py", + "vllm/request_generator/kv_sync.py", + "vllm/request_generator/prompt_generator.py", + "vllm/request_generator/vllm_request_generator.py", + "vllm/v1/attention/backends/flash_attn.py", + "vllm/v1/attention/backends/flashinfer.py", + "vllm/v1/attention/backends/mla/common.py", + "vllm/v1/attention/backends/mla/flashinfer_mla.py", + "vllm/v1/attention/backends/utils.py", + "vllm/v1/core/sched/scheduler.py", + "vllm/v1/engine/coordinator.py", + "vllm/v1/engine/core.py", + "vllm/v1/engine/core_client.py", + "vllm/v1/engine/output_processor.py", + "vllm/v1/engine/processor.py", + "vllm/v1/frontier_trace.py", + "vllm/v1/metrics/stats.py", + "vllm/v1/spec_decode/eagle.py", + "vllm/v1/utils.py", + "vllm/v1/worker/gpu_model_runner.py", + "vllm/v1/worker/gpu_worker.py", + "vllm/worker/model_runner.py", + "vllm/worker/worker.py" + ], + "unexpected": [], + "missing": [], + "accepted": true, + "patch": { + "path": "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch", + "sha256": "8d47678911b7b689bc644ea81bd9d2200ea296d773ee9c07a1c5a9bc9b3a9c81", + "files": [ + "vllm/_custom_ops.py", + "vllm/model_executor/layers/fused_moe/fused_moe.py" + ], + "equal_to_image_after_patch": { + "vllm/_custom_ops.py": true, + "vllm/model_executor/layers/fused_moe/fused_moe.py": false + } + } +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/replica_log.txt b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/replica_log.txt new file mode 100644 index 00000000..32857af2 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/replica_log.txt @@ -0,0 +1,14 @@ +# Replica log tail of RJob exp-0923-024146-345158 (codesign, H800 x4, creator i-fengyicheng). +# Platform init lines (node addresses, NCCL interface settings) are removed; workload output is verbatim. +{"python": "3.12.11", "torch": "2.8.0+cu128", "cuda_available": true, "device_count": 4, "devices": ["NVIDIA H800", "NVIDIA H800", "NVIDIA H800", "NVIDIA H800"]} +{"accepted": true, "unexpected": [], "missing": []} +/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. + import pynvml # type: ignore[import] +VLLM_IMPORT 0.10.2 /tmp/stage_admission_pp/sa-pp-20260923b/overlay/vllm/__init__.py /tmp/stage_admission_pp/sa-pp-20260923b/overlay/vllm/v1/frontier_trace.py +SCENARIO_PASS moe +SCENARIO_PASS dense +moe DRIVER_DONE {"rounds": 7, "num_gpu_blocks": 600666} +152 /tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/pp_boundary.jsonl +dense DRIVER_DONE {"rounds": 7, "num_gpu_blocks": 304854} +152 /tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/pp_boundary.jsonl +WORKER_STATUS=0 RUN_TAG=sa-pp-20260923b diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_844.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_844.jsonl new file mode 100644 index 00000000..c04d76a6 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_844.jsonl @@ -0,0 +1,22 @@ +{"kind": "frontend_snapshot", "pid": 844, "seq": 0, "monotonic": 9120624.62208326, "snapshot": 2, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 1, "monotonic": 9120626.021095267, "snapshot": 3, "counts": [[1, 1], [1, 1]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 2, "monotonic": 9120626.18612816, "snapshot": 4, "counts": [[0, 1], [0, 1]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 3, "monotonic": 9120626.285689717, "snapshot": 5, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 4, "monotonic": 9120627.350581912, "snapshot": 6, "counts": [[0, 1], [0, 1]], "wave": 1, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 5, "monotonic": 9120627.45029127, "snapshot": 7, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 844, "seq": 6, "monotonic": 9120627.550534572, "snapshot": 8, "counts": [[0, 0], [0, 0]], "wave": 2, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 844, "seq": 7, "monotonic": 9120628.489587313, "snapshot": 9, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 8, "monotonic": 9120628.589672543, "snapshot": 10, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 9, "monotonic": 9120628.689554924, "snapshot": 11, "counts": [[0, 0], [0, 0]], "wave": 3, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 844, "seq": 10, "monotonic": 9120629.62920076, "snapshot": 12, "counts": [[0, 1], [0, 1]], "wave": 3, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 11, "monotonic": 9120629.729838043, "snapshot": 13, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 844, "seq": 12, "monotonic": 9120629.83007456, "snapshot": 14, "counts": [[0, 0], [0, 0]], "wave": 4, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 844, "seq": 13, "monotonic": 9120630.766499234, "snapshot": 15, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 14, "monotonic": 9120630.866801092, "snapshot": 16, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 844, "seq": 15, "monotonic": 9120630.967000738, "snapshot": 17, "counts": [[0, 0], [0, 0]], "wave": 5, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 844, "seq": 16, "monotonic": 9120631.86492882, "snapshot": 18, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 17, "monotonic": 9120631.965198906, "snapshot": 19, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 18, "monotonic": 9120632.065105148, "snapshot": 20, "counts": [[0, 0], [0, 0]], "wave": 6, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 844, "seq": 19, "monotonic": 9120633.018878696, "snapshot": 21, "counts": [[0, 1], [0, 1]], "wave": 6, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 844, "seq": 20, "monotonic": 9120633.118211748, "snapshot": 22, "counts": [[0, 1], [0, 1]], "wave": 7, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 844, "seq": 21, "monotonic": 9120633.218477592, "snapshot": 23, "counts": [[0, 0], [0, 0]], "wave": 7, "engines_running": false} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_919.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_919.jsonl new file mode 100644 index 00000000..d5cba488 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_919.jsonl @@ -0,0 +1,65 @@ +{"kind": "engine_iteration", "pid": 919, "seq": 0, "monotonic": 9120625.96791757, "engine": 0, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 1, "monotonic": 9120626.180983996, "engine": 0, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 2, "monotonic": 9120626.18561802, "engine": 0, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 3, "monotonic": 9120626.189165356, "engine": 0, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 4, "monotonic": 9120626.191278495, "engine": 0, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 5, "monotonic": 9120627.338840736, "engine": 0, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 6, "monotonic": 9120627.35003926, "engine": 0, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 7, "monotonic": 9120627.352656804, "engine": 0, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 8, "monotonic": 9120627.362038797, "engine": 0, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 9, "monotonic": 9120627.368717212, "engine": 0, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 10, "monotonic": 9120627.37527868, "engine": 0, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 11, "monotonic": 9120627.378601518, "engine": 0, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 12, "monotonic": 9120627.380811714, "engine": 0, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 13, "monotonic": 9120628.47956062, "engine": 0, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 14, "monotonic": 9120628.4902294, "engine": 0, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 15, "monotonic": 9120628.493471997, "engine": 0, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 16, "monotonic": 9120628.502409011, "engine": 0, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 17, "monotonic": 9120628.507723143, "engine": 0, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 18, "monotonic": 9120628.513031827, "engine": 0, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 19, "monotonic": 9120628.516112473, "engine": 0, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 20, "monotonic": 9120628.519086625, "engine": 0, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 21, "monotonic": 9120629.616881263, "engine": 0, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 22, "monotonic": 9120629.62966153, "engine": 0, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 23, "monotonic": 9120629.632997084, "engine": 0, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 24, "monotonic": 9120629.642491208, "engine": 0, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 25, "monotonic": 9120629.647711592, "engine": 0, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 26, "monotonic": 9120629.652978007, "engine": 0, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 27, "monotonic": 9120629.656049391, "engine": 0, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 28, "monotonic": 9120629.658983247, "engine": 0, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 29, "monotonic": 9120630.756324528, "engine": 0, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 30, "monotonic": 9120630.765886173, "engine": 0, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 31, "monotonic": 9120630.7685595, "engine": 0, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 32, "monotonic": 9120630.776217248, "engine": 0, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 33, "monotonic": 9120630.780365208, "engine": 0, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 34, "monotonic": 9120630.784431065, "engine": 0, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q8"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 35, "monotonic": 9120630.788409028, "engine": 0, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q10"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 36, "monotonic": 9120630.79243354, "engine": 0, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q12"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 37, "monotonic": 9120630.796628807, "engine": 0, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q14"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 38, "monotonic": 9120630.800747246, "engine": 0, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 39, "monotonic": 9120630.803301813, "engine": 0, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 40, "monotonic": 9120630.80553382, "engine": 0, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 41, "monotonic": 9120631.854623005, "engine": 0, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 42, "monotonic": 9120631.864230804, "engine": 0, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 43, "monotonic": 9120631.866910912, "engine": 0, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 44, "monotonic": 9120631.87423912, "engine": 0, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 45, "monotonic": 9120631.87832478, "engine": 0, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 46, "monotonic": 9120631.882377015, "engine": 0, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q8"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 47, "monotonic": 9120631.888180677, "engine": 0, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q10"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 48, "monotonic": 9120631.89380424, "engine": 0, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q12"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 49, "monotonic": 9120631.899139255, "engine": 0, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q14"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 50, "monotonic": 9120631.90448134, "engine": 0, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 51, "monotonic": 9120631.906846043, "engine": 0, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 52, "monotonic": 9120631.910034545, "engine": 0, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 53, "monotonic": 9120633.008425832, "engine": 0, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 54, "monotonic": 9120633.019671384, "engine": 0, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 55, "monotonic": 9120633.022307867, "engine": 0, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 56, "monotonic": 9120633.029821588, "engine": 0, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 57, "monotonic": 9120633.03408219, "engine": 0, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 58, "monotonic": 9120633.038265277, "engine": 0, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q8"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 59, "monotonic": 9120633.042499, "engine": 0, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q10"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 60, "monotonic": 9120633.049114825, "engine": 0, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q12"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 61, "monotonic": 9120633.053355124, "engine": 0, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q14"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 919, "seq": 62, "monotonic": 9120633.057866, "engine": 0, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 63, "monotonic": 9120633.060146462, "engine": 0, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 919, "seq": 64, "monotonic": 9120633.063857228, "engine": 0, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_920.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_920.jsonl new file mode 100644 index 00000000..8318fd1b --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_920.jsonl @@ -0,0 +1,65 @@ +{"kind": "engine_iteration", "pid": 920, "seq": 0, "monotonic": 9120625.970102616, "engine": 1, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 1, "monotonic": 9120626.180614032, "engine": 1, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 2, "monotonic": 9120626.185660796, "engine": 1, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 3, "monotonic": 9120626.18911583, "engine": 1, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 4, "monotonic": 9120626.19126469, "engine": 1, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 5, "monotonic": 9120627.338931331, "engine": 1, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 6, "monotonic": 9120627.350064572, "engine": 1, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 7, "monotonic": 9120627.354388159, "engine": 1, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 8, "monotonic": 9120627.364628045, "engine": 1, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 9, "monotonic": 9120627.371056518, "engine": 1, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 10, "monotonic": 9120627.376390046, "engine": 1, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 11, "monotonic": 9120627.378577605, "engine": 1, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 12, "monotonic": 9120627.380741952, "engine": 1, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 13, "monotonic": 9120628.479725206, "engine": 1, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 14, "monotonic": 9120628.489064472, "engine": 1, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 15, "monotonic": 9120628.492970873, "engine": 1, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 16, "monotonic": 9120628.501319665, "engine": 1, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 17, "monotonic": 9120628.506610187, "engine": 1, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 18, "monotonic": 9120628.511955587, "engine": 1, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 19, "monotonic": 9120628.51537628, "engine": 1, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 20, "monotonic": 9120628.518301548, "engine": 1, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 21, "monotonic": 9120629.617205078, "engine": 1, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 22, "monotonic": 9120629.628686544, "engine": 1, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 23, "monotonic": 9120629.632389316, "engine": 1, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 24, "monotonic": 9120629.641484171, "engine": 1, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 25, "monotonic": 9120629.646681543, "engine": 1, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 26, "monotonic": 9120629.651882049, "engine": 1, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 27, "monotonic": 9120629.65544816, "engine": 1, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 28, "monotonic": 9120629.658302251, "engine": 1, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 29, "monotonic": 9120630.756586129, "engine": 1, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 30, "monotonic": 9120630.765814532, "engine": 1, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 31, "monotonic": 9120630.768750964, "engine": 1, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 32, "monotonic": 9120630.7762622, "engine": 1, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 33, "monotonic": 9120630.78039026, "engine": 1, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 34, "monotonic": 9120630.78437535, "engine": 1, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q9"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 35, "monotonic": 9120630.788444983, "engine": 1, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q11"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 36, "monotonic": 9120630.7924497, "engine": 1, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q13"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 37, "monotonic": 9120630.796446582, "engine": 1, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q15"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 38, "monotonic": 9120630.800872909, "engine": 1, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 39, "monotonic": 9120630.80312862, "engine": 1, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 40, "monotonic": 9120630.8055922, "engine": 1, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 41, "monotonic": 9120631.854982505, "engine": 1, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 42, "monotonic": 9120631.864391916, "engine": 1, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 43, "monotonic": 9120631.867039468, "engine": 1, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 44, "monotonic": 9120631.874338027, "engine": 1, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 45, "monotonic": 9120631.87846945, "engine": 1, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 46, "monotonic": 9120631.882465012, "engine": 1, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q9"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 47, "monotonic": 9120631.886601208, "engine": 1, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q11"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 48, "monotonic": 9120631.892491497, "engine": 1, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q13"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 49, "monotonic": 9120631.897972204, "engine": 1, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q15"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 50, "monotonic": 9120631.903372144, "engine": 1, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 51, "monotonic": 9120631.90681644, "engine": 1, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 52, "monotonic": 9120631.90912707, "engine": 1, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 53, "monotonic": 9120633.008795032, "engine": 1, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 54, "monotonic": 9120633.018359303, "engine": 1, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 55, "monotonic": 9120633.02237447, "engine": 1, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 56, "monotonic": 9120633.029993236, "engine": 1, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 57, "monotonic": 9120633.034136882, "engine": 1, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 58, "monotonic": 9120633.038214054, "engine": 1, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q9"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 59, "monotonic": 9120633.044925317, "engine": 1, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q11"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 60, "monotonic": 9120633.049206803, "engine": 1, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q13"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 61, "monotonic": 9120633.053342188, "engine": 1, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q15"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 920, "seq": 62, "monotonic": 9120633.057768637, "engine": 1, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 63, "monotonic": 9120633.061733505, "engine": 1, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 920, "seq": 64, "monotonic": 9120633.064055149, "engine": 1, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/model/config.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/model/config.json new file mode 100644 index 00000000..e9b20534 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/model/config.json @@ -0,0 +1,95 @@ +{ + "vocab_size": 128256, + "max_position_embeddings": 131072, + "hidden_size": 2048, + "intermediate_size": 8192, + "num_hidden_layers": 16, + "num_attention_heads": 32, + "num_key_value_heads": 8, + "hidden_act": "silu", + "initializer_range": 0.02, + "rms_norm_eps": 1e-05, + "pretraining_tp": 1, + "use_cache": true, + "rope_theta": 500000.0, + "rope_scaling": { + "factor": 32.0, + "high_freq_factor": 4.0, + "low_freq_factor": 1.0, + "original_max_position_embeddings": 8192, + "rope_type": "llama3" + }, + "attention_bias": false, + "attention_dropout": 0.0, + "mlp_bias": false, + "head_dim": 64, + "return_dict": true, + "output_hidden_states": false, + "torchscript": false, + "dtype": "bfloat16", + "torch_dtype": "bfloat16", + "pruned_heads": {}, + "tie_word_embeddings": true, + "chunk_size_feed_forward": 0, + "is_encoder_decoder": false, + "is_decoder": false, + "cross_attention_hidden_size": null, + "add_cross_attention": false, + "tie_encoder_decoder": false, + "architectures": [ + "LlamaForCausalLM" + ], + "finetuning_task": null, + "id2label": { + "0": "LABEL_0", + "1": "LABEL_1" + }, + "label2id": { + "LABEL_0": 0, + "LABEL_1": 1 + }, + "task_specific_params": null, + "problem_type": null, + "tokenizer_class": null, + "prefix": null, + "bos_token_id": 128000, + "pad_token_id": null, + "eos_token_id": [ + 128001, + 128008, + 128009 + ], + "sep_token_id": null, + "decoder_start_token_id": null, + "max_length": 20, + "min_length": 0, + "do_sample": false, + "early_stopping": false, + "num_beams": 1, + "temperature": 1.0, + "top_k": 50, + "top_p": 1.0, + "typical_p": 1.0, + "repetition_penalty": 1.0, + "length_penalty": 1.0, + "no_repeat_ngram_size": 0, + "encoder_no_repeat_ngram_size": 0, + "bad_words_ids": null, + "num_return_sequences": 1, + "output_scores": false, + "return_dict_in_generate": false, + "forced_bos_token_id": null, + "forced_eos_token_id": null, + "remove_invalid_values": false, + "exponential_decay_length_penalty": null, + "suppress_tokens": null, + "begin_suppress_tokens": null, + "num_beam_groups": 1, + "diversity_penalty": 0.0, + "_name_or_path": "meta-llama/Llama-3.2-1B-Instruct", + "transformers_version": "4.57.3", + "model_type": "llama", + "tf_legacy_loss": false, + "use_bfloat16": false, + "output_attentions": false +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/pp_boundary.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/pp_boundary.jsonl new file mode 100644 index 00000000..0b955342 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/pp_boundary.jsonl @@ -0,0 +1,152 @@ +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120625.968609303, "preprocess_end_ts": 9120625.97053415, "forward_start_ts": 9120625.970763369, "forward_end_ts": 9120625.980689062, "timestamp": 1790102647.1819882, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120625.980833208, "send_end_ts": 9120626.148909405} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120625.970645007, "preprocess_end_ts": 9120625.972136268, "forward_start_ts": 9120625.972334806, "forward_end_ts": 9120625.980693843, "timestamp": 1790102647.186707, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120625.98082598, "send_end_ts": 9120626.153632233} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120626.154382523, "preprocess_end_ts": 9120626.155663108, "forward_start_ts": 9120626.155793374, "forward_end_ts": 9120626.16366869, "timestamp": 1790102647.2130804, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120625.970227623, "recv_end_ts": 9120626.154076628, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120626.149046848, "preprocess_end_ts": 9120626.15055543, "forward_start_ts": 9120626.150740927, "forward_end_ts": 9120626.164248995, "timestamp": 1790102647.2134495, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120625.968175467, "recv_end_ts": 9120626.148630315, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120626.154215973, "preprocess_end_ts": 9120626.154697996, "forward_start_ts": 9120626.154713957, "forward_end_ts": 9120626.157916266, "timestamp": 1790102647.2138479, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120626.15802846, "send_end_ts": 9120626.180778168} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120626.149568712, "preprocess_end_ts": 9120626.150059056, "forward_start_ts": 9120626.150077468, "forward_end_ts": 9120626.158014348, "timestamp": 1790102647.2142947, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120626.158127574, "send_end_ts": 9120626.181224626} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120626.181134202, "preprocess_end_ts": 9120626.181580093, "forward_start_ts": 9120626.1815955, "forward_end_ts": 9120626.184787473, "timestamp": 1790102647.2182894, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120626.180380125, "recv_end_ts": 9120626.18090347, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120626.181630112, "preprocess_end_ts": 9120626.182067346, "forward_start_ts": 9120626.18208268, "forward_end_ts": 9120626.184812859, "timestamp": 1790102647.2183256, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120626.18079424, "recv_end_ts": 9120626.181401048, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.339258784, "preprocess_end_ts": 9120627.339764776, "forward_start_ts": 9120627.339779196, "forward_end_ts": 9120627.342864996, "timestamp": 1790102648.376574, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.342970168, "send_end_ts": 9120627.343508072} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.339179536, "preprocess_end_ts": 9120627.33978918, "forward_start_ts": 9120627.339804886, "forward_end_ts": 9120627.342916146, "timestamp": 1790102648.3766828, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.343029773, "send_end_ts": 9120627.343614984} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.344056806, "preprocess_end_ts": 9120627.344689254, "forward_start_ts": 9120627.344704311, "forward_end_ts": 9120627.348888468, "timestamp": 1790102648.3826468, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.33892894, "recv_end_ts": 9120627.343743356, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.344097933, "preprocess_end_ts": 9120627.344858024, "forward_start_ts": 9120627.344889121, "forward_end_ts": 9120627.349071456, "timestamp": 1790102648.3826966, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.339267079, "recv_end_ts": 9120627.343679167, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.354537774, "preprocess_end_ts": 9120627.354850387, "forward_start_ts": 9120627.35486039, "forward_end_ts": 9120627.357288308, "timestamp": 1790102648.390863, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.357378667, "send_end_ts": 9120627.357797865} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.354125634, "preprocess_end_ts": 9120627.354576396, "forward_start_ts": 9120627.354588164, "forward_end_ts": 9120627.357361272, "timestamp": 1790102648.390979, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.357461236, "send_end_ts": 9120627.357911903} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.358045336, "preprocess_end_ts": 9120627.358420543, "forward_start_ts": 9120627.358431606, "forward_end_ts": 9120627.361312632, "timestamp": 1790102648.3947983, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.35269102, "recv_end_ts": 9120627.35790194, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.358344212, "preprocess_end_ts": 9120627.358774984, "forward_start_ts": 9120627.358787036, "forward_end_ts": 9120627.36326022, "timestamp": 1790102648.396899, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.363368012, "send_end_ts": 9120627.363832705} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.35815126, "preprocess_end_ts": 9120627.35879615, "forward_start_ts": 9120627.358808013, "forward_end_ts": 9120627.36377378, "timestamp": 1790102648.3973336, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.35445616, "recv_end_ts": 9120627.35796816, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.358132698, "preprocess_end_ts": 9120627.358457213, "forward_start_ts": 9120627.358466452, "forward_end_ts": 9120627.361300139, "timestamp": 1790102648.3980007, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.361378863, "send_end_ts": 9120627.364935782} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.36394886, "preprocess_end_ts": 9120627.364315215, "forward_start_ts": 9120627.364325054, "forward_end_ts": 9120627.368024694, "timestamp": 1790102648.4014964, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.361983083, "recv_end_ts": 9120627.36381411, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.365165556, "preprocess_end_ts": 9120627.365513904, "forward_start_ts": 9120627.365523864, "forward_end_ts": 9120627.370255912, "timestamp": 1790102648.4037986, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.364590388, "recv_end_ts": 9120627.365023067, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.364283687, "preprocess_end_ts": 9120627.364721015, "forward_start_ts": 9120627.364732655, "forward_end_ts": 9120627.37016959, "timestamp": 1790102648.403803, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.370273894, "send_end_ts": 9120627.370736608} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.36527259, "preprocess_end_ts": 9120627.365592506, "forward_start_ts": 9120627.365601173, "forward_end_ts": 9120627.368131137, "timestamp": 1790102648.404424, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.368207466, "send_end_ts": 9120627.371359697} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.370907487, "preprocess_end_ts": 9120627.371254314, "forward_start_ts": 9120627.37126446, "forward_end_ts": 9120627.374588244, "timestamp": 1790102648.4080584, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.368684333, "recv_end_ts": 9120627.370726237, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.3716548, "preprocess_end_ts": 9120627.37212815, "forward_start_ts": 9120627.3721393, "forward_end_ts": 9120627.375675013, "timestamp": 1790102648.4091454, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.371039772, "recv_end_ts": 9120627.371491624, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.480072675, "preprocess_end_ts": 9120628.480590845, "forward_start_ts": 9120628.48060594, "forward_end_ts": 9120628.48369596, "timestamp": 1790102649.5173385, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.483807502, "send_end_ts": 9120628.48427284} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.479916755, "preprocess_end_ts": 9120628.480465902, "forward_start_ts": 9120628.480479784, "forward_end_ts": 9120628.48361477, "timestamp": 1790102649.5173392, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.483726276, "send_end_ts": 9120628.484273072} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.484656258, "preprocess_end_ts": 9120628.485159677, "forward_start_ts": 9120628.485177867, "forward_end_ts": 9120628.488222428, "timestamp": 1790102649.5217633, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.47985394, "recv_end_ts": 9120628.484396309, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.484680768, "preprocess_end_ts": 9120628.48525294, "forward_start_ts": 9120628.485268185, "forward_end_ts": 9120628.489254864, "timestamp": 1790102649.522823, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.479656564, "recv_end_ts": 9120628.484431809, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.49364094, "preprocess_end_ts": 9120628.494016288, "forward_start_ts": 9120628.494027358, "forward_end_ts": 9120628.496669484, "timestamp": 1790102649.5301967, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.496759748, "send_end_ts": 9120628.497131933} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.493183425, "preprocess_end_ts": 9120628.493563408, "forward_start_ts": 9120628.493574992, "forward_end_ts": 9120628.49667986, "timestamp": 1790102649.5302074, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.496780356, "send_end_ts": 9120628.497143047} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.497340364, "preprocess_end_ts": 9120628.49771906, "forward_start_ts": 9120628.497730436, "forward_end_ts": 9120628.500560196, "timestamp": 1790102649.5340424, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.492983911, "recv_end_ts": 9120628.4971987, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.497483142, "preprocess_end_ts": 9120628.497825736, "forward_start_ts": 9120628.49783498, "forward_end_ts": 9120628.500394216, "timestamp": 1790102649.5346537, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.500477197, "send_end_ts": 9120628.501588022} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.497430827, "preprocess_end_ts": 9120628.497908764, "forward_start_ts": 9120628.49792198, "forward_end_ts": 9120628.501545314, "timestamp": 1790102649.5350845, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.4935184, "recv_end_ts": 9120628.497251464, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.497464197, "preprocess_end_ts": 9120628.497778434, "forward_start_ts": 9120628.49778684, "forward_end_ts": 9120628.500323832, "timestamp": 1790102649.535703, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.500402601, "send_end_ts": 9120628.502637304} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.50173274, "preprocess_end_ts": 9120628.502162267, "forward_start_ts": 9120628.502172364, "forward_end_ts": 9120628.505890612, "timestamp": 1790102649.5393727, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.50123235, "recv_end_ts": 9120628.501591649, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.501935309, "preprocess_end_ts": 9120628.502257, "forward_start_ts": 9120628.502265956, "forward_end_ts": 9120628.505892256, "timestamp": 1790102649.5399516, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.505970024, "send_end_ts": 9120628.506887212} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.502906872, "preprocess_end_ts": 9120628.503386345, "forward_start_ts": 9120628.503397947, "forward_end_ts": 9120628.506884232, "timestamp": 1790102649.5404081, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.502334353, "recv_end_ts": 9120628.502734972, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.50298448, "preprocess_end_ts": 9120628.503298933, "forward_start_ts": 9120628.503308216, "forward_end_ts": 9120628.50581078, "timestamp": 1790102649.5410109, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.505886668, "send_end_ts": 9120628.507945262} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.50709637, "preprocess_end_ts": 9120628.507446185, "forward_start_ts": 9120628.507455656, "forward_end_ts": 9120628.511246327, "timestamp": 1790102649.5447104, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.506540043, "recv_end_ts": 9120628.50692816, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.508232096, "preprocess_end_ts": 9120628.508691614, "forward_start_ts": 9120628.508702967, "forward_end_ts": 9120628.51224555, "timestamp": 1790102649.545758, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.507640127, "recv_end_ts": 9120628.508053012, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.617235024, "preprocess_end_ts": 9120629.617765486, "forward_start_ts": 9120629.617780456, "forward_end_ts": 9120629.62183598, "timestamp": 1790102650.6555102, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.62194472, "send_end_ts": 9120629.62244408} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.617889555, "preprocess_end_ts": 9120629.618670886, "forward_start_ts": 9120629.61870918, "forward_end_ts": 9120629.623213852, "timestamp": 1790102650.6569853, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.623365313, "send_end_ts": 9120629.62391574} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.624280203, "preprocess_end_ts": 9120629.624772113, "forward_start_ts": 9120629.624787886, "forward_end_ts": 9120629.627719384, "timestamp": 1790102650.6612575, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.617380893, "recv_end_ts": 9120629.624033524, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.622827092, "preprocess_end_ts": 9120629.623382833, "forward_start_ts": 9120629.623396477, "forward_end_ts": 9120629.628656372, "timestamp": 1790102650.6622326, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.616972368, "recv_end_ts": 9120629.62258668, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.633139703, "preprocess_end_ts": 9120629.633528728, "forward_start_ts": 9120629.63354104, "forward_end_ts": 9120629.636019705, "timestamp": 1790102650.6695786, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.63611123, "send_end_ts": 9120629.63651369} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.632777745, "preprocess_end_ts": 9120629.633213265, "forward_start_ts": 9120629.633225307, "forward_end_ts": 9120629.637058595, "timestamp": 1790102650.670677, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.637166584, "send_end_ts": 9120629.637611376} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.637793612, "preprocess_end_ts": 9120629.63816856, "forward_start_ts": 9120629.638179317, "forward_end_ts": 9120629.640670473, "timestamp": 1790102650.6741621, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.632422255, "recv_end_ts": 9120629.63764468, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.636803849, "preprocess_end_ts": 9120629.637284296, "forward_start_ts": 9120629.6372973, "forward_end_ts": 9120629.641628912, "timestamp": 1790102650.6751661, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.63300281, "recv_end_ts": 9120629.63662434, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.638032388, "preprocess_end_ts": 9120629.638448147, "forward_start_ts": 9120629.63845889, "forward_end_ts": 9120629.641865157, "timestamp": 1790102650.6754727, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.641960869, "send_end_ts": 9120629.642407075} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.636854624, "preprocess_end_ts": 9120629.637174541, "forward_start_ts": 9120629.637183553, "forward_end_ts": 9120629.64094268, "timestamp": 1790102650.67577, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.641025716, "send_end_ts": 9120629.642704451} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.642587366, "preprocess_end_ts": 9120629.643015442, "forward_start_ts": 9120629.64302696, "forward_end_ts": 9120629.645904802, "timestamp": 1790102650.679381, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.641346332, "recv_end_ts": 9120629.642447297, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.642962169, "preprocess_end_ts": 9120629.643422045, "forward_start_ts": 9120629.64343342, "forward_end_ts": 9120629.646858996, "timestamp": 1790102650.6803827, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.64240166, "recv_end_ts": 9120629.642798074, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.642824026, "preprocess_end_ts": 9120629.643243168, "forward_start_ts": 9120629.643254532, "forward_end_ts": 9120629.646820208, "timestamp": 1790102650.6804247, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.646915665, "send_end_ts": 9120629.647358976} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.643039608, "preprocess_end_ts": 9120629.643353136, "forward_start_ts": 9120629.643361958, "forward_end_ts": 9120629.645853216, "timestamp": 1790102650.6810222, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.645927992, "send_end_ts": 9120629.647957291} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.647541158, "preprocess_end_ts": 9120629.647900688, "forward_start_ts": 9120629.64791027, "forward_end_ts": 9120629.651127145, "timestamp": 1790102650.6845968, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.64656526, "recv_end_ts": 9120629.647406936, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.648230966, "preprocess_end_ts": 9120629.648682158, "forward_start_ts": 9120629.648692459, "forward_end_ts": 9120629.652173012, "timestamp": 1790102650.6856806, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.647617482, "recv_end_ts": 9120629.648054553, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.756728144, "preprocess_end_ts": 9120630.757250639, "forward_start_ts": 9120630.75726482, "forward_end_ts": 9120630.760731801, "timestamp": 1790102651.794396, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.760846628, "send_end_ts": 9120630.761330284} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.757091507, "preprocess_end_ts": 9120630.757696709, "forward_start_ts": 9120630.757711885, "forward_end_ts": 9120630.76082835, "timestamp": 1790102651.7944815, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.760945473, "send_end_ts": 9120630.761415496} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.761716532, "preprocess_end_ts": 9120630.762154792, "forward_start_ts": 9120630.762168357, "forward_end_ts": 9120630.764944864, "timestamp": 1790102651.7984564, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.756694091, "recv_end_ts": 9120630.761507204, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.761652008, "preprocess_end_ts": 9120630.762110276, "forward_start_ts": 9120630.762123177, "forward_end_ts": 9120630.765052913, "timestamp": 1790102651.7985692, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.756420648, "recv_end_ts": 9120630.761439895, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.76871565, "preprocess_end_ts": 9120630.769057296, "forward_start_ts": 9120630.769067828, "forward_end_ts": 9120630.771755356, "timestamp": 1790102651.8052824, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.77184165, "send_end_ts": 9120630.772216788} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.768905576, "preprocess_end_ts": 9120630.769261034, "forward_start_ts": 9120630.7692735, "forward_end_ts": 9120630.771849427, "timestamp": 1790102651.8054075, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.771936612, "send_end_ts": 9120630.772342429} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.772571083, "preprocess_end_ts": 9120630.772931213, "forward_start_ts": 9120630.77294206, "forward_end_ts": 9120630.775472678, "timestamp": 1790102651.8089557, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.768785909, "recv_end_ts": 9120630.772411592, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.772432297, "preprocess_end_ts": 9120630.772792043, "forward_start_ts": 9120630.772801967, "forward_end_ts": 9120630.775467785, "timestamp": 1790102651.8089523, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.76860636, "recv_end_ts": 9120630.772278393, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.772544274, "preprocess_end_ts": 9120630.772870028, "forward_start_ts": 9120630.772879288, "forward_end_ts": 9120630.775547277, "timestamp": 1790102651.8095398, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.775630755, "send_end_ts": 9120630.776473988} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.772706363, "preprocess_end_ts": 9120630.77307342, "forward_start_ts": 9120630.773083623, "forward_end_ts": 9120630.7756091, "timestamp": 1790102651.809545, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.775689287, "send_end_ts": 9120630.776480244} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.776726035, "preprocess_end_ts": 9120630.777065463, "forward_start_ts": 9120630.777075876, "forward_end_ts": 9120630.779607631, "timestamp": 1790102651.8130927, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.77616563, "recv_end_ts": 9120630.776593987, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.776674774, "preprocess_end_ts": 9120630.777029077, "forward_start_ts": 9120630.777038317, "forward_end_ts": 9120630.779634157, "timestamp": 1790102651.8131168, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.776156606, "recv_end_ts": 9120630.776536943, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.77682408, "preprocess_end_ts": 9120630.777144736, "forward_start_ts": 9120630.777153444, "forward_end_ts": 9120630.779752117, "timestamp": 1790102651.81366, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.779828325, "send_end_ts": 9120630.780594528} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.776808364, "preprocess_end_ts": 9120630.777129512, "forward_start_ts": 9120630.777138796, "forward_end_ts": 9120630.779630387, "timestamp": 1790102651.8136709, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.779704709, "send_end_ts": 9120630.78060522} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.780836565, "preprocess_end_ts": 9120630.781162376, "forward_start_ts": 9120630.78117098, "forward_end_ts": 9120630.78364742, "timestamp": 1790102651.8171222, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.780266693, "recv_end_ts": 9120630.780703856, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.780801186, "preprocess_end_ts": 9120630.781138828, "forward_start_ts": 9120630.781147344, "forward_end_ts": 9120630.783756781, "timestamp": 1790102651.8172321, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.780290967, "recv_end_ts": 9120630.780661805, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.780920591, "preprocess_end_ts": 9120630.781235833, "forward_start_ts": 9120630.781244667, "forward_end_ts": 9120630.783854818, "timestamp": 1790102651.8177342, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.783930436, "send_end_ts": 9120630.78466948} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.78094024, "preprocess_end_ts": 9120630.781254206, "forward_start_ts": 9120630.781264195, "forward_end_ts": 9120630.783766752, "timestamp": 1790102651.8177555, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.783840356, "send_end_ts": 9120630.784690293} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.78488378, "preprocess_end_ts": 9120630.785230393, "forward_start_ts": 9120630.785238812, "forward_end_ts": 9120630.787740665, "timestamp": 1790102651.8211982, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.784309767, "recv_end_ts": 9120630.784752062, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.784940084, "preprocess_end_ts": 9120630.78528827, "forward_start_ts": 9120630.785297353, "forward_end_ts": 9120630.787742209, "timestamp": 1790102651.8212004, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.784412878, "recv_end_ts": 9120630.78479814, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.785016892, "preprocess_end_ts": 9120630.785338355, "forward_start_ts": 9120630.785348654, "forward_end_ts": 9120630.787786445, "timestamp": 1790102651.8217356, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.787860107, "send_end_ts": 9120630.788669506} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.78500206, "preprocess_end_ts": 9120630.785324523, "forward_start_ts": 9120630.78533306, "forward_end_ts": 9120630.787884472, "timestamp": 1790102651.821771, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.787958836, "send_end_ts": 9120630.788706576} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.788912522, "preprocess_end_ts": 9120630.789256187, "forward_start_ts": 9120630.789264748, "forward_end_ts": 9120630.791709485, "timestamp": 1790102651.8251615, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.788366668, "recv_end_ts": 9120630.78875686, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.788930383, "preprocess_end_ts": 9120630.789263789, "forward_start_ts": 9120630.789272215, "forward_end_ts": 9120630.791742334, "timestamp": 1790102651.8252108, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.78837028, "recv_end_ts": 9120630.78879814, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.78899092, "preprocess_end_ts": 9120630.789308038, "forward_start_ts": 9120630.789316673, "forward_end_ts": 9120630.791789923, "timestamp": 1790102651.82569, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.791863224, "send_end_ts": 9120630.792625263} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.789033458, "preprocess_end_ts": 9120630.789348353, "forward_start_ts": 9120630.78935704, "forward_end_ts": 9120630.791831916, "timestamp": 1790102651.8257744, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.79190594, "send_end_ts": 9120630.792709809} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.792934729, "preprocess_end_ts": 9120630.793263268, "forward_start_ts": 9120630.793271432, "forward_end_ts": 9120630.795742974, "timestamp": 1790102651.82921, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.792379472, "recv_end_ts": 9120630.792802677, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.792865507, "preprocess_end_ts": 9120630.793207586, "forward_start_ts": 9120630.793215957, "forward_end_ts": 9120630.795875024, "timestamp": 1790102651.8293643, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.792329932, "recv_end_ts": 9120630.792715462, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.793027883, "preprocess_end_ts": 9120630.793340776, "forward_start_ts": 9120630.79334936, "forward_end_ts": 9120630.795806972, "timestamp": 1790102651.8297832, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.795879817, "send_end_ts": 9120630.79671858} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.79295564, "preprocess_end_ts": 9120630.793271327, "forward_start_ts": 9120630.793279773, "forward_end_ts": 9120630.795758668, "timestamp": 1790102651.8299458, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.795833847, "send_end_ts": 9120630.796881536} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.797101848, "preprocess_end_ts": 9120630.797448287, "forward_start_ts": 9120630.797457129, "forward_end_ts": 9120630.799931336, "timestamp": 1790102651.8334818, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.796561232, "recv_end_ts": 9120630.796963716, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.796984555, "preprocess_end_ts": 9120630.797322156, "forward_start_ts": 9120630.797330884, "forward_end_ts": 9120630.800157882, "timestamp": 1790102651.8336236, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.79638398, "recv_end_ts": 9120630.796833448, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.855282338, "preprocess_end_ts": 9120631.855767962, "forward_start_ts": 9120631.855781684, "forward_end_ts": 9120631.858707948, "timestamp": 1790102652.8924625, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.858814804, "send_end_ts": 9120631.859396908} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.854982182, "preprocess_end_ts": 9120631.855524568, "forward_start_ts": 9120631.855538204, "forward_end_ts": 9120631.858833825, "timestamp": 1790102652.892465, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.858942743, "send_end_ts": 9120631.859399797} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.85971038, "preprocess_end_ts": 9120631.86018892, "forward_start_ts": 9120631.86020302, "forward_end_ts": 9120631.86336512, "timestamp": 1790102652.896879, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.85471212, "recv_end_ts": 9120631.859493257, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.859894043, "preprocess_end_ts": 9120631.860471463, "forward_start_ts": 9120631.86049034, "forward_end_ts": 9120631.863395637, "timestamp": 1790102652.896951, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.855249437, "recv_end_ts": 9120631.85961009, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.867067054, "preprocess_end_ts": 9120631.8673978, "forward_start_ts": 9120631.867408087, "forward_end_ts": 9120631.869902749, "timestamp": 1790102652.9034307, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.86998835, "send_end_ts": 9120631.870366186} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.867175924, "preprocess_end_ts": 9120631.867486104, "forward_start_ts": 9120631.867495667, "forward_end_ts": 9120631.869951831, "timestamp": 1790102652.9034617, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.870038316, "send_end_ts": 9120631.87039729} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.870574372, "preprocess_end_ts": 9120631.870930824, "forward_start_ts": 9120631.87094088, "forward_end_ts": 9120631.873476159, "timestamp": 1790102652.9069595, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.866951348, "recv_end_ts": 9120631.87042774, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.870622411, "preprocess_end_ts": 9120631.87098984, "forward_start_ts": 9120631.871000435, "forward_end_ts": 9120631.873544944, "timestamp": 1790102652.9070256, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.86708166, "recv_end_ts": 9120631.870477917, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.8706924, "preprocess_end_ts": 9120631.871004255, "forward_start_ts": 9120631.87101292, "forward_end_ts": 9120631.873535637, "timestamp": 1790102652.9075143, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.873615613, "send_end_ts": 9120631.874449918} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.87072502, "preprocess_end_ts": 9120631.871036137, "forward_start_ts": 9120631.871044472, "forward_end_ts": 9120631.873580053, "timestamp": 1790102652.9075842, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.873659866, "send_end_ts": 9120631.874519236} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.874674369, "preprocess_end_ts": 9120631.875019744, "forward_start_ts": 9120631.87502963, "forward_end_ts": 9120631.877606152, "timestamp": 1790102652.9110744, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.874144344, "recv_end_ts": 9120631.874530405, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.874781609, "preprocess_end_ts": 9120631.875126224, "forward_start_ts": 9120631.875135522, "forward_end_ts": 9120631.877671083, "timestamp": 1790102652.9111435, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.874217808, "recv_end_ts": 9120631.87465144, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.874851827, "preprocess_end_ts": 9120631.875212194, "forward_start_ts": 9120631.875220904, "forward_end_ts": 9120631.87776462, "timestamp": 1790102652.9116647, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.877839718, "send_end_ts": 9120631.878598472} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.874852149, "preprocess_end_ts": 9120631.875159081, "forward_start_ts": 9120631.875167888, "forward_end_ts": 9120631.877867987, "timestamp": 1790102652.9117095, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.877956215, "send_end_ts": 9120631.87864424} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.878800401, "preprocess_end_ts": 9120631.879158728, "forward_start_ts": 9120631.879168024, "forward_end_ts": 9120631.881685087, "timestamp": 1790102652.915156, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.878245916, "recv_end_ts": 9120631.878645755, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.878901208, "preprocess_end_ts": 9120631.879240762, "forward_start_ts": 9120631.87924943, "forward_end_ts": 9120631.881735718, "timestamp": 1790102652.9152062, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.878322909, "recv_end_ts": 9120631.8787669, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.878932904, "preprocess_end_ts": 9120631.879247816, "forward_start_ts": 9120631.879257228, "forward_end_ts": 9120631.881905263, "timestamp": 1790102652.915742, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.881980268, "send_end_ts": 9120631.882677028} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.878995804, "preprocess_end_ts": 9120631.87943838, "forward_start_ts": 9120631.879449736, "forward_end_ts": 9120631.881915934, "timestamp": 1790102652.9157922, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.881994404, "send_end_ts": 9120631.882726965} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.882963836, "preprocess_end_ts": 9120631.883310096, "forward_start_ts": 9120631.883319356, "forward_end_ts": 9120631.885828119, "timestamp": 1790102652.9193022, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.882381609, "recv_end_ts": 9120631.882829992, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.883066133, "preprocess_end_ts": 9120631.883382568, "forward_start_ts": 9120631.883393092, "forward_end_ts": 9120631.88592888, "timestamp": 1790102652.9198632, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.88600756, "send_end_ts": 9120631.886799075} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.8829099, "preprocess_end_ts": 9120631.883262, "forward_start_ts": 9120631.88327184, "forward_end_ts": 9120631.887270372, "timestamp": 1790102652.920824, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.882331923, "recv_end_ts": 9120631.882742755, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.88300686, "preprocess_end_ts": 9120631.88332234, "forward_start_ts": 9120631.883331155, "forward_end_ts": 9120631.8858272, "timestamp": 1790102652.9214623, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.885903863, "send_end_ts": 9120631.888396887} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.88706456, "preprocess_end_ts": 9120631.887404244, "forward_start_ts": 9120631.887412826, "forward_end_ts": 9120631.891734196, "timestamp": 1790102652.9252076, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.886480557, "recv_end_ts": 9120631.886926131, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.887137188, "preprocess_end_ts": 9120631.88746548, "forward_start_ts": 9120631.887475893, "forward_end_ts": 9120631.891716296, "timestamp": 1790102652.9257696, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.891795672, "send_end_ts": 9120631.892704656} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.888714574, "preprocess_end_ts": 9120631.889211528, "forward_start_ts": 9120631.889223566, "forward_end_ts": 9120631.892954983, "timestamp": 1790102652.9264822, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.888097525, "recv_end_ts": 9120631.888519298, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.888762549, "preprocess_end_ts": 9120631.889092516, "forward_start_ts": 9120631.889102204, "forward_end_ts": 9120631.891640492, "timestamp": 1790102652.92707, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.891717687, "send_end_ts": 9120631.894005127} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.892962996, "preprocess_end_ts": 9120631.893312791, "forward_start_ts": 9120631.893321082, "forward_end_ts": 9120631.897233916, "timestamp": 1790102652.930702, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.892387312, "recv_end_ts": 9120631.892826892, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.89305972, "preprocess_end_ts": 9120631.893376531, "forward_start_ts": 9120631.893385472, "forward_end_ts": 9120631.897246273, "timestamp": 1790102652.9312675, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.897326227, "send_end_ts": 9120631.898203159} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.894286577, "preprocess_end_ts": 9120631.894758344, "forward_start_ts": 9120631.894769112, "forward_end_ts": 9120631.898311712, "timestamp": 1790102652.9318323, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.893723847, "recv_end_ts": 9120631.894104771, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.894357976, "preprocess_end_ts": 9120631.894676585, "forward_start_ts": 9120631.8946855, "forward_end_ts": 9120631.897187008, "timestamp": 1790102652.9324157, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.897261884, "send_end_ts": 9120631.899351345} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.898422357, "preprocess_end_ts": 9120631.89876118, "forward_start_ts": 9120631.898769883, "forward_end_ts": 9120631.902630253, "timestamp": 1790102652.9360933, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.897881148, "recv_end_ts": 9120631.898289489, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.899629433, "preprocess_end_ts": 9120631.900082408, "forward_start_ts": 9120631.900092972, "forward_end_ts": 9120631.903661069, "timestamp": 1790102652.9371703, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.89906192, "recv_end_ts": 9120631.899461376, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.00912462, "preprocess_end_ts": 9120633.009628447, "forward_start_ts": 9120633.009642484, "forward_end_ts": 9120633.012897952, "timestamp": 1790102654.046612, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.013004847, "send_end_ts": 9120633.013546484} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.009092316, "preprocess_end_ts": 9120633.009735657, "forward_start_ts": 9120633.0097653, "forward_end_ts": 9120633.013069568, "timestamp": 1790102654.0467336, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.01319092, "send_end_ts": 9120633.013664292} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.013865676, "preprocess_end_ts": 9120633.014310487, "forward_start_ts": 9120633.014323035, "forward_end_ts": 9120633.017461952, "timestamp": 1790102654.050983, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.008865193, "recv_end_ts": 9120633.013653051, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.014084011, "preprocess_end_ts": 9120633.014665373, "forward_start_ts": 9120633.014681084, "forward_end_ts": 9120633.01861633, "timestamp": 1790102654.0522833, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.008523732, "recv_end_ts": 9120633.013835423, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.022484371, "preprocess_end_ts": 9120633.022825, "forward_start_ts": 9120633.022836223, "forward_end_ts": 9120633.02539309, "timestamp": 1790102654.0589721, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.025478587, "send_end_ts": 9120633.025907433} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.022576105, "preprocess_end_ts": 9120633.022932714, "forward_start_ts": 9120633.0229432, "forward_end_ts": 9120633.025504356, "timestamp": 1790102654.0590165, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.02559088, "send_end_ts": 9120633.025951508} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.026155168, "preprocess_end_ts": 9120633.026519824, "forward_start_ts": 9120633.0265308, "forward_end_ts": 9120633.0290985, "timestamp": 1790102654.0625827, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.022371592, "recv_end_ts": 9120633.02599124, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.026150355, "preprocess_end_ts": 9120633.026512844, "forward_start_ts": 9120633.026522849, "forward_end_ts": 9120633.02926104, "timestamp": 1790102654.0627458, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.022381136, "recv_end_ts": 9120633.026005764, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.026241688, "preprocess_end_ts": 9120633.026547017, "forward_start_ts": 9120633.02655548, "forward_end_ts": 9120633.029101554, "timestamp": 1790102654.0631752, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.029178979, "send_end_ts": 9120633.030110529} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.026288347, "preprocess_end_ts": 9120633.026598904, "forward_start_ts": 9120633.026607355, "forward_end_ts": 9120633.029218858, "timestamp": 1790102654.0632923, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.029298743, "send_end_ts": 9120633.030227378} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.030323096, "preprocess_end_ts": 9120633.030673208, "forward_start_ts": 9120633.030682083, "forward_end_ts": 9120633.033343269, "timestamp": 1790102654.0668237, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.029770184, "recv_end_ts": 9120633.03016674, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.030455012, "preprocess_end_ts": 9120633.030802865, "forward_start_ts": 9120633.030814027, "forward_end_ts": 9120633.033471644, "timestamp": 1790102654.0669456, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.029932916, "recv_end_ts": 9120633.030308735, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.030438948, "preprocess_end_ts": 9120633.030747175, "forward_start_ts": 9120633.03075576, "forward_end_ts": 9120633.03336456, "timestamp": 1790102654.0674243, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.033438902, "send_end_ts": 9120633.034359409} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.030559953, "preprocess_end_ts": 9120633.03087182, "forward_start_ts": 9120633.030880343, "forward_end_ts": 9120633.03336346, "timestamp": 1790102654.0674803, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.033438088, "send_end_ts": 9120633.034415359} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.03465004, "preprocess_end_ts": 9120633.03498366, "forward_start_ts": 9120633.03499256, "forward_end_ts": 9120633.037458057, "timestamp": 1790102654.0709217, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.03410994, "recv_end_ts": 9120633.034518484, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.03459124, "preprocess_end_ts": 9120633.034921931, "forward_start_ts": 9120633.034930164, "forward_end_ts": 9120633.037582994, "timestamp": 1790102654.0710557, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.034022192, "recv_end_ts": 9120633.034451906, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.034739578, "preprocess_end_ts": 9120633.035048965, "forward_start_ts": 9120633.035057332, "forward_end_ts": 9120633.037575962, "timestamp": 1790102654.07146, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.037650784, "send_end_ts": 9120633.038394252} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.034679515, "preprocess_end_ts": 9120633.034986053, "forward_start_ts": 9120633.034994485, "forward_end_ts": 9120633.037597232, "timestamp": 1790102654.0717065, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.037673317, "send_end_ts": 9120633.038641788} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.038858248, "preprocess_end_ts": 9120633.039213931, "forward_start_ts": 9120633.039223202, "forward_end_ts": 9120633.041816534, "timestamp": 1790102654.0752876, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.038227772, "recv_end_ts": 9120633.038721519, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.038574662, "preprocess_end_ts": 9120633.0389408, "forward_start_ts": 9120633.038949773, "forward_end_ts": 9120633.044092435, "timestamp": 1790102654.0776374, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.03808926, "recv_end_ts": 9120633.03842598, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.038994944, "preprocess_end_ts": 9120633.039339826, "forward_start_ts": 9120633.039349278, "forward_end_ts": 9120633.044071345, "timestamp": 1790102654.0777137, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.044175034, "send_end_ts": 9120633.04464744} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.038739506, "preprocess_end_ts": 9120633.039062949, "forward_start_ts": 9120633.039071944, "forward_end_ts": 9120633.041868236, "timestamp": 1790102654.078329, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.041944608, "send_end_ts": 9120633.04526319} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.044770325, "preprocess_end_ts": 9120633.045113716, "forward_start_ts": 9120633.045122268, "forward_end_ts": 9120633.048448041, "timestamp": 1790102654.0819178, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.042455737, "recv_end_ts": 9120633.044629203, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.045538375, "preprocess_end_ts": 9120633.045897225, "forward_start_ts": 9120633.045906844, "forward_end_ts": 9120633.04848914, "timestamp": 1790102654.0819652, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.044870075, "recv_end_ts": 9120633.045367245, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.045021882, "preprocess_end_ts": 9120633.045462469, "forward_start_ts": 9120633.045473741, "forward_end_ts": 9120633.048544453, "timestamp": 1790102654.0824502, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.04862099, "send_end_ts": 9120633.0493857} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.045603164, "preprocess_end_ts": 9120633.045917977, "forward_start_ts": 9120633.04592652, "forward_end_ts": 9120633.04844103, "timestamp": 1790102654.0825758, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.048513452, "send_end_ts": 9120633.049511343} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.049668692, "preprocess_end_ts": 9120633.050129907, "forward_start_ts": 9120633.050141344, "forward_end_ts": 9120633.052645463, "timestamp": 1790102654.0861177, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.049130924, "recv_end_ts": 9120633.049530424, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.049608896, "preprocess_end_ts": 9120633.049943777, "forward_start_ts": 9120633.049952548, "forward_end_ts": 9120633.052688764, "timestamp": 1790102654.086164, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.049085557, "recv_end_ts": 9120633.049453532, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.049826369, "preprocess_end_ts": 9120633.05013533, "forward_start_ts": 9120633.050143477, "forward_end_ts": 9120633.052648343, "timestamp": 1790102654.086692, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.052720623, "send_end_ts": 9120633.053626252} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.049700836, "preprocess_end_ts": 9120633.050009804, "forward_start_ts": 9120633.050018644, "forward_end_ts": 9120633.052618377, "timestamp": 1790102654.0867481, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.052692603, "send_end_ts": 9120633.053681875} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.053981254, "preprocess_end_ts": 9120633.054455565, "forward_start_ts": 9120633.054466892, "forward_end_ts": 9120633.057019856, "timestamp": 1790102654.0904927, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.05328541, "recv_end_ts": 9120633.0537842, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.053851355, "preprocess_end_ts": 9120633.054199465, "forward_start_ts": 9120633.054208232, "forward_end_ts": 9120633.057175398, "timestamp": 1790102654.0906513, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.05333501, "recv_end_ts": 9120633.053714588, "send_start_ts": null, "send_end_ts": null} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/requests.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/requests.jsonl new file mode 100644 index 00000000..a32e0ab6 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/requests.jsonl @@ -0,0 +1,76 @@ +{"request_id": "warmup-q0", "burst": "warmup", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120619.693064611, "finish_monotonic": 9120626.1815256, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "warmup-q1", "burst": "warmup", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120619.695883848, "finish_monotonic": 9120626.181534523, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "warmup-q2", "burst": "warmup", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120619.69638608, "finish_monotonic": 9120626.185938066, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "warmup-q3", "burst": "warmup", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120619.696744837, "finish_monotonic": 9120626.186357344, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q0", "burst": "b8-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120627.33764526, "finish_monotonic": 9120627.350517169, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q1", "burst": "b8-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120627.338142376, "finish_monotonic": 9120627.35052416, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q2", "burst": "b8-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120627.338400908, "finish_monotonic": 9120627.3626592, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q3", "burst": "b8-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120627.3387301, "finish_monotonic": 9120627.365025744, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q4", "burst": "b8-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9120627.33905247, "finish_monotonic": 9120627.369012823, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q5", "burst": "b8-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9120627.339336585, "finish_monotonic": 9120627.37133308, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q6", "burst": "b8-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9120627.339580867, "finish_monotonic": 9120627.375533052, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q7", "burst": "b8-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9120627.339853652, "finish_monotonic": 9120627.376606883, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q0", "burst": "b8-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9120628.478456767, "finish_monotonic": 9120628.490733864, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q1", "burst": "b8-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9120628.478942016, "finish_monotonic": 9120628.490742048, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q2", "burst": "b8-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9120628.479174543, "finish_monotonic": 9120628.502861716, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q3", "burst": "b8-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9120628.479384871, "finish_monotonic": 9120628.502868343, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q4", "burst": "b8-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9120628.4796049, "finish_monotonic": 9120628.508141924, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q5", "burst": "b8-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9120628.479847856, "finish_monotonic": 9120628.508147068, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q6", "burst": "b8-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9120628.480164504, "finish_monotonic": 9120628.513450736, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q7", "burst": "b8-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9120628.480518768, "finish_monotonic": 9120628.51345778, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q0", "burst": "b8-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9120629.615764981, "finish_monotonic": 9120629.630154692, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q1", "burst": "b8-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9120629.616310723, "finish_monotonic": 9120629.6301628, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q2", "burst": "b8-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9120629.616553552, "finish_monotonic": 9120629.642988376, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q3", "burst": "b8-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9120629.616755284, "finish_monotonic": 9120629.642997924, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q4", "burst": "b8-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9120629.61694975, "finish_monotonic": 9120629.648127552, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q5", "burst": "b8-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9120629.617157232, "finish_monotonic": 9120629.648135507, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q6", "burst": "b8-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9120629.617386969, "finish_monotonic": 9120629.653310588, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q7", "burst": "b8-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9120629.617586398, "finish_monotonic": 9120629.653318169, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q0", "burst": "b16-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120630.755274627, "finish_monotonic": 9120630.766395576, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q1", "burst": "b16-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120630.755743297, "finish_monotonic": 9120630.766404217, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q2", "burst": "b16-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120630.7559694, "finish_monotonic": 9120630.776710898, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q3", "burst": "b16-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120630.756184224, "finish_monotonic": 9120630.777303033, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q4", "burst": "b16-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9120630.756377677, "finish_monotonic": 9120630.780823816, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q5", "burst": "b16-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9120630.756564396, "finish_monotonic": 9120630.780830188, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q6", "burst": "b16-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9120630.756747931, "finish_monotonic": 9120630.78486779, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q7", "burst": "b16-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9120630.756918237, "finish_monotonic": 9120630.784872321, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q8", "burst": "b16-r0", "round": 0, "index": 8, "rank": 0, "submit_monotonic": 9120630.757167717, "finish_monotonic": 9120630.788809026, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q9", "burst": "b16-r0", "round": 0, "index": 9, "rank": 1, "submit_monotonic": 9120630.757407872, "finish_monotonic": 9120630.788894637, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q10", "burst": "b16-r0", "round": 0, "index": 10, "rank": 0, "submit_monotonic": 9120630.75773546, "finish_monotonic": 9120630.792928468, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q11", "burst": "b16-r0", "round": 0, "index": 11, "rank": 1, "submit_monotonic": 9120630.758017642, "finish_monotonic": 9120630.7929341, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q12", "burst": "b16-r0", "round": 0, "index": 12, "rank": 0, "submit_monotonic": 9120630.758321553, "finish_monotonic": 9120630.79972709, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q13", "burst": "b16-r0", "round": 0, "index": 13, "rank": 1, "submit_monotonic": 9120630.758614428, "finish_monotonic": 9120630.799734142, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q14", "burst": "b16-r0", "round": 0, "index": 14, "rank": 0, "submit_monotonic": 9120630.75890563, "finish_monotonic": 9120630.801104853, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q15", "burst": "b16-r0", "round": 0, "index": 15, "rank": 1, "submit_monotonic": 9120630.759184089, "finish_monotonic": 9120630.801354105, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q0", "burst": "b16-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9120631.853388796, "finish_monotonic": 9120631.864818912, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q1", "burst": "b16-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9120631.854072178, "finish_monotonic": 9120631.864826111, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q2", "burst": "b16-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9120631.854456872, "finish_monotonic": 9120631.87491754, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q3", "burst": "b16-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9120631.854810825, "finish_monotonic": 9120631.874924596, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q4", "burst": "b16-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9120631.855116468, "finish_monotonic": 9120631.878823621, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q5", "burst": "b16-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9120631.85540741, "finish_monotonic": 9120631.878926156, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q6", "burst": "b16-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9120631.855626736, "finish_monotonic": 9120631.883009087, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q7", "burst": "b16-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9120631.855811985, "finish_monotonic": 9120631.88301812, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q8", "burst": "b16-r1", "round": 1, "index": 8, "rank": 0, "submit_monotonic": 9120631.855988791, "finish_monotonic": 9120631.888617085, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q9", "burst": "b16-r1", "round": 1, "index": 9, "rank": 1, "submit_monotonic": 9120631.856176008, "finish_monotonic": 9120631.888627496, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q10", "burst": "b16-r1", "round": 1, "index": 10, "rank": 0, "submit_monotonic": 9120631.856351316, "finish_monotonic": 9120631.894228024, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q11", "burst": "b16-r1", "round": 1, "index": 11, "rank": 1, "submit_monotonic": 9120631.856521145, "finish_monotonic": 9120631.894245336, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q12", "burst": "b16-r1", "round": 1, "index": 12, "rank": 0, "submit_monotonic": 9120631.856704928, "finish_monotonic": 9120631.899613872, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q13", "burst": "b16-r1", "round": 1, "index": 13, "rank": 1, "submit_monotonic": 9120631.856886072, "finish_monotonic": 9120631.899623835, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q14", "burst": "b16-r1", "round": 1, "index": 14, "rank": 0, "submit_monotonic": 9120631.857065404, "finish_monotonic": 9120631.904759064, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q15", "burst": "b16-r1", "round": 1, "index": 15, "rank": 1, "submit_monotonic": 9120631.857236683, "finish_monotonic": 9120631.90476597, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q0", "burst": "b16-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9120633.007311763, "finish_monotonic": 9120633.019998591, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q1", "burst": "b16-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9120633.007788422, "finish_monotonic": 9120633.02001751, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q2", "burst": "b16-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9120633.008083, "finish_monotonic": 9120633.03034932, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q3", "burst": "b16-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9120633.00830188, "finish_monotonic": 9120633.030445123, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q4", "burst": "b16-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9120633.008509327, "finish_monotonic": 9120633.034537788, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q5", "burst": "b16-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9120633.008716583, "finish_monotonic": 9120633.034543687, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q6", "burst": "b16-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9120633.00893061, "finish_monotonic": 9120633.038714863, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q7", "burst": "b16-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9120633.009198193, "finish_monotonic": 9120633.03872045, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q8", "burst": "b16-r2", "round": 2, "index": 8, "rank": 0, "submit_monotonic": 9120633.009473508, "finish_monotonic": 9120633.042995991, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q9", "burst": "b16-r2", "round": 2, "index": 9, "rank": 1, "submit_monotonic": 9120633.00972221, "finish_monotonic": 9120633.045476284, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q10", "burst": "b16-r2", "round": 2, "index": 10, "rank": 0, "submit_monotonic": 9120633.009998403, "finish_monotonic": 9120633.049505532, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q11", "burst": "b16-r2", "round": 2, "index": 11, "rank": 1, "submit_monotonic": 9120633.010321893, "finish_monotonic": 9120633.049663324, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q12", "burst": "b16-r2", "round": 2, "index": 12, "rank": 0, "submit_monotonic": 9120633.0106315, "finish_monotonic": 9120633.053712623, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q13", "burst": "b16-r2", "round": 2, "index": 13, "rank": 1, "submit_monotonic": 9120633.010876952, "finish_monotonic": 9120633.053716576, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q14", "burst": "b16-r2", "round": 2, "index": 14, "rank": 0, "submit_monotonic": 9120633.011186015, "finish_monotonic": 9120633.058281144, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q15", "burst": "b16-r2", "round": 2, "index": 15, "rank": 1, "submit_monotonic": 9120633.011494512, "finish_monotonic": 9120633.058289863, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/summary.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/summary.json new file mode 100644 index 00000000..e03d6e86 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/summary.json @@ -0,0 +1,186 @@ +{ + "model_config": "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/data/config/models/Llama-3.2-1B-Instruct.json", + "num_gpu_blocks": 304854, + "block_size": 16, + "engine_args": { + "model": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", + "served_model_name": null, + "tokenizer": null, + "hf_config_path": null, + "runner": "auto", + "convert": "auto", + "task": null, + "skip_tokenizer_init": true, + "enable_prompt_embeds": false, + "tokenizer_mode": "auto", + "trust_remote_code": false, + "allowed_local_media_path": "", + "download_dir": null, + "safetensors_load_strategy": "lazy", + "load_format": "dummy", + "config_format": "auto", + "dtype": "bfloat16", + "kv_cache_dtype": "auto", + "seed": 0, + "max_model_len": 512, + "distributed_executor_backend": null, + "pipeline_parallel_size": 2, + "tensor_parallel_size": 1, + "decode_context_parallel_size": 1, + "data_parallel_size": 2, + "data_parallel_rank": null, + "data_parallel_start_rank": null, + "data_parallel_size_local": null, + "data_parallel_address": null, + "data_parallel_rpc_port": null, + "data_parallel_hybrid_lb": false, + "data_parallel_backend": "mp", + "enable_expert_parallel": false, + "enable_eplb": false, + "num_redundant_experts": 0, + "eplb_window_size": 1000, + "eplb_step_interval": 3000, + "eplb_log_balancedness": false, + "max_parallel_loading_workers": null, + "block_size": 16, + "enable_prefix_caching": false, + "prefix_caching_hash_algo": "sha256", + "disable_sliding_window": false, + "disable_cascade_attn": false, + "swap_space": 4, + "cpu_offload_gb": 0, + "gpu_memory_utilization": 0.5, + "kv_cache_memory_bytes": null, + "max_num_batched_tokens": 256, + "max_num_partial_prefills": 1, + "max_long_partial_prefills": 1, + "long_prefill_token_threshold": 0, + "max_num_seqs": 4, + "max_logprobs": 20, + "disable_log_stats": true, + "revision": null, + "code_revision": null, + "rope_theta": null, + "hf_token": null, + "tokenizer_revision": null, + "quantization": null, + "enforce_eager": true, + "max_seq_len_to_capture": 8192, + "disable_custom_all_reduce": false, + "interleave_mm_strings": false, + "mm_processor_kwargs": null, + "disable_mm_preprocessor_cache": false, + "mm_processor_cache_gb": 4, + "mm_encoder_tp_mode": "weights", + "io_processor_plugin": null, + "skip_mm_profiling": false, + "enable_lora": false, + "enable_lora_bias": false, + "max_loras": 1, + "max_lora_rank": 16, + "default_mm_loras": null, + "fully_sharded_loras": false, + "max_cpu_loras": null, + "lora_dtype": "auto", + "lora_extra_vocab_size": 256, + "ray_workers_use_nsight": false, + "num_gpu_blocks_override": null, + "num_lookahead_slots": 0, + "ignore_patterns": null, + "preemption_mode": null, + "scheduler_delay_factor": 0.0, + "enable_chunked_prefill": true, + "disable_chunked_mm_input": false, + "disable_hybrid_kv_cache_manager": false, + "guided_decoding_backend": "auto", + "guided_decoding_disable_fallback": false, + "guided_decoding_disable_any_whitespace": false, + "guided_decoding_disable_additional_properties": false, + "logits_processor_pattern": null, + "speculative_config": null, + "show_hidden_metrics_for_version": null, + "otlp_traces_endpoint": null, + "collect_detailed_traces": null, + "disable_async_output_proc": false, + "scheduling_policy": "fcfs", + "scheduler_cls": "vllm.v1.core.sched.scheduler.Scheduler", + "override_pooler_config": null, + "worker_cls": "auto", + "worker_extension_cls": "", + "kv_transfer_config": null, + "kv_events_config": null, + "generation_config": "auto", + "enable_sleep_mode": false, + "model_impl": "auto", + "override_attention_dtype": null, + "calculate_kv_scales": false, + "mamba_cache_dtype": "auto", + "mamba_ssm_cache_dtype": "auto", + "reasoning_parser": "", + "use_tqdm_on_load": true, + "pt_load_map_location": "cpu", + "enable_multimodal_encoder_data_parallel": false, + "logits_processors": null, + "async_scheduling": false, + "kv_sharing_fast_prefill": false, + "enable_log_requests": false + }, + "rounds": [ + { + "label": "warmup", + "round": 0, + "num_requests": 4, + "wall_minus_monotonic_before": 1780982021.0330577, + "wall_minus_monotonic_after": 1780982021.0330582, + "idle_wait_s": 1.1511429883539677 + }, + { + "label": "b8-r0", + "round": 0, + "num_requests": 8, + "wall_minus_monotonic_before": 1780982021.0330582, + "wall_minus_monotonic_after": 1780982021.0330582, + "idle_wait_s": 1.1017107591032982 + }, + { + "label": "b8-r1", + "round": 1, + "num_requests": 8, + "wall_minus_monotonic_before": 1780982021.0330584, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.1021708622574806 + }, + { + "label": "b8-r2", + "round": 2, + "num_requests": 8, + "wall_minus_monotonic_before": 1780982021.0330582, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.101815516129136 + }, + { + "label": "b16-r0", + "round": 0, + "num_requests": 16, + "wall_minus_monotonic_before": 1780982021.0330582, + "wall_minus_monotonic_after": 1780982021.0330582, + "idle_wait_s": 1.0517608132213354 + }, + { + "label": "b16-r1", + "round": 1, + "num_requests": 16, + "wall_minus_monotonic_before": 1780982021.0330582, + "wall_minus_monotonic_after": 1780982021.0330582, + "idle_wait_s": 1.1024115402251482 + }, + { + "label": "b16-r2", + "round": 2, + "num_requests": 16, + "wall_minus_monotonic_before": 1780982021.0330584, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.05128800496459 + } + ] +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_157.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_157.jsonl new file mode 100644 index 00000000..596b4858 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_157.jsonl @@ -0,0 +1,22 @@ +{"kind": "frontend_snapshot", "pid": 157, "seq": 0, "monotonic": 9120574.796476487, "snapshot": 2, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 1, "monotonic": 9120576.119036447, "snapshot": 3, "counts": [[1, 1], [1, 1]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 2, "monotonic": 9120576.294565408, "snapshot": 4, "counts": [[0, 1], [0, 1]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 3, "monotonic": 9120576.395176036, "snapshot": 5, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 4, "monotonic": 9120577.567102993, "snapshot": 6, "counts": [[0, 1], [0, 1]], "wave": 1, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 5, "monotonic": 9120577.666602153, "snapshot": 7, "counts": [[0, 1], [0, 1]], "wave": 1, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 6, "monotonic": 9120577.76681236, "snapshot": 8, "counts": [[0, 0], [0, 0]], "wave": 1, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 7, "monotonic": 9120578.830534104, "snapshot": 9, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 8, "monotonic": 9120578.930689773, "snapshot": 10, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 9, "monotonic": 9120579.030739984, "snapshot": 11, "counts": [[0, 0], [0, 0]], "wave": 2, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 10, "monotonic": 9120580.093352964, "snapshot": 12, "counts": [[0, 1], [0, 1]], "wave": 3, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 11, "monotonic": 9120580.192300623, "snapshot": 13, "counts": [[0, 1], [0, 1]], "wave": 3, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 12, "monotonic": 9120580.292534402, "snapshot": 14, "counts": [[0, 0], [0, 0]], "wave": 3, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 13, "monotonic": 9120581.361761319, "snapshot": 15, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 14, "monotonic": 9120581.46223673, "snapshot": 16, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 15, "monotonic": 9120581.599077467, "snapshot": 17, "counts": [[0, 0], [0, 0]], "wave": 5, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 157, "seq": 16, "monotonic": 9120582.59408684, "snapshot": 18, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 17, "monotonic": 9120582.694060272, "snapshot": 19, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 18, "monotonic": 9120582.833616564, "snapshot": 20, "counts": [[0, 0], [0, 0]], "wave": 6, "engines_running": false} +{"kind": "frontend_snapshot", "pid": 157, "seq": 19, "monotonic": 9120583.839656929, "snapshot": 21, "counts": [[0, 1], [0, 1]], "wave": 6, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 20, "monotonic": 9120583.939854259, "snapshot": 22, "counts": [[0, 1], [0, 1]], "wave": 6, "engines_running": true} +{"kind": "frontend_snapshot", "pid": 157, "seq": 21, "monotonic": 9120584.040139776, "snapshot": 23, "counts": [[0, 0], [0, 0]], "wave": 6, "engines_running": true} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_232.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_232.jsonl new file mode 100644 index 00000000..41c8042c --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_232.jsonl @@ -0,0 +1,65 @@ +{"kind": "engine_iteration", "pid": 232, "seq": 0, "monotonic": 9120576.068102015, "engine": 0, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 1, "monotonic": 9120576.285050265, "engine": 0, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 2, "monotonic": 9120576.293990524, "engine": 0, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 3, "monotonic": 9120576.302651672, "engine": 0, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 4, "monotonic": 9120576.309567677, "engine": 0, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 5, "monotonic": 9120577.548942052, "engine": 0, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 6, "monotonic": 9120577.56650086, "engine": 0, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 7, "monotonic": 9120577.57367626, "engine": 0, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 8, "monotonic": 9120577.588618556, "engine": 0, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 9, "monotonic": 9120577.597501721, "engine": 0, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 10, "monotonic": 9120577.606690852, "engine": 0, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 11, "monotonic": 9120577.613504106, "engine": 0, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 12, "monotonic": 9120577.620328449, "engine": 0, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 13, "monotonic": 9120578.811790982, "engine": 0, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 14, "monotonic": 9120578.829836285, "engine": 0, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 15, "monotonic": 9120578.837146323, "engine": 0, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 16, "monotonic": 9120578.851954928, "engine": 0, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 17, "monotonic": 9120578.860749573, "engine": 0, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 18, "monotonic": 9120578.869652415, "engine": 0, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 19, "monotonic": 9120578.876468524, "engine": 0, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 20, "monotonic": 9120578.883193335, "engine": 0, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 21, "monotonic": 9120580.0741665, "engine": 0, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 22, "monotonic": 9120580.092560664, "engine": 0, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 23, "monotonic": 9120580.099539263, "engine": 0, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 24, "monotonic": 9120580.119030692, "engine": 0, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 25, "monotonic": 9120580.129233615, "engine": 0, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 26, "monotonic": 9120580.138761727, "engine": 0, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 27, "monotonic": 9120580.145595407, "engine": 0, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 28, "monotonic": 9120580.152127512, "engine": 0, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 29, "monotonic": 9120581.343008349, "engine": 0, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 30, "monotonic": 9120581.36114861, "engine": 0, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 31, "monotonic": 9120581.366927866, "engine": 0, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 32, "monotonic": 9120581.379694436, "engine": 0, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 33, "monotonic": 9120581.386357982, "engine": 0, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 34, "monotonic": 9120581.39295362, "engine": 0, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q8"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 35, "monotonic": 9120581.401882611, "engine": 0, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q10"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 36, "monotonic": 9120581.409054143, "engine": 0, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q12"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 37, "monotonic": 9120581.415643968, "engine": 0, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q14"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 38, "monotonic": 9120581.422455283, "engine": 0, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 39, "monotonic": 9120581.427875658, "engine": 0, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 40, "monotonic": 9120581.432896864, "engine": 0, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 41, "monotonic": 9120582.576324183, "engine": 0, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 42, "monotonic": 9120582.593326459, "engine": 0, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 43, "monotonic": 9120582.598984815, "engine": 0, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 44, "monotonic": 9120582.615470257, "engine": 0, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 45, "monotonic": 9120582.62383548, "engine": 0, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 46, "monotonic": 9120582.632251784, "engine": 0, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q8"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 47, "monotonic": 9120582.64085373, "engine": 0, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q10"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 48, "monotonic": 9120582.649460929, "engine": 0, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q12"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 49, "monotonic": 9120582.657748772, "engine": 0, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q14"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 50, "monotonic": 9120582.667302229, "engine": 0, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 51, "monotonic": 9120582.672339192, "engine": 0, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 52, "monotonic": 9120582.677481571, "engine": 0, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 53, "monotonic": 9120583.821056273, "engine": 0, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q0"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 54, "monotonic": 9120583.83902434, "engine": 0, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 55, "monotonic": 9120583.844306864, "engine": 0, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q2"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 56, "monotonic": 9120583.860620424, "engine": 0, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q4"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 57, "monotonic": 9120583.869644647, "engine": 0, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q6"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 58, "monotonic": 9120583.878017789, "engine": 0, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q8"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 59, "monotonic": 9120583.886359224, "engine": 0, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q10"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 60, "monotonic": 9120583.894910123, "engine": 0, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q12"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 61, "monotonic": 9120583.903448792, "engine": 0, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q14"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 232, "seq": 62, "monotonic": 9120583.911947114, "engine": 0, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 63, "monotonic": 9120583.918588044, "engine": 0, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 232, "seq": 64, "monotonic": 9120583.925187727, "engine": 0, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_233.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_233.jsonl new file mode 100644 index 00000000..c7e40079 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_233.jsonl @@ -0,0 +1,65 @@ +{"kind": "engine_iteration", "pid": 233, "seq": 0, "monotonic": 9120576.066573633, "engine": 1, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 1, "monotonic": 9120576.28477715, "engine": 1, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 2, "monotonic": 9120576.294229764, "engine": 1, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 3, "monotonic": 9120576.302748293, "engine": 1, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 4, "monotonic": 9120576.30969718, "engine": 1, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 5, "monotonic": 9120577.549137808, "engine": 1, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 6, "monotonic": 9120577.566702828, "engine": 1, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 7, "monotonic": 9120577.573778512, "engine": 1, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 8, "monotonic": 9120577.588767529, "engine": 1, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 9, "monotonic": 9120577.597639225, "engine": 1, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 10, "monotonic": 9120577.606814198, "engine": 1, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 11, "monotonic": 9120577.613607325, "engine": 1, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 12, "monotonic": 9120577.62049405, "engine": 1, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 13, "monotonic": 9120578.812228687, "engine": 1, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 14, "monotonic": 9120578.829954091, "engine": 1, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 15, "monotonic": 9120578.837080324, "engine": 1, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 16, "monotonic": 9120578.851976847, "engine": 1, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 17, "monotonic": 9120578.860928042, "engine": 1, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 18, "monotonic": 9120578.869871182, "engine": 1, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 19, "monotonic": 9120578.876708878, "engine": 1, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 20, "monotonic": 9120578.883417822, "engine": 1, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 21, "monotonic": 9120580.074600577, "engine": 1, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 22, "monotonic": 9120580.09266292, "engine": 1, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 23, "monotonic": 9120580.099672109, "engine": 1, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 24, "monotonic": 9120580.119215572, "engine": 1, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 25, "monotonic": 9120580.129176484, "engine": 1, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 26, "monotonic": 9120580.138748785, "engine": 1, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 27, "monotonic": 9120580.145636436, "engine": 1, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 28, "monotonic": 9120580.152310977, "engine": 1, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 29, "monotonic": 9120581.343315285, "engine": 1, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 30, "monotonic": 9120581.361277947, "engine": 1, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 31, "monotonic": 9120581.367144477, "engine": 1, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 32, "monotonic": 9120581.379747193, "engine": 1, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 33, "monotonic": 9120581.386451261, "engine": 1, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 34, "monotonic": 9120581.392984452, "engine": 1, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q9"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 35, "monotonic": 9120581.401938003, "engine": 1, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q11"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 36, "monotonic": 9120581.409002103, "engine": 1, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q13"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 37, "monotonic": 9120581.415707905, "engine": 1, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q15"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 38, "monotonic": 9120581.422552591, "engine": 1, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 39, "monotonic": 9120581.427866183, "engine": 1, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 40, "monotonic": 9120581.432943664, "engine": 1, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 41, "monotonic": 9120582.57642316, "engine": 1, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 42, "monotonic": 9120582.593373183, "engine": 1, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 43, "monotonic": 9120582.598950867, "engine": 1, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 44, "monotonic": 9120582.615441436, "engine": 1, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 45, "monotonic": 9120582.62390378, "engine": 1, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 46, "monotonic": 9120582.632348128, "engine": 1, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q9"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 47, "monotonic": 9120582.640770746, "engine": 1, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q11"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 48, "monotonic": 9120582.649319585, "engine": 1, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q13"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 49, "monotonic": 9120582.657647757, "engine": 1, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q15"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 50, "monotonic": 9120582.667399248, "engine": 1, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 51, "monotonic": 9120582.672459846, "engine": 1, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 52, "monotonic": 9120582.677348372, "engine": 1, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 53, "monotonic": 9120583.821293969, "engine": 1, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q1"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 54, "monotonic": 9120583.839159656, "engine": 1, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 55, "monotonic": 9120583.844510851, "engine": 1, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q3"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 56, "monotonic": 9120583.860668816, "engine": 1, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q5"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 57, "monotonic": 9120583.86950854, "engine": 1, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q7"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 58, "monotonic": 9120583.878138375, "engine": 1, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q9"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 59, "monotonic": 9120583.886439929, "engine": 1, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q11"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 60, "monotonic": 9120583.894956497, "engine": 1, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q13"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 61, "monotonic": 9120583.903405605, "engine": 1, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q15"], "num_scheduled_tokens": 256} +{"kind": "engine_iteration", "pid": 233, "seq": 62, "monotonic": 9120583.911890052, "engine": 1, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 63, "monotonic": 9120583.918713111, "engine": 1, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} +{"kind": "engine_iteration", "pid": 233, "seq": 64, "monotonic": 9120583.925296413, "engine": 1, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/model/config.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/model/config.json new file mode 100644 index 00000000..9b897034 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/model/config.json @@ -0,0 +1,95 @@ +{ + "_comment": "Qwen3-30B-A3B-tiny: Modified Qwen3-30B-A3B for MoE profiling testing (reduced layers and experts)", + "vocab_size": 151936, + "max_position_embeddings": 40960, + "hidden_size": 2048, + "intermediate_size": 6144, + "num_hidden_layers": 8, + "num_attention_heads": 32, + "use_sliding_window": false, + "sliding_window": null, + "num_key_value_heads": 4, + "hidden_act": "silu", + "initializer_range": 0.02, + "rms_norm_eps": 1e-06, + "use_cache": true, + "rope_theta": 1000000.0, + "rope_scaling": null, + "attention_bias": false, + "attention_dropout": 0.0, + "decoder_sparse_step": 1, + "moe_intermediate_size": 768, + "num_experts_per_tok": 8, + "num_experts": 16, + "norm_topk_prob": true, + "output_router_logits": false, + "router_aux_loss_coef": 0.001, + "mlp_only_layers": [], + "return_dict": true, + "output_hidden_states": false, + "torchscript": false, + "dtype": "bfloat16", + "pruned_heads": {}, + "tie_word_embeddings": false, + "chunk_size_feed_forward": 0, + "is_encoder_decoder": false, + "is_decoder": false, + "cross_attention_hidden_size": null, + "add_cross_attention": false, + "tie_encoder_decoder": false, + "architectures": [ + "Qwen3MoeForCausalLM" + ], + "finetuning_task": null, + "id2label": { + "0": "LABEL_0", + "1": "LABEL_1" + }, + "label2id": { + "LABEL_0": 0, + "LABEL_1": 1 + }, + "task_specific_params": null, + "problem_type": null, + "tokenizer_class": null, + "prefix": null, + "bos_token_id": 151643, + "pad_token_id": null, + "eos_token_id": 151645, + "sep_token_id": null, + "decoder_start_token_id": null, + "max_length": 20, + "min_length": 0, + "do_sample": false, + "early_stopping": false, + "num_beams": 1, + "temperature": 1.0, + "top_k": 50, + "top_p": 1.0, + "typical_p": 1.0, + "repetition_penalty": 1.0, + "length_penalty": 1.0, + "no_repeat_ngram_size": 0, + "encoder_no_repeat_ngram_size": 0, + "bad_words_ids": null, + "num_return_sequences": 1, + "output_scores": false, + "return_dict_in_generate": false, + "forced_bos_token_id": null, + "forced_eos_token_id": null, + "remove_invalid_values": false, + "exponential_decay_length_penalty": null, + "suppress_tokens": null, + "begin_suppress_tokens": null, + "num_beam_groups": 1, + "diversity_penalty": 0.0, + "_name_or_path": "Qwen/Qwen3-30B-A3B", + "transformers_version": "4.57.3", + "head_dim": 128, + "max_window_layers": 48, + "model_type": "qwen3_moe", + "tf_legacy_loss": false, + "use_bfloat16": false, + "output_attentions": false, + "use_qk_norm": true +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/pp_boundary.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/pp_boundary.jsonl new file mode 100644 index 00000000..c99eea4d --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/pp_boundary.jsonl @@ -0,0 +1,152 @@ +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120576.06912624, "preprocess_end_ts": 9120576.08141513, "forward_start_ts": 9120576.081611509, "forward_end_ts": 9120576.09329179, "timestamp": 1790102597.2836564, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120576.093549645, "send_end_ts": 9120576.25057326} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120576.067281932, "preprocess_end_ts": 9120576.078339692, "forward_start_ts": 9120576.078541664, "forward_end_ts": 9120576.093035089, "timestamp": 1790102597.2917824, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120576.093350155, "send_end_ts": 9120576.258700026} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120576.258467624, "preprocess_end_ts": 9120576.271626204, "forward_start_ts": 9120576.271777231, "forward_end_ts": 9120576.283202868, "timestamp": 1790102597.3172412, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120576.06675498, "recv_end_ts": 9120576.25815104, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120576.251813464, "preprocess_end_ts": 9120576.266894476, "forward_start_ts": 9120576.267087674, "forward_end_ts": 9120576.283423033, "timestamp": 1790102597.3174233, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120576.068382105, "recv_end_ts": 9120576.251342716, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120576.25950386, "preprocess_end_ts": 9120576.260119552, "forward_start_ts": 9120576.26014196, "forward_end_ts": 9120576.268356636, "timestamp": 1790102597.3179822, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120576.269240877, "send_end_ts": 9120576.284913171} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120576.251260724, "preprocess_end_ts": 9120576.251849292, "forward_start_ts": 9120576.251871116, "forward_end_ts": 9120576.2692128, "timestamp": 1790102597.3182423, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120576.269356001, "send_end_ts": 9120576.28517143} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120576.285398638, "preprocess_end_ts": 9120576.28580193, "forward_start_ts": 9120576.285816776, "forward_end_ts": 9120576.291115416, "timestamp": 1790102597.326646, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120576.284717713, "recv_end_ts": 9120576.28519154, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120576.285317533, "preprocess_end_ts": 9120576.285914203, "forward_start_ts": 9120576.285933616, "forward_end_ts": 9120576.293196166, "timestamp": 1790102597.3267689, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120576.284539368, "recv_end_ts": 9120576.28504768, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.549571201, "preprocess_end_ts": 9120577.55009034, "forward_start_ts": 9120577.550104069, "forward_end_ts": 9120577.55616991, "timestamp": 1790102598.5898101, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.55628498, "send_end_ts": 9120577.556742346} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.549495123, "preprocess_end_ts": 9120577.55013278, "forward_start_ts": 9120577.55014999, "forward_end_ts": 9120577.55611502, "timestamp": 1790102598.589807, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.556241311, "send_end_ts": 9120577.556741873} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.55712839, "preprocess_end_ts": 9120577.557572547, "forward_start_ts": 9120577.557585603, "forward_end_ts": 9120577.563604295, "timestamp": 1790102598.5991333, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.549080389, "recv_end_ts": 9120577.55690697, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.557268629, "preprocess_end_ts": 9120577.557913676, "forward_start_ts": 9120577.557930704, "forward_end_ts": 9120577.565687869, "timestamp": 1790102598.599265, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.549293175, "recv_end_ts": 9120577.556970704, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.573831668, "preprocess_end_ts": 9120577.574154207, "forward_start_ts": 9120577.574164923, "forward_end_ts": 9120577.579463609, "timestamp": 1790102598.6130702, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.579588724, "send_end_ts": 9120577.579996692} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.573992858, "preprocess_end_ts": 9120577.574350936, "forward_start_ts": 9120577.574361622, "forward_end_ts": 9120577.579558125, "timestamp": 1790102598.6131058, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.579642821, "send_end_ts": 9120577.580039855} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.580262372, "preprocess_end_ts": 9120577.580620212, "forward_start_ts": 9120577.580630852, "forward_end_ts": 9120577.585889503, "timestamp": 1790102598.6212263, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.57371594, "recv_end_ts": 9120577.580101142, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.580354996, "preprocess_end_ts": 9120577.580813946, "forward_start_ts": 9120577.580826651, "forward_end_ts": 9120577.587808212, "timestamp": 1790102598.6213608, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.573786428, "recv_end_ts": 9120577.580179015, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.580370976, "preprocess_end_ts": 9120577.580712583, "forward_start_ts": 9120577.580723124, "forward_end_ts": 9120577.58584904, "timestamp": 1790102598.6218507, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.585956562, "send_end_ts": 9120577.588786367} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.580392718, "preprocess_end_ts": 9120577.58073172, "forward_start_ts": 9120577.580740644, "forward_end_ts": 9120577.585923836, "timestamp": 1790102598.6219738, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.586004073, "send_end_ts": 9120577.588908568} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.589121113, "preprocess_end_ts": 9120577.589516431, "forward_start_ts": 9120577.589526488, "forward_end_ts": 9120577.594744284, "timestamp": 1790102598.6301506, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.588429969, "recv_end_ts": 9120577.58895459, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.589128325, "preprocess_end_ts": 9120577.589584732, "forward_start_ts": 9120577.58959636, "forward_end_ts": 9120577.596727813, "timestamp": 1790102598.630274, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.588598276, "recv_end_ts": 9120577.588972116, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.589193283, "preprocess_end_ts": 9120577.589508563, "forward_start_ts": 9120577.589517279, "forward_end_ts": 9120577.594660422, "timestamp": 1790102598.6308415, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.594835784, "send_end_ts": 9120577.597777065} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.589247884, "preprocess_end_ts": 9120577.589569096, "forward_start_ts": 9120577.589578193, "forward_end_ts": 9120577.594807643, "timestamp": 1790102598.630937, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.594884492, "send_end_ts": 9120577.597870873} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.598051693, "preprocess_end_ts": 9120577.59839024, "forward_start_ts": 9120577.598399421, "forward_end_ts": 9120577.603558876, "timestamp": 1790102598.6393318, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.597388487, "recv_end_ts": 9120577.597895443, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.59808894, "preprocess_end_ts": 9120577.5985329, "forward_start_ts": 9120577.598543633, "forward_end_ts": 9120577.605920406, "timestamp": 1790102598.6394594, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.597512793, "recv_end_ts": 9120577.597917935, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.812697122, "preprocess_end_ts": 9120578.813320901, "forward_start_ts": 9120578.813337032, "forward_end_ts": 9120578.819544006, "timestamp": 1790102599.853156, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.819649665, "send_end_ts": 9120578.82009076} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.812466595, "preprocess_end_ts": 9120578.813126571, "forward_start_ts": 9120578.813147675, "forward_end_ts": 9120578.819582999, "timestamp": 1790102599.8532457, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.819706036, "send_end_ts": 9120578.82017994} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.820711149, "preprocess_end_ts": 9120578.821310576, "forward_start_ts": 9120578.821343388, "forward_end_ts": 9120578.827582017, "timestamp": 1790102599.8624983, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.812198428, "recv_end_ts": 9120578.82035111, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.820530899, "preprocess_end_ts": 9120578.821104528, "forward_start_ts": 9120578.821120372, "forward_end_ts": 9120578.829012496, "timestamp": 1790102599.8625872, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.812379096, "recv_end_ts": 9120578.820289545, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.83731334, "preprocess_end_ts": 9120578.837618664, "forward_start_ts": 9120578.83762973, "forward_end_ts": 9120578.842705376, "timestamp": 1790102599.8762634, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.842833433, "send_end_ts": 9120578.843199044} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.837259185, "preprocess_end_ts": 9120578.837583508, "forward_start_ts": 9120578.837593429, "forward_end_ts": 9120578.84280425, "timestamp": 1790102599.8763123, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.842886532, "send_end_ts": 9120578.843248315} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.843512638, "preprocess_end_ts": 9120578.843970332, "forward_start_ts": 9120578.843982011, "forward_end_ts": 9120578.851015653, "timestamp": 1790102599.8845687, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.837185057, "recv_end_ts": 9120578.84332666, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.8435567, "preprocess_end_ts": 9120578.844025772, "forward_start_ts": 9120578.844038067, "forward_end_ts": 9120578.851041503, "timestamp": 1790102599.8845901, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.837132009, "recv_end_ts": 9120578.843378464, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.843535822, "preprocess_end_ts": 9120578.84385769, "forward_start_ts": 9120578.843867462, "forward_end_ts": 9120578.848940914, "timestamp": 1790102599.8852592, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.849080913, "send_end_ts": 9120578.852194704} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.843580656, "preprocess_end_ts": 9120578.843910933, "forward_start_ts": 9120578.843920348, "forward_end_ts": 9120578.849048804, "timestamp": 1790102599.885258, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.849123884, "send_end_ts": 9120578.85219327} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.852500973, "preprocess_end_ts": 9120578.85295746, "forward_start_ts": 9120578.852968963, "forward_end_ts": 9120578.858358746, "timestamp": 1790102599.8935063, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.851837687, "recv_end_ts": 9120578.85232474, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.852441452, "preprocess_end_ts": 9120578.85289666, "forward_start_ts": 9120578.852908885, "forward_end_ts": 9120578.86007354, "timestamp": 1790102599.8936193, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.85182794, "recv_end_ts": 9120578.85227706, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.852539623, "preprocess_end_ts": 9120578.852858113, "forward_start_ts": 9120578.852866942, "forward_end_ts": 9120578.857989244, "timestamp": 1790102599.8940766, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.858322257, "send_end_ts": 9120578.861012325} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.852627993, "preprocess_end_ts": 9120578.85298128, "forward_start_ts": 9120578.852990692, "forward_end_ts": 9120578.8582934, "timestamp": 1790102599.8943467, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.858367149, "send_end_ts": 9120578.861281207} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.861280115, "preprocess_end_ts": 9120578.861627065, "forward_start_ts": 9120578.861636132, "forward_end_ts": 9120578.86709386, "timestamp": 1790102599.9024386, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.860707372, "recv_end_ts": 9120578.861127583, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.8614549, "preprocess_end_ts": 9120578.86190434, "forward_start_ts": 9120578.861915732, "forward_end_ts": 9120578.868989771, "timestamp": 1790102599.9025333, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.860852864, "recv_end_ts": 9120578.861284569, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.074596796, "preprocess_end_ts": 9120580.075139388, "forward_start_ts": 9120580.075154155, "forward_end_ts": 9120580.081509704, "timestamp": 1790102601.1157746, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.082217831, "send_end_ts": 9120580.082709515} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.075100802, "preprocess_end_ts": 9120580.075675512, "forward_start_ts": 9120580.075690024, "forward_end_ts": 9120580.082179496, "timestamp": 1790102601.1159906, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.082398407, "send_end_ts": 9120580.082924727} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.083210835, "preprocess_end_ts": 9120580.083756076, "forward_start_ts": 9120580.083771244, "forward_end_ts": 9120580.08955698, "timestamp": 1790102601.1250973, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.074461987, "recv_end_ts": 9120580.082938297, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.083296858, "preprocess_end_ts": 9120580.083885018, "forward_start_ts": 9120580.083901592, "forward_end_ts": 9120580.09166343, "timestamp": 1790102601.125262, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.074712954, "recv_end_ts": 9120580.083052542, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.100881828, "preprocess_end_ts": 9120580.101360453, "forward_start_ts": 9120580.101374147, "forward_end_ts": 9120580.109860908, "timestamp": 1790102601.1435313, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.109983295, "send_end_ts": 9120580.11046472} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.100881828, "preprocess_end_ts": 9120580.101360446, "forward_start_ts": 9120580.101374116, "forward_end_ts": 9120580.109896155, "timestamp": 1790102601.1435313, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.110006873, "send_end_ts": 9120580.110464765} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.110640671, "preprocess_end_ts": 9120580.111033125, "forward_start_ts": 9120580.111044472, "forward_end_ts": 9120580.11642792, "timestamp": 1790102601.1517575, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.099583574, "recv_end_ts": 9120580.110478189, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.1107548, "preprocess_end_ts": 9120580.111248905, "forward_start_ts": 9120580.11126213, "forward_end_ts": 9120580.11832074, "timestamp": 1790102601.151885, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.099723669, "recv_end_ts": 9120580.110559914, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.110935118, "preprocess_end_ts": 9120580.111394485, "forward_start_ts": 9120580.111406164, "forward_end_ts": 9120580.119771758, "timestamp": 1790102601.153413, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.119873416, "send_end_ts": 9120580.120346544} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.110935053, "preprocess_end_ts": 9120580.111394396, "forward_start_ts": 9120580.111406185, "forward_end_ts": 9120580.119771767, "timestamp": 1790102601.153413, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.119873255, "send_end_ts": 9120580.120346576} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.120589023, "preprocess_end_ts": 9120580.12105302, "forward_start_ts": 9120580.12106498, "forward_end_ts": 9120580.128097888, "timestamp": 1790102601.1617668, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.119136358, "recv_end_ts": 9120580.12041668, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.120535705, "preprocess_end_ts": 9120580.120907266, "forward_start_ts": 9120580.120917823, "forward_end_ts": 9120580.128306828, "timestamp": 1790102601.161878, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.118960636, "recv_end_ts": 9120580.120388944, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.120777884, "preprocess_end_ts": 9120580.121225763, "forward_start_ts": 9120580.121237168, "forward_end_ts": 9120580.12957196, "timestamp": 1790102601.1631815, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.129670313, "send_end_ts": 9120580.130115215} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.120777883, "preprocess_end_ts": 9120580.121225769, "forward_start_ts": 9120580.121237092, "forward_end_ts": 9120580.129571958, "timestamp": 1790102601.1631815, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.12967029, "send_end_ts": 9120580.130115697} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.130371591, "preprocess_end_ts": 9120580.130897123, "forward_start_ts": 9120580.130908811, "forward_end_ts": 9120580.137896962, "timestamp": 1790102601.1714482, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.129156964, "recv_end_ts": 9120580.130191972, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.130379148, "preprocess_end_ts": 9120580.130824532, "forward_start_ts": 9120580.13083534, "forward_end_ts": 9120580.137938311, "timestamp": 1790102601.1714747, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.12900894, "recv_end_ts": 9120580.13020648, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.343732748, "preprocess_end_ts": 9120581.344221508, "forward_start_ts": 9120581.344235841, "forward_end_ts": 9120581.35028834, "timestamp": 1790102602.3839042, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.350392705, "send_end_ts": 9120581.350838909} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.343441563, "preprocess_end_ts": 9120581.34409297, "forward_start_ts": 9120581.344110794, "forward_end_ts": 9120581.350297496, "timestamp": 1790102602.3839097, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.350408033, "send_end_ts": 9120581.350844346} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.351336911, "preprocess_end_ts": 9120581.351899972, "forward_start_ts": 9120581.351921292, "forward_end_ts": 9120581.35798718, "timestamp": 1790102602.3936594, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.343240311, "recv_end_ts": 9120581.351054829, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.351373885, "preprocess_end_ts": 9120581.351970855, "forward_start_ts": 9120581.35198758, "forward_end_ts": 9120581.360200204, "timestamp": 1790102602.393783, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.343579505, "recv_end_ts": 9120581.351054737, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.367097467, "preprocess_end_ts": 9120581.367441077, "forward_start_ts": 9120581.367452266, "forward_end_ts": 9120581.372749446, "timestamp": 1790102602.4062917, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.37283898, "send_end_ts": 9120581.373226961} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.36734962, "preprocess_end_ts": 9120581.367687093, "forward_start_ts": 9120581.367698174, "forward_end_ts": 9120581.37280371, "timestamp": 1790102602.4063165, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.372886997, "send_end_ts": 9120581.373252112} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.373437852, "preprocess_end_ts": 9120581.373789055, "forward_start_ts": 9120581.373799719, "forward_end_ts": 9120581.378876328, "timestamp": 1790102602.41244, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.366961967, "recv_end_ts": 9120581.37329134, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.373489227, "preprocess_end_ts": 9120581.373844985, "forward_start_ts": 9120581.373855025, "forward_end_ts": 9120581.378997438, "timestamp": 1790102602.4124928, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.367181359, "recv_end_ts": 9120581.373334873, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.37357514, "preprocess_end_ts": 9120581.373903705, "forward_start_ts": 9120581.373913374, "forward_end_ts": 9120581.379154231, "timestamp": 1790102602.413001, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.379235808, "send_end_ts": 9120581.379936537} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.373586047, "preprocess_end_ts": 9120581.373905532, "forward_start_ts": 9120581.373914763, "forward_end_ts": 9120581.37915932, "timestamp": 1790102602.4131227, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.379233835, "send_end_ts": 9120581.380057868} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.380174685, "preprocess_end_ts": 9120581.380519679, "forward_start_ts": 9120581.380529135, "forward_end_ts": 9120581.38564956, "timestamp": 1790102602.419137, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.379627496, "recv_end_ts": 9120581.380037796, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.380258484, "preprocess_end_ts": 9120581.380620003, "forward_start_ts": 9120581.38062962, "forward_end_ts": 9120581.3856865, "timestamp": 1790102602.4191911, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.379681192, "recv_end_ts": 9120581.38012081, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.380323807, "preprocess_end_ts": 9120581.380649032, "forward_start_ts": 9120581.380658597, "forward_end_ts": 9120581.385759557, "timestamp": 1790102602.4196768, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.38585794, "send_end_ts": 9120581.386612376} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.380388295, "preprocess_end_ts": 9120581.380705131, "forward_start_ts": 9120581.380713452, "forward_end_ts": 9120581.385826873, "timestamp": 1790102602.419787, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.385898268, "send_end_ts": 9120581.386721654} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.386851596, "preprocess_end_ts": 9120581.38718201, "forward_start_ts": 9120581.387190906, "forward_end_ts": 9120581.392265217, "timestamp": 1790102602.42575, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.386318544, "recv_end_ts": 9120581.386711184, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.38691968, "preprocess_end_ts": 9120581.387254061, "forward_start_ts": 9120581.387263125, "forward_end_ts": 9120581.392278733, "timestamp": 1790102602.4257653, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.386369605, "recv_end_ts": 9120581.386778977, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.38695468, "preprocess_end_ts": 9120581.38727, "forward_start_ts": 9120581.387279348, "forward_end_ts": 9120581.392546862, "timestamp": 1790102602.426297, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.39261858, "send_end_ts": 9120581.393232038} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.38705204, "preprocess_end_ts": 9120581.387368888, "forward_start_ts": 9120581.387377515, "forward_end_ts": 9120581.392576084, "timestamp": 1790102602.426374, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.392646756, "send_end_ts": 9120581.393308524} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.39347056, "preprocess_end_ts": 9120581.393800288, "forward_start_ts": 9120581.393808791, "forward_end_ts": 9120581.398831991, "timestamp": 1790102602.434644, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.392939206, "recv_end_ts": 9120581.393334216, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.39349049, "preprocess_end_ts": 9120581.393821685, "forward_start_ts": 9120581.393830648, "forward_end_ts": 9120581.401207073, "timestamp": 1790102602.434714, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.3929638, "recv_end_ts": 9120581.393355276, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.3935938, "preprocess_end_ts": 9120581.393916002, "forward_start_ts": 9120581.39392472, "forward_end_ts": 9120581.398994233, "timestamp": 1790102602.4352026, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.399140792, "send_end_ts": 9120581.402138196} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.393645296, "preprocess_end_ts": 9120581.393961037, "forward_start_ts": 9120581.393969523, "forward_end_ts": 9120581.39911096, "timestamp": 1790102602.4353406, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.399182292, "send_end_ts": 9120581.402275685} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.402473414, "preprocess_end_ts": 9120581.402806476, "forward_start_ts": 9120581.402815038, "forward_end_ts": 9120581.408110779, "timestamp": 1790102602.4417818, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.401888557, "recv_end_ts": 9120581.402333334, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.402364124, "preprocess_end_ts": 9120581.402693668, "forward_start_ts": 9120581.40270258, "forward_end_ts": 9120581.408346636, "timestamp": 1790102602.441835, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.401829578, "recv_end_ts": 9120581.40222036, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.402505113, "preprocess_end_ts": 9120581.402833944, "forward_start_ts": 9120581.402842766, "forward_end_ts": 9120581.408678753, "timestamp": 1790102602.442369, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.408749774, "send_end_ts": 9120581.40930469} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.4026157, "preprocess_end_ts": 9120581.40293946, "forward_start_ts": 9120581.402949404, "forward_end_ts": 9120581.40806486, "timestamp": 1790102602.4423795, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.408707408, "send_end_ts": 9120581.409314485} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.40956396, "preprocess_end_ts": 9120581.409890601, "forward_start_ts": 9120581.40989973, "forward_end_ts": 9120581.414940232, "timestamp": 1790102602.4484506, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.409017527, "recv_end_ts": 9120581.409412911, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.40952547, "preprocess_end_ts": 9120581.40986102, "forward_start_ts": 9120581.409869485, "forward_end_ts": 9120581.415017493, "timestamp": 1790102602.4485135, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.40897754, "recv_end_ts": 9120581.40938676, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.409655448, "preprocess_end_ts": 9120581.409974145, "forward_start_ts": 9120581.409983275, "forward_end_ts": 9120581.41501236, "timestamp": 1790102602.4489937, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.415114151, "send_end_ts": 9120581.415927997} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.409677612, "preprocess_end_ts": 9120581.40999598, "forward_start_ts": 9120581.410004305, "forward_end_ts": 9120581.415084803, "timestamp": 1790102602.449092, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.41515612, "send_end_ts": 9120581.41602733} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.416166749, "preprocess_end_ts": 9120581.416485762, "forward_start_ts": 9120581.416494308, "forward_end_ts": 9120581.421585016, "timestamp": 1790102602.4552574, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.415618919, "recv_end_ts": 9120581.416028755, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.41623896, "preprocess_end_ts": 9120581.416565629, "forward_start_ts": 9120581.416574404, "forward_end_ts": 9120581.421821417, "timestamp": 1790102602.4553607, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.415697642, "recv_end_ts": 9120581.416099109, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.576923266, "preprocess_end_ts": 9120582.577481993, "forward_start_ts": 9120582.577497356, "forward_end_ts": 9120582.583782172, "timestamp": 1790102603.6189995, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.58547205, "send_end_ts": 9120582.585934443} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.576914553, "preprocess_end_ts": 9120582.577515438, "forward_start_ts": 9120582.57753102, "forward_end_ts": 9120582.585449988, "timestamp": 1790102603.6191525, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.58558134, "send_end_ts": 9120582.58608655} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.586269628, "preprocess_end_ts": 9120582.586710135, "forward_start_ts": 9120582.586722473, "forward_end_ts": 9120582.592280349, "timestamp": 1790102603.6259403, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.576488996, "recv_end_ts": 9120582.586054899, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.586409232, "preprocess_end_ts": 9120582.586853774, "forward_start_ts": 9120582.586867034, "forward_end_ts": 9120582.592480345, "timestamp": 1790102603.626008, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.576604463, "recv_end_ts": 9120582.58619712, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.599168986, "preprocess_end_ts": 9120582.599562293, "forward_start_ts": 9120582.599573622, "forward_end_ts": 9120582.605908327, "timestamp": 1790102603.6415508, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.608063184, "send_end_ts": 9120582.608485227} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.600425657, "preprocess_end_ts": 9120582.600854224, "forward_start_ts": 9120582.60086626, "forward_end_ts": 9120582.60805398, "timestamp": 1790102603.6417098, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.608168932, "send_end_ts": 9120582.60864294} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.608761417, "preprocess_end_ts": 9120582.60915175, "forward_start_ts": 9120582.609163187, "forward_end_ts": 9120582.614419485, "timestamp": 1790102603.6480691, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.599062407, "recv_end_ts": 9120582.608579684, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.608863711, "preprocess_end_ts": 9120582.609250143, "forward_start_ts": 9120582.609261172, "forward_end_ts": 9120582.614605544, "timestamp": 1790102603.6481156, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.59895246, "recv_end_ts": 9120582.608695596, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.608876169, "preprocess_end_ts": 9120582.60922144, "forward_start_ts": 9120582.609231612, "forward_end_ts": 9120582.614570571, "timestamp": 1790102603.650187, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.616727713, "send_end_ts": 9120582.617122507} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.609102508, "preprocess_end_ts": 9120582.609549047, "forward_start_ts": 9120582.609561363, "forward_end_ts": 9120582.61670548, "timestamp": 1790102603.650329, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.61681186, "send_end_ts": 9120582.61726228} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.617367228, "preprocess_end_ts": 9120582.617727084, "forward_start_ts": 9120582.617737848, "forward_end_ts": 9120582.622994997, "timestamp": 1790102603.6565177, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.615312329, "recv_end_ts": 9120582.617211921, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.61746808, "preprocess_end_ts": 9120582.617837433, "forward_start_ts": 9120582.617848467, "forward_end_ts": 9120582.623067038, "timestamp": 1790102603.6565673, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.615323745, "recv_end_ts": 9120582.617316635, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.617499232, "preprocess_end_ts": 9120582.617834708, "forward_start_ts": 9120582.617844084, "forward_end_ts": 9120582.623157457, "timestamp": 1790102603.6586914, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.625230229, "send_end_ts": 9120582.62562664} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.617709432, "preprocess_end_ts": 9120582.618145688, "forward_start_ts": 9120582.618157173, "forward_end_ts": 9120582.625210725, "timestamp": 1790102603.6588135, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.625308787, "send_end_ts": 9120582.625747615} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.625877107, "preprocess_end_ts": 9120582.626224307, "forward_start_ts": 9120582.626234315, "forward_end_ts": 9120582.631413292, "timestamp": 1790102603.664929, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.623729687, "recv_end_ts": 9120582.62571517, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.625945903, "preprocess_end_ts": 9120582.626318509, "forward_start_ts": 9120582.62632906, "forward_end_ts": 9120582.63149246, "timestamp": 1790102603.6649878, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.623764066, "recv_end_ts": 9120582.62579982, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.625981929, "preprocess_end_ts": 9120582.626311673, "forward_start_ts": 9120582.626321288, "forward_end_ts": 9120582.63160772, "timestamp": 1790102603.6671853, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.633735837, "send_end_ts": 9120582.634120768} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.626183568, "preprocess_end_ts": 9120582.62661848, "forward_start_ts": 9120582.626630077, "forward_end_ts": 9120582.6337102, "timestamp": 1790102603.6673136, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.633808337, "send_end_ts": 9120582.634248195} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.634447675, "preprocess_end_ts": 9120582.63480125, "forward_start_ts": 9120582.634811662, "forward_end_ts": 9120582.639880544, "timestamp": 1790102603.6734767, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.63219121, "recv_end_ts": 9120582.634303868, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.63435824, "preprocess_end_ts": 9120582.63472431, "forward_start_ts": 9120582.634734532, "forward_end_ts": 9120582.640016876, "timestamp": 1790102603.6735172, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.63213902, "recv_end_ts": 9120582.634207767, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.634462172, "preprocess_end_ts": 9120582.63478484, "forward_start_ts": 9120582.634794144, "forward_end_ts": 9120582.640136449, "timestamp": 1790102603.6756687, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.642227378, "send_end_ts": 9120582.642603584} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.634682417, "preprocess_end_ts": 9120582.635108972, "forward_start_ts": 9120582.635119941, "forward_end_ts": 9120582.64221128, "timestamp": 1790102603.6758075, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.64230869, "send_end_ts": 9120582.642742233} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.642930498, "preprocess_end_ts": 9120582.64326802, "forward_start_ts": 9120582.6432767, "forward_end_ts": 9120582.648492845, "timestamp": 1790102603.6820428, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.640663384, "recv_end_ts": 9120582.642789513, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.642838014, "preprocess_end_ts": 9120582.64318108, "forward_start_ts": 9120582.64319026, "forward_end_ts": 9120582.648611909, "timestamp": 1790102603.6821089, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.640733434, "recv_end_ts": 9120582.642695213, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.642951105, "preprocess_end_ts": 9120582.64327012, "forward_start_ts": 9120582.643279066, "forward_end_ts": 9120582.648580177, "timestamp": 1790102603.6842473, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.650807079, "send_end_ts": 9120582.651182815} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.64317249, "preprocess_end_ts": 9120582.643593661, "forward_start_ts": 9120582.643604832, "forward_end_ts": 9120582.650789414, "timestamp": 1790102603.6843874, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.650882209, "send_end_ts": 9120582.65132139} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.651511695, "preprocess_end_ts": 9120582.651844472, "forward_start_ts": 9120582.651853641, "forward_end_ts": 9120582.656878868, "timestamp": 1790102603.690367, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.649217356, "recv_end_ts": 9120582.6513709, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.65140676, "preprocess_end_ts": 9120582.651739815, "forward_start_ts": 9120582.651748355, "forward_end_ts": 9120582.656938508, "timestamp": 1790102603.6904323, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.64931462, "recv_end_ts": 9120582.651264912, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.651514748, "preprocess_end_ts": 9120582.651831552, "forward_start_ts": 9120582.651840601, "forward_end_ts": 9120582.65721628, "timestamp": 1790102603.6933012, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.65985598, "send_end_ts": 9120582.660236675} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.651739929, "preprocess_end_ts": 9120582.652174477, "forward_start_ts": 9120582.65218502, "forward_end_ts": 9120582.659841204, "timestamp": 1790102603.6934776, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.659936965, "send_end_ts": 9120582.660411868} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.66045627, "preprocess_end_ts": 9120582.660783246, "forward_start_ts": 9120582.660792032, "forward_end_ts": 9120582.666339248, "timestamp": 1790102603.7000058, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.657644669, "recv_end_ts": 9120582.66031364, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.660600586, "preprocess_end_ts": 9120582.660947189, "forward_start_ts": 9120582.660958167, "forward_end_ts": 9120582.666573424, "timestamp": 1790102603.7000759, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.657538248, "recv_end_ts": 9120582.660460543, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.821541695, "preprocess_end_ts": 9120583.822123745, "forward_start_ts": 9120583.822139362, "forward_end_ts": 9120583.828476297, "timestamp": 1790102604.8637803, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.830264147, "send_end_ts": 9120583.830715194} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.82172466, "preprocess_end_ts": 9120583.822294204, "forward_start_ts": 9120583.82230907, "forward_end_ts": 9120583.830244573, "timestamp": 1790102604.8639398, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.830370191, "send_end_ts": 9120583.83087368} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.831101157, "preprocess_end_ts": 9120583.831546476, "forward_start_ts": 9120583.831559015, "forward_end_ts": 9120583.837595148, "timestamp": 1790102604.8716521, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.821289523, "recv_end_ts": 9120583.830886656, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.831336629, "preprocess_end_ts": 9120583.83196254, "forward_start_ts": 9120583.831993213, "forward_end_ts": 9120583.838179354, "timestamp": 1790102604.8717523, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.821598507, "recv_end_ts": 9120583.83101284, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.844497563, "preprocess_end_ts": 9120583.844813723, "forward_start_ts": 9120583.844823988, "forward_end_ts": 9120583.851622906, "timestamp": 1790102604.8871188, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.853661515, "send_end_ts": 9120583.854054332} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.84607402, "preprocess_end_ts": 9120583.84649125, "forward_start_ts": 9120583.846503396, "forward_end_ts": 9120583.853645235, "timestamp": 1790102604.887264, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.853750424, "send_end_ts": 9120583.854198288} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.854285724, "preprocess_end_ts": 9120583.854634887, "forward_start_ts": 9120583.854645245, "forward_end_ts": 9120583.859757915, "timestamp": 1790102604.8933077, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.844375689, "recv_end_ts": 9120583.854136454, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.854376545, "preprocess_end_ts": 9120583.854727993, "forward_start_ts": 9120583.854738573, "forward_end_ts": 9120583.859866332, "timestamp": 1790102604.8933706, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.844526729, "recv_end_ts": 9120583.854231462, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.85440786, "preprocess_end_ts": 9120583.854735203, "forward_start_ts": 9120583.854744412, "forward_end_ts": 9120583.860074464, "timestamp": 1790102604.895916, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.862467133, "send_end_ts": 9120583.86285158} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.854629343, "preprocess_end_ts": 9120583.855060486, "forward_start_ts": 9120583.855072329, "forward_end_ts": 9120583.862451376, "timestamp": 1790102604.896046, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.862548715, "send_end_ts": 9120583.86298042} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.863156855, "preprocess_end_ts": 9120583.863527492, "forward_start_ts": 9120583.863537077, "forward_end_ts": 9120583.868671449, "timestamp": 1790102604.9022155, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.860549659, "recv_end_ts": 9120583.863022398, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.863093857, "preprocess_end_ts": 9120583.863435747, "forward_start_ts": 9120583.863445677, "forward_end_ts": 9120583.868781999, "timestamp": 1790102604.902278, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.860511964, "recv_end_ts": 9120583.862957884, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.863200076, "preprocess_end_ts": 9120583.863517843, "forward_start_ts": 9120583.863527136, "forward_end_ts": 9120583.868867451, "timestamp": 1790102604.9044716, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.871042717, "send_end_ts": 9120583.871406928} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.86340592, "preprocess_end_ts": 9120583.863826012, "forward_start_ts": 9120583.863836592, "forward_end_ts": 9120583.871030165, "timestamp": 1790102604.9046118, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.871124, "send_end_ts": 9120583.87154668} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.871732265, "preprocess_end_ts": 9120583.872077122, "forward_start_ts": 9120583.872086916, "forward_end_ts": 9120583.87720402, "timestamp": 1790102604.9106915, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.869396372, "recv_end_ts": 9120583.871590275, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.87163014, "preprocess_end_ts": 9120583.871973757, "forward_start_ts": 9120583.871982671, "forward_end_ts": 9120583.877214683, "timestamp": 1790102604.910703, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.869494393, "recv_end_ts": 9120583.871493684, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.871741671, "preprocess_end_ts": 9120583.87206858, "forward_start_ts": 9120583.872077484, "forward_end_ts": 9120583.877404189, "timestamp": 1790102604.91298, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.879556447, "send_end_ts": 9120583.879915643} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.871962866, "preprocess_end_ts": 9120583.872391593, "forward_start_ts": 9120583.872401793, "forward_end_ts": 9120583.879540468, "timestamp": 1790102604.9131334, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.879632792, "send_end_ts": 9120583.880067756} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.880156256, "preprocess_end_ts": 9120583.880481124, "forward_start_ts": 9120583.880489904, "forward_end_ts": 9120583.88560634, "timestamp": 1790102604.9191005, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.877898745, "recv_end_ts": 9120583.880015442, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.880242497, "preprocess_end_ts": 9120583.88057212, "forward_start_ts": 9120583.880580816, "forward_end_ts": 9120583.885655653, "timestamp": 1790102604.9191403, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.877860919, "recv_end_ts": 9120583.880106712, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.880259756, "preprocess_end_ts": 9120583.88057252, "forward_start_ts": 9120583.880582193, "forward_end_ts": 9120583.885980275, "timestamp": 1790102604.9214914, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.8880694, "send_end_ts": 9120583.888426863} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.880488636, "preprocess_end_ts": 9120583.880906789, "forward_start_ts": 9120583.880917268, "forward_end_ts": 9120583.88806401, "timestamp": 1790102604.921652, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.888157409, "send_end_ts": 9120583.888586646} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.888661517, "preprocess_end_ts": 9120583.888987511, "forward_start_ts": 9120583.8889958, "forward_end_ts": 9120583.894107789, "timestamp": 1790102604.9276292, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.886290956, "recv_end_ts": 9120583.88852129, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.888765827, "preprocess_end_ts": 9120583.889099708, "forward_start_ts": 9120583.889109552, "forward_end_ts": 9120583.894193923, "timestamp": 1790102604.9276834, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.88630878, "recv_end_ts": 9120583.888627024, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.888755564, "preprocess_end_ts": 9120583.889065951, "forward_start_ts": 9120583.889074624, "forward_end_ts": 9120583.894493733, "timestamp": 1790102604.9300442, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.896602778, "send_end_ts": 9120583.896979593} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.889009971, "preprocess_end_ts": 9120583.88942914, "forward_start_ts": 9120583.88943924, "forward_end_ts": 9120583.896590112, "timestamp": 1790102604.9301734, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.89668142, "send_end_ts": 9120583.897108141} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.897287816, "preprocess_end_ts": 9120583.89762028, "forward_start_ts": 9120583.897630377, "forward_end_ts": 9120583.90267408, "timestamp": 1790102604.936159, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.894849937, "recv_end_ts": 9120583.897149628, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.897259109, "preprocess_end_ts": 9120583.89759168, "forward_start_ts": 9120583.897600047, "forward_end_ts": 9120583.902698973, "timestamp": 1790102604.936189, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.894804897, "recv_end_ts": 9120583.897117823, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.897304306, "preprocess_end_ts": 9120583.897619875, "forward_start_ts": 9120583.897628644, "forward_end_ts": 9120583.902977347, "timestamp": 1790102604.9385567, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.905150736, "send_end_ts": 9120583.905492732} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.897520913, "preprocess_end_ts": 9120583.89793452, "forward_start_ts": 9120583.897944763, "forward_end_ts": 9120583.905138481, "timestamp": 1790102604.9387238, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.905231489, "send_end_ts": 9120583.905658696} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.905695103, "preprocess_end_ts": 9120583.906016463, "forward_start_ts": 9120583.906024937, "forward_end_ts": 9120583.911124228, "timestamp": 1790102604.9446628, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.903391425, "recv_end_ts": 9120583.905563416, "send_start_ts": null, "send_end_ts": null} +{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.905837096, "preprocess_end_ts": 9120583.90618202, "forward_start_ts": 9120583.906192401, "forward_end_ts": 9120583.911191944, "timestamp": 1790102604.9446793, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.903329002, "recv_end_ts": 9120583.905698072, "send_start_ts": null, "send_end_ts": null} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/requests.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/requests.jsonl new file mode 100644 index 00000000..0a6f40d0 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/requests.jsonl @@ -0,0 +1,76 @@ +{"request_id": "warmup-q0", "burst": "warmup", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120569.869602332, "finish_monotonic": 9120576.285742093, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "warmup-q1", "burst": "warmup", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120569.872307455, "finish_monotonic": 9120576.285754615, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "warmup-q2", "burst": "warmup", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120569.872824574, "finish_monotonic": 9120576.294476116, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "warmup-q3", "burst": "warmup", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120569.873199455, "finish_monotonic": 9120576.294771325, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q0", "burst": "b8-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120577.547693444, "finish_monotonic": 9120577.567003388, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q1", "burst": "b8-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120577.548223112, "finish_monotonic": 9120577.56724617, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q2", "burst": "b8-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120577.54847057, "finish_monotonic": 9120577.58936014, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q3", "burst": "b8-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120577.548694393, "finish_monotonic": 9120577.589365425, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q4", "burst": "b8-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9120577.548904004, "finish_monotonic": 9120577.598108912, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q5", "burst": "b8-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9120577.54911402, "finish_monotonic": 9120577.598117024, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q6", "burst": "b8-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9120577.549561707, "finish_monotonic": 9120577.60704437, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r0-q7", "burst": "b8-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9120577.54983673, "finish_monotonic": 9120577.607169196, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q0", "burst": "b8-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9120578.810612688, "finish_monotonic": 9120578.830455095, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q1", "burst": "b8-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9120578.811313707, "finish_monotonic": 9120578.830462607, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q2", "burst": "b8-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9120578.81171338, "finish_monotonic": 9120578.852622977, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q3", "burst": "b8-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9120578.812109131, "finish_monotonic": 9120578.852632107, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q4", "burst": "b8-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9120578.812452069, "finish_monotonic": 9120578.861199869, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q5", "burst": "b8-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9120578.812766358, "finish_monotonic": 9120578.861310275, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q6", "burst": "b8-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9120578.813098883, "finish_monotonic": 9120578.869919628, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r1-q7", "burst": "b8-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9120578.813434564, "finish_monotonic": 9120578.870209333, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q0", "burst": "b8-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9120580.07300282, "finish_monotonic": 9120580.0932383, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q1", "burst": "b8-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9120580.073717771, "finish_monotonic": 9120580.093249517, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q2", "burst": "b8-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9120580.074123846, "finish_monotonic": 9120580.119620783, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q3", "burst": "b8-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9120580.07445321, "finish_monotonic": 9120580.119723072, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q4", "burst": "b8-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9120580.074752178, "finish_monotonic": 9120580.129679734, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q5", "burst": "b8-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9120580.074970948, "finish_monotonic": 9120580.12968696, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q6", "burst": "b8-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9120580.075345183, "finish_monotonic": 9120580.139080467, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b8-r2-q7", "burst": "b8-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9120580.075559553, "finish_monotonic": 9120580.139085893, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q0", "burst": "b16-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120581.341815123, "finish_monotonic": 9120581.361618357, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q1", "burst": "b16-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120581.342365772, "finish_monotonic": 9120581.36186946, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q2", "burst": "b16-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120581.342633724, "finish_monotonic": 9120581.380279476, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q3", "burst": "b16-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120581.342859657, "finish_monotonic": 9120581.380283996, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q4", "burst": "b16-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9120581.343072204, "finish_monotonic": 9120581.386880303, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q5", "burst": "b16-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9120581.3432637, "finish_monotonic": 9120581.386887154, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q6", "burst": "b16-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9120581.343454594, "finish_monotonic": 9120581.393417716, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q7", "burst": "b16-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9120581.343649356, "finish_monotonic": 9120581.393422365, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q8", "burst": "b16-r0", "round": 0, "index": 8, "rank": 0, "submit_monotonic": 9120581.343870228, "finish_monotonic": 9120581.402437443, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q9", "burst": "b16-r0", "round": 0, "index": 9, "rank": 1, "submit_monotonic": 9120581.344084216, "finish_monotonic": 9120581.40244758, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q10", "burst": "b16-r0", "round": 0, "index": 10, "rank": 0, "submit_monotonic": 9120581.344304737, "finish_monotonic": 9120581.409558967, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q11", "burst": "b16-r0", "round": 0, "index": 11, "rank": 1, "submit_monotonic": 9120581.34456862, "finish_monotonic": 9120581.409563392, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q12", "burst": "b16-r0", "round": 0, "index": 12, "rank": 0, "submit_monotonic": 9120581.344810119, "finish_monotonic": 9120581.416050447, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q13", "burst": "b16-r0", "round": 0, "index": 13, "rank": 1, "submit_monotonic": 9120581.345005594, "finish_monotonic": 9120581.416054724, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q14", "burst": "b16-r0", "round": 0, "index": 14, "rank": 0, "submit_monotonic": 9120581.345214237, "finish_monotonic": 9120581.42277076, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r0-q15", "burst": "b16-r0", "round": 0, "index": 15, "rank": 1, "submit_monotonic": 9120581.345402837, "finish_monotonic": 9120581.422895985, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q0", "burst": "b16-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9120582.575048696, "finish_monotonic": 9120582.593977828, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q1", "burst": "b16-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9120582.575551683, "finish_monotonic": 9120582.593985632, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q2", "burst": "b16-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9120582.575873053, "finish_monotonic": 9120582.616082978, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q3", "burst": "b16-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9120582.57612324, "finish_monotonic": 9120582.616088783, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q4", "burst": "b16-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9120582.576336274, "finish_monotonic": 9120582.624345195, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q5", "burst": "b16-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9120582.576600255, "finish_monotonic": 9120582.624350388, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q6", "burst": "b16-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9120582.576868795, "finish_monotonic": 9120582.6327847, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q7", "burst": "b16-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9120582.577115236, "finish_monotonic": 9120582.63279038, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q8", "burst": "b16-r1", "round": 1, "index": 8, "rank": 0, "submit_monotonic": 9120582.57734158, "finish_monotonic": 9120582.641252786, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q9", "burst": "b16-r1", "round": 1, "index": 9, "rank": 1, "submit_monotonic": 9120582.577593436, "finish_monotonic": 9120582.64126444, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q10", "burst": "b16-r1", "round": 1, "index": 10, "rank": 0, "submit_monotonic": 9120582.577844236, "finish_monotonic": 9120582.649838036, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q11", "burst": "b16-r1", "round": 1, "index": 11, "rank": 1, "submit_monotonic": 9120582.578143867, "finish_monotonic": 9120582.649843188, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q12", "burst": "b16-r1", "round": 1, "index": 12, "rank": 0, "submit_monotonic": 9120582.578371843, "finish_monotonic": 9120582.658064105, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q13", "burst": "b16-r1", "round": 1, "index": 13, "rank": 1, "submit_monotonic": 9120582.578631695, "finish_monotonic": 9120582.658071209, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q14", "burst": "b16-r1", "round": 1, "index": 14, "rank": 0, "submit_monotonic": 9120582.578840362, "finish_monotonic": 9120582.667599283, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r1-q15", "burst": "b16-r1", "round": 1, "index": 15, "rank": 1, "submit_monotonic": 9120582.579079762, "finish_monotonic": 9120582.667743009, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q0", "burst": "b16-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9120583.81993872, "finish_monotonic": 9120583.839589516, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q1", "burst": "b16-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9120583.82045228, "finish_monotonic": 9120583.839597132, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q2", "burst": "b16-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9120583.820687912, "finish_monotonic": 9120583.86100596, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q3", "burst": "b16-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9120583.820890976, "finish_monotonic": 9120583.861143965, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q4", "burst": "b16-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9120583.821091052, "finish_monotonic": 9120583.869980576, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q5", "burst": "b16-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9120583.821283503, "finish_monotonic": 9120583.869986843, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q6", "burst": "b16-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9120583.82154948, "finish_monotonic": 9120583.878445651, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q7", "burst": "b16-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9120583.82179376, "finish_monotonic": 9120583.878554093, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q8", "burst": "b16-r2", "round": 2, "index": 8, "rank": 0, "submit_monotonic": 9120583.822033968, "finish_monotonic": 9120583.886686856, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q9", "burst": "b16-r2", "round": 2, "index": 9, "rank": 1, "submit_monotonic": 9120583.822236164, "finish_monotonic": 9120583.886816649, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q10", "burst": "b16-r2", "round": 2, "index": 10, "rank": 0, "submit_monotonic": 9120583.822420392, "finish_monotonic": 9120583.8952481, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q11", "burst": "b16-r2", "round": 2, "index": 11, "rank": 1, "submit_monotonic": 9120583.82260614, "finish_monotonic": 9120583.895373551, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q12", "burst": "b16-r2", "round": 2, "index": 12, "rank": 0, "submit_monotonic": 9120583.822795508, "finish_monotonic": 9120583.903785357, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q13", "burst": "b16-r2", "round": 2, "index": 13, "rank": 1, "submit_monotonic": 9120583.82299076, "finish_monotonic": 9120583.903789992, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q14", "burst": "b16-r2", "round": 2, "index": 14, "rank": 0, "submit_monotonic": 9120583.82317668, "finish_monotonic": 9120583.912484672, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} +{"request_id": "b16-r2-q15", "burst": "b16-r2", "round": 2, "index": 15, "rank": 1, "submit_monotonic": 9120583.823371433, "finish_monotonic": 9120583.912491629, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/summary.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/summary.json new file mode 100644 index 00000000..637f420b --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/summary.json @@ -0,0 +1,186 @@ +{ + "model_config": "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/data/config/models/Qwen3-30B-A3B-tiny.json", + "num_gpu_blocks": 600666, + "block_size": 16, + "engine_args": { + "model": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", + "served_model_name": null, + "tokenizer": null, + "hf_config_path": null, + "runner": "auto", + "convert": "auto", + "task": null, + "skip_tokenizer_init": true, + "enable_prompt_embeds": false, + "tokenizer_mode": "auto", + "trust_remote_code": false, + "allowed_local_media_path": "", + "download_dir": null, + "safetensors_load_strategy": "lazy", + "load_format": "dummy", + "config_format": "auto", + "dtype": "bfloat16", + "kv_cache_dtype": "auto", + "seed": 0, + "max_model_len": 512, + "distributed_executor_backend": null, + "pipeline_parallel_size": 2, + "tensor_parallel_size": 1, + "decode_context_parallel_size": 1, + "data_parallel_size": 2, + "data_parallel_rank": null, + "data_parallel_start_rank": null, + "data_parallel_size_local": null, + "data_parallel_address": null, + "data_parallel_rpc_port": null, + "data_parallel_hybrid_lb": false, + "data_parallel_backend": "mp", + "enable_expert_parallel": true, + "enable_eplb": false, + "num_redundant_experts": 0, + "eplb_window_size": 1000, + "eplb_step_interval": 3000, + "eplb_log_balancedness": false, + "max_parallel_loading_workers": null, + "block_size": 16, + "enable_prefix_caching": false, + "prefix_caching_hash_algo": "sha256", + "disable_sliding_window": false, + "disable_cascade_attn": false, + "swap_space": 4, + "cpu_offload_gb": 0, + "gpu_memory_utilization": 0.5, + "kv_cache_memory_bytes": null, + "max_num_batched_tokens": 256, + "max_num_partial_prefills": 1, + "max_long_partial_prefills": 1, + "long_prefill_token_threshold": 0, + "max_num_seqs": 4, + "max_logprobs": 20, + "disable_log_stats": true, + "revision": null, + "code_revision": null, + "rope_theta": null, + "hf_token": null, + "tokenizer_revision": null, + "quantization": null, + "enforce_eager": true, + "max_seq_len_to_capture": 8192, + "disable_custom_all_reduce": false, + "interleave_mm_strings": false, + "mm_processor_kwargs": null, + "disable_mm_preprocessor_cache": false, + "mm_processor_cache_gb": 4, + "mm_encoder_tp_mode": "weights", + "io_processor_plugin": null, + "skip_mm_profiling": false, + "enable_lora": false, + "enable_lora_bias": false, + "max_loras": 1, + "max_lora_rank": 16, + "default_mm_loras": null, + "fully_sharded_loras": false, + "max_cpu_loras": null, + "lora_dtype": "auto", + "lora_extra_vocab_size": 256, + "ray_workers_use_nsight": false, + "num_gpu_blocks_override": null, + "num_lookahead_slots": 0, + "ignore_patterns": null, + "preemption_mode": null, + "scheduler_delay_factor": 0.0, + "enable_chunked_prefill": true, + "disable_chunked_mm_input": false, + "disable_hybrid_kv_cache_manager": false, + "guided_decoding_backend": "auto", + "guided_decoding_disable_fallback": false, + "guided_decoding_disable_any_whitespace": false, + "guided_decoding_disable_additional_properties": false, + "logits_processor_pattern": null, + "speculative_config": null, + "show_hidden_metrics_for_version": null, + "otlp_traces_endpoint": null, + "collect_detailed_traces": null, + "disable_async_output_proc": false, + "scheduling_policy": "fcfs", + "scheduler_cls": "vllm.v1.core.sched.scheduler.Scheduler", + "override_pooler_config": null, + "worker_cls": "auto", + "worker_extension_cls": "", + "kv_transfer_config": null, + "kv_events_config": null, + "generation_config": "auto", + "enable_sleep_mode": false, + "model_impl": "auto", + "override_attention_dtype": null, + "calculate_kv_scales": false, + "mamba_cache_dtype": "auto", + "mamba_ssm_cache_dtype": "auto", + "reasoning_parser": "", + "use_tqdm_on_load": true, + "pt_load_map_location": "cpu", + "enable_multimodal_encoder_data_parallel": false, + "logits_processors": null, + "async_scheduling": false, + "kv_sharing_fast_prefill": false, + "enable_log_requests": false + }, + "rounds": [ + { + "label": "warmup", + "round": 0, + "num_requests": 4, + "wall_minus_monotonic_before": 1780982021.0330575, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.2527571953833103 + }, + { + "label": "b8-r0", + "round": 0, + "num_requests": 8, + "wall_minus_monotonic_before": 1780982021.0330584, + "wall_minus_monotonic_after": 1780982021.0330582, + "idle_wait_s": 1.203191703185439 + }, + { + "label": "b8-r1", + "round": 1, + "num_requests": 8, + "wall_minus_monotonic_before": 1780982021.0330582, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.2025362998247147 + }, + { + "label": "b8-r2", + "round": 2, + "num_requests": 8, + "wall_minus_monotonic_before": 1780982021.0330584, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.202582010999322 + }, + { + "label": "b16-r0", + "round": 0, + "num_requests": 16, + "wall_minus_monotonic_before": 1780982021.0330582, + "wall_minus_monotonic_after": 1780982021.0330586, + "idle_wait_s": 1.1520026791840792 + }, + { + "label": "b16-r1", + "round": 1, + "num_requests": 16, + "wall_minus_monotonic_before": 1780982021.0330584, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.1520404387265444 + }, + { + "label": "b16-r2", + "round": 2, + "num_requests": 16, + "wall_minus_monotonic_before": 1780982021.0330582, + "wall_minus_monotonic_after": 1780982021.0330584, + "idle_wait_s": 1.152070851996541 + } + ] +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/vllm_import.txt b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/vllm_import.txt new file mode 100644 index 00000000..cd9a481f --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/vllm_import.txt @@ -0,0 +1 @@ +VLLM_IMPORT 0.10.2 /tmp/stage_admission_pp/sa-pp-20260923b/overlay/vllm/__init__.py /tmp/stage_admission_pp/sa-pp-20260923b/overlay/vllm/v1/frontier_trace.py diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/worker_env.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/worker_env.json new file mode 100644 index 00000000..00b7f124 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/worker_env.json @@ -0,0 +1 @@ +{"python": "3.12.11", "torch": "2.8.0+cu128", "cuda_available": true, "device_count": 4, "devices": ["NVIDIA H800", "NVIDIA H800", "NVIDIA H800", "NVIDIA H800"]} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py new file mode 100644 index 00000000..7e9aae2a --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py @@ -0,0 +1,46 @@ +"""Split vLLM stage-0 non-overlap into start and end offsets of paired forwards. + +For each M3 pair (lane-0 forward, lane-1 forward) of each round, the union +minus the overlap equals |start difference| + |end difference|. The start +part is where admission could act; the end part comes from per-rank forward +duration variation. Also reports M5 with every lane-1 end aligned to the +lane-0 duration, i.e. the co-execution vLLM would show with the equal +per-forward durations of Frontier's dummy predictor. + +Usage: python decompose_co_execution.py [ ...] +""" +import json +import statistics +import sys +from pathlib import Path + +from tests.comparison.stage_admission_pp.compare_lanes import vllm_forwards +from tests.e2e.stage_admission_matrix import interval_overlap + +run_dir = Path(sys.argv[1]) +summary = {} +for model in sys.argv[2:]: + runs = vllm_forwards(run_dir / "runs" / model) + for (burst, round_index), run in sorted(runs.items()): + stage0 = [f for f in run["forwards"] if f["stage"] == 0] + lanes = {lane: sorted((f for f in stage0 if f["lane"] == lane), key=lambda f: f["start"]) for lane in (0, 1)} + start_part = end_part = 0.0 + equal_duration = [] + for first, second in zip(lanes[0], lanes[1]): + start_part += abs(first["start"] - second["start"]) + end_part += abs(first["end"] - second["end"]) + equal_duration.append((first["start"], first["end"], 0)) + equal_duration.append((second["start"], second["start"] + first["end"] - first["start"], 1)) + observed = interval_overlap([(f["start"], f["end"], f["lane"]) for f in stage0]) + aligned = interval_overlap(equal_duration) + durations = [f["end"] - f["start"] for f in stage0] + summary[f"{model}/n{burst}/r{round_index}"] = { + "pairs": min(len(lanes[0]), len(lanes[1])), + "M5_observed": round(observed["multi_lane_busy_time"] / observed["busy_time"], 4), + "M5_equal_durations": round(aligned["multi_lane_busy_time"] / aligned["busy_time"], 4), + "non_overlap_ms_from_start_offsets": round(1e3 * start_part, 3), + "non_overlap_ms_from_end_offsets": round(1e3 * end_part, 3), + "stage0_duration_ms_median": round(1e3 * statistics.median(durations), 3), + "stage0_duration_ms_cv": round(statistics.pstdev(durations) / statistics.mean(durations), 3), + } +print(json.dumps(summary, indent=1)) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py new file mode 100644 index 00000000..2a4865db --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py @@ -0,0 +1,68 @@ +"""Explain every path-T difference of the P3 comparison from the stage ledger. + +For each T case whose metrics differ, check that every (stage, lane) runs the +same ordered batches with the same component durations before and after, so +the difference is start times only; report the §4.5 metric as absolute and as +a fraction of stage busy time. + +Usage: python explain_t_path.py +""" +import json +import sys +from collections import defaultdict +from pathlib import Path + +root, compare_path, output = Path(sys.argv[1]), Path(sys.argv[2]), Path(sys.argv[3]) + + +def lane_rows(set_name, case_id): + ledger = next((root / set_name / case_id / "metrics").rglob("frontier_stage_batch_ledger.jsonl")) + rows = defaultdict(list) + for line in ledger.read_text().splitlines(): + row = json.loads(line) + if row["execution_scope"] == "ATTN_DP_LANE": + rows[(row["stage_id"], row["replica_local_id"])].append(row) + for key in rows: + rows[key].sort(key=lambda row: row["stage_start_ts"]) + return rows + + +def signature(row): + return (tuple(row["request_ids"]), round(row["stage_end_ts"] - row["stage_start_ts"], 9), + json.dumps(row["execution_time"], sort_keys=True)) + + +def fraction(metric): + return {stage: round(m["multi_lane_busy_time"] / m["busy_time"], 4) for stage, m in metric.items()} + + +report = [] +for row in json.load(open(compare_path)): + if row["path"] != "T" or row["verdict"] == "PASS": + continue + before, after = lane_rows("base", row["case_id"]), lane_rows("after", row["case_id"]) + same_work = before.keys() == after.keys() and all( + [signature(r) for r in before[key]] == [signature(r) for r in after[key]] for key in before + ) + first_start = {side: {lane: rows[(0, lane)][0]["stage_start_ts"] for (stage, lane) in rows if stage == 0} + for side, rows in (("before", before), ("after", after))} + report.append({ + "case_id": row["case_id"], "verdict": row["verdict"], + "same_batches_and_component_durations": same_work, + "differing_files": row.get("differing_files"), + "witness_increase": row.get("witness_increase"), + "multi_lane_busy_time": {side: {s: m["multi_lane_busy_time"] for s, m in row[f"lane_metric_{side}"].items()} + for side in ("before", "after")}, + "co_execution_fraction": {side: fraction(row[f"lane_metric_{side}"]) for side in ("before", "after")}, + "peak_lanes": {side: {s: m["peak_lanes"] for s, m in row[f"lane_metric_{side}"].items()} + for side in ("before", "after")}, + "first_stage0_start": first_start, + }) +output.write_text(json.dumps(report, indent=1, sort_keys=True)) +for item in report: + print(item["case_id"], item["verdict"], "same_work=", item["same_batches_and_component_durations"], + "frac", item["co_execution_fraction"]["before"].get("MONOLITHIC/0/0"), "->", + item["co_execution_fraction"]["after"].get("MONOLITHIC/0/0"), + "peak", item["peak_lanes"]["before"].get("MONOLITHIC/0/0"), "->", + item["peak_lanes"]["after"].get("MONOLITHIC/0/0"), + "starts", item["first_stage0_start"]["before"], "->", item["first_stage0_start"]["after"]) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_integration_compare.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_integration_compare.json new file mode 100644 index 00000000..61d33632 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_integration_compare.json @@ -0,0 +1,12 @@ +{ + "regressions": [], + "new_failures": [], + "now_passing": [], + "skip_changes": [], + "only_before": [], + "only_after": [ + "tests.integration.test_stage_admission_pipeline_lanes::test_dense_lanes_start_in_the_same_first_forward", + "tests.integration.test_stage_admission_pipeline_lanes::test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4-expected0]", + "tests.integration.test_stage_admission_pipeline_lanes::test_moe_lanes_complete_every_request[G3a-moe-dp4-pp2-n8-expected1]" + ] +} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_unit_compare.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_unit_compare.json new file mode 100644 index 00000000..6615fe2f --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_unit_compare.json @@ -0,0 +1,15 @@ +{ + "regressions": [], + "new_failures": [], + "now_passing": [], + "skip_changes": [], + "only_before": [], + "only_after": [ + "tests.unit.test_mixed_layer_decode_ffn_scheduling::test_decode_ffn_dense_groups_keep_counter_order_around_a_queued_ep_wave", + "tests.unit.test_shared_forward_group_admission::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0]", + "tests.unit.test_shared_forward_group_admission::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[1]", + "tests.unit.test_stage_execution_context::test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave", + "tests.unit.test_stage_execution_context::test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket", + "tests.unit.test_stage_execution_context::test_queued_ep_wave_orders_full_stage_work_on_both_sides" + ] +} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/p3_t_path_explanation.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/p3_t_path_explanation.json new file mode 100644 index 00000000..6a7451c5 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/p3_t_path_explanation.json @@ -0,0 +1,436 @@ +[ + { + "case_id": "G4-dense-dp2-pp2-n4", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.7143, + "MONOLITHIC/0/1": 0.7143 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.05, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.3, + "MONOLITHIC/0/1": 0.3 + }, + "before": { + "MONOLITHIC/0/0": 0.25, + "MONOLITHIC/0/1": 0.24999999999999997 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + }, + { + "case_id": "G4-dense-dp2-pp2-n8", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.8182, + "MONOLITHIC/0/1": 0.8182 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.05, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.49999999999999994, + "MONOLITHIC/0/1": 0.49999999999999994 + }, + "before": { + "MONOLITHIC/0/0": 0.44999999999999996, + "MONOLITHIC/0/1": 0.4499999999999999 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": true + }, + { + "case_id": "G4-dense-dp2-pp3-n4", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0, + "MONOLITHIC/0/2": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.3333, + "MONOLITHIC/0/1": 0.3333, + "MONOLITHIC/0/2": 0.3333 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.036000000000000004, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.21600000000000003, + "MONOLITHIC/0/1": 0.21600000000000008, + "MONOLITHIC/0/2": 0.21600000000000005 + }, + "before": { + "MONOLITHIC/0/0": 0.10800000000000004, + "MONOLITHIC/0/1": 0.10800000000000004, + "MONOLITHIC/0/2": 0.10800000000000001 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + }, + { + "case_id": "G4-dense-dp2-pp3-n8", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0, + "MONOLITHIC/0/2": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.375, + "MONOLITHIC/0/1": 0.375, + "MONOLITHIC/0/2": 0.375 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.07200000000000001, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.39600000000000013, + "MONOLITHIC/0/1": 0.39600000000000013, + "MONOLITHIC/0/2": 0.3960000000000002 + }, + "before": { + "MONOLITHIC/0/0": 0.21600000000000008, + "MONOLITHIC/0/1": 0.21600000000000014, + "MONOLITHIC/0/2": 0.2160000000000001 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": true + }, + { + "case_id": "G4-dense-dp4-pp2-n8", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.8462, + "MONOLITHIC/0/1": 0.8462 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0, + "2": 0.0, + "3": 0.0 + }, + "before": { + "0": 0.15000000000000002, + "1": 0.0, + "2": 0.05, + "3": 0.1 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.3, + "MONOLITHIC/0/1": 0.3 + }, + "before": { + "MONOLITHIC/0/0": 0.55, + "MONOLITHIC/0/1": 0.55 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 4, + "MONOLITHIC/0/1": 4 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "STOP", + "witness_increase": false + }, + { + "case_id": "G4-dense-dp4-pp3-n8", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0, + "MONOLITHIC/0/2": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.8462, + "MONOLITHIC/0/1": 0.8462, + "MONOLITHIC/0/2": 0.8462 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0, + "2": 0.0, + "3": 0.0 + }, + "before": { + "0": 0.10800000000000001, + "1": 0.0, + "2": 0.036000000000000004, + "3": 0.07200000000000001 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.21600000000000003, + "MONOLITHIC/0/1": 0.21600000000000008, + "MONOLITHIC/0/2": 0.21600000000000005 + }, + "before": { + "MONOLITHIC/0/0": 0.39600000000000013, + "MONOLITHIC/0/1": 0.3960000000000002, + "MONOLITHIC/0/2": 0.39600000000000024 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 4, + "MONOLITHIC/0/1": 4, + "MONOLITHIC/0/2": 4 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "STOP", + "witness_increase": false + }, + { + "case_id": "G7-dense-dp2-pp2-n8", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.6, + "MONOLITHIC/0/1": 0.6 + } + }, + "differing_files": [ + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/frontier_stage_batch_ledger.jsonl", + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/request_metrics.csv", + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.12000000000000001, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.48000000000000004, + "MONOLITHIC/0/1": 0.4800000000000001 + }, + "before": { + "MONOLITHIC/0/0": 0.36000000000000004, + "MONOLITHIC/0/1": 0.3600000000000001 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + }, + { + "case_id": "G7-dense-dp2-pp2-n16", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.7778, + "MONOLITHIC/0/1": 0.7778 + } + }, + "differing_files": [ + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/frontier_stage_batch_ledger.jsonl", + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/request_metrics.csv", + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.12000000000000001, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.9600000000000001, + "MONOLITHIC/0/1": 0.9600000000000001 + }, + "before": { + "MONOLITHIC/0/0": 0.8400000000000001, + "MONOLITHIC/0/1": 0.8400000000000001 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + } +] \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_dense_dp1_pp2.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_dense_dp1_pp2.json new file mode 100644 index 00000000..06c42922 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_dense_dp1_pp2.json @@ -0,0 +1,10 @@ +{ + "shape": { + "is_moe": false, + "attn_dp": 1, + "moe_ep": 1, + "stages": 2 + }, + "completed": 6, + "requests": 6 +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp1.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp1.json new file mode 100644 index 00000000..1bec72f1 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp1.json @@ -0,0 +1,10 @@ +{ + "shape": { + "is_moe": true, + "attn_dp": 2, + "moe_ep": 2, + "stages": 1 + }, + "completed": 6, + "requests": 6 +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp2.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp2.json new file mode 100644 index 00000000..3f52dce5 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp2.json @@ -0,0 +1,10 @@ +{ + "shape": { + "is_moe": true, + "attn_dp": 2, + "moe_ep": 2, + "stages": 2 + }, + "completed": 6, + "requests": 6 +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp3.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp3.json new file mode 100644 index 00000000..67b94dac --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp3.json @@ -0,0 +1,9 @@ +{ + "shape": { + "is_moe": true, + "attn_dp": 2, + "moe_ep": 2, + "stages": 3 + }, + "error": "ValueError('collective-sim physical topology requires cluster_total_devices 6 to be divisible by node size 4')" +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/base_moe_dp2_pp2.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/base_moe_dp2_pp2.json new file mode 100644 index 00000000..a1c86beb --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/base_moe_dp2_pp2.json @@ -0,0 +1,9 @@ +{ + "shape": { + "is_moe": true, + "attn_dp": 2, + "moe_ep": 2, + "stages": 2 + }, + "error": "RuntimeError('Sequential simulation ended with non-empty scheduler state: ...')" +} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_completion.py b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_completion.py new file mode 100644 index 00000000..67b8b468 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_completion.py @@ -0,0 +1,33 @@ +"""C6: Step 9 boundary probe shapes on a branch without the PR 35 batch-end seam. + +Uses probe_main.build_config unchanged and reports completion only. +""" +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from probe_main import build_config # noqa: E402 + +SHAPES = { + "moe_dp2_pp1": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=1), + "moe_dp2_pp2": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=2), + "moe_dp2_pp3": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=3), + "dense_dp1_pp2": dict(is_moe=False, attn_dp=1, moe_ep=1, stages=2), +} + +root, label = Path(sys.argv[1]), sys.argv[2] +case_root = root / label +case_root.mkdir(parents=True, exist_ok=True) +from frontier.simulator import Simulator # noqa: E402 + +result = {"shape": SHAPES[label]} +try: + simulator = Simulator(build_config(case_root, **SHAPES[label])) + simulator.run() + requests = list(simulator._all_requests) + result.update(completed=sum(1 for r in requests if r.completed), requests=len(requests)) +except Exception as exc: + result.update(error=repr(exc)[:800]) +(case_root / "result.json").write_text(json.dumps(result, indent=1)) +print(label, json.dumps(result)) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py new file mode 100644 index 00000000..f99c9900 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py @@ -0,0 +1,179 @@ +"""P1(b): record Frontier's admission and completion boundaries under PP. + +No source change. The probe wraps `_get_next_batch` (one call per admission, +after `_running_requests` has grown) and the inert `on_replica_batch_end` seam, +and reads the candidate report key -- the Replica's next forward id held by +`ForwardSyncState` -- at each boundary. +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +ROOT = "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering" +sys.path.insert(0, ROOT) + + +def build_config(root: Path, *, is_moe: bool, attn_dp: int, moe_ep: int, stages: int): + from frontier.config import ( + BaseModelConfig, ClusterConfig, FixedRequestLengthGeneratorConfig, + MetricsConfig, PoissonRequestIntervalGeneratorConfig, + RandomForrestExecutionTimePredictorConfig, ReplicaConfig, + RoundRobinClusterSchedulerConfig, SimulationConfig, + SyntheticRequestGeneratorConfig, VllmV1SchedulerConfig, + ) + from frontier.types import ActivationType, NormType + + model = BaseModelConfig( + num_layers=6, num_q_heads=4, num_kv_heads=2, embedding_dim=256, + mlp_hidden_dim=64, max_position_embeddings=4096, use_gated_mlp=True, + use_bias=False, use_qkv_bias=False, activation=ActivationType.SILU, + norm=NormType.RMS_NORM, post_attn_norm=True, vocab_size=1024, + is_moe=is_moe, num_experts=8 if is_moe else 0, + num_experts_per_tok=2 if is_moe else 0, torch_dtype="bfloat16", + ) + model._model_name = f"w9_probe_{'moe' if is_moe else 'dense'}" + original = BaseModelConfig.create_from_name + BaseModelConfig.create_from_name = classmethod( + lambda cls, name: model if name == model._model_name else original(name) + ) + moe_fields = dict( + moe_tensor_parallel_size=1, moe_expert_parallel_size=moe_ep, + total_expert_num=8, router_topk=2, + ) if is_moe else {} + replica = ReplicaConfig( + model_name=model._model_name, device="a100", + network_device="a100_pairwise_nvlink", num_pipeline_stages=stages, + attn_tensor_parallel_size=1, attn_dp=attn_dp, + memory_margin_fraction=0.1, **moe_fields, + ) + cluster = ClusterConfig( + replica_config=replica, + replica_scheduler_config=VllmV1SchedulerConfig( + num_blocks=128, block_size=16, batch_size_cap=4, + max_tokens_in_batch=16, enable_chunked_prefill=True, + ), + cluster_scheduler_config=RoundRobinClusterSchedulerConfig(), + execution_time_predictor_config=RandomForrestExecutionTimePredictorConfig( + enable_dummy_mode=True + ), + ) + return SimulationConfig( + simulation_mode="offline", sys_arch="co-location", + enable_parallel_clusters=False, decode_cuda_graph_mode="none", + cluster_config=cluster, + metrics_config=MetricsConfig( + output_dir=str(root / "metrics"), cache_dir=str(root / "cache"), + run_id="w9_probe", write_metrics=False, store_request_metrics=False, + store_batch_metrics=False, store_operation_metrics=False, + store_utilization_metrics=False, store_plots=False, + enable_chrome_trace=False, write_json_trace=False, + ), + request_generator_config=SyntheticRequestGeneratorConfig( + num_requests=6, + length_generator_config=FixedRequestLengthGeneratorConfig( + prefill_tokens=16, decode_tokens=3 + ), + interval_generator_config=PoissonRequestIntervalGeneratorConfig(qps=1e6), + ), + ) + + +def run(root: Path, *, is_moe: bool, attn_dp: int, moe_ep: int, stages: int): + from frontier.scheduler.cluster_scheduler.base_cluster_scheduler import ( + BaseClusterScheduler, + ) + from frontier.scheduler.replica_scheduler.vllm_v1_engine_replica_scheduler import ( + VLLMv1EngineReplicaScheduler, + ) + from frontier.scheduler.utils.forward_sync_state import ForwardSyncState + from frontier.simulator import Simulator + + events: list[dict] = [] + schedulers: dict = {} + + def next_forward_id(cluster_scheduler, replica_id): + state = cluster_scheduler._forward_sync_state + return int(state._next_step_id_by_replica.get(replica_id, 0)) + + original_next_batch = VLLMv1EngineReplicaScheduler._get_next_batch + + def observed_next_batch(self, is_micro_batch=False): + batch = original_next_batch(self, is_micro_batch=is_micro_batch) + if batch is not None: + schedulers[(self._replica_id, self._replica_local_id)] = self + events.append({ + "kind": "admit", + "time": round(float(self._current_schedule_time), 9), + "lane": self._replica_local_id, + "batch": batch.id, + "provisional": batch._forward_cohort_provisional_id, + "running_batches_before": self._num_running_batches, + "stages": self._num_stages, + "load": list(self.get_request_load()), + "candidate_key": next_forward_id(self._cluster_scheduler, self._replica_id), + }) + return batch + + original_batch_end = BaseClusterScheduler.on_replica_batch_end + + def observed_batch_end(self, time, replica_id, replica_local_id, batch): + result = original_batch_end(self, time, replica_id, replica_local_id, batch) + lane = self.get_replica_scheduler(replica_id, replica_local_id) + events.append({ + "kind": "complete", + "time": round(float(time), 9), + "lane": replica_local_id, + "batch": batch.id, + "provisional": batch._forward_cohort_provisional_id, + "resolved": ForwardSyncState.get_step_id(batch), + "running_batches_after": lane.num_running_batches, + "load": list(lane.get_request_load()), + "candidate_key": next_forward_id(self, replica_id), + }) + return result + + VLLMv1EngineReplicaScheduler._get_next_batch = observed_next_batch + BaseClusterScheduler.on_replica_batch_end = observed_batch_end + try: + config = build_config(root, is_moe=is_moe, attn_dp=attn_dp, + moe_ep=moe_ep, stages=stages) + simulator = Simulator(config) + simulator.run() + requests = list(simulator._all_requests) + finally: + VLLMv1EngineReplicaScheduler._get_next_batch = original_next_batch + BaseClusterScheduler.on_replica_batch_end = original_batch_end + return { + "completed": sum(1 for r in requests if r.completed), + "requests": len(requests), + "events": events, + } + + +if __name__ == "__main__": + root = Path(sys.argv[1]) + summary = {} + for label, shape in { + "moe_dp2_pp1": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=1), + "moe_dp2_pp2": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=2), + "moe_dp2_pp3": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=3), + "dense_dp1_pp2": dict(is_moe=False, attn_dp=1, moe_ep=1, stages=2), + }.items(): + case_root = root / label + case_root.mkdir(parents=True, exist_ok=True) + result = run(case_root, **shape) + summary[label] = result + events = result["events"] + print(f"\n=== {label}: {result['completed']}/{result['requests']} completed, " + f"{len(events)} boundaries") + print(f"{'time':>9} {'kind':>8} {'lane':>4} {'batch':>5} {'prov':>4} " + f"{'resolved':>8} {'load':>8} {'key':>4} {'slots':>6}") + for e in events[:28]: + slots = (f"{e['running_batches_before']}/{e['stages']}" if e["kind"] == "admit" + else f"{e['running_batches_after']}") + print(f"{e['time']:>9.5f} {e['kind']:>8} {str(e['lane']):>4} {e['batch']:>5} " + f"{e['provisional']:>4} {str(e.get('resolved', '')):>8} " + f"{str(tuple(e['load'])):>8} {e['candidate_key']:>4} {slots:>6}") + (root / "frontier_boundaries.json").write_text(json.dumps(summary, indent=1)) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/plan.md b/task_memory/task_2026-09-22_stage_admission_ordering/plan.md index cfcc4070..7251a5d5 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/plan.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/plan.md @@ -4,6 +4,8 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-7: the MoE retry job runs the ground truth with one recorded overlay patch (four-argument `topk_softmax`); §4.7 notes it. | +| 2026-09-23 | R-6: execution started. Added package P5 (vLLM comparison on a GPU worker), criterion C7, the vLLM-aligned group G7, §4.7 and D-8. | | 2026-09-23 | Applied the round-1 plan review (`review.md`). Changes: C1 now targets confirmed admission-deadlock witnesses from a phase-controlled group; C3 uses the stage ledger and overlap duration; C4 takes the reviewer's wording; P0 lists its artifacts and outcome classes; P2 covers both sides of the EP boundary, the `DECODE_FFN` dense-group control, a second admission round and per-fixture base expectations, and the dense fixture asserts a same-start condition that discriminates on the base; P3 has three acceptance paths. The matrix now publishes a concrete case list on the analytical backend and keeps `attn_dp=2, PP=3`. Added D-6 and D-7. Not executed. | | 2026-09-23 | D-1..D-5 adopted by the user; D-3 executed now (push + draft PR for remote review); D-5 adjusted so the records travel with the branch. Work packages P0–P4 unblocked. | | 2026-09-22 | Created for user review. Scope, acceptance criteria, work packages P0–P4 with dependencies, verification matrix, decisions D-1..D-4. No source change yet. | @@ -23,6 +25,8 @@ own branch: the full-stage admission predicate in `try_acquire`, removal of the admitted ticket with `remove(ticket)`, and the two docstrings that describe admission as "FIFO-head". +- Validation against vLLM on a GPU worker (R-6, package P5): tests and + comparison scripts only, no source change. - Out of scope: `full_stage_capacity`, the forward-group seal, EP wave protocol, sync rooms, wake-up helper, any configuration field, the Step 9 report key, and mixed-phase forward failures on `main` (PR 35 W3; see @@ -37,6 +41,7 @@ own branch: | C3 | Timing change (path T). Base-successful cases with `attn_dp > 1` and `PP > 1` (all G4 `PP > 1` cells, and G3a/G3b cells P0 classifies as `success`) are either byte-identical, or their difference is explained with the stage-ledger metric of §4.5 plus batch membership and component durations. The designated contention witnesses (§4.2, marked W) show a strictly larger `multi_lane_busy_time` after P1. In every case: no lane overlaps itself, and `peak_lanes ≤ attn_dp`. | P3, §4.5 metric from `frontier_stage_batch_ledger.jsonl`. | | C4 | Existing passing tests must remain passing without assertion changes. Existing failures, collection errors, and skips must be compared against a fresh run of the exact base revision in the same environment. Any new failure or required change to an ordering assertion stops implementation for review. | G2, §4.6. | | C5 | The predicate is one readable condition, and the module and method docstrings state the ordering contract as implemented. The change adds no flag, config field, `getattr` fallback, lane field on tickets, acquisition wake-up, PP-specific branch, second queue or capacity-1 special case. | Review of the diff against the quality gates. | +| C7 | vLLM comparison (path V, R-6). On the vLLM-aligned shapes of §4.7, the P1 revision matches vLLM on completion, per-lane batch sequences, stage-0 lane pairing, first-forward co-start and stage-0 co-execution, and the base revision fails the negative controls stated there. | P5, §4.7. | | C6 | Informational. The Step 9 boundary probe on MoE `attn_dp=2, moe_ep=2, PP=2` runs to completion on this branch, or its remaining failure is classified. Composition with PR 35 (W3 mixed-phase forward) is validated in the parent task after merge-forward, before Step 9 is declared unblocked. | P3 probe rerun; parent-task follow-up (§6). | ## 3. Work packages @@ -46,14 +51,23 @@ P0 evidence and baseline (no source change) -> P1 rule change -> {P2 tests, P3 rerun and comparison} -> P4 records, commit, push + +P5a vLLM driver, extraction and comparison scripts (independent of P1) + -> P5b GPU ground-truth run + -> P5c comparison against P0 (base) and P3 (after) G7 outputs + -> P4 ``` +P5a and P5b run in parallel with P0–P3; P5c needs the G7 outputs of P0 and +P3. + | Package | Content | Acceptance | | --- | --- | --- | | P0 Evidence and baseline | (1) Move the reproduction into `tests/e2e/stage_admission_matrix.py`, following the `tests/e2e/moe_ep_non_dummy_matrix.py` precedent. The module holds: the §4.1 fixture builder; the §4.2 case table; a runner with one child process per case, because `IS_MOE` is process-global; the outcome classifier and state-report writer of §4.3, which replace the session scripts `drain_state.py`/`drain_lanes.py`; and the §4.5 ledger metric. Outputs go to `resolve_scratch_root()/stage_admission_ordering/base//` (`tests/scratch_root.py`). (2) Confirm that the branch source equals `1f694f7` (`git diff --stat 1f694f7 -- . ':!task_memory' ':!.gitignore'` is empty). (3) Run R0, G1, G3a, G3b, G4, G5 and the G2 suites. (4) Write the §4.3 artifacts for every case. (5) Rerun two success cases (one G1 recipe, one G4 `PP=2` cell); an unstable file is named and excluded from C2, with the reason. (6) Record the classification table in the test report. | Every case has `case.json`, `run.json` and its class artifact. R0 reproduces the author-reported table, or each difference is explained. No `other_failure`. C1's per-pair witness condition holds. Every successful `attn_dp>1` case has `ATTN_DP_LANE` ledger rows for each of its lanes, otherwise §4.5 cannot be computed and P0 stops. | | P1 Rule | Implement the `design.md` rule. Full-stage tickets are refused only by an EP wave queued ahead; EP waves keep the strict head rule; the admitted ticket leaves the FIFO by `remove(ticket)`. Update the `StageExecutionContext` class docstring ("A complete operation first enters the ready FIFO, then the owner admits it atomically") and the `try_acquire` docstring ("Acquire the FIFO-head ticket if this stage is currently idle") to state the implemented contract. Queued full-stage work may pass other full-stage work but not an earlier queued EP wave. Queued EP waves keep FIFO admission. Active layer-to-layer scope transitions remain a separate mechanism. | The diff touches one source file. `python -m pytest tests/unit/test_stage_execution_context.py tests/unit/test_shared_forward_group_admission.py -q` passes with no assertion change. | | P2 Tests | **(a)** Contract tests in `tests/unit/test_stage_execution_context.py`. *Bypass* and *EP boundary* use a capacity-2 context (`ep_size=2`) with FIFO `full0, full1, wave0, full2`. *Bypass*: `full1` acquires before `full0`, and afterwards `queued_tickets == (full0, wave0, full2)`; `full2` is then refused although capacity remains, because `wave0` is ahead; `full0` acquires. *EP boundary* (acquisitions in head order, so it also runs on the base): `full0` and `full1` acquire; after `full1` releases, `full2` is still refused, because `wave0` is ahead; `wave0` is refused while `full0` is active and acquires once it releases; `full2` is refused while `wave0` is active and acquires after `wave0` releases. *Capacity 1*: on an idle context with FIFO `[full0, full1]`, `try_acquire(full1)` succeeds, pinning the API-level change stated in `design.md`. The two existing EP-order tests stay unchanged. **(a′)** A `DECODE_FFN` control in `tests/unit/test_mixed_layer_decode_ffn_scheduling.py` with its mixed-layer fixture. It materializes two successive `DenseFFNBatchGroup`s and a neighbouring EP group on one target replica and stage, through `_schedule_dense_ffn_from_m2n_group` and the real full-stage `ReplicaStageScheduler`. It asserts that FIFO order and heap order both follow the group counter, that the dense groups are admitted in counter order, and that neither dense group crosses an EP wave queued ahead of it. **(b)** A scheduler-level test in `tests/unit/test_shared_forward_group_admission.py` using its `make_stage`/`make_batch` helpers, parametrized over which lane enqueues first. It rebuilds the drain state: the first lane is active with a second ticket queued, and the other lane has two queued tickets. It asserts that the other lane's `pop_batch_if_not_busy` returns its heap head and binds the same forward group. It then continues through promotion to an EP wave and restoration to full-stage owners (`replace_full_stage_owners_with_ep_wave`, `replace_ep_wave_with_full_stage_owners`), release of both owners and `on_stage_end` of both lanes, and it asserts that both lanes admit their next queued batch into a later forward group, leaving the FIFO empty. **(c)** Simulator-level tests in `tests/integration/test_stage_admission_pipeline_lanes.py`, importing the §4.1 builder from `tests.e2e.stage_admission_matrix`, one child process per case. MoE witnesses `G3a-moe-dp2-pp2-n4` and `G3a-moe-dp4-pp2-n8` assert completion and conservation. The dense fixture `G4-dense-dp2-pp2-n8` asserts completion, and that the first stage-0 ledger rows of both lanes start at the same simulated time, because every request arrives at `t=0` and capacity admits both lanes into the first forward. A bare `multi_lane_busy_time > 0` would not discriminate: from source, the base already overlaps the lanes after the first release. Expected values are written from the scenario, not copied from a run. | Expected on `1f694f7`: (a) *bypass* fails at its first assertion, *capacity 1* fails, and *EP boundary* passes; (a′) passes; (b) fails at the other lane's first admission; (c) each MoE witness fails through the documented `admission_deadlock` signature, and the dense fixture completes but fails only its same-start assertion, because the second lane's first row starts at the first lane's first stage-0 end. After P1 all of them pass. The base failures are recorded as negative controls. | | P3 Rerun | Rerun every P0 case on the P1 revision into `.../after//`, then apply the acceptance path of each case (§4.2). **U**: hashes identical. **L**: the case completes with conservation; there is no base metrics hash to compare. **T**: hashes identical, or the difference is explained by the §4.5 metric (before and after), batch membership and component durations; W cases must show a strict increase. Stop and report, adjusting nothing, on any of these: a U difference (including `attn_dp=4, PP=1`); an L case that fails in any other way, such as a mixed-phase failure in G3b; a T difference that the ledger does not explain; a class change outside these paths; a failure of the self-overlap or `peak_lanes ≤ attn_dp` checks. Rerun the Step 9 boundary probe for C6. | C1–C4 and C6 tables in `test_report__stage_admission_ordering.md`. | +| P5 vLLM comparison | **(a)** Scripts under `tests/comparison/stage_admission_pp/`: a burst driver that runs inside the vLLM image, an extractor that turns the vLLM trace files into per-forward lane rows, and a comparison that computes the §4.7 metrics for vLLM and for Frontier ledgers. They are checked on the CPU host against synthetic inputs before any GPU job. **(b)** One GPU job (§4.7 "GPU job"), recorded in the case directory `calibration/stage_admission_case_001/`. **(c)** Comparison of the vLLM rows with the base (P0) and after (P3) runs of G7, written as a workflow-gap table with one `MATCH`/`MISMATCH` row per metric and burst. | C7 per §4.7. A `MISMATCH` on the after revision stops before P4 and is reported with its Frontier owner; nothing is adjusted to make it match. | | P4 Records | Test report, `progress.md`, `summary.md`. Commit P0's harness, P1 and P2 as code commits (harness separately from the rule, so the rule commit stays one file plus its tests), and the records as a docs commit. Push the branch and update the draft PR body with the C1–C3 tables. Note in the parent task (`issues.md` W9-01) the branch and commits. | Pushed and verified. | ## 4. Verification matrix @@ -100,9 +114,10 @@ path U; any other class in P0 is handled by §4.3. | G5 single lane | `attn_dp=1`, `PP ∈ {1,2,3}`, MoE (`moe_ep=1`) and dense, `n=6` | PD | 6 | success | U | | G6 PD-AF | The 10 PD-AF recipes inside G1, plus `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`, `test_decode_ep_wave_materialization.py` and `test_prefill_ep_wave_materialization.py` | — | (in G1) | success; tests pass | U, C4 | | G2 suites | `tests/unit` and `tests/integration` | — | 2 | base identities from P0 | C4 | +| G7 vLLM-aligned | §4.7 shapes: MoE `Qwen3-30B-A3B-tiny` and dense `Llama-3.2-1B-Instruct`, `attn_dp=2, PP=2`, `n ∈ {8, 16}` | V (256/1) | 4 | MoE: deadlock; dense: success with a delayed second lane | V (§4.7), plus L or T by P0 class | -Simulator runs: 30 + 18 + 12 + 12 + 6 = 78, plus 16 R0 record runs and the two -pytest suites. This satisfies the AGENTS.md gate of at least 50 concrete +Simulator runs: 30 + 18 + 12 + 12 + 6 + 4 = 82, plus 16 R0 record runs and the +two pytest suites. This satisfies the AGENTS.md gate of at least 50 concrete settings. ### 4.3 Outcome classes and artifacts @@ -176,7 +191,106 @@ fails or errors; collection errors and skips are unchanged. A base failure that now passes is reported, not treated as a stop. No failure count from another checkpoint is used. -## 5. Decisions +### 4.7 vLLM comparison (P5, C7) + +**Why vLLM is a valid reference for this rule.** In vLLM 0.10.2 (the +`vLLM-BS` checkout below): + +| Fact | Source | +| --- | --- | +| With `data_parallel_size > 1`, every DP rank is a `DPEngineCoreProc` with its own scheduler and its own PP workers on its own GPUs. Queued work on one rank cannot refuse admission to another rank's stage. | `vllm/v1/engine/core.py:773-779` | +| At PP > 1 each rank keeps up to PP batches in flight through `step_with_batch_queue`. | `core.py:152-158`, `core.py:364-420` | +| DP ranks at one PP stage meet once per forward in the `DPMetadata` token-count all-reduce, and a rank without runnable work executes a dummy batch, so stage forwards pair one to one across ranks. MoE layers add EP collectives inside the forward. | `vllm/forward_context.py:84,216`; `core.py:1185-1193` | +| A request can be pinned to a DP rank with `data_parallel_rank`. | `vllm/v1/engine/async_llm.py:275`; `vllm/v1/engine/core_client.py:1148` | + +Frontier's attention-DP lanes model those ranks (`AGENTS.md` "vLLM Parallel +Semantics and Frontier Mapping"). The fix claims that a lane with runnable work +is no longer refused at a shared stage by another lane's queued work, so the +comparison measures exactly the behaviours that claim predicts. + +**Shapes.** vLLM DP=2, PP=2, TP=1 on 4×H800; Frontier `attn_dp=2`, +`num_pipeline_stages=2`, `attn_tensor_parallel_size=1`, one Replica, +`RoundRobinClusterSchedulerConfig`, `vllm_v1` replica scheduler. + +| Item | MoE | Dense | +| --- | --- | --- | +| Model config (vLLM `config.json` and Frontier `model_name`) | `data/config/models/Qwen3-30B-A3B-tiny.json`: 8 layers, 16 experts, top-8 | `data/config/models/Llama-3.2-1B-Instruct.json`: 16 layers | +| Expert parallelism | vLLM `enable_expert_parallel` (EP=2 per stage); Frontier `moe_tensor_parallel_size=1`, `moe_expert_parallel_size=2` | — | +| Frontier device | `h800` / `h800_dgx`, dummy predictor, analytical CC backend | same | + +**Workload.** Prompt 256 tokens (distinct token ids per request), one output +token (`max_tokens=1`, `ignore_eos`), so every request completes at its prefill +boundary, as in G3a. Token budget 256 and at most 4 sequences per batch on both +sides, so every batch holds one request. Bursts of `n ∈ {8, 16}` requests; +request `i` goes to lane `i mod 2` (vLLM `data_parallel_rank`; Frontier's +round robin produces the same assignment, `round_robin_cluster_scheduler.py:381-387`, +checked from the ledger). vLLM runs each burst three times after four warmup +requests (two per rank), with the engines idle between rounds; Frontier runs each +burst once (deterministic). Common settings: eager mode, chunked prefill on, +prefix caching off, block size 16, FCFS. Frontier `num_blocks=1024`; vLLM's +`num_gpu_blocks` is recorded. Neither side is expected to preempt; a +preemption on either side is a `MISMATCH`. + +**Evidence.** vLLM runs in the instrumented mode of the calibration contract +(`VLLM_FRONTIER_INSTRUMENTATION=1`, which synchronizes after each forward), +with `VLLM_FRONTIER_PP_BOUNDARY_LOG_PATH` (one row per real forward: request +ids, `pp_rank`, monotonic `forward_start_ts` and `send_start_ts`) and +`VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR` (engine iterations). The driver records the +rank, submit and finish times of every request. No clean E2E run is made: the +Frontier side uses the dummy predictor, so latency is not compared. Frontier +evidence is the stage ledger of the G7 cases. + +**Interval per forward.** vLLM stage 0: `[forward_start_ts, send_start_ts]` +(`send_start_ts` follows the post-forward synchronize). vLLM stage 1: +`[forward_start_ts, timestamp − offset]`, with `offset = time.time() − +time.monotonic()` sampled by the driver before and after each round; stage-1 +metrics are informational. Frontier: `[stage_start_ts, stage_end_ts)` of +`ATTN_DP_LANE` rows. Dummy forwards are not logged by vLLM and have no ledger +row in Frontier; both sides compare real forwards only. + +**Metrics** (same definitions on both sides, per burst and round): + +| Id | Metric | +| --- | --- | +| M1 | Completed formal requests / submitted. | +| M2 | Per `(lane, stage)`, the ordered list of request-id tuples of its forwards. | +| M3 | Stage-0 pairing: for each lane-0 forward, the lane-1 forward with the largest overlap (or none). | +| M4 | First-forward co-start: `|start(lane 0) − start(lane 1)|` of each lane's first stage-0 forward, divided by the median stage-0 forward duration of that run. | +| M5 | Stage-0 co-execution: `multi_lane_busy_time / union_busy_time` over stage-0 intervals (§4.5 definitions). | + +**Pass conditions (C7).** + +| Id | Condition | +| --- | --- | +| V1 | vLLM completes every formal request of every round; after-revision G7 cases complete. MoE negative control: P0 classifies the G7 MoE cases as `admission_deadlock`. If P0 finds them `success`, the aligned shape does not exercise the defect: record it, and V1 rests on the G3a witnesses only. | +| V2 | After-revision M2 equals vLLM M2 in every round. vLLM rounds that disagree with one another are reported, with the cause. | +| V3 | After-revision M3 equals vLLM M3 in every round. A vLLM round in which a dummy forward shifts the pairing is named and reported, not dropped. | +| V4 | vLLM M4 < 0.5 in every round and after-revision M4 < 0.5: both lanes start in the same forward slot. Dense negative control: base M4 ≥ 0.5. | +| V5 | `|M5(after) − mean M5(vLLM)| ≤ 0.10`; for dense also `|M5(base) − mean M5(vLLM)| > |M5(after) − mean M5(vLLM)|`. The 0.10 bound reuses the calibration contract's tolerance; it is applied to a fraction, not to latency. | + +The comparison writes `analysis/workflow_gap_table.csv` (one row per metric, +burst and round, with `MATCH`/`MISMATCH`, values, source and Frontier owner), +`workflow_gap_summary.md` and `workflow_gap_status.json` in the case directory. + +**GPU job.** StepMind Python `RJobBackend`, `i-fengyicheng` personal auth, +`charged_group="codesign"`, `positive_tags=["H800"]`, `gpu=4, cpu=16, +mem_gb=128`, image `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`, +`code_mount_point=/data/ycfeng/Frontier` (covers this worktree and the vLLM +checkout), cloud volume mounted with outputs under +`/mnt/codesign-exp/ycfeng/frontier/stage_admission_pp//`, and the extracted +evidence copied to the case directory. The worker repairs the libcuda loader +path (runbook §10), builds the overlay (the image's installed `vllm` package with +every `vllm/**/*.py` of the checkout copied over it, keeping the image's compiled +extensions), and verifies before the workload that the files where image and +checkout differ are exactly the fork's changes over its upstream base +`01efc7ef7`. Ground truth: `/data/ycfeng/Frontier/.real-engine/vLLM-BS`, +branch `feature/frontier-comparison-instrumentation`, commit `494b9f327`, +clean. Budget: at most two jobs of at most one hour (one run, one retry after +an environment failure). +The retry (R-7) applies `calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch` +to the accepted overlay: the fork passes a fifth `renormalize` argument to +`_moe_C::topk_softmax`, which its own csrc and the image declare with four. The launcher stays alive for the job; no resubmission +while queued. Adopted by the user on 2026-09-23 ("采纳你d1-d5的推荐决策"): @@ -196,6 +310,13 @@ Adopted from the round-1 review on 2026-09-23, under the user's instruction | D-6 | The synthetic groups select `AnalyticalCCBackendConfig` explicitly and keep `attn_dp=2, PP=3`, instead of substituting `attn_dp=4, PP=3`. | The change is admission-only. The node-size rule belongs to the `collective_sim`/`astra_sim_analytical` backends, and substituting the shape would leave part of C1 untested. | | D-7 | C1 witnesses come from the phase-controlled prefill-only group G3a. Mixed-phase failures are out of scope: stop, report, diagnose separately. Composition with PR 35 is checked in the parent task. | `main` lacks PR 35 W3; this keeps the admission repair separable from the mixed-phase lifecycle. | +Adopted under R-6 on 2026-09-23 (execution request; routine choices made from +evidence and recorded here for review): + +| Id | Decision | Reason | +| --- | --- | --- | +| D-8 | The vLLM comparison is structural (M1–M5), in vLLM's instrumented mode, with a prefill-only workload on one MoE and one dense model already in `data/config/models/`. No E2E latency gate. | The fix changes admission, not durations; the Frontier side runs the dummy predictor. Prefill-only keeps the comparison inside the D-7 scope boundary. Both models exist on both sides without new assets. | + ## 6. Dependencies and risks - The reproduction scripts still live in the session scratchpad @@ -210,6 +331,13 @@ Adopted from the round-1 review on 2026-09-23, under the user's instruction If either differs, stop and report before adjusting anything. - Risk: after P1, a G3b case may reach a mixed-phase cohort and fail (scope boundary). Stop and report; it is not repaired on this branch. +- Risk (P5): at a burst start, a vLLM rank may run a dummy forward before its + first request is visible, which shifts the stage-0 pairing by one forward. + V3 names such a round; the driver submits every request of a burst before + yielding to the event loop to make it unlikely. +- Risk (P5): vLLM stage-0 intervals start before the per-forward DP + all-reduce, so a rank that arrives early records its wait as busy time. M4 + uses start times only, and V5 has the stated 0.10 bound. - The parent task's Step 9 resumes only after this branch is merged into `main` and merged forward into `fix/issue26-correctness-pr`. The parent task then reruns G3b on that branch, where W3 is present, as the composition check diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md index 604dcc01..1115f55c 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md @@ -4,6 +4,8 @@ | Date | Change | | --- | --- | +| 2026-09-23 | P0–P3 and P5 executed. Rule committed (`dac4e69`). Two plan stop conditions reached (C3 witness metric, C7 V5 dense); P4 push held for the user. | +| 2026-09-23 | R-6 received: execution started; P5 (vLLM comparison) added to the plan. | | 2026-09-23 | Round-1 plan review verified against source and applied to `design.md`, `plan.md` and `requirements.md`; `review.md` created; resume prompt updated for round 2. Not executed. | | 2026-09-23 | Decisions adopted; records pushed for remote review. | | 2026-09-22 | Created. Worktree and branch created; defect reproduced on `origin/main`; plan and design written for review. No source change. | @@ -19,7 +21,12 @@ | Plan for review | completed; round-1 review applied | `plan.md`, `review.md` | | Records published for remote review (`.gitignore` exception, docs commit, push, draft PR) | completed 2026-09-23 | commit and PR recorded below | | Round-1 plan review (10 findings) verified and applied | completed 2026-09-23 | `review.md`; plan D-6, D-7 | -| P0–P4 | pending | P0 is defined (`plan.md` P0, §4); it waits for the owner's start signal (R-5: "暂不执行") | +| P0 evidence and baseline | completed 2026-09-23 | 98 cases classified as designed; rerun hashes stable; see "Execution" | +| P1 rule | completed | `dac4e69` (with P2 tests) | +| P2 tests and base negative controls | completed | `evidence/base_negative_controls.log`; all base outcomes as planned | +| P3 rerun and comparison | completed; **stopped** on C3 witness rule at `attn_dp=4` | `test_report_2026-09-23_stage_admission_ordering.md` §4 | +| P5 vLLM comparison | completed; **stopped** on V5 dense | case `calibration/stage_admission_case_001/`, report §5 | +| P4 records, commit, push | in-progress: records written and committed locally; push and PR body held for the user's decision | report §6 | ## Commands run (2026-09-22) @@ -45,3 +52,28 @@ worktree, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`. | Draft PR | https://github.com/NetX-lab/Frontier/pull/36 | | Reviewer resume prompt | `review_prompt.md` in this directory | | Round-1 review applied | the commit that adds `review.md` (`git log -- task_memory/task_2026-09-22_stage_admission_ordering/review.md`) | + +## Execution (2026-09-23) + +| Step | Command / action | Evidence | Result | +| --- | --- | --- | --- | +| P0 base set | `python -m tests.e2e.stage_admission_matrix run --set base --jobs 8` at `a054d87` | `/data/ycfeng/tmp/stage_admission_ordering/base/` | G1 30 success; G3a 10 deadlock + 8 success; G3b 6 + 6; G4 12 success; G5 6 success; G7 MoE 2 deadlock, dense 2 success; R0 as `design.md` | +| P0 rerun stability | set `base-rerun`, 4 cases | same | hashes identical; no file excluded | +| P0 pytest | unit and integration with `--junitxml` | `/data/ycfeng/tmp/stage_admission_ordering/base-pytest/` | unit 84 failed / 3644 passed / 49 skipped / 10 errors; integration 11 / 21 skipped / 5 errors | +| P1 | `try_acquire` rule and docstrings | `dac4e69` | 34 passed on the two contract files, no assertion change | +| P2 | tests (a), (a′), (b), (c) | `dac4e69` | all pass after P1 | +| P2 base controls | new tests in a `git archive 799ccb4` export | `evidence/base_negative_controls.log` | 7 failed, 2 passed, each at the planned assertion | +| P3 after set | `run --set after --jobs 8` at `dac4e69` | `/data/ycfeng/tmp/stage_admission_ordering/after/` | 97 success, 1 configuration_rejection (R0 dp2-pp3) | +| P3 compare | `compare --before base --after after` | `compare_base_after.json` | U 50 PASS; L 18 PASS; T 6 PASS, 6 EXPLAIN, 2 STOP (dp4 witnesses) | +| P3 explain | `evidence/explain_t_path.py` | `evidence/p3_t_path_explanation.json` | all 8 differing T cases: same batches and component durations, start times only | +| P3 G2 | unit and integration after-runs | `evidence/g2_*_compare.json` | no regression, no new failure, skips and errors unchanged | +| C6 probe | `evidence/step9_probe/probe_completion.py` | `evidence/step9_probe/` | MoE dp2-ep2-pp2 6/6 (base drains); pp3 W9-02 rejection | +| P5b run a | RJob `exp-0923-022226-151935` | `runs/vllm-instrumented/sa-pp-20260923a/` | dense complete; MoE failed on `_moe_C::topk_softmax` 5 vs 4 args | +| Decision | user: "topk_softmax 统一修复为4 个参数的版本" | `requirements.md` R-7 | recorded overlay patch, checkout unchanged | +| Overlay patch support | `vllm_burst_driver.py overlay --patch`, worker `OVERLAY_PATCH` | `a1b9819`; CPU dry run against an `upstream-v0.10.2` export | accepted; `_custom_ops.py` equals upstream after patch; second application fails loudly | +| P5b run b | RJob `exp-0923-024146-345158` | `runs/vllm-instrumented/sa-pp-20260923b/` | MoE and dense complete, status 0 | +| P5c | `compare_lanes --vllm-run …/sa-pp-20260923b` | `calibration/stage_admission_case_001/analysis/` | 50/52 MATCH; V5 dense n8/n16 MISMATCH (vLLM 0.706/0.865 vs 1.0) | + +Open decisions (report §6): the C3 witness metric at `attn_dp=4`, and V5 for +the dense shape. P4 push, the PR 36 body update and the parent-task note wait +for them. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md index 375a4341..1cf2281c 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md @@ -4,6 +4,8 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-7: ground-truth `topk_softmax` fixed to the four-argument version for the MoE retry. | +| 2026-09-23 | R-6: execute P0–P4 and validate the fix against vLLM on a GPU worker (package P5). | | 2026-09-23 | R-5: round-1 plan review verified and applied to the records; execution deferred. | | 2026-09-23 | R-4: D-1..D-5 adopted; push and draft PR authorized. | | 2026-09-22 | Created from the Step 9 finding W9-01 in `task_2026-09-21_issue26_correctness_pr`; recorded the user's scope decision and the request for a reviewable plan. | @@ -32,6 +34,15 @@ a recommendation to fix it as a separate correctness item): > 以下是最新review结果，请你核实每个comments，采纳高价值和必要决策，修复完善docs，暂不执行。 +`[Original Request]` (2026-09-23, after round 1 was applied and pushed): + +> 按照已有plan执行上述修复（该修复需要和在gpu worker上运行的vllm进行合理的对比验证，确保修改的有效性） + +`[Original Request]` (2026-09-23, during P5, after the first GPU run failed on the +MoE `topk_softmax` ABI): + +> 我先提前决策，避免中断任务：topk_softmax 统一修复为4 个参数的版本 + Quality gates the user repeated for every core-module change in this line of work, carried over verbatim: @@ -46,6 +57,8 @@ work, carried over verbatim: | R-3 | No source change before the user reviews `plan.md` and `design.md`. | user, 2026-09-22 | | R-4 | Plan decisions D-1..D-5 adopted as recommended. Push the branch and open a draft PR so the review happens on the remote; the reviewer resumes from a prepared prompt. | user, 2026-09-23 | | R-5 | Verify every review finding against the source, adopt the high-value and necessary corrections into the records (dispositions in `review.md`, new decisions D-6 and D-7 in `plan.md`), and do not execute: no P0 run, no source change. The docs commit is pushed to the draft PR under R-4. | user, 2026-09-23 | +| R-6 | Execute P0–P4 as planned. The fix must also be validated against vLLM running on a GPU worker, in a comparison designed to show whether the change is effective (package P5 in `plan.md`). The request authorizes the GPU job within the standing GPU rules below. | user, 2026-09-23 | +| R-7 | The vLLM ground truth uses the four-argument `topk_softmax` (wrapper and call). Applied as the recorded overlay patch `calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch` on the one retry job; the vLLM-BS checkout is unchanged. | user, 2026-09-23 | ## Constraints carried from the parent task @@ -58,3 +71,10 @@ work, carried over verbatim: - Temporary files under `/data/ycfeng/tmp`; the simulator interpreter is `/data/ycfeng/envs/frontier-py310/bin/python`. - Never `cd` into the original repository root; use `git -C` and absolute paths. +- GPU work (R-6): StepMind Python `RJobBackend` only, `i-fengyicheng` personal + auth, `charged_group="codesign"` only (`steptron_ci` paused until the user + allows it again), `positive_tags=["H800"]`, submitted from this machine with + local NFS mounts, launcher kept alive, no resubmission while queued. Cloud + volume access is confined to `/mnt/codesign-exp/ycfeng`. Credential values + stay in restricted files and process environments; never print or record + them, and keep shell tracing off. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md b/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md new file mode 100644 index 00000000..08b52b35 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md @@ -0,0 +1,260 @@ +# Test report — stage admission ordering (P0–P3, P5) + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-23 | Created. P0–P3 and P5 executed; two plan stop conditions reached (C3 witness metric at `attn_dp=4`, C7 V5 on the dense shape). P4 push held for the user's decision. | + +## 1. Result + +| Criterion | Result | Section | +| --- | --- | --- | +| C1 repaired liveness | PASS: all 10 G3a `admission_deadlock` cases complete with conservation; so do the 6 G3b and 2 G7 MoE deadlocks. | §4.1 | +| C2 unchanged controls | PASS: 50 of 50 U cases byte-identical. | §4.2 | +| C3 timing change | **STOP (plan §3 P3)**: 6 T cases identical, 8 differ. All 8 differences are start times only (same batches, same component durations), with no self-overlap and `peak_lanes ≤ attn_dp`. The two `attn_dp=4` contention witnesses fail the stated rule "strictly larger `multi_lane_busy_time`". | §4.3 | +| C4 existing tests | PASS: no base-passed node regresses, no new failure or error, skips and collection errors unchanged. | §4.4 | +| C5 rule shape | PASS by review: one predicate, docstrings state the contract, no flag, field, fallback, wake-up, PP branch, second queue or capacity-1 case. | §3 | +| C6 Step 9 probe | Informational: MoE `attn_dp=2, moe_ep=2, PP=2` completes 6/6 (base drains). `PP=3` stops on the known W9-02 node-size rejection. | §4.5 | +| C7 vLLM comparison | **STOP (plan §3 P5)**: 50 of 52 rows MATCH. The two `MISMATCH` rows are V5 (stage-0 co-execution) on the dense shape. MoE matches on all 26 rows, V5 included. The negative controls fail on the base as planned. | §5 | + +Observed facts are separated from inferences. Inferences are marked +"Inference". + +## 2. Environment and commits + +| Item | Value | +| --- | --- | +| Host | `kun-workspace-vgen2` (CPU) | +| Interpreter | `/data/ycfeng/envs/frontier-py310/bin/python`, Python 3.10.6; distribution digest `ecd50ea8…1902620` for both sets | +| Environment | `PYTHONPATH=`, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`, `TMPDIR=/data/ycfeng/tmp/stage_admission_ordering/pytest-tmp` | +| Base set `base` | run at `a054d87` (harness only; `frontier/` identical to `1f694f7`) | +| After set `after` | run at `dac4e69`, tree clean outside `task_memory/`; 98 cases in 90 s with `--jobs 8` | +| Rule commit | `dac4e69` fix(scheduler): order full-stage admission only behind queued EP waves | +| Harness commits | `a054d87`, `5ade853` (matrix), `799ccb4` (vLLM comparison), `a1b9819` (recorded overlay patch) | +| Scratch root | `/data/ycfeng/tmp/stage_admission_ordering/{base,after,base-rerun,base-pytest,after-pytest,step9_probe}` | + +Commands: + +```bash +python -m tests.e2e.stage_admission_matrix run --set after --jobs 8 +python -m tests.e2e.stage_admission_matrix compare --before base --after after \ + --output /data/ycfeng/tmp/stage_admission_ordering/compare_base_after.json +python -m pytest tests/ -q -p no:cacheprovider --continue-on-collection-errors \ + --junitxml=/.xml # suite in {unit, integration}, base and after +python task_memory/.../evidence/explain_t_path.py +python -m tests.comparison.stage_admission_pp.compare_lanes \ + --vllm-run calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b \ + --before base --after after --output calibration/stage_admission_case_001/analysis +``` + +## 3. P1 rule + +`StageExecutionContext.try_acquire` (`frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`): +an EP wave must be the FIFO head; a full-stage ticket is refused only by an EP +wave queued ahead of it; the admitted ticket leaves the FIFO by +`remove(ticket)`. `_validate_ticket` already rejects a ticket that is neither +queued nor active, so the scan always finds the ticket or an earlier wave. +One file, +21/−7 lines. P1 acceptance: `tests/unit/test_stage_execution_context.py` +and `tests/unit/test_shared_forward_group_admission.py` gave 34 passed with no +assertion change. + +## 4. P2 and P3 + +### 4.0 P2 tests and base negative controls + +The new tests were copied into a `git archive 799ccb4` export (rule as on +`1f694f7`) and run there; the log is `evidence/base_negative_controls.log`. + +| Test | Expected on base | Observed on base | After P1 | +| --- | --- | --- | --- | +| (a) `test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave` | fails at first assertion | fails at line 111, `try_acquire(full1)` is False | pass | +| (a) `test_queued_ep_wave_orders_full_stage_work_on_both_sides` | pass | pass | pass | +| (a) `test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket` | fails | fails at line 141 | pass | +| (a′) `test_decode_ffn_dense_groups_keep_counter_order_around_a_queued_ep_wave` | pass | pass | pass | +| (b) `test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0,1]` | fails at the other lane's first admission | both fail at line 71, `pop_batch_if_not_busy()` is None | pass | +| (c) `test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4, G3a-moe-dp4-pp2-n8]` | `admission_deadlock` | both `admission_deadlock` | pass: (4, 64, 4) and (8, 128, 8) | +| (c) `test_dense_lanes_start_in_the_same_first_forward` | fails only the same-start assertion | fails `0.05 == 0.0`; lane 1 runs `[0, 0.05]`, lane 0 starts at `0.05` | pass: both lanes start at 0.0 | + +### 4.1 C1 — path L (18 cases, all PASS) + +Observed `(requests, prefill tokens, decode tokens)` after P1 equals the +generated workload in every case. + +| Cases | Observed | +| --- | --- | +| G3a `dp2-pp{2,3}-n{4,8,12}`, `dp4-pp{2,3}-n{8,12}` (10) | n4: (4, 64, 4); n8: (8, 128, 8); n12: (12, 192, 12) | +| G3b `dp2-pp{2,3}-n{4,8}`, `dp4-pp{2,3}-n8` (6) | n4: (4, 64, 12); n8: (8, 128, 24); no mixed-phase failure | +| G7 MoE `dp2-pp2-n{8,16}` (2) | (8, 2048, 8); (16, 4096, 16) | + +R0 (informational): the three base deadlocks `moe-dp2-pp2-n4`, `moe-dp2-pp2-n6`, +`moe-dp4-pp2-n8` now succeed; `moe-dp2-pp3-n6` remains +`configuration_rejection` (node-size rule, D-6); the other 12 stay `success`. + +### 4.2 C2 — path U (50 cases, all PASS) + +Byte-identical `sha256sums.txt`: G1 30 recipes (10 PD-AF included), `PP=1` +cells of G3a (6), G3b (4) and G4 (4, `attn_dp=4, PP=1` included), G5 6. + +### 4.3 C3 — path T (14 cases) + +Identical hashes (6): G3a/G3b/G4 `dp4-pp{2,3}-n4` (one batch per lane). + +Differing (8). `evidence/explain_t_path.py` checks, per stage and lane, that +the ordered batch list, the forward duration and the `execution_time` +component ledger are equal before and after; output +`evidence/p3_t_path_explanation.json`. All 8: `same_batches_and_component_durations = true`, +no self-overlap, `peak_lanes ≤ attn_dp`. Differing files are the ledger, +`request_metrics.csv` and `system_metrics.json` only. Stage 0: + +| Case | W | multi-lane time before → after | co-execution fraction before → after | peak lanes | first stage-0 starts before → after | Verdict | +| --- | --- | --- | --- | --- | --- | --- | +| G4-dense-dp2-pp2-n4 | | 0.25 → 0.30 | 0.714 → 1.0 | 2 → 2 | {1: 0, 0: 0.05} → all 0 | EXPLAIN | +| G4-dense-dp2-pp2-n8 | W | 0.45 → 0.50 | 0.818 → 1.0 | 2 → 2 | {1: 0, 0: 0.05} → all 0 | EXPLAIN | +| G4-dense-dp2-pp3-n4 | | 0.108 → 0.216 | 0.333 → 1.0 | 2 → 2 | {1: 0, 0: 0.036} → all 0 | EXPLAIN | +| G4-dense-dp2-pp3-n8 | W | 0.216 → 0.396 | 0.375 → 1.0 | 2 → 2 | {1: 0, 0: 0.072} → all 0 | EXPLAIN | +| G4-dense-dp4-pp2-n8 | W | **0.55 → 0.30** | 0.846 → 1.0 | 2 → 4 | {1: 0, 2: 0.05, 3: 0.10, 0: 0.15} → all 0 | **STOP** | +| G4-dense-dp4-pp3-n8 | W | **0.396 → 0.216** | 0.846 → 1.0 | 2 → 4 | {1: 0, 2: 0.036, 3: 0.072, 0: 0.108} → all 0 | **STOP** | +| G7-dense-dp2-pp2-n8 | | 0.36 → 0.48 | 0.60 → 1.0 | 2 → 2 | {1: 0, 0: 0.12} → all 0 | EXPLAIN | +| G7-dense-dp2-pp2-n16 | | 0.84 → 0.96 | 0.778 → 1.0 | 2 → 2 | {1: 0, 0: 0.12} → all 0 | EXPLAIN | + +Why the two witnesses fail the stated rule (observed from the ledgers): at +`attn_dp=4` the base admits the lanes two at a time; after P1 all four lanes +start every forward together. The stage busy period shrinks from 0.65 to 0.30 +(PP=2) and from 0.468 to 0.216 (PP=3), so the time with two or more lanes busy +shrinks with it, although it is now the whole busy period. Request E2E for +`dp4-pp2-n8` drops from 500–700 ms to 300–350 ms with identical batches. +Inference: absolute `multi_lane_busy_time` measures overlap only while the +busy period stays the same length; it cannot express "more overlap" when the +fix compresses the timeline, which happens whenever the base serialized more +than two lanes. The plan requires a stop here, with nothing adjusted. + +### 4.4 C4 — G2 test identities + +| Suite | Base | After | Regressions | New failures | Skip / collection changes | Only after | +| --- | --- | --- | --- | --- | --- | --- | +| `tests/unit` | 84 failed, 3644 passed, 49 skipped, 10 errors | 84 failed, 3650 passed, 49 skipped, 10 errors | 0 | 0 | none; `ERROR` lines identical | the 6 new P2 unit tests, all passed | +| `tests/integration` | 11 passed, 21 skipped, 5 errors | 14 passed, 21 skipped, 5 errors | 0 | 0 | none; `ERROR` lines identical | the 3 new P2(c) tests, all passed | + +Evidence: `evidence/g2_unit_compare.json`, `evidence/g2_integration_compare.json`; +junit XML under the scratch root. + +### 4.5 C6 — Step 9 boundary probe + +The original `probe_main.py` wraps `BaseClusterScheduler.on_replica_batch_end`, +a seam that exists only on the PR 35 branch, so it raises `AttributeError` +on this branch. `evidence/step9_probe/probe_completion.py` reuses its +`build_config` unchanged (a100, 6 requests, 16/3 tokens, Poisson) and reports +completion, one process per shape. + +| Shape | Base | After | +| --- | --- | --- | +| MoE `attn_dp=2, moe_ep=2, PP=1` | — | 6/6 | +| MoE `attn_dp=2, moe_ep=2, PP=2` | drain, "Sequential simulation ended with non-empty scheduler state" | 6/6 | +| MoE `attn_dp=2, moe_ep=2, PP=3` | — | `ValueError`: collective-sim node-size rule (W9-02, unchanged) | +| dense `attn_dp=1, PP=2` | — | 6/6 | + +Composition with PR 35 W3 stays a parent-task check after merge-forward. + +## 5. C7 — vLLM comparison (P5) + +### 5.1 Ground-truth runs + +| Run | RJob | Result | +| --- | --- | --- | +| `sa-pp-20260923a` | `exp-0923-022226-151935`, codesign, 4×H800, creator `i-fengyicheng`, NFS `100.96.128.195:/data/ycfeng/Frontier` | dense complete; MoE failed in `profile_run`: `_moe_C::topk_softmax() expected at most 4 argument(s) but received 5`. Job `Failed`. | +| `sa-pp-20260923b` | `exp-0923-024146-345158`, same shape and mount | MoE and dense complete; job `Succeeded`; worker status 0 | + +Cause of the run-a failure (observed): fork commit `1109c4f16` changed +`vllm/_custom_ops.py::topk_softmax` and the `vllm_topk_softmax` call in +`fused_moe.py` to pass a fifth `renormalize` argument, but the fork's own +`csrc/moe/torch_bindings.cpp` (unchanged from `upstream-v0.10.2`) and the +v0.10.2 image both declare the four-argument op. The user decided on +2026-09-23: "topk_softmax 统一修复为4 个参数的版本". Run b applies +`inputs/groundtruth_overlay.patch` (SHA-256 `8d476789…3a9c81`) to the accepted +overlay: it restores the upstream four-argument wrapper and call. The worker +records `_custom_ops.py` as byte-identical to the image's after the patch. +Numerics are unchanged: `vllm_topk_softmax` renormalizes in Python after the +call in both versions. The checkout `494b9f327` is not modified. + +vLLM run b: 7 rounds per model (1 warmup + 2 bursts × 3), 152 `pp_boundary` records per +model, no preemption, placement records for every request with none +misplaced; `num_gpu_blocks` 600666 (MoE) and 304854 (dense). + +### 5.2 Workflow-gap table (run b) + +`analysis/workflow_gap_table.csv`, `analysis/lane_metrics.json`, +`analysis/workflow_gap_status.json`. + +| Check | MoE n8 | MoE n16 | Dense n8 | Dense n16 | +| --- | --- | --- | --- | --- | +| V1 completion | 3/3 MATCH; base `admission_deadlock` | 3/3 MATCH; base `admission_deadlock` | 3/3 MATCH | 3/3 MATCH | +| V2 lane sequences | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | +| V3 stage-0 pairing | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH; base pairs 0↔3, 2↔5, …, 6↔none | 3/3 MATCH; base shifted by one forward | +| V4 co-start (vLLM / after / base) | 0.009–0.063 / 0.0 / — | 0.005–0.024 / 0.0 / — | 0.008–0.248 / 0.0 / 1.0 | 0.046–0.171 / 0.0 / 1.0 | +| V5 co-execution (vLLM mean / after / base) | 0.976 / 1.0 / — MATCH | 0.948 / 1.0 / — MATCH | **0.706 / 1.0 / 0.600 MISMATCH** | **0.865 / 1.0 / 0.778 MISMATCH** | + +Run a (dense only, same scripts): V1–V4 all MATCH; V5 vLLM mean 0.714 (n8) +and 0.685 (n16), also MISMATCH. + +### 5.3 The V5 dense mismatch + +`evidence/decompose_co_execution.py` splits the stage-0 non-overlap of each +M3 pair into `|Δstart| + |Δend|` +(`analysis/co_execution_decomposition_sa-pp-20260923{a,b}.json`). + +| Shape (run b) | vLLM M5 per round | Σ start offsets (ms) | Σ end offsets (ms) | stage-0 duration median (ms), CV | +| --- | --- | --- | --- | --- | +| MoE n8 | 0.977, 0.974, 0.977 | 0.32–0.54 | 0.19–0.20 | 5.3–8.5, 0.07–0.09 | +| MoE n16 | 0.978, 0.937, 0.928 | 0.70–3.83 | 0.26–0.69 | 5.3–7.5, 0.05–0.07 | +| Dense n8 | 0.657, 0.851, 0.609 | 1.49–2.63 | 0.26–4.40 | 3.0–3.8, 0.13–0.29 | +| Dense n16 | 0.926, 0.833, 0.837 | 1.02–3.60 | 0.60–2.81 | 2.6–2.7, 0.10–0.19 | + +Observed: + +- vLLM's dense M5 varies between rounds more than the V5 bound: 0.537–0.926 + across the 12 dense rounds of runs a and b. The n16 means of the two runs + differ by 0.18. +- In every vLLM round the pairing (V3) and the one-to-one lane sequences (V2) + match the after revision, and the first forwards co-start (V4). +- The non-overlap consists of per-pair start offsets of up to about 1.5 ms + (`forward_start_ts` is taken before the per-forward DP metadata exchange) + and end offsets from per-rank duration variation. +- MoE ends align within 0.2–0.7 ms in total. + +Inference: in MoE the EP collectives inside each forward hold the two ranks +together, so vLLM's co-execution is close to Frontier's 1.0. The dense ranks +meet once per forward and then run host-bound forwards of about 3 ms whose +durations vary per rank. The dummy predictor gives both lanes the same +duration, so Frontier's co-execution is exactly 1.0 whenever the lanes +co-start. The residual is a duration-variance property of the ground truth +that the dummy predictor does not model. It is not an admission difference: +admission is what V1–V4 measure, and they match. The dense base (0.600, +0.778) is numerically closer to vLLM only because base serialization removes +overlap; its pairing (V3) and co-start (V4) are wrong in every round. + +`compare_lanes.py` labels every `MISMATCH` with the admission owner +`stage_execution_context.py`; on the evidence above, these two rows belong to +the execution-time model instead. The plan (§3 P5) requires a stop with +nothing adjusted. + +## 6. Decisions needed before P4 + +1. **C3 witness metric.** + - Observed: at `attn_dp=4` the rule "strictly larger `multi_lane_busy_time`" fails, although overlap becomes complete. + - Proposal: define the witness condition on the co-execution fraction `multi_lane_busy_time / busy_time`, the same quantity as M5. It strictly increases in all four witnesses (0.818, 0.375, 0.846, 0.846 → 1.0). Keep the self-overlap and `peak_lanes` checks unchanged. +2. **C7 V5 on the dense shape.** + - Observed: vLLM's own round-to-round spread exceeds the 0.10 bound, and the gap comes from per-rank duration variance. + - Proposal: report V5 for dense as informational, with the decomposition above, and keep V5 as a gate for MoE, where it passes. C7 then rests on V1–V4 for both models, V5 for MoE, and the base negative controls. + +## 7. Verification limits + +- The Frontier side runs the dummy predictor; no latency or duration + parity is claimed (D-8). +- vLLM instrumented mode synchronizes after each forward; stage-1 intervals + use a wall/monotonic offset and are informational. +- The vLLM ground truth runs with one recorded overlay patch (§5.1); the fork + checkout still carries the five-argument call and its fork test + `tests/model_executor/test_enabled_custom_ops.py::test_topk_softmax_wrapper_forwards_renormalize`. +- C6 was measured with a completion-only probe because the boundary seam is + on PR 35; the PR 35 composition check is pending in the parent task. From aeeca933826b41e56aaa07079b00eaab140e5dc0 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 10:40:22 +0800 Subject: [PATCH 10/19] tests: judge admission witnesses and dense co-execution per plan D-9 Contention witnesses now pass on a strictly larger co-execution fraction (multi_lane_busy_time / busy_time). At attn_dp=4 the fix admits all lanes together and shortens the busy period, so the absolute overlap time fell while the overlap became complete. V5 of the vLLM comparison gates the MoE shape only. vLLM's dense DP ranks meet once per forward and vary in duration per rank (co-execution 0.54-0.93 across rounds), which the dummy predictor does not model; the dense value is reported as INFORMATIONAL and the status counts only MISMATCH rows. --- .../stage_admission_pp/compare_lanes.py | 17 ++++++++++++----- tests/e2e/stage_admission_matrix.py | 7 +++++-- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/tests/comparison/stage_admission_pp/compare_lanes.py b/tests/comparison/stage_admission_pp/compare_lanes.py index 035eb68a..1ff9264a 100644 --- a/tests/comparison/stage_admission_pp/compare_lanes.py +++ b/tests/comparison/stage_admission_pp/compare_lanes.py @@ -37,6 +37,11 @@ BURSTS = (8, 16) CO_START_BOUND = 0.5 CO_EXECUTION_BOUND = 0.10 +# Dense DP ranks meet once per forward and then vary in duration per rank, +# which the dummy predictor does not model; their co-execution is reported, +# not gated (plan §4.7 V5, D-9). MoE ranks stay aligned by EP collectives. +CO_EXECUTION_GATED = {"moe": True, "dense": False} +INFORMATIONAL = "INFORMATIONAL" FRONTIER_OWNER = "frontier/scheduler/replica_stage_scheduler/stage_execution_context.py" @@ -200,11 +205,13 @@ def compare(vllm_run: Path, frontier_root: Path, before: str, after: str) -> tup gt_m5 = statistics.mean(vllm_metrics[r]["stage0"]["M5_co_execution"] for r in rounds) after_m5 = new_metrics["stage0"]["M5_co_execution"] if new_metrics else None base_m5 = base_metrics["stage0"]["M5_co_execution"] if base_metrics else None - m5_ok = after_m5 is not None and abs(after_m5 - gt_m5) <= CO_EXECUTION_BOUND - if model == "dense": - m5_ok = m5_ok and base_m5 is not None and abs(base_m5 - gt_m5) > abs(after_m5 - gt_m5) + if CO_EXECUTION_GATED[model]: + m5_ok = after_m5 is not None and abs(after_m5 - gt_m5) <= CO_EXECUTION_BOUND + m5_status = "MATCH" if m5_ok else "MISMATCH" + else: + m5_status = INFORMATIONAL rows.append(_row("V5", model, burst, "mean", "M5 stage-0 co-execution", gt_m5, after_m5, base_m5, - "MATCH" if m5_ok else "MISMATCH")) + m5_status)) return rows, details @@ -223,7 +230,7 @@ def main(argv=None) -> int: writer.writeheader() writer.writerows(rows) (args.output / "lane_metrics.json").write_text(json.dumps(details, indent=1, sort_keys=True)) - mismatches = [row for row in rows if row["status"] != "MATCH"] + mismatches = [row for row in rows if row["status"] == "MISMATCH"] placement_ok = all(p["ok"] for p in details["placement"].values()) placement_unseen = sum(len(p["unseen"]) for p in details["placement"].values()) status = { diff --git a/tests/e2e/stage_admission_matrix.py b/tests/e2e/stage_admission_matrix.py index e08a63cc..6f952a45 100644 --- a/tests/e2e/stage_admission_matrix.py +++ b/tests/e2e/stage_admission_matrix.py @@ -692,8 +692,11 @@ def compare_sets(before: str, after: str) -> list[dict]: if not identical: row["differing_files"] = _differing_files(base["sha256sums"], new["sha256sums"]) if case.contention_witness: - total = lambda metric: sum(stage["multi_lane_busy_time"] for stage in metric.values()) - row["witness_increase"] = total(after_metric) > total(before_metric) + # The fraction, not the absolute overlap: admitting more lanes + # together also shortens the busy period. + fraction = lambda metric: (sum(stage["multi_lane_busy_time"] for stage in metric.values()) + / sum(stage["busy_time"] for stage in metric.values())) + row["witness_increase"] = fraction(after_metric) > fraction(before_metric) checks_ok = checks_ok and row["witness_increase"] row["verdict"] = ("PASS" if identical and checks_ok else "EXPLAIN" if checks_ok else "STOP") From fc34341441ffc563a8ff62211e9f87aa548c6889 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 10:42:32 +0800 Subject: [PATCH 11/19] docs(stage-admission): adopt D-9 and close the comparisons Records the user's decision to judge contention witnesses by the co-execution fraction and to gate V5 on the MoE shape only (plan D-9, requirements R-8), the reruns of both comparisons under those rules (no STOP, no MISMATCH), the updated test report and calibration case, and the completion summary. --- .../analysis/workflow_gap_status.json | 6 +- .../analysis/workflow_gap_summary.md | 16 +++--- .../analysis/workflow_gap_table.csv | 4 +- .../stage_admission_case_001/manifest.yaml | 13 +++-- .../plan.md | 20 +++++-- .../progress.md | 14 +++-- .../requirements.md | 6 ++ .../summary.md | 57 +++++++++++++++++++ ...ort_2026-09-23_stage_admission_ordering.md | 52 ++++++++++------- 9 files changed, 142 insertions(+), 46 deletions(-) create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/summary.md diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json index 2475be98..a7c1edd6 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json @@ -1,10 +1,10 @@ { "analysis_state": "COMPLETE", - "status": "FAIL", + "status": "PASS", "correction_state": "not_applicable", "rows": 52, - "mismatches": 2, + "mismatches": 0, "vllm_placement_ok": true, "vllm_placement_unseen_requests": 0, - "next_action": "report each MISMATCH row with its cause before P4; adjust nothing" + "next_action": "record C7 in the test report" } \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md index 3c6a5768..09661073 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | D-9 adopted: dense V5 reported, not gated; rerun status PASS, 0 mismatches. | | 2026-09-23 | Created from run `sa-pp-20260923b` against Frontier sets `base` (`1f694f7` rule) and `after` (`dac4e69`). | ## Inputs @@ -17,8 +18,8 @@ ## Result -52 rows: 50 `MATCH`, 2 `MISMATCH`. `vllm_placement_ok = true`, no unseen -request. +52 rows: 50 `MATCH`, 0 `MISMATCH`, 2 `INFORMATIONAL` (dense V5 under D-9); +status PASS. `vllm_placement_ok = true`, no unseen request. | Metric | MoE (n8, n16) | Dense (n8, n16) | | --- | --- | --- | @@ -26,9 +27,9 @@ request. | V2 lane sequences | MATCH 6/6 | MATCH 6/6 | | V3 stage-0 pairing | MATCH 6/6 | MATCH 6/6; base pairs are shifted by one forward | | V4 first-forward co-start | MATCH 6/6 (vLLM ≤ 0.063, after 0.0) | MATCH 6/6 (vLLM ≤ 0.248, after 0.0, base 1.0: negative control holds) | -| V5 stage-0 co-execution | MATCH: vLLM 0.976 / 0.948, after 1.0 | **MISMATCH**: vLLM 0.706 / 0.865, after 1.0, base 0.600 / 0.778 | +| V5 stage-0 co-execution | MATCH: vLLM 0.976 / 0.948, after 1.0 | INFORMATIONAL (D-9): vLLM 0.706 / 0.865, after 1.0, base 0.600 / 0.778 | -## The two MISMATCH rows +## Dense V5 (MISMATCH before D-9) - Observed: vLLM's dense co-execution varies from round to round by more than the 0.10 bound: 0.537–0.926 over the 12 dense rounds of runs a and b. The @@ -46,6 +47,7 @@ request. owner is the execution-time model: the dummy predictor gives equal durations. It is not `stage_execution_context.py`, the default owner label written into the table. -- Plan §3 P5 requires a stop before P4 with nothing adjusted. Proposed - resolution, pending the user: V5 becomes informational for the dense shape - and stays a gate for MoE. +- The first analysis stopped here with nothing adjusted. The user adopted + D-9: V5 is informational for the dense shape and stays a gate for MoE. + C7 rests on V1–V4 for both models, V5 for MoE, and the base negative + controls. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv index 4396e801..bd5e9e32 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv @@ -37,7 +37,7 @@ V1,dense,8,2,M1 completion,"""8/8""","""success""","""success""",MATCH,, V2,dense,8,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",MATCH,, V3,dense,8,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], null]]",MATCH,, V4,dense,8,2,M4 stage-0 co-start,0.24755095323041817,0.0,1.0,MATCH,, -V5,dense,8,mean,M5 stage-0 co-execution,0.7055180242540143,1.0,0.6,MISMATCH,frontier/scheduler/replica_stage_scheduler/stage_execution_context.py, +V5,dense,8,mean,M5 stage-0 co-execution,0.7055180242540143,1.0,0.6,INFORMATIONAL,, V1,dense,16,0,M1 completion,"""16/16""","""success""","""success""",MATCH,, V2,dense,16,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",MATCH,, V3,dense,16,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], [9]], [[8], [11]], [[10], [13]], [[12], [15]], [[14], null]]",MATCH,, @@ -50,4 +50,4 @@ V1,dense,16,2,M1 completion,"""16/16""","""success""","""success""",MATCH,, V2,dense,16,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",MATCH,, V3,dense,16,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], [9]], [[8], [11]], [[10], [13]], [[12], [15]], [[14], null]]",MATCH,, V4,dense,16,2,M4 stage-0 co-start,0.04588904145721386,0.0,1.0000000000000002,MATCH,, -V5,dense,16,mean,M5 stage-0 co-execution,0.8653912262138098,1.0,0.7777777777777778,MISMATCH,frontier/scheduler/replica_stage_scheduler/stage_execution_context.py, +V5,dense,16,mean,M5 stage-0 co-execution,0.8653912262138098,1.0,0.7777777777777778,INFORMATIONAL,, diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml index 1b634e51..b448e41b 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml @@ -140,9 +140,14 @@ decisions: question: How to run the MoE ground truth after the topk_softmax ABI failure of run a? answer: "topk_softmax 统一修复为4 个参数的版本" decided_at_utc: "2026-09-23" + - decision_id: D-9 + question: V5 on the dense shape, after the first analysis found vLLM's own dense co-execution spread wider than 0.10. + answer: "采纳你的推荐，继续" + outcome: V5 gates MoE only; dense M5 is reported with its start/end decomposition. + decided_at_utc: "2026-09-23" analysis_result: >- - 50 of 52 rows MATCH. V5 (stage-0 co-execution) MISMATCH on dense n8 and n16; - cause and proposal in analysis/workflow_gap_summary.md. Plan §3 P5 stop: - nothing adjusted, awaiting the user's decision. -status: STOPPED_FOR_DECISION + 52 rows: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5, D-9). The first + analysis, before D-9, had these two rows as MISMATCH; cause in + analysis/workflow_gap_summary.md. +status: COMPLETE diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/plan.md b/task_memory/task_2026-09-22_stage_admission_ordering/plan.md index 7251a5d5..75ba7d5f 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/plan.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/plan.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-8 / D-9: the C3 witness condition uses the co-execution fraction, and V5 gates MoE only (dense is reported). Adopted after the P3/P5 stops, before P4. | | 2026-09-23 | R-7: the MoE retry job runs the ground truth with one recorded overlay patch (four-argument `topk_softmax`); §4.7 notes it. | | 2026-09-23 | R-6: execution started. Added package P5 (vLLM comparison on a GPU worker), criterion C7, the vLLM-aligned group G7, §4.7 and D-8. | | 2026-09-23 | Applied the round-1 plan review (`review.md`). Changes: C1 now targets confirmed admission-deadlock witnesses from a phase-controlled group; C3 uses the stage ledger and overlap duration; C4 takes the reviewer's wording; P0 lists its artifacts and outcome classes; P2 covers both sides of the EP boundary, the `DECODE_FFN` dense-group control, a second admission round and per-fixture base expectations, and the dense fixture asserts a same-start condition that discriminates on the base; P3 has three acceptance paths. The matrix now publishes a concrete case list on the analytical backend and keeps `attn_dp=2, PP=3`. Added D-6 and D-7. Not executed. | @@ -38,7 +39,7 @@ own branch: | --- | --- | --- | | C1 | Repaired liveness (path L). Every G3a case that P0 classifies as `admission_deadlock` completes after P1, with request count, prefill tokens and decode tokens conserved. G3a is the phase-controlled, prefill-only group. P0 must find at least one such case for each of `(attn_dp, PP)` ∈ {2, 4} × {2, 3}; if a pair has none, stop and report before P1, because the case list does not exercise the defect there. | P0 classification; P2(c); P3 path L. | | C2 | Unchanged controls (path U). Every run-to-run-stable metrics file is byte-identical before and after for G1 (all 30 release recipes, including the 10 PD-AF recipes), every `PP = 1` cell of G3a, G3b and G4, and every G5 cell. | P0 vs P3 `sha256sums.txt`. | -| C3 | Timing change (path T). Base-successful cases with `attn_dp > 1` and `PP > 1` (all G4 `PP > 1` cells, and G3a/G3b cells P0 classifies as `success`) are either byte-identical, or their difference is explained with the stage-ledger metric of §4.5 plus batch membership and component durations. The designated contention witnesses (§4.2, marked W) show a strictly larger `multi_lane_busy_time` after P1. In every case: no lane overlaps itself, and `peak_lanes ≤ attn_dp`. | P3, §4.5 metric from `frontier_stage_batch_ledger.jsonl`. | +| C3 | Timing change (path T). Base-successful cases with `attn_dp > 1` and `PP > 1` (all G4 `PP > 1` cells, and G3a/G3b cells P0 classifies as `success`) are either byte-identical, or their difference is explained with the stage-ledger metric of §4.5 plus batch membership and component durations. The designated contention witnesses (§4.2, marked W) show a strictly larger co-execution fraction `multi_lane_busy_time / busy_time` (§4.5, summed over the witness's stages) after P1 (D-9). In every case: no lane overlaps itself, and `peak_lanes ≤ attn_dp`. | P3, §4.5 metric from `frontier_stage_batch_ledger.jsonl`. | | C4 | Existing passing tests must remain passing without assertion changes. Existing failures, collection errors, and skips must be compared against a fresh run of the exact base revision in the same environment. Any new failure or required change to an ordering assertion stops implementation for review. | G2, §4.6. | | C5 | The predicate is one readable condition, and the module and method docstrings state the ordering contract as implemented. The change adds no flag, config field, `getattr` fallback, lane field on tickets, acquisition wake-up, PP-specific branch, second queue or capacity-1 special case. | Review of the diff against the quality gates. | | C7 | vLLM comparison (path V, R-6). On the vLLM-aligned shapes of §4.7, the P1 revision matches vLLM on completion, per-lane batch sequences, stage-0 lane pairing, first-forward co-start and stage-0 co-execution, and the base revision fails the negative controls stated there. | P5, §4.7. | @@ -154,7 +155,7 @@ scratch root; the test report keeps the classification table and the metrics. | --- | --- | --- | | U unchanged | G1, G5, every `PP=1` cell, G6 recipes | Identical `sha256sums.txt` (run-to-run-unstable files excluded by P0 with a reason). | | L repaired liveness | Cases P0 classifies as `admission_deadlock` | `success` after P1; completed requests = generated requests; the sums of prefill and decode tokens over `request_metrics.csv` equal the generated lengths. | -| T timing | Cases P0 classifies as `success` with `attn_dp>1, PP>1` | Identical hashes, or a ledger-explained difference (§4.5). W cases: strictly larger `multi_lane_busy_time`. | +| T timing | Cases P0 classifies as `success` with `attn_dp>1, PP>1` | Identical hashes, or a ledger-explained difference (§4.5). W cases: strictly larger co-execution fraction (D-9). | ### 4.5 Lane-overlap metric (C3) @@ -170,6 +171,8 @@ with `execution_scope == "ATTN_DP_LANE"` as half-open intervals | Output | Definition | | --- | --- | | `multi_lane_busy_time` | Total simulated time during which at least two distinct lanes have an open interval. Touching endpoints overlap for zero time; zero-length rows contribute nothing. | +| `busy_time` | Total simulated time during which at least one lane has an open interval. | +| Co-execution fraction | `multi_lane_busy_time / busy_time`; over several stages, the sums of both. | | `peak_lanes` | The largest number of distinct lanes open at one instant. | | `makespan` | The largest `stage_end_ts` in the ledger. | | Checks | No lane's intervals overlap one another. `peak_lanes ≤ attn_dp`. | @@ -266,7 +269,7 @@ row in Frontier; both sides compare real forwards only. | V2 | After-revision M2 equals vLLM M2 in every round. vLLM rounds that disagree with one another are reported, with the cause. | | V3 | After-revision M3 equals vLLM M3 in every round. A vLLM round in which a dummy forward shifts the pairing is named and reported, not dropped. | | V4 | vLLM M4 < 0.5 in every round and after-revision M4 < 0.5: both lanes start in the same forward slot. Dense negative control: base M4 ≥ 0.5. | -| V5 | `|M5(after) − mean M5(vLLM)| ≤ 0.10`; for dense also `|M5(base) − mean M5(vLLM)| > |M5(after) − mean M5(vLLM)|`. The 0.10 bound reuses the calibration contract's tolerance; it is applied to a fraction, not to latency. | +| V5 | MoE: `|M5(after) − mean M5(vLLM)| ≤ 0.10`. The 0.10 bound reuses the calibration contract's tolerance; it is applied to a fraction, not to latency. Dense: M5 is reported with its start/end decomposition, not gated (D-9). | The comparison writes `analysis/workflow_gap_table.csv` (one row per metric, burst and round, with `MATCH`/`MISMATCH`, values, source and Frontier owner), @@ -317,6 +320,13 @@ evidence and recorded here for review): | --- | --- | --- | | D-8 | The vLLM comparison is structural (M1–M5), in vLLM's instrumented mode, with a prefill-only workload on one MoE and one dense model already in `data/config/models/`. No E2E latency gate. | The fix changes admission, not durations; the Frontier side runs the dummy predictor. Prefill-only keeps the comparison inside the D-7 scope boundary. Both models exist on both sides without new assets. | +Adopted after the P3 and P5 stops on 2026-09-23 ("采纳你的推荐，继续"; evidence in +`test_report_2026-09-23_stage_admission_ordering.md` §4.3, §5.3): + +| Id | Decision | Reason | +| --- | --- | --- | +| D-9 | (a) A contention witness passes when its co-execution fraction strictly increases; the self-overlap and `peak_lanes` checks are unchanged. (b) V5 gates the MoE shape only; for the dense shape M5 is reported with its start/end decomposition. | (a) At `attn_dp=4` the fix makes all four lanes co-execute and shortens the busy period, so absolute `multi_lane_busy_time` falls (0.55 → 0.30) while overlap becomes complete; the fraction measures overlap independently of that compression. (b) vLLM's dense ranks meet once per forward and vary in duration per rank (M5 0.54–0.93 across rounds, wider than 0.10), a property the dummy predictor does not model; admission is covered by V1–V4. MoE ranks stay aligned by in-forward EP collectives (M5 0.93–0.98). | + ## 6. Dependencies and risks - The reproduction scripts still live in the session scratchpad @@ -337,7 +347,9 @@ evidence and recorded here for review): yielding to the event loop to make it unlikely. - Risk (P5): vLLM stage-0 intervals start before the per-forward DP all-reduce, so a rank that arrives early records its wait as busy time. M4 - uses start times only, and V5 has the stated 0.10 bound. + uses start times only, and V5 has the stated 0.10 bound. Observed in P5: + on the dense shape the vLLM ranks' own M5 varies by more than 0.10 between + rounds, so dense V5 is reported, not gated (D-9). - The parent task's Step 9 resumes only after this branch is merged into `main` and merged forward into `fix/issue26-correctness-pr`. The parent task then reruns G3b on that branch, where W3 is present, as the composition check diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md index 1115f55c..fabeeeac 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-8 / D-9 adopted; both comparisons rerun and pass (`aeeca93`); P4 in progress. | | 2026-09-23 | P0–P3 and P5 executed. Rule committed (`dac4e69`). Two plan stop conditions reached (C3 witness metric, C7 V5 dense); P4 push held for the user. | | 2026-09-23 | R-6 received: execution started; P5 (vLLM comparison) added to the plan. | | 2026-09-23 | Round-1 plan review verified against source and applied to `design.md`, `plan.md` and `requirements.md`; `review.md` created; resume prompt updated for round 2. Not executed. | @@ -24,9 +25,9 @@ | P0 evidence and baseline | completed 2026-09-23 | 98 cases classified as designed; rerun hashes stable; see "Execution" | | P1 rule | completed | `dac4e69` (with P2 tests) | | P2 tests and base negative controls | completed | `evidence/base_negative_controls.log`; all base outcomes as planned | -| P3 rerun and comparison | completed; **stopped** on C3 witness rule at `attn_dp=4` | `test_report_2026-09-23_stage_admission_ordering.md` §4 | -| P5 vLLM comparison | completed; **stopped** on V5 dense | case `calibration/stage_admission_case_001/`, report §5 | -| P4 records, commit, push | in-progress: records written and committed locally; push and PR body held for the user's decision | report §6 | +| P3 rerun and comparison | completed; the first comparison stopped on the C3 witness rule, passes under D-9 | `test_report_2026-09-23_stage_admission_ordering.md` §4 | +| P5 vLLM comparison | completed; the first analysis stopped on dense V5, passes under D-9 | case `calibration/stage_admission_case_001/`, report §5 | +| P4 records, commit, push | in-progress | see below | ## Commands run (2026-09-22) @@ -74,6 +75,7 @@ worktree, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`. | P5b run b | RJob `exp-0923-024146-345158` | `runs/vllm-instrumented/sa-pp-20260923b/` | MoE and dense complete, status 0 | | P5c | `compare_lanes --vllm-run …/sa-pp-20260923b` | `calibration/stage_admission_case_001/analysis/` | 50/52 MATCH; V5 dense n8/n16 MISMATCH (vLLM 0.706/0.865 vs 1.0) | -Open decisions (report §6): the C3 witness metric at `attn_dp=4`, and V5 for -the dense shape. P4 push, the PR 36 body update and the parent-task note wait -for them. +| D-9 rules | witness by co-execution fraction; V5 gated on MoE only | `aeeca93` | — | +| P3 compare rerun | `compare --before base --after after --output …/compare_base_after_d9.json` | scratch root | U 50 PASS; L 18 PASS; T 6 PASS, 8 EXPLAIN; no STOP | +| P5c rerun | `compare_lanes --vllm-run …/sa-pp-20260923b` | `analysis/` | status PASS: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL | +| Checks after D-9 | P5a synthetic check; P2(c) integration test | — | synthetic planted round still caught by V3/V4; 3 passed | diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md index 1cf2281c..3d50238d 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-8: D-9 adopted for the C3 witness rule and dense V5; P4 authorized to continue. | | 2026-09-23 | R-7: ground-truth `topk_softmax` fixed to the four-argument version for the MoE retry. | | 2026-09-23 | R-6: execute P0–P4 and validate the fix against vLLM on a GPU worker (package P5). | | 2026-09-23 | R-5: round-1 plan review verified and applied to the records; execution deferred. | @@ -43,6 +44,10 @@ MoE `topk_softmax` ABI): > 我先提前决策，避免中断任务：topk_softmax 统一修复为4 个参数的版本 +`[Original Request]` (2026-09-23, answering the two stops of the test report §6): + +> 采纳你的推荐，继续 + Quality gates the user repeated for every core-module change in this line of work, carried over verbatim: @@ -59,6 +64,7 @@ work, carried over verbatim: | R-5 | Verify every review finding against the source, adopt the high-value and necessary corrections into the records (dispositions in `review.md`, new decisions D-6 and D-7 in `plan.md`), and do not execute: no P0 run, no source change. The docs commit is pushed to the draft PR under R-4. | user, 2026-09-23 | | R-6 | Execute P0–P4 as planned. The fix must also be validated against vLLM running on a GPU worker, in a comparison designed to show whether the change is effective (package P5 in `plan.md`). The request authorizes the GPU job within the standing GPU rules below. | user, 2026-09-23 | | R-7 | The vLLM ground truth uses the four-argument `topk_softmax` (wrapper and call). Applied as the recorded overlay patch `calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch` on the one retry job; the vLLM-BS checkout is unchanged. | user, 2026-09-23 | +| R-8 | Adopt both recommendations (plan D-9): contention witnesses pass on a strictly larger co-execution fraction; V5 gates MoE only and reports dense. Continue to P4. | user, 2026-09-23 | ## Constraints carried from the parent task diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/summary.md b/task_memory/task_2026-09-22_stage_admission_ordering/summary.md new file mode 100644 index 00000000..13979b22 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/summary.md @@ -0,0 +1,57 @@ +# Stage admission ordering under pipeline parallelism — Summary + +## Modification History + +| Date | Change | +| --- | --- | +| 2026-09-23 | Created at P4: fix, tests, P0–P3 and the vLLM comparison complete under D-9. | + +## Overview + +W9-01: with `attn_dp > 1` and `num_pipeline_stages > 1`, a busy lane's queued +ticket at the head of a stage's ready FIFO refused another lane's runnable +batch. MoE runs drained with requests unfinished (admission deadlock). Dense +runs completed but started the lanes one forward apart. + +The fix (plan D-1, option B) changes one predicate in +`StageExecutionContext.try_acquire`. A full-stage ticket is refused only by an +EP wave queued ahead of it. EP waves keep the strict FIFO-head rule. The +admitted ticket leaves the FIFO by `remove(ticket)`. + +## Deliverables + +| Item | Path / commit | +| --- | --- | +| Rule and P2 tests | `dac4e69`: `frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`, `tests/unit/test_stage_execution_context.py`, `tests/unit/test_shared_forward_group_admission.py`, `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`, `tests/integration/test_stage_admission_pipeline_lanes.py` | +| Case matrix | `tests/e2e/stage_admission_matrix.py` (`a054d87`, `5ade853`, `aeeca93`) | +| vLLM comparison | `tests/comparison/stage_admission_pp/{vllm_burst_driver.py,run_vllm_worker.sh,compare_lanes.py}` (`799ccb4`, `a1b9819`, `aeeca93`) | +| Test report | `test_report_2026-09-23_stage_admission_ordering.md` | +| Calibration case | `calibration/stage_admission_case_001/` (manifest, inputs incl. `groundtruth_overlay.patch`, two vLLM runs, `analysis/`) | +| Evidence | `evidence/` (base negative controls, G2 comparisons, path-T explanation, Step 9 probe, co-execution decomposition script) | +| Branch / PR | `fix/stage-admission-ordering`, draft PR https://github.com/NetX-lab/Frontier/pull/36 | + +## Validation (observed) + +| Criterion | Result | +| --- | --- | +| C1 | 18 base admission deadlocks (G3a 10, G3b 6, G7 2) complete with requests and tokens conserved | +| C2 | 50/50 unchanged cases byte-identical (30 release recipes, every `PP=1` cell, G5) | +| C3 | 6 T cases identical. The other 8 change start times only: same batches, same component durations, no self-overlap, `peak_lanes ≤ attn_dp`. All 4 witnesses have a strictly larger co-execution fraction (D-9). | +| C4 | `tests/unit` and `tests/integration`: no regression, no new failure, skips and collection errors unchanged | +| C5 | One predicate plus docstrings; no flag, field, fallback, wake-up or special case | +| C6 | Step 9 probe shape MoE `attn_dp=2, moe_ep=2, PP=2` completes 6/6 (base drains) | +| C7 | vLLM DP=2/PP=2 on 4×H800, run `sa-pp-20260923b`: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5, D-9). MoE: 26/26 rows MATCH. Dense: V1–V4 MATCH in every round. The base fails its negative controls: MoE deadlock, dense pairing and co-start. | + +Decisions taken during execution: + +| Id | Decision | +| --- | --- | +| R-7 | The vLLM ground truth uses the four-argument `topk_softmax`, applied as a recorded overlay patch. The checkout is unchanged. | +| R-8 / D-9 | Witnesses are judged by co-execution fraction. V5 gates MoE only. | + +## Open and deferred work + +- PR 35 (`fix/issue26-correctness-pr`) merges this branch forward after it lands and reruns G3b as the composition check with W3. Only then does Step 9 resume (C6). +- `PP=3` with `attn_dp=2` stays rejected by the node-size rule on the default backends (W9-02), outside this fix. +- vLLM-BS: the fork's Python `topk_softmax` still passes five arguments. So does its test `tests/model_executor/test_enabled_custom_ops.py::test_topk_softmax_wrapper_forwards_renormalize`. The four-argument form was applied only as this case's overlay patch. +- Dense per-rank duration variance, which vLLM shows and the dummy predictor lacks, is an execution-time-model topic. It is not part of admission. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md b/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md index 08b52b35..239ff593 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | D-9 adopted ("采纳你的推荐，继续"): C3 witnesses judged by co-execution fraction, V5 gated on MoE only. Both comparisons rerun (`aeeca93`); all criteria pass. | | 2026-09-23 | Created. P0–P3 and P5 executed; two plan stop conditions reached (C3 witness metric at `attn_dp=4`, C7 V5 on the dense shape). P4 push held for the user's decision. | ## 1. Result @@ -12,11 +13,11 @@ | --- | --- | --- | | C1 repaired liveness | PASS: all 10 G3a `admission_deadlock` cases complete with conservation; so do the 6 G3b and 2 G7 MoE deadlocks. | §4.1 | | C2 unchanged controls | PASS: 50 of 50 U cases byte-identical. | §4.2 | -| C3 timing change | **STOP (plan §3 P3)**: 6 T cases identical, 8 differ. All 8 differences are start times only (same batches, same component durations), with no self-overlap and `peak_lanes ≤ attn_dp`. The two `attn_dp=4` contention witnesses fail the stated rule "strictly larger `multi_lane_busy_time`". | §4.3 | +| C3 timing change | PASS (D-9): 6 T cases identical, 8 differ. All 8 differences are start times only (same batches, same component durations), with no self-overlap and `peak_lanes ≤ attn_dp`. All 4 contention witnesses have a strictly larger co-execution fraction. The first comparison stopped on the original absolute-overlap rule; see §4.3. | §4.3 | | C4 existing tests | PASS: no base-passed node regresses, no new failure or error, skips and collection errors unchanged. | §4.4 | | C5 rule shape | PASS by review: one predicate, docstrings state the contract, no flag, field, fallback, wake-up, PP branch, second queue or capacity-1 case. | §3 | | C6 Step 9 probe | Informational: MoE `attn_dp=2, moe_ep=2, PP=2` completes 6/6 (base drains). `PP=3` stops on the known W9-02 node-size rejection. | §4.5 | -| C7 vLLM comparison | **STOP (plan §3 P5)**: 50 of 52 rows MATCH. The two `MISMATCH` rows are V5 (stage-0 co-execution) on the dense shape. MoE matches on all 26 rows, V5 included. The negative controls fail on the base as planned. | §5 | +| C7 vLLM comparison | PASS (D-9): 50 rows MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5). MoE matches on all 26 rows, V5 included; dense matches V1–V4 in every round. The negative controls fail on the base as planned. The first comparison stopped on dense V5; see §5.3. | §5 | Observed facts are separated from inferences. Inferences are marked "Inference". @@ -31,7 +32,7 @@ Observed facts are separated from inferences. Inferences are marked | Base set `base` | run at `a054d87` (harness only; `frontier/` identical to `1f694f7`) | | After set `after` | run at `dac4e69`, tree clean outside `task_memory/`; 98 cases in 90 s with `--jobs 8` | | Rule commit | `dac4e69` fix(scheduler): order full-stage admission only behind queued EP waves | -| Harness commits | `a054d87`, `5ade853` (matrix), `799ccb4` (vLLM comparison), `a1b9819` (recorded overlay patch) | +| Harness commits | `a054d87`, `5ade853` (matrix), `799ccb4` (vLLM comparison), `a1b9819` (recorded overlay patch), `aeeca93` (D-9 witness and V5 rules) | | Scratch root | `/data/ycfeng/tmp/stage_admission_ordering/{base,after,base-rerun,base-pytest,after-pytest,step9_probe}` | Commands: @@ -39,7 +40,7 @@ Commands: ```bash python -m tests.e2e.stage_admission_matrix run --set after --jobs 8 python -m tests.e2e.stage_admission_matrix compare --before base --after after \ - --output /data/ycfeng/tmp/stage_admission_ordering/compare_base_after.json + --output /data/ycfeng/tmp/stage_admission_ordering/compare_base_after_d9.json # first run: compare_base_after.json python -m pytest tests/ -q -p no:cacheprovider --continue-on-collection-errors \ --junitxml=/.xml # suite in {unit, integration}, base and after python task_memory/.../evidence/explain_t_path.py @@ -113,8 +114,8 @@ no self-overlap, `peak_lanes ≤ attn_dp`. Differing files are the ledger, | G4-dense-dp2-pp2-n8 | W | 0.45 → 0.50 | 0.818 → 1.0 | 2 → 2 | {1: 0, 0: 0.05} → all 0 | EXPLAIN | | G4-dense-dp2-pp3-n4 | | 0.108 → 0.216 | 0.333 → 1.0 | 2 → 2 | {1: 0, 0: 0.036} → all 0 | EXPLAIN | | G4-dense-dp2-pp3-n8 | W | 0.216 → 0.396 | 0.375 → 1.0 | 2 → 2 | {1: 0, 0: 0.072} → all 0 | EXPLAIN | -| G4-dense-dp4-pp2-n8 | W | **0.55 → 0.30** | 0.846 → 1.0 | 2 → 4 | {1: 0, 2: 0.05, 3: 0.10, 0: 0.15} → all 0 | **STOP** | -| G4-dense-dp4-pp3-n8 | W | **0.396 → 0.216** | 0.846 → 1.0 | 2 → 4 | {1: 0, 2: 0.036, 3: 0.072, 0: 0.108} → all 0 | **STOP** | +| G4-dense-dp4-pp2-n8 | W | 0.55 → 0.30 | 0.846 → 1.0 | 2 → 4 | {1: 0, 2: 0.05, 3: 0.10, 0: 0.15} → all 0 | EXPLAIN (first run: STOP) | +| G4-dense-dp4-pp3-n8 | W | 0.396 → 0.216 | 0.846 → 1.0 | 2 → 4 | {1: 0, 2: 0.036, 3: 0.072, 0: 0.108} → all 0 | EXPLAIN (first run: STOP) | | G7-dense-dp2-pp2-n8 | | 0.36 → 0.48 | 0.60 → 1.0 | 2 → 2 | {1: 0, 0: 0.12} → all 0 | EXPLAIN | | G7-dense-dp2-pp2-n16 | | 0.84 → 0.96 | 0.778 → 1.0 | 2 → 2 | {1: 0, 0: 0.12} → all 0 | EXPLAIN | @@ -127,7 +128,11 @@ shrinks with it, although it is now the whole busy period. Request E2E for Inference: absolute `multi_lane_busy_time` measures overlap only while the busy period stays the same length; it cannot express "more overlap" when the fix compresses the timeline, which happens whenever the base serialized more -than two lanes. The plan requires a stop here, with nothing adjusted. +than two lanes. The first comparison stopped here with nothing adjusted. +Under D-9 the witness condition is the co-execution fraction, which strictly +increases in all four witnesses (0.818, 0.375, 0.846, 0.846 → 1.0); the rerun +(`compare_base_after_d9.json`) gives U 50 PASS, L 18 PASS, T 6 PASS and +8 EXPLAIN, and no STOP. ### 4.4 C4 — G2 test identities @@ -192,12 +197,12 @@ misplaced; `num_gpu_blocks` 600666 (MoE) and 304854 (dense). | V2 lane sequences | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | | V3 stage-0 pairing | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH; base pairs 0↔3, 2↔5, …, 6↔none | 3/3 MATCH; base shifted by one forward | | V4 co-start (vLLM / after / base) | 0.009–0.063 / 0.0 / — | 0.005–0.024 / 0.0 / — | 0.008–0.248 / 0.0 / 1.0 | 0.046–0.171 / 0.0 / 1.0 | -| V5 co-execution (vLLM mean / after / base) | 0.976 / 1.0 / — MATCH | 0.948 / 1.0 / — MATCH | **0.706 / 1.0 / 0.600 MISMATCH** | **0.865 / 1.0 / 0.778 MISMATCH** | +| V5 co-execution (vLLM mean / after / base) | 0.976 / 1.0 / — MATCH | 0.948 / 1.0 / — MATCH | 0.706 / 1.0 / 0.600 INFORMATIONAL (first run: MISMATCH) | 0.865 / 1.0 / 0.778 INFORMATIONAL (first run: MISMATCH) | Run a (dense only, same scripts): V1–V4 all MATCH; V5 vLLM mean 0.714 (n8) -and 0.685 (n16), also MISMATCH. +and 0.685 (n16): MISMATCH under the first rule, INFORMATIONAL under D-9. -### 5.3 The V5 dense mismatch +### 5.3 V5 on the dense shape `evidence/decompose_co_execution.py` splits the stage-0 non-overlap of each M3 pair into `|Δstart| + |Δend|` @@ -235,22 +240,29 @@ overlap; its pairing (V3) and co-start (V4) are wrong in every round. `compare_lanes.py` labels every `MISMATCH` with the admission owner `stage_execution_context.py`; on the evidence above, these two rows belong to -the execution-time model instead. The plan (§3 P5) requires a stop with -nothing adjusted. +the execution-time model instead. The first comparison stopped here with +nothing adjusted. Under D-9 dense V5 is reported, not gated; the rerun gives +`workflow_gap_status.json` status PASS with 0 mismatches. The P5a synthetic +check (`analysis/synthetic_check.py`) still flags its planted dummy-shifted +dense round through V3 and V4. -## 6. Decisions needed before P4 +## 6. Decisions -1. **C3 witness metric.** - - Observed: at `attn_dp=4` the rule "strictly larger `multi_lane_busy_time`" fails, although overlap becomes complete. - - Proposal: define the witness condition on the co-execution fraction `multi_lane_busy_time / busy_time`, the same quantity as M5. It strictly increases in all four witnesses (0.818, 0.375, 0.846, 0.846 → 1.0). Keep the self-overlap and `peak_lanes` checks unchanged. -2. **C7 V5 on the dense shape.** - - Observed: vLLM's own round-to-round spread exceeds the 0.10 bound, and the gap comes from per-rank duration variance. - - Proposal: report V5 for dense as informational, with the decomposition above, and keep V5 as a gate for MoE, where it passes. C7 then rests on V1–V4 for both models, V5 for MoE, and the base negative controls. +Both stops were resolved by D-9 (`plan.md`), adopted by the user on +2026-09-23 ("采纳你的推荐，继续"): + +1. C3: a contention witness passes on a strictly larger co-execution fraction + `multi_lane_busy_time / busy_time`; the self-overlap and `peak_lanes` checks + are unchanged. +2. C7: V5 gates the MoE shape only; the dense value is reported with the + decomposition of §5.3. C7 rests on V1–V4 for both models, V5 for MoE, and the + base negative controls. ## 7. Verification limits - The Frontier side runs the dummy predictor; no latency or duration - parity is claimed (D-8). + parity is claimed (D-8). Dense co-execution against vLLM is therefore not a + gate (D-9). - vLLM instrumented mode synchronizes after each forward; stage-1 intervals use a wall/monotonic offset and are informational. - The vLLM ground truth runs with one recorded overlay patch (§5.1); the fork From 4bcd61635344d764d63580ce3bdb4d4535bc8dc5 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 10:44:55 +0800 Subject: [PATCH 12/19] docs(stage-admission): track the base negative-control log The test report cites evidence/base_negative_controls.log, but the repository-wide *.log ignore rule kept it out of the tree. --- .../evidence/base_negative_controls.log | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/base_negative_controls.log diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/base_negative_controls.log b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/base_negative_controls.log new file mode 100644 index 00000000..c44f32a2 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/base_negative_controls.log @@ -0,0 +1,59 @@ +F.F.FFFFF [100%] +=================================== FAILURES =================================== +__ test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave __ +tests/unit/test_stage_execution_context.py:111: in test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave + assert context.try_acquire(full1) is True +E AssertionError: assert False is True +E + where False = try_acquire(StageAdmissionTicket(replica_id=0, stage_id=0, admission_seq=1, operation_id='full1', scope='FULL_STAGE_WORLD', participant_ep_ids=())) +E + where try_acquire = .try_acquire +_____ test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket _____ +tests/unit/test_stage_execution_context.py:141: in test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket + assert context.try_acquire(full1) is True +E AssertionError: assert False is True +E + where False = try_acquire(StageAdmissionTicket(replica_id=0, stage_id=0, admission_seq=1, operation_id='full1', scope='FULL_STAGE_WORLD', participant_ep_ids=())) +E + where try_acquire = .try_acquire +________ test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0] ________ +tests/unit/test_shared_forward_group_admission.py:71: in test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket + assert stages[other_lane].pop_batch_if_not_busy() is other_now +E assert None is batch id = 10\ndecode_attn_original_replica_id=None, decode_attn_original_replica_local_id=None\nnum req = 1, [8]\n------...8\nnum_prefill_tokens=16\nnum_decode_tokens=4\nnum_processed_tokens=0\ncurrent_decode_token_index=1\ncompleted_layer_count=0 +E + where None = pop_batch_if_not_busy() +E + where pop_batch_if_not_busy = .pop_batch_if_not_busy +________ test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[1] ________ +tests/unit/test_shared_forward_group_admission.py:71: in test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket + assert stages[other_lane].pop_batch_if_not_busy() is other_now +E assert None is batch id = 14\ndecode_attn_original_replica_id=None, decode_attn_original_replica_local_id=None\nnum req = 1, [12]\n-----...2\nnum_prefill_tokens=16\nnum_decode_tokens=4\nnum_processed_tokens=0\ncurrent_decode_token_index=1\ncompleted_layer_count=0 +E + where None = pop_batch_if_not_busy() +E + where pop_batch_if_not_busy = .pop_batch_if_not_busy +_____ test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4-expected0] ______ +tests/integration/test_stage_admission_pipeline_lanes.py:41: in test_moe_lanes_complete_every_request + metrics_dir = run_case(tmp_path, case_id) +tests/integration/test_stage_admission_pipeline_lanes.py:31: in run_case + assert outcome["outcome"] == SUCCESS, outcome +E AssertionError: {'exception': 'Sequential simulation ended with non-empty scheduler state', 'outcome': 'admission_deadlock', 'simulation_time': 0.007, 'wall_end': 1790101999.9712346, ...} +E assert 'admission_deadlock' == 'success' +E +E - success +E + admission_deadlock +_____ test_moe_lanes_complete_every_request[G3a-moe-dp4-pp2-n8-expected1] ______ +tests/integration/test_stage_admission_pipeline_lanes.py:41: in test_moe_lanes_complete_every_request + metrics_dir = run_case(tmp_path, case_id) +tests/integration/test_stage_admission_pipeline_lanes.py:31: in run_case + assert outcome["outcome"] == SUCCESS, outcome +E AssertionError: {'exception': 'Sequential simulation ended with non-empty scheduler state', 'outcome': 'admission_deadlock', 'simulation_time': 0.007, 'wall_end': 1790102001.4043047, ...} +E assert 'admission_deadlock' == 'success' +E +E - success +E + admission_deadlock +_______________ test_dense_lanes_start_in_the_same_first_forward _______________ +tests/integration/test_stage_admission_pipeline_lanes.py:60: in test_dense_lanes_start_in_the_same_first_forward + assert first_start[0] == first_start[1] +E assert 0.05 == 0.0 +=========================== short test summary info ============================ +FAILED tests/unit/test_stage_execution_context.py::test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave +FAILED tests/unit/test_stage_execution_context.py::test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket +FAILED tests/unit/test_shared_forward_group_admission.py::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0] +FAILED tests/unit/test_shared_forward_group_admission.py::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[1] +FAILED tests/integration/test_stage_admission_pipeline_lanes.py::test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4-expected0] +FAILED tests/integration/test_stage_admission_pipeline_lanes.py::test_moe_lanes_complete_every_request[G3a-moe-dp4-pp2-n8-expected1] +FAILED tests/integration/test_stage_admission_pipeline_lanes.py::test_dense_lanes_start_in_the_same_first_forward +7 failed, 2 passed in 6.74s From ecff89a2d21ea4cb74de190359c18400386a1595 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 10:46:54 +0800 Subject: [PATCH 13/19] docs(stage-admission): record P4 completion --- .../task_2026-09-22_stage_admission_ordering/progress.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md index fabeeeac..66509c51 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | P4 completed: branch pushed at `4bcd616`, PR 36 body updated (still draft), W9-01 resolution recorded in the parent task (`4c2d573`). | | 2026-09-23 | R-8 / D-9 adopted; both comparisons rerun and pass (`aeeca93`); P4 in progress. | | 2026-09-23 | P0–P3 and P5 executed. Rule committed (`dac4e69`). Two plan stop conditions reached (C3 witness metric, C7 V5 dense); P4 push held for the user. | | 2026-09-23 | R-6 received: execution started; P5 (vLLM comparison) added to the plan. | @@ -27,7 +28,7 @@ | P2 tests and base negative controls | completed | `evidence/base_negative_controls.log`; all base outcomes as planned | | P3 rerun and comparison | completed; the first comparison stopped on the C3 witness rule, passes under D-9 | `test_report_2026-09-23_stage_admission_ordering.md` §4 | | P5 vLLM comparison | completed; the first analysis stopped on dense V5, passes under D-9 | case `calibration/stage_admission_case_001/`, report §5 | -| P4 records, commit, push | in-progress | see below | +| P4 records, commit, push | completed 2026-09-23 | "Execution" P4 rows | ## Commands run (2026-09-22) @@ -74,8 +75,10 @@ worktree, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`. | Overlay patch support | `vllm_burst_driver.py overlay --patch`, worker `OVERLAY_PATCH` | `a1b9819`; CPU dry run against an `upstream-v0.10.2` export | accepted; `_custom_ops.py` equals upstream after patch; second application fails loudly | | P5b run b | RJob `exp-0923-024146-345158` | `runs/vllm-instrumented/sa-pp-20260923b/` | MoE and dense complete, status 0 | | P5c | `compare_lanes --vllm-run …/sa-pp-20260923b` | `calibration/stage_admission_case_001/analysis/` | 50/52 MATCH; V5 dense n8/n16 MISMATCH (vLLM 0.706/0.865 vs 1.0) | - | D-9 rules | witness by co-execution fraction; V5 gated on MoE only | `aeeca93` | — | | P3 compare rerun | `compare --before base --after after --output …/compare_base_after_d9.json` | scratch root | U 50 PASS; L 18 PASS; T 6 PASS, 8 EXPLAIN; no STOP | | P5c rerun | `compare_lanes --vllm-run …/sa-pp-20260923b` | `analysis/` | status PASS: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL | | Checks after D-9 | P5a synthetic check; P2(c) integration test | — | synthetic planted round still caught by V3/V4; 3 passed | +| P4 push | `git push origin fix/stage-admission-ordering` | remote head `4bcd616` | records `df7868e`, `fc34341`; `4bcd616` force-adds `evidence/base_negative_controls.log`, which the repository-wide `*.log` rule had kept out of the tree | +| P4 PR body | REST `PATCH repos/NetX-lab/Frontier/pulls/36` (`gh pr edit` fails on the retired Projects classic query) | PR 36 | body carries the rule, commits, C1–C4/C7 and C3 tables, R-7/D-9 and open items; body read back identical; still draft | +| P4 parent note | parent `issues.md` W9-01 Resolution, `progress.md`, case manifest decision `W9-01-scope`; `summary.md` and the test report copied to `w9_01_stage_admission_ordering/` (D-5) | `fix/issue26-correctness-pr` `4c2d573`, pushed | PR 35 still draft | From 1218ba6b3ebecc69e339166981012b6df5856375 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 11:16:49 +0800 Subject: [PATCH 14/19] docs(stage-admission): record the round-2 code review of PR 36 Fifteen findings on ecff89a, each re-checked against source and posted to PR 36 as inline comments. Fixes are deferred by the owner. --- .../progress.md | 2 ++ .../requirements.md | 8 +++++ .../review.md | 35 +++++++++++++++++++ 3 files changed, 45 insertions(+) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md index 66509c51..1564cdd0 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | Round-2 code review posted to PR 36 (15 inline comments, `review.md`); fixes deferred by the owner. | | 2026-09-23 | P4 completed: branch pushed at `4bcd616`, PR 36 body updated (still draft), W9-01 resolution recorded in the parent task (`4c2d573`). | | 2026-09-23 | R-8 / D-9 adopted; both comparisons rerun and pass (`aeeca93`); P4 in progress. | | 2026-09-23 | P0–P3 and P5 executed. Rule committed (`dac4e69`). Two plan stop conditions reached (C3 witness metric, C7 V5 dense); P4 push held for the user. | @@ -29,6 +30,7 @@ | P3 rerun and comparison | completed; the first comparison stopped on the C3 witness rule, passes under D-9 | `test_report_2026-09-23_stage_admission_ordering.md` §4 | | P5 vLLM comparison | completed; the first analysis stopped on dense V5, passes under D-9 | case `calibration/stage_admission_case_001/`, report §5 | | P4 records, commit, push | completed 2026-09-23 | "Execution" P4 rows | +| Round-2 code review (R2-01..R2-15) | posted; fixes pending the owner's decision | `review.md` Round 2; https://github.com/NetX-lab/Frontier/pull/36#pullrequestreview-5286523149 | ## Commands run (2026-09-22) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md index 3d50238d..eaec8bc6 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-9: code review of PR 36 posted as inline comments; fixes deferred. | | 2026-09-23 | R-8: D-9 adopted for the C3 witness rule and dense V5; P4 authorized to continue. | | 2026-09-23 | R-7: ground-truth `topk_softmax` fixed to the four-argument version for the MoE retry. | | 2026-09-23 | R-6: execute P0–P4 and validate the fix against vLLM on a GPU worker (package P5). | @@ -84,3 +85,10 @@ work, carried over verbatim: volume access is confined to `/mnt/codesign-exp/ycfeng`. Credential values stay in restricted files and process environments; never print or record them, and keep shell tracing off. + +`[Original Request]` R-9 (2026-09-23, after P4): + +> review pr36，将review comments提交到该remote repo的pr36上，暂不执行修复。 + +Outcome: round-2 review recorded in `review.md` and posted to PR 36 as one +`COMMENT` review with 15 inline comments. No source or test change. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/review.md b/task_memory/task_2026-09-22_stage_admission_ordering/review.md index aa67897f..8228678a 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/review.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/review.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | Round 2: code review of the implementation at `ecff89a` recorded and posted to PR 36; findings verified, fixes deferred by the owner. | | 2026-09-23 | Created. First external plan review of PR 36 at `a6ec6a6` recorded; each finding re-checked against `1f694f7` source, with a disposition and the place it was applied. | ## Round 1: plan review of PR 36 at `a6ec6a6` @@ -46,3 +47,37 @@ ### Status after round 1 Docs corrected. P0 has not started, per the owner's "暂不执行". The next step is the owner's decision to start P0. + +## Round 2: code review of PR 36 at `ecff89a` + +| Item | Value | +| --- | --- | +| Component / phase | Implementation after P4: the rule, P2 tests, case matrix, vLLM comparison tools, and task evidence scripts; diff `1f694f7..ecff89a` | +| Reviewer | `/code-review` skill, run as a forked review agent | +| Inspected by the reviewer | The full PR diff; three touched unit files run on the branch (180 passed); a short script reproduced R2-01 on both trees | +| Re-check | Each finding re-read against the cited source on `ecff89a`, 2026-09-23. R2-01 was reproduced again here. R2-02 numbers were read from `analysis/co_execution_decomposition_sa-pp-20260923b.json`. R2-09 ordering was checked in Python. | +| Owner instruction | "review pr36，将review comments提交到该remote repo的pr36上，暂不执行修复。" (requirements R-9) | +| Posted | https://github.com/NetX-lab/Frontier/pull/36#pullrequestreview-5286523149 (event `COMMENT`, 15 inline comments on `ecff89a`) | + +### Findings (disposition pending; no fix applied) + +| Id | Anchor | Verdict | Finding | +| --- | --- | --- | --- | +| R2-01 | `stage_execution_context.py:351` | confirmed, reproduced | `try_acquire` on an already-active full-stage ticket runs off the scan and `remove` raises `ValueError` ("not in deque"); base returned `False`. The only production caller checks `owns()` first. | +| R2-02 | `compare_lanes.py:43` | confirmed | The D-9 rationale ("duration variance, not admission") is not fully supported. Start offsets exceed end offsets in 3 of 6 dense rounds, and `M5_equal_durations` is 0.667–0.911. The union-minus-overlap identity in `decompose_co_execution.py` overcounts disjoint pairs. | +| R2-03 | `stage_admission_matrix.py:255` | plausible | Before/after cases are offline co-location only. There are no PDD or online cells at `attn_dp>1, PP>1`. | +| R2-04 | `compare_lanes.py:99` | confirmed | `vllm_placement` `ok` ignores `unseen`, so missing placement logs still pass. | +| R2-05 | `compare_lanes.py:182` | confirmed | V1 and dense V4 fold the base negative control into the vLLM MATCH status, so a rerun against a fixed base reports MISMATCH. | +| R2-06 | `.gitignore:173` | confirmed | The task-directory exception publishes records into `main` on merge, reversing `26b490a`. D-5 has no pre-merge removal step. | +| R2-07 | `stage_admission_matrix.py:581` | confirmed | No `subprocess.run` timeout in `_run_one` / `_run_recipe_case`. | +| R2-08 | `stage_admission_matrix.py:445` | confirmed | `work/` is shared across sets, so concurrent sets delete each other's outputs. | +| R2-09 | `vllm_burst_driver.py:71` | confirmed | `differing` (sorted as `Path`) is compared with `expected` (sorted as `str`), so some identical sets are rejected. | +| R2-10 | `vllm_burst_driver.py:95` | confirmed | `apply_patch` skips unknown-tag lines without counting them, and `hunks[target] = []` drops a repeated file's earlier hunks. | +| R2-11 | `stage_execution_context.py:342` | confirmed | The scope branch is repeated and the FIFO is scanned twice (simplification). | +| R2-12 | `test_stage_execution_context.py:136` | plausible | Capacity-1 contexts lose the context-level full-stage insertion order. This should be stated as a contract change, not as "unaffected". | +| R2-13 | `stage_execution_context.py:345` | design note | On shared-lane contexts the FIFO no longer orders admission. `queued_tickets` / `admission_seq` still read as an ordered queue there. | +| R2-14 | `evidence/step9_probe/probe_main.py:15` | confirmed | A hard-coded worktree `ROOT` is put first on `sys.path`, so a #35 rerun would import this tree. | +| R2-15 | `calibration/.../analysis/synthetic_check.py:7` | confirmed | A hard-coded scratch `BASE` bypasses `matrix_root()`, and reusable probes live under `task_memory/` rather than `tests/`. | + +The owner deferred fixes ("暂不执行修复"). Dispositions will be recorded +here when the owner decides which findings to adopt. From 1661bf1bd529c712a9f7a7cbcfc31134a27dce16 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 11:42:36 +0800 Subject: [PATCH 15/19] fix(stage-admission): refuse an active ticket in try_acquire Round-2 review R2-01: after dac4e69, try_acquire on a full-stage ticket that is already active ran off the FIFO scan and remove() raised ValueError; the base rule returned False. Each scope now has one branch. An EP wave leaves the FIFO by popleft. A full-stage ticket is found in one pass and deleted by position, and a ticket that is not queued is refused (R2-11). The class docstring now states that full-stage order comes from the lane stage schedulers and that admission_seq records enqueue order only (R2-13). A unit test re-acquires an active ticket and checks the context is unchanged. --- .../stage_execution_context.py | 46 +++++++++++-------- tests/unit/test_stage_execution_context.py | 17 +++++++ 2 files changed, 43 insertions(+), 20 deletions(-) diff --git a/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py b/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py index 1523586f..1dcda539 100644 --- a/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py +++ b/frontier/scheduler/replica_stage_scheduler/stage_execution_context.py @@ -38,9 +38,11 @@ class StageExecutionContext: admits it atomically. An EP wave is admitted only from the FIFO head, so it waits for every operation queued before it. A full-stage operation may be admitted ahead of earlier queued full-stage operations, but never ahead - of an EP wave queued before it. EP child schedulers may start only after - their wave's ticket has been acquired, and the ticket remains active - through the wave-level combine/cleanup boundary. + of an EP wave queued before it. Full-stage operations are therefore + admitted in the order their lane stage schedulers present them, and + ``admission_seq`` records enqueue order only. EP child schedulers may + start only after their wave's ticket has been acquired, and the ticket + remains active through the wave-level combine/cleanup boundary. """ def __init__( @@ -325,33 +327,37 @@ def _validate_ticket(self, ticket: StageAdmissionTicket) -> None: def try_acquire(self, ticket: StageAdmissionTicket) -> bool: """Acquire ``ticket`` if the stage can admit it now. - An EP wave must be the FIFO head. A full-stage ticket must have no EP - wave queued ahead of it. + An EP wave must be the FIFO head of an idle stage. A full-stage ticket + must have no EP wave queued ahead of it. A ticket that is already + active is not queued and is refused. """ self._validate_ticket(ticket) if ticket.scope == EP_WAVE: - if self._active_ep_ticket is not None or self._active_full_stage_tickets: - return False - elif self._active_ep_ticket is not None: - return False - elif len(self._active_full_stage_tickets) >= self._full_stage_capacity: - return False - elif self._forward_group_sealed: - return False - if ticket.scope == EP_WAVE: - if not self._ready_fifo or self._ready_fifo[0] != ticket: + if ( + self._active_ep_ticket is not None + or self._active_full_stage_tickets + or not self._ready_fifo + or self._ready_fifo[0] != ticket + ): return False + self._ready_fifo.popleft() + self._active_ep_ticket = ticket else: - for queued in self._ready_fifo: + if ( + self._active_ep_ticket is not None + or len(self._active_full_stage_tickets) >= self._full_stage_capacity + or self._forward_group_sealed + ): + return False + for position, queued in enumerate(self._ready_fifo): if queued == ticket: break if queued.scope == EP_WAVE: return False - self._ready_fifo.remove(ticket) - if ticket.scope == EP_WAVE: - self._active_ep_ticket = ticket - else: + else: + return False + del self._ready_fifo[position] self._active_full_stage_tickets.add(ticket) self._refresh_active_ticket_view() return True diff --git a/tests/unit/test_stage_execution_context.py b/tests/unit/test_stage_execution_context.py index 6777cdb5..0e471f5c 100644 --- a/tests/unit/test_stage_execution_context.py +++ b/tests/unit/test_stage_execution_context.py @@ -142,6 +142,23 @@ def test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket() -> No assert context.queued_tickets == (full0,) +def test_active_full_stage_ticket_is_refused_without_changing_the_stage() -> None: + context = StageExecutionContext( + replica_id=0, + stage_id=0, + ep_size=1, + full_stage_capacity=2, + ) + first = context.enqueue_full_stage(operation_id=("lane", 0)) + second = context.enqueue_full_stage(operation_id=("lane", 1)) + assert context.try_acquire(first) is True + + assert context.try_acquire(first) is False + assert context.is_active(first) + assert context.queued_tickets == (second,) + assert context.try_acquire(second) is True + + def test_release_requires_the_active_operation_ticket() -> None: context = StageExecutionContext(replica_id=0, stage_id=0, ep_size=1) wave = context.enqueue_ep_wave(operation_id=30, participant_ep_ids=(0,)) From a8e8d8aad254636261c94d3b62763b8c235c160b Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 11:45:17 +0800 Subject: [PATCH 16/19] tests: add PDD, online and PD-AF cells to the stage-admission matrix Round-2 review R2-03, R2-07 and R2-08. - Cases carry sys_arch, simulation_mode, a Poisson rate and recipe environment overrides. New groups: G8 PDD offline, G9 PDD online, G10 co-location online, G11 PD-AF recipes with PREFILL_PP=2. Dense PDD requires attn_dp=1 and PD-AF DECODE_ATTN requires attn_dp=1, so those cells are unchanged-path controls. On main, MONOLITHIC and PREFILL place incremental online arrivals on lane 0, so G9 and G10 add burst cells that reach several lanes. - The drain state report and the deadlock signature read every cluster type, keyed by cluster. - Each child runs in its own session under --case-timeout (default 600 s); a timed-out session is killed and recorded as other_failure. - run holds an exclusive lock on the matrix root, because every set shares work/ for byte-identical outputs. --- tests/e2e/stage_admission_matrix.py | 159 ++++++++++++++++++++++------ 1 file changed, 124 insertions(+), 35 deletions(-) diff --git a/tests/e2e/stage_admission_matrix.py b/tests/e2e/stage_admission_matrix.py index 6f952a45..628bd2bb 100644 --- a/tests/e2e/stage_admission_matrix.py +++ b/tests/e2e/stage_admission_matrix.py @@ -11,10 +11,13 @@ objects after the sequential run ends with work left; * ``configuration_rejection`` / ``other_failure``: ``error.txt``. -Each case runs in its own child process because ``IS_MOE`` is process-global. -Every child writes its simulator output under ``/work/``, a path -shared by all sets, so that files embedding the output path compare byte for -byte between a set run before a change and one run after it. +Each case runs in its own child process because ``IS_MOE`` is process-global, +and each child runs in its own session so a case that exceeds +``--case-timeout`` is killed with everything it started. Every child writes +its simulator output under ``/work/``, a path shared by all +sets, so that files embedding the output path compare byte for byte between a +set run before a change and one run after it. Sets therefore run one at a +time: ``run`` holds an exclusive lock on the matrix root. Usage:: @@ -28,8 +31,10 @@ import csv import hashlib import json +import fcntl import os import shutil +import signal import subprocess import sys import time @@ -62,6 +67,9 @@ PREFILL_ONLY = (16, 1) PREFILL_DECODE = (16, 3) VLLM_ALIGNED_PREFILL_ONLY = (256, 1) +ONLINE_QPS = 20.0 +ONLINE_QPS_SWEEP = (5.0, 80.0) +DEFAULT_CASE_TIMEOUT_S = 600 @dataclass(frozen=True) @@ -76,8 +84,12 @@ class Case: prefill_tokens: int = 0 decode_tokens: int = 0 arrival: str = "static" + qps: float = 1e6 + sys_arch: str = "co-location" + simulation_mode: str = "offline" cc_backend: str = "analytical" recipe: str | None = None + recipe_env: tuple[tuple[str, str], ...] = () contention_witness: bool = False @property @@ -94,9 +106,9 @@ def _shape_id(group: str, is_moe: bool, attn_dp: int, stages: int, num_requests: def _synthetic(group: str, is_moe: bool, attn_dp: int, stages: int, num_requests: int, - lengths: tuple[int, int], **fields) -> Case: + lengths: tuple[int, int], suffix: str = "", **fields) -> Case: return Case( - case_id=_shape_id(group, is_moe, attn_dp, stages, num_requests), + case_id=_shape_id(group, is_moe, attn_dp, stages, num_requests) + suffix, group=group, is_moe=is_moe, attn_dp=attn_dp, @@ -166,6 +178,47 @@ def build_cases() -> list[Case]: _synthetic("G7", is_moe, 2, 2, num_requests, VLLM_ALIGNED_PREFILL_ONLY, fixture=VLLM_ALIGNED) ) + # Plan §7 (R2-03). PDD admits multi-lane contexts only for MoE: dense PDD + # requires attn_dp == 1. Online cells use Poisson arrivals at ONLINE_QPS. + # On main, MONOLITHIC and PREFILL place every request of one scheduling + # call from lane 0 on, so incremental online arrivals all land on lane 0; + # the "-burst" cells deliver all requests at t=0 to reach several lanes. + pdd_shapes = [(True, attn_dp, stages) for attn_dp in (2, 4) for stages in (1, 2, 3)] + pdd_shapes.append((False, 1, 2)) + online = dict(simulation_mode="online", arrival="poisson", qps=ONLINE_QPS) + burst = dict(simulation_mode="online", suffix="-burst") + for group, timing in (("G8", {}), ("G9", online)): + for is_moe, attn_dp, stages in pdd_shapes: + cases.append( + _synthetic(group, is_moe, attn_dp, stages, 8, PREFILL_DECODE, + sys_arch="pd-disaggregation", **timing) + ) + for attn_dp in (2, 4): + for stages in (2, 3): + cases.append( + _synthetic("G9", True, attn_dp, stages, 8, PREFILL_DECODE, + sys_arch="pd-disaggregation", **burst) + ) + for is_moe, lengths in ((True, PREFILL_ONLY), (False, PREFILL_DECODE)): + for attn_dp in (2, 4): + for stages in (1, 2, 3): + cases.append(_synthetic("G10", is_moe, attn_dp, stages, 8, lengths, **online)) + cases.append(_synthetic("G10", is_moe, attn_dp, stages, 8, lengths, **burst)) + for qps in ONLINE_QPS_SWEEP: + cases.append( + _synthetic("G10", is_moe, 2, 2, 8, lengths, suffix=f"-q{qps:g}", + **dict(online, qps=qps)) + ) + # PD-AF contexts have capacity 1 (DECODE_ATTN requires attn_dp == 1), so + # these are unchanged controls for PREFILL pipeline stages. + for mode in ("offline", "online"): + suffix = "_online" if mode == "online" else "" + for stem in ("dense_model_basic", "moe_model_basic"): + script = f"examples/architecture/pd-af-disagg/{mode}/{stem}{suffix}.sh" + cases.append( + Case(case_id=f"G11-pd-af-disagg-{mode}-{stem}-pp2", group="G11", + recipe=script, recipe_env=(("PREFILL_PP", "2"),)) + ) return cases @@ -236,6 +289,9 @@ def build_config(case: Case, output_dir: Path, cache_dir: Path): cluster_fields["cc_backend_config"] = AnalyticalCCBackendConfig() elif case.cc_backend != "default": raise ValueError(f"unknown CC backend selector {case.cc_backend!r}") + if case.sys_arch == "pd-disaggregation": + # One Replica per role; both roles take the fixture's replica config. + cluster_fields.update(prefill_cluster_num_replicas=1, decode_cluster_num_replicas=1) cluster = ClusterConfig( replica_config=replica, replica_scheduler_config=scheduler, @@ -248,11 +304,11 @@ def build_config(case: Case, output_dir: Path, cache_dir: Path): if case.arrival == "static": interval = StaticRequestIntervalGeneratorConfig() elif case.arrival == "poisson": - interval = PoissonRequestIntervalGeneratorConfig(qps=1e6) + interval = PoissonRequestIntervalGeneratorConfig(qps=case.qps) else: raise ValueError(f"unknown arrival process {case.arrival!r}") return SimulationConfig( - simulation_mode="offline", sys_arch="co-location", + simulation_mode=case.simulation_mode, sys_arch=case.sys_arch, enable_parallel_clusters=False, decode_cuda_graph_mode="none", cluster_config=cluster, metrics_config=MetricsConfig( @@ -284,11 +340,22 @@ def _ticket_view(ticket) -> dict: def build_state_report(simulator) -> dict: - """Read stage contexts, lane queues and sync rooms after a drain.""" - from frontier.types import ClusterType - - cluster_scheduler = simulator.scheduler.get_cluster_scheduler(ClusterType.MONOLITHIC) + """Read stage contexts, lane queues and sync rooms of every cluster after a drain.""" lanes = {} + contexts = [] + rooms = [] + for cluster_type, cluster_scheduler in simulator.scheduler._cluster_schedulers.items(): + _read_cluster_state(cluster_type.name, cluster_scheduler, lanes, contexts, rooms) + return { + "simulation_time": simulator._time, + "contexts": contexts, + "lanes": lanes, + "sync_rooms": rooms, + } + + +def _read_cluster_state(cluster: str, cluster_scheduler, lanes: dict, + contexts: list, rooms: list) -> None: queued_owner = {} for (replica_id, lane_id), replica_scheduler in sorted( cluster_scheduler._replica_schedulers.items(), key=lambda item: str(item[0]) @@ -303,13 +370,14 @@ def build_state_report(simulator) -> dict: heap.append({"batch_id": batch.id, "global_id": batch.global_id, **_ticket_view(ticket)}) stage_views.append({"busy": stage.is_busy, "heap": heap}) - lanes[f"{replica_id}/{lane_id}"] = { - "replica_id": replica_id, "lane": lane_id, "stages": stage_views, + lanes[f"{cluster}/{replica_id}/{lane_id}"] = { + "cluster": cluster, "replica_id": replica_id, "lane": lane_id, + "stages": stage_views, } - contexts = [] for (replica_id, stage_id), context in sorted(cluster_scheduler._stage_execution_contexts.items()): contexts.append({ + "cluster": cluster, "replica_id": replica_id, "stage_id": stage_id, "capacity": context.full_stage_capacity, @@ -327,7 +395,6 @@ def build_state_report(simulator) -> dict: ], }) - rooms = [] for room_name in ("_prefill_sync_waiting_room", "_decode_sync_waiting_room"): by_replica = getattr(cluster_scheduler, room_name) or {} for replica_id, by_stage in by_replica.items(): @@ -338,17 +405,11 @@ def build_state_report(simulator) -> dict: if not room["batches"]: continue rooms.append({ - "room": room_name.strip("_"), + "cluster": cluster, "room": room_name.strip("_"), "replica_id": replica_id, "stage_id": stage_id, "step": step, "layer": layer, "sync_stage": str(sync_stage), "lanes_present": sorted(room["batches"]), }) - return { - "simulation_time": simulator._time, - "contexts": contexts, - "lanes": lanes, - "sync_rooms": rooms, - } def has_admission_deadlock_signature(report: dict) -> bool: @@ -362,12 +423,13 @@ def has_admission_deadlock_signature(report: dict) -> bool: head = context["fifo"][0] if head["scope"] != "FULL_STAGE_WORLD" or head["lane"] is None: continue - replica_id, stage_id = context["replica_id"], context["stage_id"] - head_stage = lanes[f"{replica_id}/{head['lane']}"]["stages"][stage_id] + cluster, replica_id, stage_id = context["cluster"], context["replica_id"], context["stage_id"] + head_stage = lanes[f"{cluster}/{replica_id}/{head['lane']}"]["stages"][stage_id] if not head_stage["busy"]: continue for lane in lanes.values(): - if lane["replica_id"] != replica_id or lane["lane"] == head["lane"]: + if ((lane["cluster"], lane["replica_id"]) != (cluster, replica_id) + or lane["lane"] == head["lane"]): continue stage = lane["stages"][stage_id] if stage["busy"] or not stage["heap"]: @@ -375,7 +437,8 @@ def has_admission_deadlock_signature(report: dict) -> bool: if stage["heap"][0]["admission_seq"] <= head["admission_seq"]: continue for room in report["sync_rooms"]: - if (room["replica_id"] == replica_id and room["stage_id"] == stage_id + if ((room["cluster"], room["replica_id"], room["stage_id"]) + == (cluster, replica_id, stage_id) and head["lane"] in room["lanes_present"] and lane["lane"] not in room["lanes_present"]): return True @@ -427,6 +490,7 @@ def _run_recipe_case(case: Case, work_dir: Path, case_dir: Path) -> dict: "PYTHON_BIN": sys.executable, "METRICS_OUTPUT_DIR": str(work_dir / "metrics"), "RUN_ID": case.case_id, + **dict(case.recipe_env), }) result = subprocess.run( ["bash", str(REPO_ROOT / case.recipe)], cwd=REPO_ROOT, env=env, @@ -573,18 +637,27 @@ def matrix_root() -> Path: return resolve_scratch_root() / MATRIX_DIR_NAME -def _run_one(case: Case, root: Path, set_name: str, provenance: dict) -> dict: +def _run_one(case: Case, root: Path, set_name: str, provenance: dict, + case_timeout: float) -> dict: command = [sys.executable, "-m", "tests.e2e.stage_admission_matrix", "child", "--set", set_name, "--case", case.case_id] env = dict(os.environ, PYTHONPATH=str(REPO_ROOT), WANDB_DISABLED="true", VIDUR_DISABLE_WANDB="1") - result = subprocess.run(command, cwd=REPO_ROOT, env=env, capture_output=True, text=True) + child = subprocess.Popen(command, cwd=REPO_ROOT, env=env, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, text=True, start_new_session=True) + try: + stdout, stderr = child.communicate(timeout=case_timeout) + failure = f"child exit code {child.returncode}" + except subprocess.TimeoutExpired: + os.killpg(child.pid, signal.SIGKILL) + stdout, stderr = child.communicate() + failure = f"case timeout after {case_timeout:g} s" case_dir = root / set_name / case.case_id case_dir.mkdir(parents=True, exist_ok=True) outcome_path = case_dir / "outcome.json" - if result.returncode != 0 or not outcome_path.exists(): - (case_dir / "error.txt").write_text(result.stdout[-50_000:] + result.stderr[-50_000:]) - outcome = {"outcome": OTHER_FAILURE, "exception": f"child exit code {result.returncode}"} + if child.returncode != 0 or not outcome_path.exists(): + (case_dir / "error.txt").write_text(stdout[-50_000:] + stderr[-50_000:]) + outcome = {"outcome": OTHER_FAILURE, "exception": failure} else: outcome = json.loads(outcome_path.read_text()) (case_dir / "case.json").write_text(json.dumps(asdict(case), indent=1, sort_keys=True)) @@ -594,12 +667,26 @@ def _run_one(case: Case, root: Path, set_name: str, provenance: dict) -> dict: "exception": outcome.get("exception")} -def run_set(set_name: str, cases: Sequence[Case], jobs: int) -> list[dict]: +def run_set(set_name: str, cases: Sequence[Case], jobs: int, case_timeout: float) -> list[dict]: root = matrix_root() (root / set_name).mkdir(parents=True, exist_ok=True) + with (root / "run.lock").open("w") as lock: + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + raise RuntimeError( + f"another set is running under {root}; sets share work/ and run one at a time" + ) from None + return _run_locked_set(root, set_name, cases, jobs, case_timeout) + + +def _run_locked_set(root: Path, set_name: str, cases: Sequence[Case], jobs: int, + case_timeout: float) -> list[dict]: provenance = set_provenance() with ThreadPoolExecutor(max_workers=jobs) as pool: - rows = list(pool.map(lambda case: _run_one(case, root, set_name, provenance), cases)) + rows = list(pool.map( + lambda case: _run_one(case, root, set_name, provenance, case_timeout), cases + )) if set_provenance()["git_head"] != provenance["git_head"]: raise RuntimeError("git HEAD changed while the set was running") index = root / set_name / "cases.jsonl" @@ -714,6 +801,8 @@ def main(argv: Sequence[str] | None = None) -> int: run_parser.add_argument("--group", action="append", default=[]) run_parser.add_argument("--case", action="append", default=[]) run_parser.add_argument("--jobs", type=int, default=8) + run_parser.add_argument("--case-timeout", type=float, default=DEFAULT_CASE_TIMEOUT_S, + help="seconds before a case's child session is killed") child_parser = commands.add_parser("child", help=argparse.SUPPRESS) child_parser.add_argument("--set", required=True) child_parser.add_argument("--case", required=True) @@ -736,7 +825,7 @@ def main(argv: Sequence[str] | None = None) -> int: selected = [case for case in cases_by_id.values() if (not args.group or case.group in args.group) and (not args.case or case.case_id in args.case)] - for row in run_set(args.set, selected, args.jobs): + for row in run_set(args.set, selected, args.jobs, args.case_timeout): print(f"{row['case_id']:<48} {row['outcome']}") return 0 From e35242f213ce8ea75cbfc5bafe75cd3e7541070d Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 11:52:41 +0800 Subject: [PATCH 17/19] tests: separate negative controls and harden the vLLM comparison tools Round-2 review R2-02, R2-04, R2-05, R2-09, R2-10, R2-14 and R2-15. - compare_lanes: V1 and V4 compare vLLM with the after revision only. The base negative controls are their own rows, N1 (MoE base admission_deadlock) and N4 (dense base co-start >= 0.5), reported as HOLDS or LOST, with negative_control_holds in the status. Placement is ok only when no request is misplaced or unseen. The D-9 comment names both sources of dense non-overlap. - vllm_burst_driver: overlay acceptance compares file sets. apply_patch reads an empty hunk line as a trimmed context line, rejects any other unknown hunk line, and keeps every section of a file that appears more than once. - tests/unit/test_stage_admission_pp_tools.py replaces the task-local synthetic_check.py and needs no scratch data. On the ecff89a tools 7 of its 9 tests fail. - decompose_co_execution.py states its identity for overlapping pairs only, counts disjoint and unpaired forwards, checks the pairing against M3, and adds M5 with both starts of a pair set to the later one (derived from where vLLM's DP all-reduce sits). Reran on runs a and b: no disjoint pair; existing fields unchanged. - explain_t_path.py takes the after set as an argument. - probe_main.py no longer puts a hard-coded worktree on sys.path. --- ...ecution_decomposition_sa-pp-20260923a.json | 30 +++ ...ecution_decomposition_sa-pp-20260923b.json | 60 +++++ .../analysis/synthetic_check.py | 55 ---- .../evidence/decompose_co_execution.py | 60 +++-- .../evidence/explain_t_path.py | 9 +- .../evidence/step9_probe/probe_main.py | 5 +- .../stage_admission_pp/compare_lanes.py | 34 ++- .../stage_admission_pp/vllm_burst_driver.py | 9 +- tests/unit/test_stage_admission_pp_tools.py | 237 ++++++++++++++++++ 9 files changed, 412 insertions(+), 87 deletions(-) delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/synthetic_check.py create mode 100644 tests/unit/test_stage_admission_pp_tools.py diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json index fda094a3..11dd7567 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json @@ -1,55 +1,85 @@ { "dense/n8/r0": { "pairs": 4, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.6421, "M5_equal_durations": 0.7101, + "M5_barrier_aligned": 0.7389, "non_overlap_ms_from_start_offsets": 2.2, "non_overlap_ms_from_end_offsets": 3.808, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 3.435, "stage0_duration_ms_cv": 0.116 }, "dense/n8/r1": { "pairs": 4, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.7393, "M5_equal_durations": 0.7521, + "M5_barrier_aligned": 0.8432, "non_overlap_ms_from_start_offsets": 1.805, "non_overlap_ms_from_end_offsets": 2.014, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 3.222, "stage0_duration_ms_cv": 0.183 }, "dense/n8/r2": { "pairs": 4, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.7604, "M5_equal_durations": 0.9323, + "M5_barrier_aligned": 0.7818, "non_overlap_ms_from_start_offsets": 0.395, "non_overlap_ms_from_end_offsets": 3.071, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 2.651, "stage0_duration_ms_cv": 0.261 }, "dense/n16/r0": { "pairs": 8, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.5372, "M5_equal_durations": 0.5704, + "M5_barrier_aligned": 0.6561, "non_overlap_ms_from_start_offsets": 12.44, "non_overlap_ms_from_end_offsets": 11.9, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 4.561, "stage0_duration_ms_cv": 0.16 }, "dense/n16/r1": { "pairs": 8, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.752, "M5_equal_durations": 0.6557, + "M5_barrier_aligned": 0.9449, "non_overlap_ms_from_start_offsets": 5.729, "non_overlap_ms_from_end_offsets": 1.232, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 3.032, "stage0_duration_ms_cv": 0.162 }, "dense/n16/r2": { "pairs": 8, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.7668, "M5_equal_durations": 0.6756, + "M5_barrier_aligned": 0.9501, "non_overlap_ms_from_start_offsets": 5.235, "non_overlap_ms_from_end_offsets": 1.092, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 2.738, "stage0_duration_ms_cv": 0.178 } diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json index 04089bd8..5c797f40 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json @@ -1,109 +1,169 @@ { "moe/n8/r0": { "pairs": 4, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.9769, "M5_equal_durations": 0.9713, + "M5_barrier_aligned": 0.9912, "non_overlap_ms_from_start_offsets": 0.321, "non_overlap_ms_from_end_offsets": 0.194, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 5.312, "stage0_duration_ms_cv": 0.066 }, "moe/n8/r1": { "pairs": 4, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.9735, "M5_equal_durations": 0.9648, + "M5_barrier_aligned": 0.9911, "non_overlap_ms_from_start_offsets": 0.402, "non_overlap_ms_from_end_offsets": 0.197, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 5.335, "stage0_duration_ms_cv": 0.091 }, "moe/n8/r2": { "pairs": 4, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.9774, "M5_equal_durations": 0.9676, + "M5_barrier_aligned": 0.9937, "non_overlap_ms_from_start_offsets": 0.536, "non_overlap_ms_from_end_offsets": 0.204, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 8.45, "stage0_duration_ms_cv": 0.088 }, "moe/n16/r0": { "pairs": 8, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.9782, "M5_equal_durations": 0.9686, + "M5_barrier_aligned": 0.994, "non_overlap_ms_from_start_offsets": 0.698, "non_overlap_ms_from_end_offsets": 0.26, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 5.294, "stage0_duration_ms_cv": 0.067 }, "moe/n16/r1": { "pairs": 8, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.9365, "M5_equal_durations": 0.9029, + "M5_barrier_aligned": 0.9884, "non_overlap_ms_from_start_offsets": 3.274, "non_overlap_ms_from_end_offsets": 0.688, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 7.424, "stage0_duration_ms_cv": 0.05 }, "moe/n16/r2": { "pairs": 8, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.9283, "M5_equal_durations": 0.8972, + "M5_barrier_aligned": 0.9885, "non_overlap_ms_from_start_offsets": 3.827, "non_overlap_ms_from_end_offsets": 0.682, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 7.483, "stage0_duration_ms_cv": 0.055 }, "dense/n8/r0": { "pairs": 4, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.6569, "M5_equal_durations": 0.832, + "M5_barrier_aligned": 0.7216, "non_overlap_ms_from_start_offsets": 1.487, "non_overlap_ms_from_end_offsets": 4.198, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 3.052, "stage0_duration_ms_cv": 0.291 }, "dense/n8/r1": { "pairs": 4, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.8508, "M5_equal_durations": 0.7401, + "M5_barrier_aligned": 0.9769, "non_overlap_ms_from_start_offsets": 1.669, "non_overlap_ms_from_end_offsets": 0.26, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 2.967, "stage0_duration_ms_cv": 0.128 }, "dense/n8/r2": { "pairs": 4, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.6089, "M5_equal_durations": 0.6668, + "M5_barrier_aligned": 0.7132, "non_overlap_ms_from_start_offsets": 2.627, "non_overlap_ms_from_end_offsets": 4.399, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 3.752, "stage0_duration_ms_cv": 0.19 }, "dense/n16/r0": { "pairs": 8, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.9259, "M5_equal_durations": 0.9111, + "M5_barrier_aligned": 0.9697, "non_overlap_ms_from_start_offsets": 1.017, "non_overlap_ms_from_end_offsets": 0.653, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 2.616, "stage0_duration_ms_cv": 0.103 }, "dense/n16/r1": { "pairs": 8, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.8332, "M5_equal_durations": 0.7156, + "M5_barrier_aligned": 0.9722, "non_overlap_ms_from_start_offsets": 3.596, "non_overlap_ms_from_end_offsets": 0.599, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 2.615, "stage0_duration_ms_cv": 0.182 }, "dense/n16/r2": { "pairs": 8, + "unpaired_forwards": 0, + "pairing_equals_M3": true, + "disjoint_pairs": 0, "M5_observed": 0.8371, "M5_equal_durations": 0.8982, + "M5_barrier_aligned": 0.8832, "non_overlap_ms_from_start_offsets": 1.324, "non_overlap_ms_from_end_offsets": 2.807, + "non_overlap_ms_of_disjoint_pairs": 0.0, "stage0_duration_ms_median": 2.676, "stage0_duration_ms_cv": 0.193 } diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/synthetic_check.py b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/synthetic_check.py deleted file mode 100644 index d26e683c..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/synthetic_check.py +++ /dev/null @@ -1,55 +0,0 @@ -"""Synthetic inputs for compare_lanes: ideal vLLM pairing, a matching Frontier -'after' set, the real P0 base G7 cases, and one vLLM round shifted by a dummy.""" -import json, shutil, sys -from pathlib import Path -from tests.comparison.stage_admission_pp import compare_lanes - -S = Path(sys.argv[1]); BASE = Path('/data/ycfeng/tmp/stage_admission_ordering/base') -D = 0.12 # forward duration -def ideal(n, shift_round=None, rnd=0): - """Stage 0: lane pair k runs [k*D, (k+1)*D); stage 1 one slot later.""" - fw = [] - for i in range(n): - lane, k = i % 2, i // 2 - s0 = k * D + (D if (shift_round == rnd and lane == 1) else 0.0) - fw.append((lane, 0, s0, s0 + D, i)); fw.append((lane, 1, s0 + D, s0 + 2 * D, i)) - return fw -def write_vllm(model, shift): - d = S / 'vllm' / 'runs' / model; (d / 'dp_placement').mkdir(parents=True) - reqs, pp, rounds, place, t0 = [], [], [], {0: [], 1: []}, 100.0 - for label, rnd, n in [('warmup', 0, 4)] + [(f'b{b}-r{r}', r, b) for b in (8, 16) for r in range(3)]: - off = 1.7e9 - for lane, stage, s, e, i in (ideal(n, shift, rnd) if label != 'warmup' else ideal(n)): - rid = f'{label}-q{i}' - rec = {'request_ids': [rid], 'pp_rank': stage, 'is_last_rank': stage == 1, - 'forward_start_ts': t0 + s, 'send_start_ts': None if stage else t0 + e, - 'timestamp': off + t0 + e} - pp.append(rec) - if stage == 0: place[lane].append({'kind': 'engine_iteration', 'engine': lane, 'scheduled_new_req_ids': [rid]}) - for i in range(n): - reqs.append({'request_id': f'{label}-q{i}', 'burst': label, 'round': rnd, 'index': i, 'rank': i % 2, - 'num_output_tokens': 1}) - rounds.append({'label': label, 'wall_minus_monotonic_before': off, 'wall_minus_monotonic_after': off}) - t0 += 10.0 - (d / 'requests.jsonl').write_text(''.join(json.dumps(r) + '\n' for r in reqs)) - (d / 'pp_boundary.jsonl').write_text(''.join(json.dumps(r) + '\n' for r in pp)) - (d / 'summary.json').write_text(json.dumps({'rounds': rounds})) - for e, rows in place.items(): - (d / 'dp_placement' / f'dp_placement_{e}.jsonl').write_text(''.join(json.dumps(r) + '\n' for r in rows)) -def write_after(model, n): - cid = f'G7-{model}-dp2-pp2-n{n}'; d = S / 'frontier' / 'after' / cid; (d / 'metrics' / 'x').mkdir(parents=True) - (d / 'run.json').write_text(json.dumps({'outcome': 'success'})) - shutil.copy(BASE / cid / 'case.json', d / 'case.json') - rows = [{'execution_scope': 'ATTN_DP_LANE', 'replica_local_id': l, 'stage_id': st, 'stage_start_ts': s, - 'stage_end_ts': e, 'request_ids': [str(i)]} for l, st, s, e, i in ideal(n)] - (d / 'metrics' / 'x' / 'frontier_stage_batch_ledger.jsonl').write_text(''.join(json.dumps(r) + '\n' for r in rows)) - (S / 'frontier' / 'base').mkdir(parents=True, exist_ok=True) - (S / 'frontier' / 'base' / cid).symlink_to(BASE / cid) -write_vllm('moe', shift=None); write_vllm('dense', shift=1) -for m in ('moe', 'dense'): - for n in (8, 16): write_after(m, n) -rows, details = compare_lanes.compare(S / 'vllm', S / 'frontier', 'base', 'after') -bad = [(r['check'], r['model'], r['burst'], r['round']) for r in rows if r['status'] != 'MATCH'] -print('rows', len(rows)); print('mismatch', bad) -print('dense base n8 stage0', {k: v for k, v in details['runs']['G7-dense-dp2-pp2-n8']['frontier_base']['stage0'].items() if k != 'M3_pairing'}) -print('placement', {m: (p['ok'], len(p['unseen'])) for m, p in details['placement'].items()}) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py index 7e9aae2a..f7102aaf 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py @@ -1,11 +1,24 @@ """Split vLLM stage-0 non-overlap into start and end offsets of paired forwards. -For each M3 pair (lane-0 forward, lane-1 forward) of each round, the union -minus the overlap equals |start difference| + |end difference|. The start -part is where admission could act; the end part comes from per-rank forward -duration variation. Also reports M5 with every lane-1 end aligned to the -lane-0 duration, i.e. the co-execution vLLM would show with the equal -per-forward durations of Frontier's dummy predictor. +Pairs are the i-th lane-0 and i-th lane-1 stage-0 forwards of a round; the +output states whether this equals the M3 pairing of ``compare_lanes``. For an +overlapping pair, union minus overlap equals |start difference| + |end +difference|, so the non-overlap splits into a start part and an end part. A +disjoint pair has no such split; it is counted and its non-overlap is +reported whole. + +Two counterfactual co-execution fractions are reported: + +* ``M5_equal_durations``: every lane-1 forward keeps its start and takes the + duration of its lane-0 partner, as with the dummy predictor's equal + durations. +* ``M5_barrier_aligned``: both forwards of a pair start at the later of the + two starts and keep their own ends. vLLM 0.10.2 without CUDA graphs runs + the per-forward DP metadata all-reduce inside ``set_forward_context``, after + ``forward_start_ts``, so neither rank computes before the later one arrives. + The traces carry no timestamp after that exchange, so this value is derived + from where the all-reduce sits, not measured. It is ``None`` when a round + has a disjoint pair. Usage: python decompose_co_execution.py [ ...] """ @@ -14,7 +27,7 @@ import sys from pathlib import Path -from tests.comparison.stage_admission_pp.compare_lanes import vllm_forwards +from tests.comparison.stage_admission_pp.compare_lanes import lane_metrics, vllm_forwards from tests.e2e.stage_admission_matrix import interval_overlap run_dir = Path(sys.argv[1]) @@ -24,22 +37,39 @@ for (burst, round_index), run in sorted(runs.items()): stage0 = [f for f in run["forwards"] if f["stage"] == 0] lanes = {lane: sorted((f for f in stage0 if f["lane"] == lane), key=lambda f: f["start"]) for lane in (0, 1)} - start_part = end_part = 0.0 - equal_duration = [] - for first, second in zip(lanes[0], lanes[1]): - start_part += abs(first["start"] - second["start"]) - end_part += abs(first["end"] - second["end"]) + pairs = list(zip(lanes[0], lanes[1])) + start_part = end_part = disjoint_part = 0.0 + disjoint = 0 + equal_duration, barrier_aligned = [], [] + for first, second in pairs: + later_start = max(first["start"], second["start"]) + if min(first["end"], second["end"]) > later_start: + start_part += abs(first["start"] - second["start"]) + end_part += abs(first["end"] - second["end"]) + else: + disjoint += 1 + disjoint_part += (first["end"] - first["start"]) + (second["end"] - second["start"]) equal_duration.append((first["start"], first["end"], 0)) equal_duration.append((second["start"], second["start"] + first["end"] - first["start"], 1)) + barrier_aligned.append((later_start, first["end"], 0)) + barrier_aligned.append((later_start, second["end"], 1)) observed = interval_overlap([(f["start"], f["end"], f["lane"]) for f in stage0]) - aligned = interval_overlap(equal_duration) + equal = interval_overlap(equal_duration) + aligned = interval_overlap(barrier_aligned) + m3 = lane_metrics(run["forwards"])["stage0"]["M3_pairing"] durations = [f["end"] - f["start"] for f in stage0] summary[f"{model}/n{burst}/r{round_index}"] = { - "pairs": min(len(lanes[0]), len(lanes[1])), + "pairs": len(pairs), + "unpaired_forwards": abs(len(lanes[0]) - len(lanes[1])), + "pairing_equals_M3": m3 == [[list(a["indices"]), list(b["indices"])] for a, b in pairs], + "disjoint_pairs": disjoint, "M5_observed": round(observed["multi_lane_busy_time"] / observed["busy_time"], 4), - "M5_equal_durations": round(aligned["multi_lane_busy_time"] / aligned["busy_time"], 4), + "M5_equal_durations": round(equal["multi_lane_busy_time"] / equal["busy_time"], 4), + "M5_barrier_aligned": (round(aligned["multi_lane_busy_time"] / aligned["busy_time"], 4) + if not disjoint else None), "non_overlap_ms_from_start_offsets": round(1e3 * start_part, 3), "non_overlap_ms_from_end_offsets": round(1e3 * end_part, 3), + "non_overlap_ms_of_disjoint_pairs": round(1e3 * disjoint_part, 3), "stage0_duration_ms_median": round(1e3 * statistics.median(durations), 3), "stage0_duration_ms_cv": round(statistics.pstdev(durations) / statistics.mean(durations), 3), } diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py index 2a4865db..be36bc2c 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py @@ -5,14 +5,17 @@ the difference is start times only; report the §4.5 metric as absolute and as a fraction of stage busy time. -Usage: python explain_t_path.py +Usage: python explain_t_path.py + +The before set is always ``base``. """ import json import sys from collections import defaultdict from pathlib import Path -root, compare_path, output = Path(sys.argv[1]), Path(sys.argv[2]), Path(sys.argv[3]) +root, after_set = Path(sys.argv[1]), sys.argv[2] +compare_path, output = Path(sys.argv[3]), Path(sys.argv[4]) def lane_rows(set_name, case_id): @@ -40,7 +43,7 @@ def fraction(metric): for row in json.load(open(compare_path)): if row["path"] != "T" or row["verdict"] == "PASS": continue - before, after = lane_rows("base", row["case_id"]), lane_rows("after", row["case_id"]) + before, after = lane_rows("base", row["case_id"]), lane_rows(after_set, row["case_id"]) same_work = before.keys() == after.keys() and all( [signature(r) for r in before[key]] == [signature(r) for r in after[key]] for key in before ) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py index f99c9900..87743805 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py @@ -4,6 +4,8 @@ after `_running_requests` has grown) and the inert `on_replica_batch_end` seam, and reads the candidate report key -- the Replica's next forward id held by `ForwardSyncState` -- at each boundary. + +Run with ``PYTHONPATH`` set to the Frontier tree under test. """ from __future__ import annotations @@ -11,9 +13,6 @@ import sys from pathlib import Path -ROOT = "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering" -sys.path.insert(0, ROOT) - def build_config(root: Path, *, is_moe: bool, attn_dp: int, moe_ep: int, stages: int): from frontier.config import ( diff --git a/tests/comparison/stage_admission_pp/compare_lanes.py b/tests/comparison/stage_admission_pp/compare_lanes.py index 1ff9264a..1d16ecc3 100644 --- a/tests/comparison/stage_admission_pp/compare_lanes.py +++ b/tests/comparison/stage_admission_pp/compare_lanes.py @@ -37,11 +37,15 @@ BURSTS = (8, 16) CO_START_BOUND = 0.5 CO_EXECUTION_BOUND = 0.10 -# Dense DP ranks meet once per forward and then vary in duration per rank, -# which the dummy predictor does not model; their co-execution is reported, -# not gated (plan §4.7 V5, D-9). MoE ranks stay aligned by EP collectives. +# Dense DP ranks meet once per forward, in the DP metadata all-reduce that +# vLLM runs after ``forward_start_ts``. A rank's recorded interval therefore +# includes its wait for the other rank (start offsets) as well as its own +# duration variation (end offsets); the dummy predictor models neither, so +# dense co-execution is reported, not gated (plan §4.7 V5, D-9). MoE ranks +# stay aligned by EP collectives. CO_EXECUTION_GATED = {"moe": True, "dense": False} INFORMATIONAL = "INFORMATIONAL" +HOLDS, LOST = "HOLDS", "LOST" FRONTIER_OWNER = "frontier/scheduler/replica_stage_scheduler/stage_execution_context.py" @@ -98,7 +102,8 @@ def vllm_placement(scenario_dir: Path) -> dict: scheduled_by[request_id].add(record["engine"]) misplaced = sorted(rid for rid, rank in pinned.items() if scheduled_by.get(rid, {rank}) != {rank}) unseen = sorted(rid for rid in pinned if rid not in scheduled_by) - return {"requests": len(pinned), "misplaced": misplaced, "unseen": unseen, "ok": not misplaced} + return {"requests": len(pinned), "misplaced": misplaced, "unseen": unseen, + "ok": not misplaced and not unseen} def frontier_run(set_dir: Path, case_id: str) -> dict: @@ -179,11 +184,22 @@ def compare(vllm_run: Path, frontier_root: Path, before: str, after: str) -> tup "frontier_after_outcome": new["outcome"], "frontier_after_placement_ok": new.get("placement_ok"), "vllm": vllm_metrics} - base_control = (base["outcome"] == ADMISSION_DEADLOCK) if model == "moe" else (base["outcome"] == SUCCESS) + base_m4 = base_metrics["stage0"]["M4_co_start"] if base_metrics else None + # Negative controls: the base rule deadlocks MoE and starts dense + # lanes one forward apart. They describe the base, not vLLM. + if model == "moe": + rows.append(_row("N1", model, burst, "base", "base outcome", None, new["outcome"], + base["outcome"], HOLDS if base["outcome"] == ADMISSION_DEADLOCK else LOST, + note=f"expected {ADMISSION_DEADLOCK}")) + else: + rows.append(_row("N4", model, burst, "base", "M4 stage-0 co-start", None, + new_metrics["stage0"]["M4_co_start"] if new_metrics else None, base_m4, + HOLDS if base_m4 is not None and base_m4 >= CO_START_BOUND else LOST, + note=f"expected >= {CO_START_BOUND}")) for r in rounds: run = vllm_runs[(burst, r)] completed = run["completed"] == run["submitted"] == burst - status = "MATCH" if completed and new["outcome"] == SUCCESS and base_control else "MISMATCH" + status = "MATCH" if completed and new["outcome"] == SUCCESS else "MISMATCH" rows.append(_row("V1", model, burst, r, "M1 completion", f"{run['completed']}/{run['submitted']}", new["outcome"], base["outcome"], status)) gt = vllm_metrics[r] @@ -196,10 +212,8 @@ def compare(vllm_run: Path, frontier_root: Path, before: str, after: str) -> tup base_metrics["stage0"]["M3_pairing"] if base_metrics else None, "MATCH" if after_m3 == gt["stage0"]["M3_pairing"] else "MISMATCH")) after_m4 = new_metrics["stage0"]["M4_co_start"] if new_metrics else None - base_m4 = base_metrics["stage0"]["M4_co_start"] if base_metrics else None m4_ok = (gt["stage0"]["M4_co_start"] < CO_START_BOUND and after_m4 is not None - and after_m4 < CO_START_BOUND - and (model == "moe" or (base_m4 is not None and base_m4 >= CO_START_BOUND))) + and after_m4 < CO_START_BOUND) rows.append(_row("V4", model, burst, r, "M4 stage-0 co-start", gt["stage0"]["M4_co_start"], after_m4, base_m4, "MATCH" if m4_ok else "MISMATCH")) gt_m5 = statistics.mean(vllm_metrics[r]["stage0"]["M5_co_execution"] for r in rounds) @@ -233,6 +247,7 @@ def main(argv=None) -> int: mismatches = [row for row in rows if row["status"] == "MISMATCH"] placement_ok = all(p["ok"] for p in details["placement"].values()) placement_unseen = sum(len(p["unseen"]) for p in details["placement"].values()) + controls = [row for row in rows if row["status"] in (HOLDS, LOST)] status = { "analysis_state": "COMPLETE", "status": "PASS" if not mismatches and placement_ok else "FAIL", @@ -241,6 +256,7 @@ def main(argv=None) -> int: "mismatches": len(mismatches), "vllm_placement_ok": placement_ok, "vllm_placement_unseen_requests": placement_unseen, + "negative_control_holds": all(row["status"] == HOLDS for row in controls), "next_action": ("record C7 in the test report" if not mismatches and placement_ok else "report each MISMATCH row with its cause before P4; adjust nothing"), } diff --git a/tests/comparison/stage_admission_pp/vllm_burst_driver.py b/tests/comparison/stage_admission_pp/vllm_burst_driver.py index 70ab06cf..21a04109 100644 --- a/tests/comparison/stage_admission_pp/vllm_burst_driver.py +++ b/tests/comparison/stage_admission_pp/vllm_burst_driver.py @@ -68,7 +68,7 @@ def build_overlay(site_vllm: Path, checkout: Path, destination: Path, expected_c "expected_py_changes": expected, "unexpected": sorted(set(differing) - set(expected)), "missing": sorted(set(expected) - set(differing)), - "accepted": differing == expected, + "accepted": set(differing) == set(expected), } @@ -77,6 +77,7 @@ def apply_patch(patch: Path, root: Path) -> list[str]: The worker image need not carry ``patch`` or ``git``, so hunks are applied here as text replacements; each hunk must match its file exactly once. + An empty hunk line is a context line whose leading space was trimmed. """ lines = patch.read_text().splitlines(keepends=True) hunks: dict[str, list[tuple[str, str]]] = {} @@ -86,7 +87,7 @@ def apply_patch(patch: Path, root: Path) -> list[str]: index += 1 if line.startswith("+++ "): target = line[4:].strip().removeprefix("b/") - hunks[target] = [] + hunks.setdefault(target, []) elif line.startswith("@@ "): header = re.match(r"@@ -\d+(?:,(\d+))? \+\d+(?:,(\d+))? @@", line) old_count, new_count = (int(count or 1) for count in header.groups()) @@ -94,6 +95,10 @@ def apply_patch(patch: Path, root: Path) -> list[str]: while old_count or new_count: tag, text = lines[index][0], lines[index][1:] index += 1 + if tag == "\n": + tag, text = " ", "\n" + if tag not in " -+": + raise ValueError(f"{patch}: unexpected hunk line {lines[index - 1]!r}") if tag in " -": old.append(text) old_count -= 1 diff --git a/tests/unit/test_stage_admission_pp_tools.py b/tests/unit/test_stage_admission_pp_tools.py new file mode 100644 index 00000000..ea88f196 --- /dev/null +++ b/tests/unit/test_stage_admission_pp_tools.py @@ -0,0 +1,237 @@ +"""Synthetic checks of the stage-admission vLLM comparison tools. + +``compare_lanes`` is driven end to end on hand-built vLLM traces and Frontier +sets: ideal lane pairing on both sides, and the base rule's two negative +controls (a MoE admission deadlock, dense lanes one forward apart). +``vllm_burst_driver`` is checked for overlay acceptance and patch parsing. +""" + +from __future__ import annotations + +import csv +import json +from pathlib import Path + +import pytest + +from tests.comparison.stage_admission_pp import compare_lanes +from tests.comparison.stage_admission_pp.vllm_burst_driver import apply_patch, build_overlay +from tests.e2e.stage_admission_matrix import ADMISSION_DEADLOCK, MATRIX_DIR_NAME, SUCCESS + + +FORWARD = 0.12 +ROUNDS = 3 + + +def forwards(num_requests: int, late_lane: bool = False) -> list[tuple[int, int, float, float, int]]: + """Two-stage forwards: pair ``k`` runs stage 0 in ``[k*F, (k+1)*F)``. + + With ``late_lane`` lane 1 starts one forward after lane 0, as under the + base rule. + """ + rows = [] + for index in range(num_requests): + lane, slot = index % 2, index // 2 + start = (slot + (1 if late_lane and lane == 1 else 0)) * FORWARD + rows.append((lane, 0, start, start + FORWARD, index)) + rows.append((lane, 1, start + FORWARD, start + 2 * FORWARD, index)) + return rows + + +def write_jsonl(path: Path, records) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("".join(json.dumps(record) + "\n" for record in records)) + + +def write_vllm_scenario(run_dir: Path, model: str, late_round: int | None = None) -> None: + scenario = run_dir / "runs" / model + requests, boundaries, rounds = [], [], [] + placement = {0: [], 1: []} + wall_offset, origin = 1.7e9, 100.0 + labels = [("warmup", 0, 4)] + [ + (f"b{burst}-r{r}", r, burst) for burst in compare_lanes.BURSTS for r in range(ROUNDS) + ] + for label, round_index, num_requests in labels: + late = label != "warmup" and round_index == late_round + for lane, stage, start, end, index in forwards(num_requests, late): + request_id = f"{label}-q{index}" + boundaries.append({ + "request_ids": [request_id], "pp_rank": stage, "is_last_rank": stage == 1, + "forward_start_ts": origin + start, + "send_start_ts": None if stage else origin + end, + "timestamp": wall_offset + origin + end, + }) + if stage == 0: + placement[lane].append({"kind": "engine_iteration", "engine": lane, + "scheduled_new_req_ids": [request_id]}) + requests.extend( + {"request_id": f"{label}-q{index}", "burst": label, "round": round_index, + "index": index, "rank": index % 2, "num_output_tokens": 1} + for index in range(num_requests) + ) + rounds.append({"label": label, "wall_minus_monotonic_before": wall_offset, + "wall_minus_monotonic_after": wall_offset}) + origin += 10.0 + write_jsonl(scenario / "requests.jsonl", requests) + write_jsonl(scenario / "pp_boundary.jsonl", boundaries) + (scenario / "summary.json").write_text(json.dumps({"rounds": rounds})) + for engine, records in placement.items(): + write_jsonl(scenario / "dp_placement" / f"dp_placement_{engine}.jsonl", records) + + +def write_frontier_case(set_dir: Path, model: str, burst: int, outcome: str, + late_lane: bool = False) -> None: + case_dir = set_dir / f"G7-{model}-dp2-pp2-n{burst}" + case_dir.mkdir(parents=True) + (case_dir / "run.json").write_text(json.dumps({"outcome": outcome})) + (case_dir / "case.json").write_text(json.dumps({"num_requests": burst})) + if outcome == SUCCESS: + write_jsonl( + case_dir / "metrics" / "run" / "frontier_stage_batch_ledger.jsonl", + ({"execution_scope": "ATTN_DP_LANE", "replica_local_id": lane, "stage_id": stage, + "stage_start_ts": start, "stage_end_ts": end, "request_ids": [str(index)]} + for lane, stage, start, end, index in forwards(burst, late_lane)), + ) + + +@pytest.fixture +def workspace(tmp_path, monkeypatch): + monkeypatch.setenv("FRONTIER_TMP_ROOT", str(tmp_path / "scratch")) + return tmp_path + + +def run_compare(workspace: Path, *, fixed_base: bool = False, late_round: int | None = None, + drop_placement_engine: int | None = None) -> tuple[dict, list[dict]]: + vllm_run = workspace / "vllm" + for model in compare_lanes.MODELS: + write_vllm_scenario(vllm_run, model, late_round if model == "dense" else None) + if drop_placement_engine is not None: + (vllm_run / "runs" / "moe" / "dp_placement" + / f"dp_placement_{drop_placement_engine}.jsonl").unlink() + frontier = workspace / "scratch" / MATRIX_DIR_NAME + for burst in compare_lanes.BURSTS: + for model in compare_lanes.MODELS: + write_frontier_case(frontier / "after", model, burst, SUCCESS) + if fixed_base: + write_frontier_case(frontier / "base", model, burst, SUCCESS) + elif model == "moe": + write_frontier_case(frontier / "base", model, burst, ADMISSION_DEADLOCK) + else: + write_frontier_case(frontier / "base", model, burst, SUCCESS, late_lane=True) + output = workspace / "analysis" + compare_lanes.main(["--vllm-run", str(vllm_run), "--output", str(output)]) + status = json.loads((output / "workflow_gap_status.json").read_text()) + with (output / "workflow_gap_table.csv").open() as handle: + rows = list(csv.DictReader(handle)) + return status, rows + + +def statuses(rows: list[dict], checks: tuple[str, ...]) -> set[str]: + return {row["status"] for row in rows if row["check"] in checks} + + +def test_ideal_after_revision_matches_and_base_controls_hold(workspace) -> None: + status, rows = run_compare(workspace) + + assert status["status"] == "PASS" + assert status["mismatches"] == 0 + assert status["negative_control_holds"] is True + assert statuses(rows, ("V1", "V2", "V3", "V4")) == {"MATCH"} + assert {(row["model"], row["status"]) for row in rows if row["check"] == "V5"} == { + ("moe", "MATCH"), ("dense", compare_lanes.INFORMATIONAL) + } + assert sorted((row["check"], row["model"], row["status"]) for row in rows + if row["check"] in ("N1", "N4")) == [ + ("N1", "moe", "HOLDS"), ("N1", "moe", "HOLDS"), + ("N4", "dense", "HOLDS"), ("N4", "dense", "HOLDS"), + ] + + +def test_fixed_base_loses_the_controls_without_a_mismatch(workspace) -> None: + status, rows = run_compare(workspace, fixed_base=True) + + assert status["status"] == "PASS" + assert status["mismatches"] == 0 + assert status["negative_control_holds"] is False + assert statuses(rows, ("V1", "V2", "V3", "V4")) == {"MATCH"} + assert statuses(rows, ("N1", "N4")) == {"LOST"} + + +def test_vllm_round_with_a_late_lane_is_reported(workspace) -> None: + status, rows = run_compare(workspace, late_round=1) + + mismatched = sorted((row["check"], row["model"], row["burst"], row["round"]) + for row in rows if row["status"] == "MISMATCH") + assert mismatched == [ + ("V3", "dense", "16", "1"), ("V3", "dense", "8", "1"), + ("V4", "dense", "16", "1"), ("V4", "dense", "8", "1"), + ] + assert status["status"] == "FAIL" + + +def test_missing_placement_log_fails_the_comparison(workspace) -> None: + status, _ = run_compare(workspace, drop_placement_engine=1) + + assert status["vllm_placement_ok"] is False + assert status["vllm_placement_unseen_requests"] > 0 + assert status["status"] == "FAIL" + + +def write_tree(root: Path, files: dict[str, str]) -> None: + for relative, text in files.items(): + path = root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text) + + +@pytest.mark.parametrize("extra_change, accepted", [(False, True), (True, False)]) +def test_overlay_acceptance_compares_file_sets(tmp_path, extra_change, accepted) -> None: + # Path order puts vllm/a/c.py first; string order puts vllm/a-b.py first. + changed = ["vllm/a-b.py", "vllm/a/c.py"] + write_tree(tmp_path / "site", {"vllm/a-b.py": "old\n", "vllm/a/c.py": "old\n", "vllm/d.py": "same\n"}) + checkout_files = {"vllm/a-b.py": "new\n", "vllm/a/c.py": "new\n", "vllm/d.py": "same\n"} + if extra_change: + checkout_files["vllm/d.py"] = "changed\n" + write_tree(tmp_path / "checkout", checkout_files) + expected = tmp_path / "expected_changes.txt" + expected.write_text("".join(f"{name}\n" for name in sorted(changed))) + + report = build_overlay(tmp_path / "site" / "vllm", tmp_path / "checkout", + tmp_path / "overlay", expected) + + assert report["accepted"] is accepted + assert report["unexpected"] == ([] if accepted else ["vllm/d.py"]) + + +def test_apply_patch_keeps_every_section_of_a_repeated_file(tmp_path) -> None: + write_tree(tmp_path, {"pkg/mod.py": "a = 1\nb = 2\nc = 3\n"}) + patch = tmp_path / "change.patch" + patch.write_text( + "--- a/pkg/mod.py\n+++ b/pkg/mod.py\n@@ -1,1 +1,1 @@\n-a = 1\n+a = 10\n" + "--- a/pkg/mod.py\n+++ b/pkg/mod.py\n@@ -3,1 +3,1 @@\n-c = 3\n+c = 30\n" + ) + + assert apply_patch(patch, tmp_path) == ["pkg/mod.py"] + assert (tmp_path / "pkg" / "mod.py").read_text() == "a = 10\nb = 2\nc = 30\n" + + +def test_apply_patch_reads_a_trimmed_context_line(tmp_path) -> None: + write_tree(tmp_path, {"pkg/mod.py": "a = 1\n\nb = 2\n"}) + patch = tmp_path / "change.patch" + patch.write_text("--- a/pkg/mod.py\n+++ b/pkg/mod.py\n@@ -1,3 +1,3 @@\n a = 1\n\n-b = 2\n+b = 20\n") + + apply_patch(patch, tmp_path) + + assert (tmp_path / "pkg" / "mod.py").read_text() == "a = 1\n\nb = 20\n" + + +def test_apply_patch_rejects_an_unknown_hunk_line(tmp_path) -> None: + write_tree(tmp_path, {"pkg/mod.py": "a = 1\n"}) + patch = tmp_path / "change.patch" + patch.write_text( + "--- a/pkg/mod.py\n+++ b/pkg/mod.py\n@@ -1,1 +1,1 @@\n-a = 1\n" + "\\ No newline at end of file\n+a = 2\n" + ) + + with pytest.raises(ValueError, match="unexpected hunk line"): + apply_patch(patch, tmp_path) From 7a7c22e0320d8f839edfe6f0f18d2717cc2be389 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 11:59:57 +0800 Subject: [PATCH 18/19] docs(stage-admission): record the round-2 remediation (R-10) Plan section 7, the D-9 (b) rationale restated with both sources of dense non-overlap, test report section 8, dispositions of R2-01..R2-15, the capacity-1 contract and FIFO notes in design.md, the calibration rerun with separate negative-control rows, and the round-2 evidence (path-T explanation, G2 comparisons, tool tests on the old tools). R2-06 is recorded as pre-merge step P6, not executed. --- .../analysis/workflow_gap_status.json | 3 +- .../analysis/workflow_gap_summary.md | 39 +- .../analysis/workflow_gap_table.csv | 4 + .../stage_admission_case_001/manifest.yaml | 12 +- .../design.md | 24 + .../evidence/r2_g2_integration_compare.json | 12 + .../evidence/r2_g2_unit_compare.json | 25 + .../evidence/r2_t_path_explanation.json | 660 ++++++++++++++++++ .../evidence/r2_tool_tests_on_ecff89a.txt | 8 + .../plan.md | 67 +- .../progress.md | 21 + .../requirements.md | 9 + .../review.md | 29 +- .../summary.md | 32 +- ...ort_2026-09-23_stage_admission_ordering.md | 160 ++++- 15 files changed, 1052 insertions(+), 53 deletions(-) create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_integration_compare.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_unit_compare.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_t_path_explanation.json create mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_tool_tests_on_ecff89a.txt diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json index a7c1edd6..5bdd6f2a 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json @@ -2,9 +2,10 @@ "analysis_state": "COMPLETE", "status": "PASS", "correction_state": "not_applicable", - "rows": 52, + "rows": 56, "mismatches": 0, "vllm_placement_ok": true, "vllm_placement_unseen_requests": 0, + "negative_control_holds": true, "next_action": "record C7 in the test report" } \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md index 09661073..d14c01b9 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-10: rerun against `after-r2` with the negative controls as rows N1/N4; dense V5 cause restated with both sources and the derived barrier-aligned M5. | | 2026-09-23 | D-9 adopted: dense V5 reported, not gated; rerun status PASS, 0 mismatches. | | 2026-09-23 | Created from run `sa-pp-20260923b` against Frontier sets `base` (`1f694f7` rule) and `after` (`dac4e69`). | @@ -13,20 +14,23 @@ | --- | --- | | vLLM | `runs/vllm-instrumented/sa-pp-20260923b/` — vLLM-BS `494b9f327` plus `inputs/groundtruth_overlay.patch` (SHA-256 `8d476789…3a9c81`), DP=2, PP=2, TP=1, MoE with EP, 4×H800 | | Frontier before | `/data/ycfeng/tmp/stage_admission_ordering/base/G7-*` | -| Frontier after | `/data/ycfeng/tmp/stage_admission_ordering/after/G7-*` | +| Frontier after | `/data/ycfeng/tmp/stage_admission_ordering/after-r2/G7-*` (R-10; byte-identical to set `after` at `dac4e69`) | | Producer | `tests/comparison/stage_admission_pp/compare_lanes.py` → `workflow_gap_table.csv`, `lane_metrics.json`, `workflow_gap_status.json` | ## Result -52 rows: 50 `MATCH`, 0 `MISMATCH`, 2 `INFORMATIONAL` (dense V5 under D-9); -status PASS. `vllm_placement_ok = true`, no unseen request. +56 rows: 50 `MATCH`, 0 `MISMATCH`, 2 `INFORMATIONAL` (dense V5 under D-9), +4 `HOLDS` (negative controls); status PASS, `negative_control_holds = true`. +`vllm_placement_ok = true`, no unseen request. Before R-10 the controls were +folded into V1 and dense V4 (52 rows). | Metric | MoE (n8, n16) | Dense (n8, n16) | | --- | --- | --- | -| V1 completion | MATCH in 6/6 rounds; base `admission_deadlock` (negative control holds) | MATCH in 6/6 rounds | +| V1 completion | MATCH in 6/6 rounds | MATCH in 6/6 rounds | +| N1 / N4 base control | N1 HOLDS (n8, n16): base `admission_deadlock` | N4 HOLDS (n8, n16): base co-start 1.0 | | V2 lane sequences | MATCH 6/6 | MATCH 6/6 | | V3 stage-0 pairing | MATCH 6/6 | MATCH 6/6; base pairs are shifted by one forward | -| V4 first-forward co-start | MATCH 6/6 (vLLM ≤ 0.063, after 0.0) | MATCH 6/6 (vLLM ≤ 0.248, after 0.0, base 1.0: negative control holds) | +| V4 first-forward co-start | MATCH 6/6 (vLLM ≤ 0.063, after 0.0) | MATCH 6/6 (vLLM ≤ 0.248, after 0.0) | | V5 stage-0 co-execution | MATCH: vLLM 0.976 / 0.948, after 1.0 | INFORMATIONAL (D-9): vLLM 0.706 / 0.865, after 1.0, base 0.600 / 0.778 | ## Dense V5 (MISMATCH before D-9) @@ -35,18 +39,25 @@ status PASS. `vllm_placement_ok = true`, no unseen request. the 0.10 bound: 0.537–0.926 over the 12 dense rounds of runs a and b. The n16 means of the two runs differ by 0.18. - Observed (`co_execution_decomposition_sa-pp-20260923{a,b}.json`): the dense - non-overlap has two sources. - - Per-pair start offsets of up to about 1.5 ms, measured before the - per-forward DP metadata exchange. + non-overlap has two sources. Every pair overlaps, so it splits exactly. + - Per-pair start offsets. `forward_start_ts` is taken before the + per-forward DP metadata all-reduce, so the rank that arrives first + records its wait as busy time. Start offsets exceed end offsets in 3 of + the 6 dense rounds of run b. - End offsets from per-rank duration variation: CV 0.10–0.29 on forwards of about 3 ms. MoE ends stay within 0.2–0.7 ms in total. -- Inference: the gap is a duration-variance property of vLLM's dense ranks, - which meet once per forward. It is not an admission difference, because - admission is measured by V1–V4, and they match in every round. The Frontier - owner is the execution-time model: the dummy predictor gives equal - durations. It is not `stage_execution_context.py`, the default owner label - written into the table. +- Derived, not measured (R-10): with both starts of each pair set to the + later one, where the all-reduce releases both ranks, M5 is 0.66–0.98 for + dense and 0.988–0.994 for MoE, higher than observed in every round. The + traces carry no timestamp after the exchange. +- Inference (restated at R-10): neither source is an admission difference. + Both ranks enter the same forward, which V1–V4 measure, and they match in + every round. The Frontier owner of the dense gap is the execution-time + model: the dummy predictor models neither the pre-exchange wait nor the + duration variation. It is not `stage_execution_context.py`, the default + owner label written into the table. The first version of this inference + named only the duration variation. - The first analysis stopped here with nothing adjusted. The user adopted D-9: V5 is informational for the dense shape and stays a gate for MoE. C7 rests on V1–V4 for both models, V5 for MoE, and the base negative diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv index bd5e9e32..cd61d56f 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv @@ -1,4 +1,5 @@ check,model,burst,round,metric,groundtruth,frontier_after,frontier_base,status,frontier_owner,note +N1,moe,8,base,base outcome,null,"""success""","""admission_deadlock""",HOLDS,,expected admission_deadlock V1,moe,8,0,M1 completion,"""8/8""","""success""","""admission_deadlock""",MATCH,, V2,moe,8,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",null,MATCH,, V3,moe,8,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]",null,MATCH,, @@ -12,6 +13,7 @@ V2,moe,8,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/s V3,moe,8,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]",null,MATCH,, V4,moe,8,2,M4 stage-0 co-start,0.06341533360159404,0.0,null,MATCH,, V5,moe,8,mean,M5 stage-0 co-execution,0.9759339956912042,1.0,null,MATCH,, +N1,moe,16,base,base outcome,null,"""success""","""admission_deadlock""",HOLDS,,expected admission_deadlock V1,moe,16,0,M1 completion,"""16/16""","""success""","""admission_deadlock""",MATCH,, V2,moe,16,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",null,MATCH,, V3,moe,16,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]",null,MATCH,, @@ -25,6 +27,7 @@ V2,moe,16,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10] V3,moe,16,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]",null,MATCH,, V4,moe,16,2,M4 stage-0 co-start,0.022678808030265958,0.0,null,MATCH,, V5,moe,16,mean,M5 stage-0 co-execution,0.947701161508853,1.0,null,MATCH,, +N4,dense,8,base,M4 stage-0 co-start,null,0.0,1.0,HOLDS,,expected >= 0.5 V1,dense,8,0,M1 completion,"""8/8""","""success""","""success""",MATCH,, V2,dense,8,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",MATCH,, V3,dense,8,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], null]]",MATCH,, @@ -38,6 +41,7 @@ V2,dense,8,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0 V3,dense,8,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], null]]",MATCH,, V4,dense,8,2,M4 stage-0 co-start,0.24755095323041817,0.0,1.0,MATCH,, V5,dense,8,mean,M5 stage-0 co-execution,0.7055180242540143,1.0,0.6,INFORMATIONAL,, +N4,dense,16,base,M4 stage-0 co-start,null,0.0,1.0000000000000002,HOLDS,,expected >= 0.5 V1,dense,16,0,M1 completion,"""16/16""","""success""","""success""",MATCH,, V2,dense,16,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",MATCH,, V3,dense,16,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], [9]], [[8], [11]], [[10], [13]], [[12], [15]], [[14], null]]",MATCH,, diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml index b448e41b..97d4d580 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml +++ b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml @@ -21,6 +21,7 @@ selected_checkout: frontier_branch: fix/stage-admission-ordering frontier_before_source: 1f694f7 # P0 set "base"; source identical to origin/main frontier_after_commit: dac4e69 # P1 rule commit; P3 set "after" + frontier_after_r2_commit: a8e8d8a # R-10 rule refactor 1661bf1 plus harness; set "after-r2", G7 byte-identical to "after" frontier_harness: tests/e2e/stage_admission_matrix.py (group G7) groundtruth_checkout_path: /data/ycfeng/Frontier/.real-engine/vLLM-BS @@ -114,8 +115,8 @@ modes: source: /data/ycfeng/tmp/stage_admission_ordering/base/G7-* status: COMPLETE # MoE n8/n16 admission_deadlock; dense n8/n16 success simulator_after: - source: /data/ycfeng/tmp/stage_admission_ordering/after/G7-* - status: COMPLETE # all four G7 cases success at dac4e69 + source: /data/ycfeng/tmp/stage_admission_ordering/after-r2/G7-* + status: COMPLETE # all four G7 cases success; set "after" (dac4e69) was used before R-10 and is byte-identical analysis: producer: tests/comparison/stage_admission_pp/compare_lanes.py @@ -147,7 +148,8 @@ decisions: decided_at_utc: "2026-09-23" analysis_result: >- - 52 rows: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5, D-9). The first - analysis, before D-9, had these two rows as MISMATCH; cause in - analysis/workflow_gap_summary.md. + 56 rows: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5, D-9), 4 HOLDS + (negative controls N1 and N4, separate rows since R-10); rerun against the + Frontier set after-r2. The first analysis, before D-9, had the two dense V5 + rows as MISMATCH; cause in analysis/workflow_gap_summary.md. status: COMPLETE diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/design.md b/task_memory/task_2026-09-22_stage_admission_ordering/design.md index 9c0ef993..300af11d 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/design.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/design.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-10: added the round-2 notes on the capacity-1 contract (R2-12), FIFO meaning on shared-lane contexts (R2-13) and the deferred EP-only queue variant. | | 2026-09-23 | Applied the round-1 plan review (`review.md`). Changes: the admission-loop anchor now points to the `MONOLITHIC`/`PREFILL` path; the drain condition is stated as a queued-ticket arrangement, not a shape; the shape table is marked author-reported until P0; added where queued EP waves exist; `remove(ticket)` made explicit; option A's stall trace labelled an unverified hypothesis; the capacity-1 section rewritten as a caller-level condition; the queue bound narrowed; added the mixed-phase scope boundary; the dense "lanes serialized" label withdrawn as unmeasured and replaced by the admission sequence read from source. | | 2026-09-22 | Created: defect restated from source on `origin/main` `1f694f7`, what the FIFO guarantees today, four options, recommended rule with its invariants, fidelity expectation. For review before implementation. | @@ -264,6 +265,29 @@ No capacity-1 or `PP=1` special case is added: no supported caller has been shown to need arbitrary cross-lane full-stage FIFO order. An unexpected difference in any of these classes stops the work and is reported. +### Round-2 notes (R-10) + +- Capacity-1 contract (R2-12). At the context API this is a contract change, + not an unaffected path: an idle capacity-1 context used to admit full-stage + tickets in enqueue order and now admits whichever one its lane stage + scheduler presents, unless an EP wave is queued ahead. Order among + full-stage tickets now comes from the callers. Measured on the callers: + every capacity-1 context in the matrix is byte-identical. That covers the + 10 PD-AF release recipes (`DECODE_ATTN`, `DECODE_FFN`, `PREFILL`) and the 4 + PD-AF recipes with `PREFILL_PP=2` (G11), offline and online. +- FIFO meaning on shared-lane contexts (R2-13). `enqueue_ep_wave` has one + caller, the `DECODE_FFN` M2N path (`round_robin_cluster_scheduler.py:1052`). + So the FIFO of a `MONOLITHIC`, `PREFILL` or unified `DECODE` context only + ever holds full-stage tickets, and under B it no longer orders admission + there. `queued_tickets` and `admission_seq` record enqueue order only, and + the class docstring says so. +- Variant considered and deferred: queue only EP waves, and keep queued + full-stage tickets as an unordered set. The data structure would then + match the rule. But it changes `queued_tickets`, `is_queued` and `cancel`, + and the drain diagnostics that read FIFO order, with no change in + behaviour. It is not pursued unless a caller needs the queue to express + admission order. + ## Scope boundary: mixed-phase forwards This branch is based on `main`, where prefill and decode source lanes still diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_integration_compare.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_integration_compare.json new file mode 100644 index 00000000..61d33632 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_integration_compare.json @@ -0,0 +1,12 @@ +{ + "regressions": [], + "new_failures": [], + "now_passing": [], + "skip_changes": [], + "only_before": [], + "only_after": [ + "tests.integration.test_stage_admission_pipeline_lanes::test_dense_lanes_start_in_the_same_first_forward", + "tests.integration.test_stage_admission_pipeline_lanes::test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4-expected0]", + "tests.integration.test_stage_admission_pipeline_lanes::test_moe_lanes_complete_every_request[G3a-moe-dp4-pp2-n8-expected1]" + ] +} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_unit_compare.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_unit_compare.json new file mode 100644 index 00000000..26aaaae2 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_unit_compare.json @@ -0,0 +1,25 @@ +{ + "regressions": [], + "new_failures": [], + "now_passing": [], + "skip_changes": [], + "only_before": [], + "only_after": [ + "tests.unit.test_mixed_layer_decode_ffn_scheduling::test_decode_ffn_dense_groups_keep_counter_order_around_a_queued_ep_wave", + "tests.unit.test_shared_forward_group_admission::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0]", + "tests.unit.test_shared_forward_group_admission::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[1]", + "tests.unit.test_stage_admission_pp_tools::test_apply_patch_keeps_every_section_of_a_repeated_file", + "tests.unit.test_stage_admission_pp_tools::test_apply_patch_reads_a_trimmed_context_line", + "tests.unit.test_stage_admission_pp_tools::test_apply_patch_rejects_an_unknown_hunk_line", + "tests.unit.test_stage_admission_pp_tools::test_fixed_base_loses_the_controls_without_a_mismatch", + "tests.unit.test_stage_admission_pp_tools::test_ideal_after_revision_matches_and_base_controls_hold", + "tests.unit.test_stage_admission_pp_tools::test_missing_placement_log_fails_the_comparison", + "tests.unit.test_stage_admission_pp_tools::test_overlay_acceptance_compares_file_sets[False-True]", + "tests.unit.test_stage_admission_pp_tools::test_overlay_acceptance_compares_file_sets[True-False]", + "tests.unit.test_stage_admission_pp_tools::test_vllm_round_with_a_late_lane_is_reported", + "tests.unit.test_stage_execution_context::test_active_full_stage_ticket_is_refused_without_changing_the_stage", + "tests.unit.test_stage_execution_context::test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave", + "tests.unit.test_stage_execution_context::test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket", + "tests.unit.test_stage_execution_context::test_queued_ep_wave_orders_full_stage_work_on_both_sides" + ] +} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_t_path_explanation.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_t_path_explanation.json new file mode 100644 index 00000000..91379ba2 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_t_path_explanation.json @@ -0,0 +1,660 @@ +[ + { + "case_id": "G4-dense-dp2-pp2-n4", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.7143, + "MONOLITHIC/0/1": 0.7143 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.05, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.3, + "MONOLITHIC/0/1": 0.3 + }, + "before": { + "MONOLITHIC/0/0": 0.25, + "MONOLITHIC/0/1": 0.24999999999999997 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + }, + { + "case_id": "G4-dense-dp2-pp2-n8", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.8182, + "MONOLITHIC/0/1": 0.8182 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.05, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.49999999999999994, + "MONOLITHIC/0/1": 0.49999999999999994 + }, + "before": { + "MONOLITHIC/0/0": 0.44999999999999996, + "MONOLITHIC/0/1": 0.4499999999999999 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": true + }, + { + "case_id": "G4-dense-dp2-pp3-n4", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0, + "MONOLITHIC/0/2": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.3333, + "MONOLITHIC/0/1": 0.3333, + "MONOLITHIC/0/2": 0.3333 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.036000000000000004, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.21600000000000003, + "MONOLITHIC/0/1": 0.21600000000000008, + "MONOLITHIC/0/2": 0.21600000000000005 + }, + "before": { + "MONOLITHIC/0/0": 0.10800000000000004, + "MONOLITHIC/0/1": 0.10800000000000004, + "MONOLITHIC/0/2": 0.10800000000000001 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + }, + { + "case_id": "G4-dense-dp2-pp3-n8", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0, + "MONOLITHIC/0/2": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.375, + "MONOLITHIC/0/1": 0.375, + "MONOLITHIC/0/2": 0.375 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.07200000000000001, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.39600000000000013, + "MONOLITHIC/0/1": 0.39600000000000013, + "MONOLITHIC/0/2": 0.3960000000000002 + }, + "before": { + "MONOLITHIC/0/0": 0.21600000000000008, + "MONOLITHIC/0/1": 0.21600000000000014, + "MONOLITHIC/0/2": 0.2160000000000001 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": true + }, + { + "case_id": "G4-dense-dp4-pp2-n8", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.8462, + "MONOLITHIC/0/1": 0.8462 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0, + "2": 0.0, + "3": 0.0 + }, + "before": { + "0": 0.15000000000000002, + "1": 0.0, + "2": 0.05, + "3": 0.1 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.3, + "MONOLITHIC/0/1": 0.3 + }, + "before": { + "MONOLITHIC/0/0": 0.55, + "MONOLITHIC/0/1": 0.55 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 4, + "MONOLITHIC/0/1": 4 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": true + }, + { + "case_id": "G4-dense-dp4-pp3-n8", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0, + "MONOLITHIC/0/2": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.8462, + "MONOLITHIC/0/1": 0.8462, + "MONOLITHIC/0/2": 0.8462 + } + }, + "differing_files": [ + "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/request_metrics.csv", + "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0, + "2": 0.0, + "3": 0.0 + }, + "before": { + "0": 0.10800000000000001, + "1": 0.0, + "2": 0.036000000000000004, + "3": 0.07200000000000001 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.21600000000000003, + "MONOLITHIC/0/1": 0.21600000000000008, + "MONOLITHIC/0/2": 0.21600000000000005 + }, + "before": { + "MONOLITHIC/0/0": 0.39600000000000013, + "MONOLITHIC/0/1": 0.3960000000000002, + "MONOLITHIC/0/2": 0.39600000000000024 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 4, + "MONOLITHIC/0/1": 4, + "MONOLITHIC/0/2": 4 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": true + }, + { + "case_id": "G7-dense-dp2-pp2-n8", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.6, + "MONOLITHIC/0/1": 0.6 + } + }, + "differing_files": [ + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/frontier_stage_batch_ledger.jsonl", + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/request_metrics.csv", + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.12000000000000001, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.48000000000000004, + "MONOLITHIC/0/1": 0.4800000000000001 + }, + "before": { + "MONOLITHIC/0/0": 0.36000000000000004, + "MONOLITHIC/0/1": 0.3600000000000001 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + }, + { + "case_id": "G7-dense-dp2-pp2-n16", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.7778, + "MONOLITHIC/0/1": 0.7778 + } + }, + "differing_files": [ + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/frontier_stage_batch_ledger.jsonl", + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/request_metrics.csv", + "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.12000000000000001, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.9600000000000001, + "MONOLITHIC/0/1": 0.9600000000000001 + }, + "before": { + "MONOLITHIC/0/0": 0.8400000000000001, + "MONOLITHIC/0/1": 0.8400000000000001 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + }, + { + "case_id": "G10-dense-dp2-pp2-n8-burst", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.8182, + "MONOLITHIC/0/1": 0.8182 + } + }, + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_burst/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_burst/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_burst/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.05, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.49999999999999994, + "MONOLITHIC/0/1": 0.49999999999999994 + }, + "before": { + "MONOLITHIC/0/0": 0.44999999999999996, + "MONOLITHIC/0/1": 0.4499999999999999 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + }, + { + "case_id": "G10-dense-dp2-pp3-n8-burst", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0, + "MONOLITHIC/0/2": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.375, + "MONOLITHIC/0/1": 0.375, + "MONOLITHIC/0/2": 0.375 + } + }, + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8_burst/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8_burst/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8_burst/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0 + }, + "before": { + "0": 0.07200000000000001, + "1": 0.0 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.39600000000000013, + "MONOLITHIC/0/1": 0.39600000000000013, + "MONOLITHIC/0/2": 0.3960000000000002 + }, + "before": { + "MONOLITHIC/0/0": 0.21600000000000008, + "MONOLITHIC/0/1": 0.21600000000000014, + "MONOLITHIC/0/2": 0.2160000000000001 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + }, + { + "case_id": "G10-dense-dp4-pp2-n8-burst", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.8462, + "MONOLITHIC/0/1": 0.8462 + } + }, + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8_burst/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8_burst/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8_burst/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0, + "2": 0.0, + "3": 0.0 + }, + "before": { + "0": 0.15000000000000002, + "1": 0.0, + "2": 0.05, + "3": 0.1 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.3, + "MONOLITHIC/0/1": 0.3 + }, + "before": { + "MONOLITHIC/0/0": 0.55, + "MONOLITHIC/0/1": 0.55 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 4, + "MONOLITHIC/0/1": 4 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + }, + { + "case_id": "G10-dense-dp4-pp3-n8-burst", + "co_execution_fraction": { + "after": { + "MONOLITHIC/0/0": 1.0, + "MONOLITHIC/0/1": 1.0, + "MONOLITHIC/0/2": 1.0 + }, + "before": { + "MONOLITHIC/0/0": 0.8462, + "MONOLITHIC/0/1": 0.8462, + "MONOLITHIC/0/2": 0.8462 + } + }, + "differing_files": [ + "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8_burst/frontier_stage_batch_ledger.jsonl", + "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8_burst/request_metrics.csv", + "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8_burst/system_metrics.json" + ], + "first_stage0_start": { + "after": { + "0": 0.0, + "1": 0.0, + "2": 0.0, + "3": 0.0 + }, + "before": { + "0": 0.10800000000000001, + "1": 0.0, + "2": 0.036000000000000004, + "3": 0.07200000000000001 + } + }, + "multi_lane_busy_time": { + "after": { + "MONOLITHIC/0/0": 0.21600000000000003, + "MONOLITHIC/0/1": 0.21600000000000008, + "MONOLITHIC/0/2": 0.21600000000000005 + }, + "before": { + "MONOLITHIC/0/0": 0.39600000000000013, + "MONOLITHIC/0/1": 0.3960000000000002, + "MONOLITHIC/0/2": 0.39600000000000024 + } + }, + "peak_lanes": { + "after": { + "MONOLITHIC/0/0": 4, + "MONOLITHIC/0/1": 4, + "MONOLITHIC/0/2": 4 + }, + "before": { + "MONOLITHIC/0/0": 2, + "MONOLITHIC/0/1": 2, + "MONOLITHIC/0/2": 2 + } + }, + "same_batches_and_component_durations": true, + "verdict": "EXPLAIN", + "witness_increase": null + } +] \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_tool_tests_on_ecff89a.txt b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_tool_tests_on_ecff89a.txt new file mode 100644 index 00000000..0e1d48c0 --- /dev/null +++ b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_tool_tests_on_ecff89a.txt @@ -0,0 +1,8 @@ +FAILED tests/unit/test_stage_admission_pp_tools.py::test_ideal_after_revision_matches_and_base_controls_hold +FAILED tests/unit/test_stage_admission_pp_tools.py::test_fixed_base_loses_the_controls_without_a_mismatch +FAILED tests/unit/test_stage_admission_pp_tools.py::test_missing_placement_log_fails_the_comparison +FAILED tests/unit/test_stage_admission_pp_tools.py::test_overlay_acceptance_compares_file_sets[False-True] +FAILED tests/unit/test_stage_admission_pp_tools.py::test_apply_patch_keeps_every_section_of_a_repeated_file +FAILED tests/unit/test_stage_admission_pp_tools.py::test_apply_patch_reads_a_trimmed_context_line +FAILED tests/unit/test_stage_admission_pp_tools.py::test_apply_patch_rejects_an_unknown_hunk_line +7 failed, 2 passed in 0.32s diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/plan.md b/task_memory/task_2026-09-22_stage_admission_ordering/plan.md index 75ba7d5f..6a1a9334 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/plan.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/plan.md @@ -4,6 +4,8 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-10 executed: §7 group table amended with the online burst cells (lane-0 placement on main), D-9 (b) rationale restated (R2-02). | +| 2026-09-23 | R-10: §7 added, the round-2 remediation (R2 findings), new groups G8–G11 and the pre-merge step P6. | | 2026-09-23 | R-8 / D-9: the C3 witness condition uses the co-execution fraction, and V5 gates MoE only (dense is reported). Adopted after the P3/P5 stops, before P4. | | 2026-09-23 | R-7: the MoE retry job runs the ground truth with one recorded overlay patch (four-argument `topk_softmax`); §4.7 notes it. | | 2026-09-23 | R-6: execution started. Added package P5 (vLLM comparison on a GPU worker), criterion C7, the vLLM-aligned group G7, §4.7 and D-8. | @@ -325,7 +327,7 @@ Adopted after the P3 and P5 stops on 2026-09-23 ("采纳你的推荐，继续"; | Id | Decision | Reason | | --- | --- | --- | -| D-9 | (a) A contention witness passes when its co-execution fraction strictly increases; the self-overlap and `peak_lanes` checks are unchanged. (b) V5 gates the MoE shape only; for the dense shape M5 is reported with its start/end decomposition. | (a) At `attn_dp=4` the fix makes all four lanes co-execute and shortens the busy period, so absolute `multi_lane_busy_time` falls (0.55 → 0.30) while overlap becomes complete; the fraction measures overlap independently of that compression. (b) vLLM's dense ranks meet once per forward and vary in duration per rank (M5 0.54–0.93 across rounds, wider than 0.10), a property the dummy predictor does not model; admission is covered by V1–V4. MoE ranks stay aligned by in-forward EP collectives (M5 0.93–0.98). | +| D-9 | (a) A contention witness passes when its co-execution fraction strictly increases; the self-overlap and `peak_lanes` checks are unchanged. (b) V5 gates the MoE shape only; for the dense shape M5 is reported with its start/end decomposition. | (a) At `attn_dp=4` the fix makes all four lanes co-execute and shortens the busy period, so absolute `multi_lane_busy_time` falls (0.55 → 0.30) while overlap becomes complete; the fraction measures overlap independently of that compression. (b) Restated under R-10 (R2-02). vLLM's dense ranks meet once per forward, in the DP metadata all-reduce that runs after `forward_start_ts`. Their stage-0 non-overlap has two sources, neither of them admission: the rank that arrives first records its wait for the other as busy time (start offsets), and per-rank durations vary (end offsets, CV 0.10–0.29). Observed dense M5 is 0.54–0.93 across rounds, wider than 0.10. With both starts of each pair set to the later one (derived from the all-reduce position, not measured), dense M5 is 0.66–0.98 and MoE 0.988–0.994; the dense residual is the end offsets. The dummy predictor models neither source; admission is covered by V1–V4. MoE ranks stay aligned by in-forward EP collectives (M5 0.93–0.98). | ## 6. Dependencies and risks @@ -354,3 +356,66 @@ Adopted after the P3 and P5 stops on 2026-09-23 ("采纳你的推荐，继续"; and merged forward into `fix/issue26-correctness-pr`. The parent task then reruns G3b on that branch, where W3 is present, as the composition check (C6). + +## 7. Round-2 remediation (R-10) + +Findings are in `review.md` Round 2. R2-06 got no answer on timing, so only +its pre-merge step is recorded (P6 below). + +| Finding | Change | Acceptance | +| --- | --- | --- | +| R2-01, R2-11 | `try_acquire` has one branch per scope. The EP wave leaves the FIFO by `popleft`. A full-stage ticket is found in one pass and deleted by index. A ticket that is not in the FIFO, because it is already active, is refused, as on the base. | A new unit test: re-acquiring an active full-stage ticket returns `False` and leaves the context unchanged. Existing contract tests pass without assertion changes. Every case of set `after` is byte-identical in set `after-r2`. | +| R2-12, R2-13 | Documentation. The class docstring names where full-stage order comes from. The PR body states the capacity-1 contract change. `design.md` records the EP-only-queue variant as considered and deferred. | Review. | +| R2-02 | The D-9 (b) rationale now names both sources. (1) Start offsets: `forward_start_ts` is taken before the per-forward DP all-reduce, so the rank that reaches it first records its wait as busy time. (2) End offsets: per-rank duration variance. `evidence/decompose_co_execution.py` states its identity for overlapping pairs only and counts disjoint pairs. It also reports M5 with both starts of each pair set to the later one, which is when the all-reduce releases both ranks. This is derived from the barrier; the traces carry no post-exchange timestamp. Dense V5 stays reported. | Rerun on runs a and b; values in the test report. No GPU job. | +| R2-04 | `vllm_placement` is `ok` only with no misplaced and no unseen request. | Unit test. | +| R2-05 | The negative controls become their own rows, `N1` (MoE base `admission_deadlock`) and `N4` (dense base M4 ≥ 0.5), each `HOLDS` or `LOST`. `workflow_gap_status.json` gains `negative_control_holds`. V1 and V4 compare vLLM with the after revision only. | Unit test: a base equal to the after revision leaves every V row `MATCH` and every N row `LOST`. The rerun on run b gives PASS with the controls holding. | +| R2-07 | Each matrix child runs in its own session under `--case-timeout` (seconds, default 600). On timeout the whole session is killed and the case is recorded as `other_failure`. | A case made to exceed a small timeout is recorded, and the set completes. | +| R2-08 | The shared `work/` path stays: files that embed the output path must compare byte for byte. `run` instead takes an exclusive lock on the matrix root, and the module docstring says sets run one at a time. | A second concurrent `run` fails at once with the lock message. | +| R2-09, R2-10 | Overlay acceptance compares file sets. `apply_patch` reads an empty hunk line as a trimmed context line, fails on any other unknown line, and accumulates hunks when one file appears in several sections. | Unit tests. | +| R2-14, R2-15 | `probe_main.py` drops its hard-coded path and imports from `PYTHONPATH`. `synthetic_check.py` is replaced by `tests/unit/test_stage_admission_pp_tools.py`. For #35, the C6 shapes are rerun as matrix group R0: `R0-moe-dp2-pp{1,2,3}-n6` and `R0-dense-dp1-pp2-n6` are the probe's configuration, apart from metrics flags and the model name. | Unit tests pass; no absolute path left in the evidence scripts. | +| R2-03 | New groups G8–G11 (below). `build_config` gains `sys_arch`, `simulation_mode` and a Poisson rate. The state report and deadlock signature are read per cluster type. Recipe cases take environment overrides. | Paths U/L/T of §4.4 on the new cells. | + +New groups (R2-03). PDD rejects dense `attn_dp > 1` at configuration +(`config.py` `_validate_replica_config`). PD-AF takes one `attn_dp` for every +role, and `DECODE_ATTN` requires 1, so neither can reach a multi-lane context. +PD-AF is therefore covered by `PP > 1` controls on its capacity-1 `PREFILL` +contexts, which exercise the R2-12 contract change. + +| Group | Cases | Profile, arrivals | Count | Path | +| --- | --- | --- | --- | --- | +| G8 PDD offline | MoE `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}`, `n=8`; dense `attn_dp=1, PP=2`, `n=8` | PD, static | 7 | U / L / T | +| G9 PDD online | the G8 shapes at Poisson 20/s; MoE `attn_dp ∈ {2,4}` × `PP ∈ {2,3}` as `-burst` cells | PD, Poisson or burst | 11 | U / L / T | +| G10 co-location online | MoE (PF) and dense (PD), `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}`, `n=8`, each at Poisson 20/s and as a `-burst` cell; plus MoE and dense `dp2-pp2` at 5/s and 80/s | PF / PD, Poisson or burst | 28 | U / L / T | +| G11 PD-AF `PP > 1` | the dense and MoE PD-AF recipes, offline and online, with `PREFILL_PP=2` | recipe | 4 | U | + +Amendment found while running (R-10). On `main`, `_schedule_batch_mode` +(MONOLITHIC and PREFILL) numbers DP lanes from 0 within each scheduling call, +so online Poisson arrivals, one per call, all land on lane 0; unified DECODE +rotates across calls. This is PR 35's W2 defect, fixed on that branch. The +Poisson cells therefore exercise one lane of a multi-lane context here. The +`-burst` cells (online mode, all requests at `t=0`, one call) reach every +lane and carry the online L/T coverage. After PR 35 merges `main` forward, +its composition check reruns G9 and G10 with lane rotation in place. + +Base runs for G8–G11 use the base rule: the one `frontier/` file differing +from `1f694f7` is swapped in the worktree for the run, and `run.json` records +the modified tree. Set `after-r2` then runs every case on the R2 revision. + +A T difference with the same batches is explained as in §4.4. Where online +arrivals let earlier admission change later batch composition, the +explanation must name the first ledger row that differs and show the base +refusal before it. Otherwise stop and report. + +Verification order: + +```text +R2-01/R2-11 rule refactor and unit test + -> {matrix harness (R2-03, R2-07, R2-08), tools and unit tests (R2-04, R2-05, R2-09, R2-10, R2-15), evidence scripts (R2-02, R2-14)} + -> base runs of G8–G11 -> after-r2 set -> compare (base, after-r2) and identity (after, after-r2) + -> compare_lanes and decomposition reruns -> G2 suites -> records, commits, push, PR body +``` + +**P6 pre-merge step (R2-06, not executed).** Before PR 36 merges, a last +commit drops the `!task_memory/task_2026-09-22_stage_admission_ordering/` +exception and untracks the directory. The archive copy stays in the parent +task. This deletes tracked records, so it runs only on the owner's go-ahead. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md index 1564cdd0..d230fb9b 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md @@ -4,6 +4,8 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-10 executed: commits `1661bf1`, `a8e8d8a`, `e35242f` and records; all checks pass; R2-06 recorded as P6. | +| 2026-09-23 | R-10: round-2 remediation started (plan §7). | | 2026-09-23 | Round-2 code review posted to PR 36 (15 inline comments, `review.md`); fixes deferred by the owner. | | 2026-09-23 | P4 completed: branch pushed at `4bcd616`, PR 36 body updated (still draft), W9-01 resolution recorded in the parent task (`4c2d573`). | | 2026-09-23 | R-8 / D-9 adopted; both comparisons rerun and pass (`aeeca93`); P4 in progress. | @@ -84,3 +86,22 @@ worktree, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`. | P4 push | `git push origin fix/stage-admission-ordering` | remote head `4bcd616` | records `df7868e`, `fc34341`; `4bcd616` force-adds `evidence/base_negative_controls.log`, which the repository-wide `*.log` rule had kept out of the tree | | P4 PR body | REST `PATCH repos/NetX-lab/Frontier/pulls/36` (`gh pr edit` fails on the retired Projects classic query) | PR 36 | body carries the rule, commits, C1–C4/C7 and C3 tables, R-7/D-9 and open items; body read back identical; still draft | | P4 parent note | parent `issues.md` W9-01 Resolution, `progress.md`, case manifest decision `W9-01-scope`; `summary.md` and the test report copied to `w9_01_stage_admission_ordering/` (D-5) | `fix/issue26-correctness-pr` `4c2d573`, pushed | PR 35 still draft | + +## Round-2 remediation (R-10, plan §7) + +| Step | Command / action | Evidence | Result | +| --- | --- | --- | --- | +| Records | `requirements.md` R-10, `plan.md` §7 | — | recorded | +| R2-01/R2-11/R2-13 | `try_acquire` one branch per scope; active ticket refused; docstring | `1661bf1` | 181 passed on the three context unit files; the new test's scenario: base `False`, `dac4e69` `ValueError`, now `False` | +| R2-03/R2-07/R2-08 | matrix: `sys_arch`, `simulation_mode`, Poisson rate, recipe env; groups G8–G11; cluster-keyed drain report; `--case-timeout`; set lock | `a8e8d8a` | probe set `r2-probe`: PDD first rejected for missing role replica counts, fixed by one Replica per role; online Poisson cells found on lane 0 only (PR 35 W2), burst cells added; lock and 2 s timeout checked | +| Base for G8–G11 | rule file swapped to `1f694f7` at `a8e8d8a`, `run --set base --group G8 … G11 --jobs 16`, then `git checkout` of the file | scratch `base/` | 12 `admission_deadlock` (G8 4, G9 burst 4, G10 MoE burst 4), 38 success | +| After set | `run --set after-r2 --jobs 16` at `a8e8d8a`, clean | scratch `after-r2/` | 147 success, 1 configuration rejection (R0 dp2-pp3, W9-02) | +| Identity | `after` vs `after-r2` | scratch `identity_after_after-r2.json` | 98/98 identical | +| Compare | `compare --before base --after after-r2` | scratch `compare_base_after-r2.json` | 120 PASS, 12 EXPLAIN, 16 informational, 0 STOP | +| Explain | `explain_t_path.py after-r2 …` | `evidence/r2_t_path_explanation.json` | 12 EXPLAIN: same batches and durations, start times only | +| Tools | compare_lanes N1/N4 rows, placement; driver overlay and patch; tool unit tests; `synthetic_check.py` removed; decomposition and probe scripts | `e35242f` | 9 passed; 7 fail on the `ecff89a` tools | +| Decomposition | `decompose_co_execution.py` on runs a and b | `analysis/co_execution_decomposition_*.json` | no disjoint pair; aligned M5 dense 0.66–0.98, MoE 0.988–0.994 | +| C7 rerun | `compare_lanes --after after-r2` on run b | `analysis/` | PASS, 56 rows, controls hold | +| C6 probe | `probe_completion.py` with `PYTHONPATH` only | scratch `r2/c6_probe/` | 6/6 for both shapes | +| G2 | unit and integration, `--junitxml` | `evidence/r2_g2_*_compare.json` | 0 regressions, 0 new failures, 0 skip changes | +| Records | test report §8, `summary.md`, `design.md`, `review.md`, manifest, workflow-gap summary, plan §7 amendment and D-9 (b) | this commit | — | diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md index eaec8bc6..6c253e02 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-10: round-2 fixes confirmed; R2-02 per the recommendation; R2-03 extended to PDD, online and PD-AF. | | 2026-09-23 | R-9: code review of PR 36 posted as inline comments; fixes deferred. | | 2026-09-23 | R-8: D-9 adopted for the C3 witness rule and dense V5; P4 authorized to continue. | | 2026-09-23 | R-7: ground-truth `topk_softmax` fixed to the four-argument version for the MoE retry. | @@ -66,6 +67,7 @@ work, carried over verbatim: | R-6 | Execute P0–P4 as planned. The fix must also be validated against vLLM running on a GPU worker, in a comparison designed to show whether the change is effective (package P5 in `plan.md`). The request authorizes the GPU job within the standing GPU rules below. | user, 2026-09-23 | | R-7 | The vLLM ground truth uses the four-argument `topk_softmax` (wrapper and call). Applied as the recorded overlay patch `calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch` on the one retry job; the vLLM-BS checkout is unchanged. | user, 2026-09-23 | | R-8 | Adopt both recommendations (plan D-9): contention witnesses pass on a strictly larger co-execution fraction; V5 gates MoE only and reports dense. Continue to P4. | user, 2026-09-23 | +| R-10 | Fix the recommended round-2 findings (R2-01, R2-04, R2-05, R2-07 to R2-11, R2-14, R2-15), with R2-12 and R2-13 as documentation. R2-02: restate the D-9 rationale with both sources and quantify the start-offset part without a new GPU job. R2-03: add PDD and online cells, plus PD-AF online if needed. R2-06 had no answer: record the pre-merge step only (plan §7). | user, 2026-09-23 | ## Constraints carried from the parent task @@ -92,3 +94,10 @@ work, carried over verbatim: Outcome: round-2 review recorded in `review.md` and posted to PR 36 as one `COMMENT` review with 15 inline comments. No source or test change. + +`[Original Request]` R-10 (2026-09-23, after the round-2 review). The question +listed the recommended fixes, two options for R2-02 (recommended: restate D-9 +with both sources and measure the post-synchronization start without a new +GPU job), and asked for the scope of R2-03 and R2-06: + +> 确认，执行上上述修复； R2-02 采纳你的推荐；R2-03需要补充 PDD+online（如果你认为pd-af+online有必要，请一并补充） diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/review.md b/task_memory/task_2026-09-22_stage_admission_ordering/review.md index 8228678a..636463df 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/review.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/review.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | Round 2 dispositions recorded (R-10): 14 findings applied and verified, R2-06 recorded as pre-merge step P6. | | 2026-09-23 | Round 2: code review of the implementation at `ecff89a` recorded and posted to PR 36; findings verified, fixes deferred by the owner. | | 2026-09-23 | Created. First external plan review of PR 36 at `a6ec6a6` recorded; each finding re-checked against `1f694f7` source, with a disposition and the place it was applied. | @@ -59,7 +60,7 @@ Docs corrected. P0 has not started, per the owner's "暂不执行". The next ste | Owner instruction | "review pr36，将review comments提交到该remote repo的pr36上，暂不执行修复。" (requirements R-9) | | Posted | https://github.com/NetX-lab/Frontier/pull/36#pullrequestreview-5286523149 (event `COMMENT`, 15 inline comments on `ecff89a`) | -### Findings (disposition pending; no fix applied) +### Findings | Id | Anchor | Verdict | Finding | | --- | --- | --- | --- | @@ -79,5 +80,27 @@ Docs corrected. P0 has not started, per the owner's "暂不执行". The next ste | R2-14 | `evidence/step9_probe/probe_main.py:15` | confirmed | A hard-coded worktree `ROOT` is put first on `sys.path`, so a #35 rerun would import this tree. | | R2-15 | `calibration/.../analysis/synthetic_check.py:7` | confirmed | A hard-coded scratch `BASE` bypasses `matrix_root()`, and reusable probes live under `task_memory/` rather than `tests/`. | -The owner deferred fixes ("暂不执行修复"). Dispositions will be recorded -here when the owner decides which findings to adopt. +The owner first deferred fixes ("暂不执行修复"), then decided (R-10): +"确认，执行上上述修复； R2-02 采纳你的推荐；R2-03需要补充 PDD+online（如果你认为pd-af+online有必要，请一并补充）". + +### Dispositions (R-10) + +Evidence for each row is in the test report §8. + +| Id | Disposition | Where | Verification | +| --- | --- | --- | --- | +| R2-01 | Fixed: an active ticket is refused, as on the base | `1661bf1` | unit test; `after` vs `after-r2` 98/98 byte-identical | +| R2-02 | Rationale restated with both sources; the script's identity is limited to overlapping pairs; a derived barrier-aligned M5 added. No GPU job. | `e35242f`, `plan.md` D-9 (b), test report §5.3 | reran on runs a and b: no disjoint pair; aligned M5 dense 0.66–0.98, MoE 0.988–0.994 | +| R2-03 | Fixed: G8 PDD offline, G9 PDD online, G10 co-location online, G11 PD-AF `PREFILL_PP=2` (50 cases). Online burst cells added because MONOLITHIC/PREFILL place incremental arrivals on lane 0 on `main` (PR 35 W2). | `a8e8d8a` | 12 more base deadlocks repaired; 0 STOP | +| R2-04 | Fixed | `e35242f` | unit test | +| R2-05 | Fixed: rows N1 and N4, `negative_control_holds` | `e35242f` | unit tests; run b rerun PASS, controls hold | +| R2-06 | Recorded as pre-merge step P6; not executed | `plan.md` §7 | waits for the owner | +| R2-07 | Fixed | `a8e8d8a` | a 2 s timeout recorded as `other_failure` | +| R2-08 | Fixed by a set lock; the shared `work/` path stays for byte identity | `a8e8d8a` | a concurrent run fails at once | +| R2-09 | Fixed | `e35242f` | unit test | +| R2-10 | Fixed | `e35242f` | unit tests | +| R2-11 | Fixed with R2-01 | `1661bf1` | as R2-01 | +| R2-12 | Stated as a contract change | `design.md` round-2 notes, PR body | G1 PD-AF and G11 byte-identical | +| R2-13 | Docstring states the FIFO meaning; the EP-only queue variant deferred | `1661bf1`, `design.md` | review | +| R2-14 | Fixed | `e35242f` | C6 probe 6/6 with `PYTHONPATH` only | +| R2-15 | Replaced by `tests/unit/test_stage_admission_pp_tools.py`; `synthetic_check.py` removed | `e35242f` | 9 passed; 7 fail on the `ecff89a` tools | diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/summary.md b/task_memory/task_2026-09-22_stage_admission_ordering/summary.md index 13979b22..2a56ada2 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/summary.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/summary.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-10: round-2 review remediation (14 findings applied, R2-06 pending as P6); PDD, online and PD-AF cells added. | | 2026-09-23 | Created at P4: fix, tests, P0–P3 and the vLLM comparison complete under D-9. | ## Overview @@ -15,19 +16,20 @@ runs completed but started the lanes one forward apart. The fix (plan D-1, option B) changes one predicate in `StageExecutionContext.try_acquire`. A full-stage ticket is refused only by an -EP wave queued ahead of it. EP waves keep the strict FIFO-head rule. The -admitted ticket leaves the FIFO by `remove(ticket)`. +EP wave queued ahead of it. EP waves keep the strict FIFO-head rule. A +ticket that is already active is refused (round 2, R2-01). ## Deliverables | Item | Path / commit | | --- | --- | +| Rule refactor (round 2) | `1661bf1`: one branch per scope; an active ticket is refused; docstring on FIFO meaning | | Rule and P2 tests | `dac4e69`: `frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`, `tests/unit/test_stage_execution_context.py`, `tests/unit/test_shared_forward_group_admission.py`, `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`, `tests/integration/test_stage_admission_pipeline_lanes.py` | -| Case matrix | `tests/e2e/stage_admission_matrix.py` (`a054d87`, `5ade853`, `aeeca93`) | -| vLLM comparison | `tests/comparison/stage_admission_pp/{vllm_burst_driver.py,run_vllm_worker.sh,compare_lanes.py}` (`799ccb4`, `a1b9819`, `aeeca93`) | +| Case matrix | `tests/e2e/stage_admission_matrix.py` (`a054d87`, `5ade853`, `aeeca93`; round 2 `a8e8d8a`: PDD, online and PD-AF groups G8–G11, case timeout, set lock), 148 cases | +| vLLM comparison | `tests/comparison/stage_admission_pp/{vllm_burst_driver.py,run_vllm_worker.sh,compare_lanes.py}` (`799ccb4`, `a1b9819`, `aeeca93`; round 2 `e35242f`: negative-control rows, placement, overlay and patch fixes) and `tests/unit/test_stage_admission_pp_tools.py` | | Test report | `test_report_2026-09-23_stage_admission_ordering.md` | | Calibration case | `calibration/stage_admission_case_001/` (manifest, inputs incl. `groundtruth_overlay.patch`, two vLLM runs, `analysis/`) | -| Evidence | `evidence/` (base negative controls, G2 comparisons, path-T explanation, Step 9 probe, co-execution decomposition script) | +| Evidence | `evidence/` (base negative controls, G2 comparisons, path-T explanations for rounds 1 and 2, tool tests on the old tools, Step 9 probe, co-execution decomposition script) | | Branch / PR | `fix/stage-admission-ordering`, draft PR https://github.com/NetX-lab/Frontier/pull/36 | ## Validation (observed) @@ -42,15 +44,33 @@ admitted ticket leaves the FIFO by `remove(ticket)`. | C6 | Step 9 probe shape MoE `attn_dp=2, moe_ep=2, PP=2` completes 6/6 (base drains) | | C7 | vLLM DP=2/PP=2 on 4×H800, run `sa-pp-20260923b`: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5, D-9). MoE: 26/26 rows MATCH. Dense: V1–V4 MATCH in every round. The base fails its negative controls: MoE deadlock, dense pairing and co-start. | +Round 2 (R-10, test report §8): + +| Check | Result | +| --- | --- | +| Rule refactor | `after` vs `after-r2`: 98/98 cases byte-identical | +| New groups G8–G11 (50) | PDD offline and online, co-location online, PD-AF `PREFILL_PP=2`: U 18 PASS, L 12 PASS (base deadlocks, conserved), T 12 PASS and 4 EXPLAIN (start times only); 0 STOP | +| G2 | no regression, no new failure, skips unchanged | +| C7 rerun (run b vs `after-r2`) | 56 rows: 50 MATCH, 2 INFORMATIONAL, 4 negative controls HOLDS, 0 MISMATCH | +| Harness and tools | timeout, lock, placement, overlay and patch checks verified; tool tests fail 7/9 on the old tools | + Decisions taken during execution: | Id | Decision | | --- | --- | | R-7 | The vLLM ground truth uses the four-argument `topk_softmax`, applied as a recorded overlay patch. The checkout is unchanged. | -| R-8 / D-9 | Witnesses are judged by co-execution fraction. V5 gates MoE only. | +| R-8 / D-9 | Witnesses are judged by co-execution fraction. V5 gates MoE only. Dense rationale restated at R-10: pre-exchange wait and duration variance, neither admission. | +| R-10 | Round-2 fixes as recommended; PDD and online cells added, plus PD-AF `PP > 1` controls. | ## Open and deferred work +- R2-06 / P6: before PR 36 merges, drop the `.gitignore` exception and + untrack this task directory (the parent task keeps copies). It deletes + tracked records, so it waits for the owner's go-ahead. +- On this branch, online Poisson arrivals reach only lane 0 of MONOLITHIC and + PREFILL contexts (PR 35 W2). PR 35's composition check after the + merge-forward reruns G9 and G10 with lane rotation, as well as G3b. + - PR 35 (`fix/issue26-correctness-pr`) merges this branch forward after it lands and reruns G3b as the composition check with W3. Only then does Step 9 resume (C6). - `PP=3` with `attn_dp=2` stays rejected by the node-size rule on the default backends (W9-02), outside this fix. - vLLM-BS: the fork's Python `topk_softmax` still passes five arguments. So does its test `tests/model_executor/test_enabled_custom_ops.py::test_topk_softmax_wrapper_forwards_renormalize`. The four-argument form was applied only as this case's overlay patch. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md b/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md index 239ff593..267c850f 100644 --- a/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md +++ b/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md @@ -4,6 +4,7 @@ | Date | Change | | --- | --- | +| 2026-09-23 | R-10 round-2 remediation: §8 added (rule refactor, PDD/online/PD-AF cells, tool fixes); §5.2 and §5.3 restated for the N rows and the barrier-aligned M5. | | 2026-09-23 | D-9 adopted ("采纳你的推荐，继续"): C3 witnesses judged by co-execution fraction, V5 gated on MoE only. Both comparisons rerun (`aeeca93`); all criteria pass. | | 2026-09-23 | Created. P0–P3 and P5 executed; two plan stop conditions reached (C3 witness metric at `attn_dp=4`, C7 V5 on the dense shape). P4 push held for the user's decision. | @@ -19,6 +20,11 @@ | C6 Step 9 probe | Informational: MoE `attn_dp=2, moe_ep=2, PP=2` completes 6/6 (base drains). `PP=3` stops on the known W9-02 node-size rejection. | §4.5 | | C7 vLLM comparison | PASS (D-9): 50 rows MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5). MoE matches on all 26 rows, V5 included; dense matches V1–V4 in every round. The negative controls fail on the base as planned. The first comparison stopped on dense V5; see §5.3. | §5 | +Round 2 (R-10, §8): the rule refactor keeps all 98 cases byte-identical; +50 new PDD, online and PD-AF cells pass (12 more base deadlocks repaired, +0 STOP); G2 shows no regression; the vLLM comparison rerun passes with the +negative controls as separate rows that hold. + Observed facts are separated from inferences. Inferences are marked "Inference". @@ -193,7 +199,8 @@ misplaced; `num_gpu_blocks` 600666 (MoE) and 304854 (dense). | Check | MoE n8 | MoE n16 | Dense n8 | Dense n16 | | --- | --- | --- | --- | --- | -| V1 completion | 3/3 MATCH; base `admission_deadlock` | 3/3 MATCH; base `admission_deadlock` | 3/3 MATCH | 3/3 MATCH | +| V1 completion | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | +| N1 / N4 base control (R-10) | N1 HOLDS: base `admission_deadlock` | N1 HOLDS: base `admission_deadlock` | N4 HOLDS: base co-start 1.0 | N4 HOLDS: base co-start 1.0 | | V2 lane sequences | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | | V3 stage-0 pairing | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH; base pairs 0↔3, 2↔5, …, 6↔none | 3/3 MATCH; base shifted by one forward | | V4 co-start (vLLM / after / base) | 0.009–0.063 / 0.0 / — | 0.005–0.024 / 0.0 / — | 0.008–0.248 / 0.0 / 1.0 | 0.046–0.171 / 0.0 / 1.0 | @@ -202,18 +209,36 @@ misplaced; `num_gpu_blocks` 600666 (MoE) and 304854 (dense). Run a (dense only, same scripts): V1–V4 all MATCH; V5 vLLM mean 0.714 (n8) and 0.685 (n16): MISMATCH under the first rule, INFORMATIONAL under D-9. +Until R-10 the base controls were folded into V1 and dense V4, so a base +without the defect would have turned those vLLM rows into MISMATCH (R2-05). +The rerun at R-10 (`--after after-r2`) writes them as rows N1 and N4: 56 rows, +50 MATCH, 2 INFORMATIONAL, 4 HOLDS, 0 MISMATCH; status PASS, +`negative_control_holds = true`, placement ok with 0 unseen requests. + ### 5.3 V5 on the dense shape `evidence/decompose_co_execution.py` splits the stage-0 non-overlap of each -M3 pair into `|Δstart| + |Δend|` -(`analysis/co_execution_decomposition_sa-pp-20260923{a,b}.json`). - -| Shape (run b) | vLLM M5 per round | Σ start offsets (ms) | Σ end offsets (ms) | stage-0 duration median (ms), CV | -| --- | --- | --- | --- | --- | -| MoE n8 | 0.977, 0.974, 0.977 | 0.32–0.54 | 0.19–0.20 | 5.3–8.5, 0.07–0.09 | -| MoE n16 | 0.978, 0.937, 0.928 | 0.70–3.83 | 0.26–0.69 | 5.3–7.5, 0.05–0.07 | -| Dense n8 | 0.657, 0.851, 0.609 | 1.49–2.63 | 0.26–4.40 | 3.0–3.8, 0.13–0.29 | -| Dense n16 | 0.926, 0.833, 0.837 | 1.02–3.60 | 0.60–2.81 | 2.6–2.7, 0.10–0.19 | +pair of overlapping forwards into `|Δstart| + |Δend|` +(`analysis/co_execution_decomposition_sa-pp-20260923{a,b}.json`). The identity +holds only for overlapping pairs; at R-10 the script counts disjoint pairs and +checks its pairing against M3. In every round of runs a and b there is no +disjoint pair and no unpaired forward, and the pairing equals M3. The last +column sets both starts of each pair to the later one (R-10, R2-02): vLLM +0.10.2 without CUDA graphs runs the per-forward DP metadata all-reduce inside +`set_forward_context`, after `forward_start_ts`, so neither rank computes +before the later one arrives. The traces carry no timestamp after that +exchange, so this column is derived, not measured. + +| Shape (run b) | vLLM M5 per round | Σ start offsets (ms) | Σ end offsets (ms) | stage-0 duration median (ms), CV | M5, starts aligned to the later one (derived) | +| --- | --- | --- | --- | --- | --- | +| MoE n8 | 0.977, 0.974, 0.977 | 0.32–0.54 | 0.19–0.20 | 5.3–8.5, 0.07–0.09 | 0.991, 0.991, 0.994 | +| MoE n16 | 0.978, 0.937, 0.928 | 0.70–3.83 | 0.26–0.69 | 5.3–7.5, 0.05–0.07 | 0.994, 0.988, 0.989 | +| Dense n8 | 0.657, 0.851, 0.609 | 1.49–2.63 | 0.26–4.40 | 3.0–3.8, 0.13–0.29 | 0.722, 0.977, 0.713 | +| Dense n16 | 0.926, 0.833, 0.837 | 1.02–3.60 | 0.60–2.81 | 2.6–2.7, 0.10–0.19 | 0.970, 0.972, 0.883 | + +Run a, dense, same columns: observed 0.642, 0.739, 0.760 (n8) and 0.537, +0.752, 0.767 (n16); starts aligned 0.739, 0.843, 0.782 and 0.656, 0.945, +0.950. Observed: @@ -222,19 +247,24 @@ Observed: differ by 0.18. - In every vLLM round the pairing (V3) and the one-to-one lane sequences (V2) match the after revision, and the first forwards co-start (V4). -- The non-overlap consists of per-pair start offsets of up to about 1.5 ms - (`forward_start_ts` is taken before the per-forward DP metadata exchange) - and end offsets from per-rank duration variation. +- The non-overlap consists of per-pair start offsets and end offsets. Start + offsets exceed end offsets in 3 of the 6 dense rounds of run b. +- With both starts aligned to the later one, M5 rises in every round (dense + 0.66–0.98, MoE 0.988–0.994). What remains in dense is the end offsets. - MoE ends align within 0.2–0.7 ms in total. -Inference: in MoE the EP collectives inside each forward hold the two ranks -together, so vLLM's co-execution is close to Frontier's 1.0. The dense ranks -meet once per forward and then run host-bound forwards of about 3 ms whose -durations vary per rank. The dummy predictor gives both lanes the same -duration, so Frontier's co-execution is exactly 1.0 whenever the lanes -co-start. The residual is a duration-variance property of the ground truth -that the dummy predictor does not model. It is not an admission difference: -admission is what V1–V4 measure, and they match. The dense base (0.600, +Inference (restated at R-10, R2-02): in MoE the EP collectives inside each +forward hold the two ranks together, so vLLM's co-execution is close to +Frontier's 1.0. The dense ranks meet once per forward, in the DP all-reduce. +The dense non-overlap has two sources. First, the rank that reaches the +all-reduce first records its wait as busy time, because `forward_start_ts` +precedes the exchange. Second, the host-bound forwards of about 3 ms vary in +duration per rank. Neither is an admission difference: both ranks enter the +same forward, which is what V1–V4 measure, and they match. The dummy +predictor models neither the wait nor the variation, so Frontier's +co-execution is exactly 1.0 whenever the lanes co-start. The first version of +this paragraph named only the duration variation; the start part was there +too. The dense base (0.600, 0.778) is numerically closer to vLLM only because base serialization removes overlap; its pairing (V3) and co-start (V4) are wrong in every round. @@ -243,8 +273,8 @@ overlap; its pairing (V3) and co-start (V4) are wrong in every round. the execution-time model instead. The first comparison stopped here with nothing adjusted. Under D-9 dense V5 is reported, not gated; the rerun gives `workflow_gap_status.json` status PASS with 0 mismatches. The P5a synthetic -check (`analysis/synthetic_check.py`) still flags its planted dummy-shifted -dense round through V3 and V4. +check, now `tests/unit/test_stage_admission_pp_tools.py` (R-10), still flags +a planted late-lane dense round through V3 and V4. ## 6. Decisions @@ -270,3 +300,87 @@ Both stops were resolved by D-9 (`plan.md`), adopted by the user on `tests/model_executor/test_enabled_custom_ops.py::test_topk_softmax_wrapper_forwards_renormalize`. - C6 was measured with a completion-only probe because the boundary seam is on PR 35; the PR 35 composition check is pending in the parent task. +- Round 2 (§8): on this branch, online Poisson arrivals reach only lane 0 of + MONOLITHIC and PREFILL contexts (PR 35's W2 defect on `main`); online + multi-lane coverage here comes from the `-burst` cells. The derived + barrier-aligned M5 of §5.3 is not a measurement. + +## 8. Round-2 remediation (R-10) + +Scope: `plan.md` §7; findings in `review.md` Round 2. Owner instruction: +"确认，执行上上述修复； R2-02 采纳你的推荐；R2-03需要补充 PDD+online（如果你认为pd-af+online有必要，请一并补充）". + +### 8.1 Commits and commands + +| Commit | Content | +| --- | --- | +| `1661bf1` | R2-01, R2-11, R2-13: `try_acquire` refactor, class docstring, unit test | +| `a8e8d8a` | R2-03, R2-07, R2-08: matrix groups G8–G11, cluster-keyed drain report, case timeout, set lock | +| `e35242f` | R2-02, R2-04, R2-05, R2-09, R2-10, R2-14, R2-15: comparison tools, tool unit tests, evidence scripts | + +```bash +# base for the new groups at a8e8d8a, with stage_execution_context.py replaced by +# its 1f694f7 version for the run (run.json: status "M frontier/.../stage_execution_context.py"), then restored +python -m tests.e2e.stage_admission_matrix run --set base --group G8 --group G9 --group G10 --group G11 --jobs 16 +python -m tests.e2e.stage_admission_matrix run --set after-r2 --jobs 16 # a8e8d8a, clean outside task_memory +python -m tests.e2e.stage_admission_matrix compare --before base --after after-r2 \ + --output /data/ycfeng/tmp/stage_admission_ordering/compare_base_after-r2.json +python task_memory/.../evidence/explain_t_path.py after-r2 /compare_base_after-r2.json \ + task_memory/.../evidence/r2_t_path_explanation.json +python -m pytest tests/ -q -p no:cacheprovider --continue-on-collection-errors \ + --junitxml=/after-r2-pytest/.xml +python -m tests.comparison.stage_admission_pp.compare_lanes \ + --vllm-run calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b \ + --before base --after after-r2 --output calibration/stage_admission_case_001/analysis +python task_memory/.../evidence/decompose_co_execution.py dense # and moe dense +``` + +Environment as §2; interpreter digest `ecd50ea8…` for every set. + +### 8.2 Results per finding + +| Finding | Check | Expected | Observed | Result | +| --- | --- | --- | --- | --- | +| R2-01, R2-11 | `try_acquire` on an active full-stage ticket, capacity 2 | `False`, context unchanged | base rule `False`; `dac4e69` raises `ValueError` ("not in deque"); `1661bf1` `False`, ticket still active, FIFO unchanged | PASS | +| R2-01, R2-11 | the three context unit files | pass, no assertion change | 181 passed | PASS | +| R2-01, R2-11 | set `after` vs `after-r2`, 98 cases | byte-identical | 97 success hash files identical; the configuration rejection has the same error (`identity_after_after-r2.json`) | PASS | +| R2-03 | groups G8–G11, 50 cases (§8.3) | §4.4 paths | U 18 PASS, L 12 PASS, T 16: 12 PASS, 4 EXPLAIN; 0 STOP | PASS | +| R2-07 | `run --case-timeout 2` on a recipe case | `other_failure`, set completes | `"case timeout after 2 s"` after 2 s; no child or simulator process left | PASS | +| R2-08 | a second `run` while one is running | fails at once | `RuntimeError: another set is running under …; sets share work/ and run one at a time`, exit 1 | PASS | +| R2-04, R2-05, R2-09, R2-10, R2-15 | `tests/unit/test_stage_admission_pp_tools.py` | pass; the ecff89a tools fail the new checks | 9 passed; on the ecff89a tools 7 failed, 2 passed (the late-lane round and the unexpected-file rejection, which the old tools already handled) (`evidence/r2_tool_tests_on_ecff89a.txt`) | PASS | +| R2-05 | `compare_lanes` rerun on run b vs `after-r2` | PASS with controls holding | 56 rows: 50 MATCH, 2 INFORMATIONAL, 4 HOLDS, 0 MISMATCH; `negative_control_holds = true` (§5.2) | PASS | +| R2-02 | decomposition rerun on runs a and b | identity stated for overlapping pairs; derived aligned M5 | no disjoint or unpaired forward in any round; pairing equals M3; existing fields unchanged; aligned M5 in §5.3 | done | +| R2-14 | `probe_completion.py` with `PYTHONPATH` only | C6 shapes complete | `moe_dp2_pp2` 6/6, `dense_dp1_pp2` 6/6; the probe's resolved config and `R0-moe-dp2-pp2-n6`'s differ only in `metrics_config` | PASS | +| G2 | `tests/unit`, `tests/integration` vs `base-pytest` | no regression | unit 84 failed / 3660 passed / 49 skipped / 10 errors (base 84 / 3644 / 49 / 10); integration 14 passed / 21 skipped / 5 errors (base 11 / 21 / 5). 0 regressions, 0 new failures, 0 skip changes; new node ids only: 16 unit, 3 integration (`evidence/r2_g2_*_compare.json`) | PASS | +| R2-12, R2-13 | documentation | contract stated | `design.md` round-2 note; PR body | done | +| R2-06 | pre-merge step P6 | recorded, not executed | `plan.md` §7 | open | + +### 8.3 New groups (R2-03) + +| Group | Cases | Base | `after-r2` | Paths | +| --- | --- | --- | --- | --- | +| G8 PDD offline | MoE `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}`, dense `dp1-pp2`, `n=8`, prefill 16 / decode 3 | 4 `admission_deadlock` (MoE `PP > 1`), 3 success | 7 success | U 3 PASS; L 4 PASS | +| G9 PDD online | the G8 shapes at Poisson 20/s; MoE `dp{2,4}-pp{2,3}` burst | 4 `admission_deadlock` (the burst cells), 7 success | 11 success | U 3 PASS; T 4 PASS, identical; L 4 PASS | +| G10 co-location online | MoE (prefill-only) and dense, `dp{2,4}-pp{1,2,3}`, Poisson 20/s and burst; `dp2-pp2` at 5/s and 80/s | 4 `admission_deadlock` (MoE burst `PP > 1`), 24 success | 28 success | U 8 PASS; T 12 PASS, identical, 4 EXPLAIN (dense burst `PP > 1`); L 4 PASS | +| G11 PD-AF | dense and MoE recipes, offline and online, `PREFILL_PP=2` | 4 success | 4 success | U 4 PASS | + +Every L case conserves requests and tokens (8 requests, 128 prefill tokens; +24 decode tokens for PD cells, 8 for prefill-only cells). The 4 EXPLAIN cases +(`evidence/r2_t_path_explanation.json`) run the same batches with the same +component durations as the base; only start times differ. Their co-execution +fraction goes from 0.818, 0.375, 0.846 and 0.846 to 1.0, `peak_lanes ≤ +attn_dp`, and no lane overlaps itself. These are the same values as the +offline G4 cells of the same shapes. + +Observed while running: in the Poisson online cells, every MONOLITHIC and +PREFILL forward runs on lane 0. PDD DECODE lanes all run. `_schedule_batch_mode` +numbers lanes from 0 within each scheduling call, and online arrivals come one +per call. That is PR 35's W2 defect on `main` (fixed on PR 35, not here). The +burst cells deliver all requests at `t=0` in online mode and reach every lane. +They carry the online L and T coverage on this branch. The Poisson cells show +that a multi-lane context with one live lane is unchanged. + +PD-AF: `DECODE_ATTN` requires `attn_dp = 1`, and PD-AF has one `attn_dp` for +every role, so no PD-AF context has more than one lane. The G11 cells are +capacity-1 `PREFILL` contexts at `PP = 2`, where R2-12's contract change +applies. They are byte-identical, offline and online. From 4d08c5d1c13850e42e5cb474721875507f8f99f5 Mon Sep 17 00:00:00 2001 From: fwyc0573 <935953068@qq.com> Date: Wed, 23 Sep 2026 13:28:34 +0800 Subject: [PATCH 19/19] chore: stop tracking the stage-admission task records before merge Restore the repository-wide task_memory/ ignore rule from main and untrack task_memory/task_2026-09-22_stage_admission_ordering, so that no task record reaches main (review finding R2-06, plan step P6). The parent task on fix/issue26-correctness-pr keeps the archive copies of the summary and the test report. The matrix module docstring now points at build_cases instead of the untracked plan file. --- .gitignore | 4 +- ...ecution_decomposition_sa-pp-20260923a.json | 86 - ...ecution_decomposition_sa-pp-20260923b.json | 170 - .../analysis/lane_metrics.json | 3568 ----------------- .../analysis/workflow_gap_status.json | 11 - .../analysis/workflow_gap_summary.md | 64 - .../analysis/workflow_gap_table.csv | 57 - .../stage_admission_case_001/case_init.md | 20 - .../inputs/fork_changed_files.txt | 61 - .../inputs/groundtruth_local_commit.diff | 412 -- .../inputs/groundtruth_overlay.patch | 24 - .../stage_admission_case_001/manifest.yaml | 155 - .../sa-pp-20260923a/COMPLETE | 1 - .../sa-pp-20260923a/overlay_report.json | 130 - .../sa-pp-20260923a/replica_log.txt | 73 - .../dense/dp_placement/dp_placement_588.jsonl | 22 - .../dense/dp_placement/dp_placement_663.jsonl | 65 - .../dense/dp_placement/dp_placement_664.jsonl | 65 - .../runs/dense/model/config.json | 95 - .../runs/dense/pp_boundary.jsonl | 152 - .../sa-pp-20260923a/runs/dense/requests.jsonl | 76 - .../sa-pp-20260923a/runs/dense/summary.json | 186 - .../runs/moe/model/config.json | 95 - .../sa-pp-20260923a/vllm_import.txt | 1 - .../sa-pp-20260923a/worker_env.json | 1 - .../sa-pp-20260923b/COMPLETE | 1 - .../sa-pp-20260923b/overlay_report.json | 142 - .../sa-pp-20260923b/replica_log.txt | 14 - .../dense/dp_placement/dp_placement_844.jsonl | 22 - .../dense/dp_placement/dp_placement_919.jsonl | 65 - .../dense/dp_placement/dp_placement_920.jsonl | 65 - .../runs/dense/model/config.json | 95 - .../runs/dense/pp_boundary.jsonl | 152 - .../sa-pp-20260923b/runs/dense/requests.jsonl | 76 - .../sa-pp-20260923b/runs/dense/summary.json | 186 - .../moe/dp_placement/dp_placement_157.jsonl | 22 - .../moe/dp_placement/dp_placement_232.jsonl | 65 - .../moe/dp_placement/dp_placement_233.jsonl | 65 - .../runs/moe/model/config.json | 95 - .../runs/moe/pp_boundary.jsonl | 152 - .../sa-pp-20260923b/runs/moe/requests.jsonl | 76 - .../sa-pp-20260923b/runs/moe/summary.json | 186 - .../sa-pp-20260923b/vllm_import.txt | 1 - .../sa-pp-20260923b/worker_env.json | 1 - .../design.md | 330 -- .../evidence/base_negative_controls.log | 59 - .../evidence/decompose_co_execution.py | 76 - .../evidence/explain_t_path.py | 71 - .../evidence/g2_integration_compare.json | 12 - .../evidence/g2_unit_compare.json | 15 - .../evidence/p3_t_path_explanation.json | 436 -- .../evidence/r2_g2_integration_compare.json | 12 - .../evidence/r2_g2_unit_compare.json | 25 - .../evidence/r2_t_path_explanation.json | 660 --- .../evidence/r2_tool_tests_on_ecff89a.txt | 8 - .../step9_probe/after_dense_dp1_pp2.json | 10 - .../step9_probe/after_moe_dp2_pp1.json | 10 - .../step9_probe/after_moe_dp2_pp2.json | 10 - .../step9_probe/after_moe_dp2_pp3.json | 9 - .../step9_probe/base_moe_dp2_pp2.json | 9 - .../evidence/step9_probe/probe_completion.py | 33 - .../evidence/step9_probe/probe_main.py | 178 - .../plan.md | 421 -- .../progress.md | 107 - .../requirements.md | 103 - .../review.md | 106 - .../review_prompt.md | 34 - .../summary.md | 77 - ...ort_2026-09-23_stage_admission_ordering.md | 386 -- tests/e2e/stage_admission_matrix.py | 7 +- 70 files changed, 4 insertions(+), 10275 deletions(-) delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/lane_metrics.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/case_init.md delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/fork_changed_files.txt delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_local_commit.diff delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/COMPLETE delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/overlay_report.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/replica_log.txt delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_588.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_663.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_664.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/model/config.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/pp_boundary.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/requests.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/summary.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/moe/model/config.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/vllm_import.txt delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/worker_env.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/COMPLETE delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/overlay_report.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/replica_log.txt delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_844.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_919.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_920.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/model/config.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/pp_boundary.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/requests.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/summary.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_157.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_232.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_233.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/model/config.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/pp_boundary.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/requests.jsonl delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/summary.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/vllm_import.txt delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/worker_env.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/design.md delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/base_negative_controls.log delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_integration_compare.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_unit_compare.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/p3_t_path_explanation.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_integration_compare.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_unit_compare.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_t_path_explanation.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_tool_tests_on_ecff89a.txt delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_dense_dp1_pp2.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp1.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp2.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp3.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/base_moe_dp2_pp2.json delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_completion.py delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/plan.md delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/progress.md delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/requirements.md delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/review.md delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/summary.md delete mode 100644 task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md diff --git a/.gitignore b/.gitignore index cffd05ad..0381dbf2 100644 --- a/.gitignore +++ b/.gitignore @@ -168,9 +168,7 @@ cache settings.json # Task memory and repair receipts are local-only; do not vendor them. -# The one exception below is reviewed on its branch, as PR 34/35 do. -task_memory/* -!task_memory/task_2026-09-22_stage_admission_ordering/ +task_memory/ repairs/ # Local linked worktrees used for isolated feature implementation. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json deleted file mode 100644 index 11dd7567..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923a.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "dense/n8/r0": { - "pairs": 4, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.6421, - "M5_equal_durations": 0.7101, - "M5_barrier_aligned": 0.7389, - "non_overlap_ms_from_start_offsets": 2.2, - "non_overlap_ms_from_end_offsets": 3.808, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 3.435, - "stage0_duration_ms_cv": 0.116 - }, - "dense/n8/r1": { - "pairs": 4, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.7393, - "M5_equal_durations": 0.7521, - "M5_barrier_aligned": 0.8432, - "non_overlap_ms_from_start_offsets": 1.805, - "non_overlap_ms_from_end_offsets": 2.014, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 3.222, - "stage0_duration_ms_cv": 0.183 - }, - "dense/n8/r2": { - "pairs": 4, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.7604, - "M5_equal_durations": 0.9323, - "M5_barrier_aligned": 0.7818, - "non_overlap_ms_from_start_offsets": 0.395, - "non_overlap_ms_from_end_offsets": 3.071, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 2.651, - "stage0_duration_ms_cv": 0.261 - }, - "dense/n16/r0": { - "pairs": 8, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.5372, - "M5_equal_durations": 0.5704, - "M5_barrier_aligned": 0.6561, - "non_overlap_ms_from_start_offsets": 12.44, - "non_overlap_ms_from_end_offsets": 11.9, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 4.561, - "stage0_duration_ms_cv": 0.16 - }, - "dense/n16/r1": { - "pairs": 8, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.752, - "M5_equal_durations": 0.6557, - "M5_barrier_aligned": 0.9449, - "non_overlap_ms_from_start_offsets": 5.729, - "non_overlap_ms_from_end_offsets": 1.232, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 3.032, - "stage0_duration_ms_cv": 0.162 - }, - "dense/n16/r2": { - "pairs": 8, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.7668, - "M5_equal_durations": 0.6756, - "M5_barrier_aligned": 0.9501, - "non_overlap_ms_from_start_offsets": 5.235, - "non_overlap_ms_from_end_offsets": 1.092, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 2.738, - "stage0_duration_ms_cv": 0.178 - } -} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json deleted file mode 100644 index 5c797f40..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/co_execution_decomposition_sa-pp-20260923b.json +++ /dev/null @@ -1,170 +0,0 @@ -{ - "moe/n8/r0": { - "pairs": 4, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.9769, - "M5_equal_durations": 0.9713, - "M5_barrier_aligned": 0.9912, - "non_overlap_ms_from_start_offsets": 0.321, - "non_overlap_ms_from_end_offsets": 0.194, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 5.312, - "stage0_duration_ms_cv": 0.066 - }, - "moe/n8/r1": { - "pairs": 4, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.9735, - "M5_equal_durations": 0.9648, - "M5_barrier_aligned": 0.9911, - "non_overlap_ms_from_start_offsets": 0.402, - "non_overlap_ms_from_end_offsets": 0.197, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 5.335, - "stage0_duration_ms_cv": 0.091 - }, - "moe/n8/r2": { - "pairs": 4, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.9774, - "M5_equal_durations": 0.9676, - "M5_barrier_aligned": 0.9937, - "non_overlap_ms_from_start_offsets": 0.536, - "non_overlap_ms_from_end_offsets": 0.204, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 8.45, - "stage0_duration_ms_cv": 0.088 - }, - "moe/n16/r0": { - "pairs": 8, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.9782, - "M5_equal_durations": 0.9686, - "M5_barrier_aligned": 0.994, - "non_overlap_ms_from_start_offsets": 0.698, - "non_overlap_ms_from_end_offsets": 0.26, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 5.294, - "stage0_duration_ms_cv": 0.067 - }, - "moe/n16/r1": { - "pairs": 8, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.9365, - "M5_equal_durations": 0.9029, - "M5_barrier_aligned": 0.9884, - "non_overlap_ms_from_start_offsets": 3.274, - "non_overlap_ms_from_end_offsets": 0.688, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 7.424, - "stage0_duration_ms_cv": 0.05 - }, - "moe/n16/r2": { - "pairs": 8, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.9283, - "M5_equal_durations": 0.8972, - "M5_barrier_aligned": 0.9885, - "non_overlap_ms_from_start_offsets": 3.827, - "non_overlap_ms_from_end_offsets": 0.682, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 7.483, - "stage0_duration_ms_cv": 0.055 - }, - "dense/n8/r0": { - "pairs": 4, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.6569, - "M5_equal_durations": 0.832, - "M5_barrier_aligned": 0.7216, - "non_overlap_ms_from_start_offsets": 1.487, - "non_overlap_ms_from_end_offsets": 4.198, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 3.052, - "stage0_duration_ms_cv": 0.291 - }, - "dense/n8/r1": { - "pairs": 4, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.8508, - "M5_equal_durations": 0.7401, - "M5_barrier_aligned": 0.9769, - "non_overlap_ms_from_start_offsets": 1.669, - "non_overlap_ms_from_end_offsets": 0.26, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 2.967, - "stage0_duration_ms_cv": 0.128 - }, - "dense/n8/r2": { - "pairs": 4, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.6089, - "M5_equal_durations": 0.6668, - "M5_barrier_aligned": 0.7132, - "non_overlap_ms_from_start_offsets": 2.627, - "non_overlap_ms_from_end_offsets": 4.399, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 3.752, - "stage0_duration_ms_cv": 0.19 - }, - "dense/n16/r0": { - "pairs": 8, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.9259, - "M5_equal_durations": 0.9111, - "M5_barrier_aligned": 0.9697, - "non_overlap_ms_from_start_offsets": 1.017, - "non_overlap_ms_from_end_offsets": 0.653, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 2.616, - "stage0_duration_ms_cv": 0.103 - }, - "dense/n16/r1": { - "pairs": 8, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.8332, - "M5_equal_durations": 0.7156, - "M5_barrier_aligned": 0.9722, - "non_overlap_ms_from_start_offsets": 3.596, - "non_overlap_ms_from_end_offsets": 0.599, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 2.615, - "stage0_duration_ms_cv": 0.182 - }, - "dense/n16/r2": { - "pairs": 8, - "unpaired_forwards": 0, - "pairing_equals_M3": true, - "disjoint_pairs": 0, - "M5_observed": 0.8371, - "M5_equal_durations": 0.8982, - "M5_barrier_aligned": 0.8832, - "non_overlap_ms_from_start_offsets": 1.324, - "non_overlap_ms_from_end_offsets": 2.807, - "non_overlap_ms_of_disjoint_pairs": 0.0, - "stage0_duration_ms_median": 2.676, - "stage0_duration_ms_cv": 0.193 - } -} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/lane_metrics.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/lane_metrics.json deleted file mode 100644 index 3333d51c..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/lane_metrics.json +++ /dev/null @@ -1,3568 +0,0 @@ -{ - "placement": { - "dense": { - "misplaced": [], - "ok": true, - "requests": 76, - "unseen": [] - }, - "moe": { - "misplaced": [], - "ok": true, - "requests": 76, - "unseen": [] - } - }, - "runs": { - "G7-dense-dp2-pp2-n16": { - "frontier_after": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.0, - "M5_co_execution": 1.0, - "median_forward_duration": 0.12, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.0, - "M5_co_execution": 1.0, - "median_forward_duration": 0.12, - "self_overlap": false - } - }, - "frontier_after_outcome": "success", - "frontier_after_placement_ok": true, - "frontier_base": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 3 - ] - ], - [ - [ - 2 - ], - [ - 5 - ] - ], - [ - [ - 4 - ], - [ - 7 - ] - ], - [ - [ - 6 - ], - [ - 9 - ] - ], - [ - [ - 8 - ], - [ - 11 - ] - ], - [ - [ - 10 - ], - [ - 13 - ] - ], - [ - [ - 12 - ], - [ - 15 - ] - ], - [ - [ - 14 - ], - null - ] - ], - "M4_co_start": 1.0000000000000002, - "M5_co_execution": 0.7777777777777778, - "median_forward_duration": 0.12, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 3 - ] - ], - [ - [ - 2 - ], - [ - 5 - ] - ], - [ - [ - 4 - ], - [ - 7 - ] - ], - [ - [ - 6 - ], - [ - 9 - ] - ], - [ - [ - 8 - ], - [ - 11 - ] - ], - [ - [ - 10 - ], - [ - 13 - ] - ], - [ - [ - 12 - ], - [ - 15 - ] - ], - [ - [ - 14 - ], - null - ] - ], - "M4_co_start": 1.0000000000000002, - "M5_co_execution": 0.7777777777777778, - "median_forward_duration": 0.12, - "self_overlap": false - } - }, - "frontier_base_outcome": "success", - "vllm": { - "0": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.17091455981041143, - "M5_co_execution": 0.9259308116414676, - "median_forward_duration": 0.00261572003364563, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.015250329608562718, - "M5_co_execution": 0.9556969257536658, - "median_forward_duration": 0.0029625799506902695, - "self_overlap": false - } - }, - "1": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.09309313158893653, - "M5_co_execution": 0.8331530258980444, - "median_forward_duration": 0.002615438774228096, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.08185507172547078, - "M5_co_execution": 0.7170726933331707, - "median_forward_duration": 0.003510111942887306, - "self_overlap": false - } - }, - "2": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.04588904145721386, - "M5_co_execution": 0.8370898411019174, - "median_forward_duration": 0.0026763956993818283, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.11631847952009687, - "M5_co_execution": 0.7891339057001696, - "median_forward_duration": 0.0030781766399741173, - "self_overlap": false - } - } - } - }, - "G7-dense-dp2-pp2-n8": { - "frontier_after": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.0, - "M5_co_execution": 1.0, - "median_forward_duration": 0.12000000000000001, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.0, - "M5_co_execution": 1.0, - "median_forward_duration": 0.12000000000000002, - "self_overlap": false - } - }, - "frontier_after_outcome": "success", - "frontier_after_placement_ok": true, - "frontier_base": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 3 - ] - ], - [ - [ - 2 - ], - [ - 5 - ] - ], - [ - [ - 4 - ], - [ - 7 - ] - ], - [ - [ - 6 - ], - null - ] - ], - "M4_co_start": 1.0, - "M5_co_execution": 0.6, - "median_forward_duration": 0.12000000000000001, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 3 - ] - ], - [ - [ - 2 - ], - [ - 5 - ] - ], - [ - [ - 4 - ], - [ - 7 - ] - ], - [ - [ - 6 - ], - null - ] - ], - "M4_co_start": 0.9999999999999999, - "M5_co_execution": 0.6000000000000001, - "median_forward_duration": 0.12000000000000002, - "self_overlap": false - } - }, - "frontier_base_outcome": "success", - "vllm": { - "0": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.008418151150125628, - "M5_co_execution": 0.6568505845035172, - "median_forward_duration": 0.0030516916885972023, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.04170587227710599, - "M5_co_execution": 0.6089605761452099, - "median_forward_duration": 0.004431265406310558, - "self_overlap": false - } - }, - "1": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.04251974485265244, - "M5_co_execution": 0.8508060484104523, - "median_forward_duration": 0.002966976724565029, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.02229866292638622, - "M5_co_execution": 0.6404995388202928, - "median_forward_duration": 0.004050368443131447, - "self_overlap": false - } - }, - "2": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.24755095323041817, - "M5_co_execution": 0.6088974398480734, - "median_forward_duration": 0.0037516485899686813, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.37008108266775924, - "M5_co_execution": 0.6159455220656315, - "median_forward_duration": 0.0037597408518195152, - "self_overlap": false - } - } - } - }, - "G7-moe-dp2-pp2-n16": { - "frontier_after": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.0, - "M5_co_execution": 1.0, - "median_forward_duration": 0.062000000000000055, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.0, - "M5_co_execution": 1.0, - "median_forward_duration": 0.062000000000000055, - "self_overlap": false - } - }, - "frontier_after_outcome": "success", - "frontier_after_placement_ok": true, - "frontier_base": null, - "frontier_base_outcome": "admission_deadlock", - "vllm": { - "0": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.023619782508997624, - "M5_co_execution": 0.9782382605035794, - "median_forward_duration": 0.005294156260788441, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.01174241455626505, - "M5_co_execution": 0.9788841920364375, - "median_forward_duration": 0.005645160563290119, - "self_overlap": false - } - }, - "1": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.004534493121171677, - "M5_co_execution": 0.9365388912624988, - "median_forward_duration": 0.007423891685903072, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.025260268381604473, - "M5_co_execution": 0.9713419954163164, - "median_forward_duration": 0.005722890608012676, - "self_overlap": false - } - }, - "2": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ], - [ - 8 - ], - [ - 10 - ], - [ - 12 - ], - [ - 14 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ], - [ - 9 - ], - [ - 11 - ], - [ - 13 - ], - [ - 15 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.022678808030265958, - "M5_co_execution": 0.9283263327604807, - "median_forward_duration": 0.0074830856174230576, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ], - [ - [ - 8 - ], - [ - 9 - ] - ], - [ - [ - 10 - ], - [ - 11 - ] - ], - [ - [ - 12 - ], - [ - 13 - ] - ], - [ - [ - 14 - ], - [ - 15 - ] - ] - ], - "M4_co_start": 0.0778888599864643, - "M5_co_execution": 0.9677368487359697, - "median_forward_duration": 0.005574577488005161, - "self_overlap": false - } - } - } - }, - "G7-moe-dp2-pp2-n8": { - "frontier_after": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.0, - "M5_co_execution": 1.0, - "median_forward_duration": 0.062000000000000055, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.0, - "M5_co_execution": 1.0, - "median_forward_duration": 0.062000000000000055, - "self_overlap": false - } - }, - "frontier_after_outcome": "success", - "frontier_after_placement_ok": true, - "frontier_base": null, - "frontier_base_outcome": "admission_deadlock", - "vllm": { - "0": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.00864423422319438, - "M5_co_execution": 0.9769112551681639, - "median_forward_duration": 0.005312402732670307, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.044595115345557684, - "M5_co_execution": 0.9602193830355867, - "median_forward_duration": 0.007738551124930382, - "self_overlap": false - } - }, - "1": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.035494719435994984, - "M5_co_execution": 0.9734773526983487, - "median_forward_duration": 0.005334779620170593, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.029321800955295463, - "M5_co_execution": 0.9702129108716677, - "median_forward_duration": 0.007605820894241333, - "self_overlap": false - } - }, - "2": { - "M2_sequences": { - "lane0/stage0": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane0/stage1": [ - [ - 0 - ], - [ - 2 - ], - [ - 4 - ], - [ - 6 - ] - ], - "lane1/stage0": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ], - "lane1/stage1": [ - [ - 1 - ], - [ - 3 - ], - [ - 5 - ], - [ - 7 - ] - ] - }, - "stage0": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.06341533360159404, - "M5_co_execution": 0.9774133792071, - "median_forward_duration": 0.00845013465732336, - "self_overlap": false - }, - "stage1": { - "M3_pairing": [ - [ - [ - 0 - ], - [ - 1 - ] - ], - [ - [ - 2 - ], - [ - 3 - ] - ], - [ - [ - 4 - ], - [ - 5 - ] - ], - [ - [ - 6 - ], - [ - 7 - ] - ] - ], - "M4_co_start": 0.01704118680493941, - "M5_co_execution": 0.968491350994991, - "median_forward_duration": 0.0076489923521876335, - "self_overlap": false - } - } - } - } - } -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json deleted file mode 100644 index 5bdd6f2a..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_status.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "analysis_state": "COMPLETE", - "status": "PASS", - "correction_state": "not_applicable", - "rows": 56, - "mismatches": 0, - "vllm_placement_ok": true, - "vllm_placement_unseen_requests": 0, - "negative_control_holds": true, - "next_action": "record C7 in the test report" -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md deleted file mode 100644 index d14c01b9..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_summary.md +++ /dev/null @@ -1,64 +0,0 @@ -# Workflow-gap summary — stage_admission_case_001 - -## Modification History - -| Date | Change | -| --- | --- | -| 2026-09-23 | R-10: rerun against `after-r2` with the negative controls as rows N1/N4; dense V5 cause restated with both sources and the derived barrier-aligned M5. | -| 2026-09-23 | D-9 adopted: dense V5 reported, not gated; rerun status PASS, 0 mismatches. | -| 2026-09-23 | Created from run `sa-pp-20260923b` against Frontier sets `base` (`1f694f7` rule) and `after` (`dac4e69`). | - -## Inputs - -| Side | Source | -| --- | --- | -| vLLM | `runs/vllm-instrumented/sa-pp-20260923b/` — vLLM-BS `494b9f327` plus `inputs/groundtruth_overlay.patch` (SHA-256 `8d476789…3a9c81`), DP=2, PP=2, TP=1, MoE with EP, 4×H800 | -| Frontier before | `/data/ycfeng/tmp/stage_admission_ordering/base/G7-*` | -| Frontier after | `/data/ycfeng/tmp/stage_admission_ordering/after-r2/G7-*` (R-10; byte-identical to set `after` at `dac4e69`) | -| Producer | `tests/comparison/stage_admission_pp/compare_lanes.py` → `workflow_gap_table.csv`, `lane_metrics.json`, `workflow_gap_status.json` | - -## Result - -56 rows: 50 `MATCH`, 0 `MISMATCH`, 2 `INFORMATIONAL` (dense V5 under D-9), -4 `HOLDS` (negative controls); status PASS, `negative_control_holds = true`. -`vllm_placement_ok = true`, no unseen request. Before R-10 the controls were -folded into V1 and dense V4 (52 rows). - -| Metric | MoE (n8, n16) | Dense (n8, n16) | -| --- | --- | --- | -| V1 completion | MATCH in 6/6 rounds | MATCH in 6/6 rounds | -| N1 / N4 base control | N1 HOLDS (n8, n16): base `admission_deadlock` | N4 HOLDS (n8, n16): base co-start 1.0 | -| V2 lane sequences | MATCH 6/6 | MATCH 6/6 | -| V3 stage-0 pairing | MATCH 6/6 | MATCH 6/6; base pairs are shifted by one forward | -| V4 first-forward co-start | MATCH 6/6 (vLLM ≤ 0.063, after 0.0) | MATCH 6/6 (vLLM ≤ 0.248, after 0.0) | -| V5 stage-0 co-execution | MATCH: vLLM 0.976 / 0.948, after 1.0 | INFORMATIONAL (D-9): vLLM 0.706 / 0.865, after 1.0, base 0.600 / 0.778 | - -## Dense V5 (MISMATCH before D-9) - -- Observed: vLLM's dense co-execution varies from round to round by more than - the 0.10 bound: 0.537–0.926 over the 12 dense rounds of runs a and b. The - n16 means of the two runs differ by 0.18. -- Observed (`co_execution_decomposition_sa-pp-20260923{a,b}.json`): the dense - non-overlap has two sources. Every pair overlaps, so it splits exactly. - - Per-pair start offsets. `forward_start_ts` is taken before the - per-forward DP metadata all-reduce, so the rank that arrives first - records its wait as busy time. Start offsets exceed end offsets in 3 of - the 6 dense rounds of run b. - - End offsets from per-rank duration variation: CV 0.10–0.29 on forwards of - about 3 ms. - MoE ends stay within 0.2–0.7 ms in total. -- Derived, not measured (R-10): with both starts of each pair set to the - later one, where the all-reduce releases both ranks, M5 is 0.66–0.98 for - dense and 0.988–0.994 for MoE, higher than observed in every round. The - traces carry no timestamp after the exchange. -- Inference (restated at R-10): neither source is an admission difference. - Both ranks enter the same forward, which V1–V4 measure, and they match in - every round. The Frontier owner of the dense gap is the execution-time - model: the dummy predictor models neither the pre-exchange wait nor the - duration variation. It is not `stage_execution_context.py`, the default - owner label written into the table. The first version of this inference - named only the duration variation. -- The first analysis stopped here with nothing adjusted. The user adopted - D-9: V5 is informational for the dense shape and stays a gate for MoE. - C7 rests on V1–V4 for both models, V5 for MoE, and the base negative - controls. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv deleted file mode 100644 index cd61d56f..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/analysis/workflow_gap_table.csv +++ /dev/null @@ -1,57 +0,0 @@ -check,model,burst,round,metric,groundtruth,frontier_after,frontier_base,status,frontier_owner,note -N1,moe,8,base,base outcome,null,"""success""","""admission_deadlock""",HOLDS,,expected admission_deadlock -V1,moe,8,0,M1 completion,"""8/8""","""success""","""admission_deadlock""",MATCH,, -V2,moe,8,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",null,MATCH,, -V3,moe,8,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]",null,MATCH,, -V4,moe,8,0,M4 stage-0 co-start,0.00864423422319438,0.0,null,MATCH,, -V1,moe,8,1,M1 completion,"""8/8""","""success""","""admission_deadlock""",MATCH,, -V2,moe,8,1,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",null,MATCH,, -V3,moe,8,1,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]",null,MATCH,, -V4,moe,8,1,M4 stage-0 co-start,0.035494719435994984,0.0,null,MATCH,, -V1,moe,8,2,M1 completion,"""8/8""","""success""","""admission_deadlock""",MATCH,, -V2,moe,8,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",null,MATCH,, -V3,moe,8,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]",null,MATCH,, -V4,moe,8,2,M4 stage-0 co-start,0.06341533360159404,0.0,null,MATCH,, -V5,moe,8,mean,M5 stage-0 co-execution,0.9759339956912042,1.0,null,MATCH,, -N1,moe,16,base,base outcome,null,"""success""","""admission_deadlock""",HOLDS,,expected admission_deadlock -V1,moe,16,0,M1 completion,"""16/16""","""success""","""admission_deadlock""",MATCH,, -V2,moe,16,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",null,MATCH,, -V3,moe,16,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]",null,MATCH,, -V4,moe,16,0,M4 stage-0 co-start,0.023619782508997624,0.0,null,MATCH,, -V1,moe,16,1,M1 completion,"""16/16""","""success""","""admission_deadlock""",MATCH,, -V2,moe,16,1,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",null,MATCH,, -V3,moe,16,1,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]",null,MATCH,, -V4,moe,16,1,M4 stage-0 co-start,0.004534493121171677,0.0,null,MATCH,, -V1,moe,16,2,M1 completion,"""16/16""","""success""","""admission_deadlock""",MATCH,, -V2,moe,16,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",null,MATCH,, -V3,moe,16,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]",null,MATCH,, -V4,moe,16,2,M4 stage-0 co-start,0.022678808030265958,0.0,null,MATCH,, -V5,moe,16,mean,M5 stage-0 co-execution,0.947701161508853,1.0,null,MATCH,, -N4,dense,8,base,M4 stage-0 co-start,null,0.0,1.0,HOLDS,,expected >= 0.5 -V1,dense,8,0,M1 completion,"""8/8""","""success""","""success""",MATCH,, -V2,dense,8,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",MATCH,, -V3,dense,8,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], null]]",MATCH,, -V4,dense,8,0,M4 stage-0 co-start,0.008418151150125628,0.0,1.0,MATCH,, -V1,dense,8,1,M1 completion,"""8/8""","""success""","""success""",MATCH,, -V2,dense,8,1,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",MATCH,, -V3,dense,8,1,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], null]]",MATCH,, -V4,dense,8,1,M4 stage-0 co-start,0.04251974485265244,0.0,1.0,MATCH,, -V1,dense,8,2,M1 completion,"""8/8""","""success""","""success""",MATCH,, -V2,dense,8,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}","{""lane0/stage0"": [[0], [2], [4], [6]], ""lane0/stage1"": [[0], [2], [4], [6]], ""lane1/stage0"": [[1], [3], [5], [7]], ""lane1/stage1"": [[1], [3], [5], [7]]}",MATCH,, -V3,dense,8,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], null]]",MATCH,, -V4,dense,8,2,M4 stage-0 co-start,0.24755095323041817,0.0,1.0,MATCH,, -V5,dense,8,mean,M5 stage-0 co-execution,0.7055180242540143,1.0,0.6,INFORMATIONAL,, -N4,dense,16,base,M4 stage-0 co-start,null,0.0,1.0000000000000002,HOLDS,,expected >= 0.5 -V1,dense,16,0,M1 completion,"""16/16""","""success""","""success""",MATCH,, -V2,dense,16,0,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",MATCH,, -V3,dense,16,0,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], [9]], [[8], [11]], [[10], [13]], [[12], [15]], [[14], null]]",MATCH,, -V4,dense,16,0,M4 stage-0 co-start,0.17091455981041143,0.0,1.0000000000000002,MATCH,, -V1,dense,16,1,M1 completion,"""16/16""","""success""","""success""",MATCH,, -V2,dense,16,1,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",MATCH,, -V3,dense,16,1,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], [9]], [[8], [11]], [[10], [13]], [[12], [15]], [[14], null]]",MATCH,, -V4,dense,16,1,M4 stage-0 co-start,0.09309313158893653,0.0,1.0000000000000002,MATCH,, -V1,dense,16,2,M1 completion,"""16/16""","""success""","""success""",MATCH,, -V2,dense,16,2,M2 lane sequences,"{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}","{""lane0/stage0"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane0/stage1"": [[0], [2], [4], [6], [8], [10], [12], [14]], ""lane1/stage0"": [[1], [3], [5], [7], [9], [11], [13], [15]], ""lane1/stage1"": [[1], [3], [5], [7], [9], [11], [13], [15]]}",MATCH,, -V3,dense,16,2,M3 stage-0 pairing,"[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [1]], [[2], [3]], [[4], [5]], [[6], [7]], [[8], [9]], [[10], [11]], [[12], [13]], [[14], [15]]]","[[[0], [3]], [[2], [5]], [[4], [7]], [[6], [9]], [[8], [11]], [[10], [13]], [[12], [15]], [[14], null]]",MATCH,, -V4,dense,16,2,M4 stage-0 co-start,0.04588904145721386,0.0,1.0000000000000002,MATCH,, -V5,dense,16,mean,M5 stage-0 co-execution,0.8653912262138098,1.0,0.7777777777777778,INFORMATIONAL,, diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/case_init.md b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/case_init.md deleted file mode 100644 index 3aea24cb..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/case_init.md +++ /dev/null @@ -1,20 +0,0 @@ -# case_init — stage_admission_case_001 - -Immutable record. Written 2026-09-23, before the first GPU run. - -| Field | Value | -| --- | --- | -| `case_id` | `stage_admission_case_001` | -| `run_generation` | 1 | -| `requesting_user` | `i-fengyicheng` | -| `reviewer_identity` | `i-fengyicheng` | -| `auto_recycle` | `false` | -| Ground-truth checkout | `/data/ycfeng/Frontier/.real-engine/vLLM-BS` | -| Ground-truth branch | `feature/frontier-comparison-instrumentation` | -| Ground-truth commit | `494b9f327036d4493034a9b37ebb343354884e01` | -| Ground-truth remote tip | `ea95f571e20937c7c908c6d59ddd1cd6bf9268f1` (local is one unpushed commit ahead) | -| Tree dirty | `false` | -| Diff artifact | `inputs/groundtruth_local_commit.diff`, SHA-256 `84fc24db0e2411268a93f8be7ca5f8e4e5072ea09d86063cac2cfb98381feb2c` | -| Fork changes over `upstream-v0.10.2` (`01efc7ef7`) | `inputs/fork_changed_files.txt`, SHA-256 `be638438ff661e1178f0c5ab68da7b9d249ebf8e641dcb1f6cd15d128c0abdaa` | -| Weight mode | `dummy`, no real weight download | -| Mode | instrumented only; no clean E2E run (plan D-8) | diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/fork_changed_files.txt b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/fork_changed_files.txt deleted file mode 100644 index 893d3620..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/fork_changed_files.txt +++ /dev/null @@ -1,61 +0,0 @@ -vllm/_C.py -vllm/_custom_ops.py -vllm/_moe_C.py -vllm/attention/layer.py -vllm/benchmarks/throughput.py -vllm/compilation/compiler_interface.py -vllm/config/__init__.py -vllm/distributed/communication_op.py -vllm/distributed/kv_transfer/kv_connector/v1/base.py -vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py -vllm/distributed/kv_transfer/kv_connector/v1/p2p/frontier_kv_transfer_logger.py -vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py -vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py -vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py -vllm/distributed/parallel_state.py -vllm/engine/arg_utils.py -vllm/engine/llm_engine.py -vllm/entrypoints/openai/frontier_request_metrics.py -vllm/entrypoints/openai/serving_chat.py -vllm/entrypoints/openai/serving_completion.py -vllm/entrypoints/openai/serving_engine.py -vllm/envs.py -vllm/model_executor/custom_op.py -vllm/model_executor/layers/fused_moe/configs/specific-README -vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py -vllm/model_executor/layers/fused_moe/deepep_ll_prepare_finalize.py -vllm/model_executor/layers/fused_moe/fused_moe.py -vllm/model_executor/layers/fused_moe/layer.py -vllm/model_executor/layers/fused_moe/pplx_prepare_finalize.py -vllm/model_executor/layers/linear.py -vllm/model_executor/layers/vocab_parallel_embedding.py -vllm/model_executor/models/llama.py -vllm/model_executor/models/phimoe.py -vllm/model_executor/models/qwen3_moe.py -vllm/model_executor/models/qwen3_moe_mtp.py -vllm/model_executor/models/registry.py -vllm/request_generator/__init__.py -vllm/request_generator/config.py -vllm/request_generator/kv_sync.py -vllm/request_generator/plan.md -vllm/request_generator/prompt_generator.py -vllm/request_generator/vllm_request_generator.py -vllm/v1/attention/backends/flash_attn.py -vllm/v1/attention/backends/flashinfer.py -vllm/v1/attention/backends/mla/common.py -vllm/v1/attention/backends/mla/flashinfer_mla.py -vllm/v1/attention/backends/utils.py -vllm/v1/core/sched/scheduler.py -vllm/v1/engine/coordinator.py -vllm/v1/engine/core.py -vllm/v1/engine/core_client.py -vllm/v1/engine/output_processor.py -vllm/v1/engine/processor.py -vllm/v1/frontier_trace.py -vllm/v1/metrics/stats.py -vllm/v1/spec_decode/eagle.py -vllm/v1/utils.py -vllm/v1/worker/gpu_model_runner.py -vllm/v1/worker/gpu_worker.py -vllm/worker/model_runner.py -vllm/worker/worker.py diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_local_commit.diff b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_local_commit.diff deleted file mode 100644 index 5d788f9a..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_local_commit.diff +++ /dev/null @@ -1,412 +0,0 @@ -diff --git a/vllm/v1/engine/coordinator.py b/vllm/v1/engine/coordinator.py -index 596edfdbe..1fd9ad16f 100644 ---- a/vllm/v1/engine/coordinator.py -+++ b/vllm/v1/engine/coordinator.py -@@ -12,6 +12,7 @@ import zmq - from vllm.config import ParallelConfig - from vllm.logger import init_logger - from vllm.utils import get_mp_context, make_zmq_socket, set_process_title -+from vllm.v1 import frontier_trace - from vllm.v1.engine import EngineCoreOutputs, EngineCoreRequestType - from vllm.v1.serial_utils import MsgpackDecoder - from vllm.v1.utils import get_engine_client_zmq_addr, shutdown -@@ -157,6 +158,10 @@ class DPCoordinatorProc: - last_stats_wave = -1 - last_step_counts: Optional[list[list[int]]] = None - -+ # Identifies each set of counts sent to the front ends, so a placement -+ # can be traced back to the engine reports it was computed from. -+ snapshot_id = 0 -+ - with make_zmq_socket( - path=front_publish_address, # IPC - ctx=self.ctx, -@@ -208,12 +213,23 @@ class DPCoordinatorProc: - if last_step_counts is not None: - engine_req_counts_list = last_step_counts - last_step_counts = None -+ counts_source = "latched_previous_step" - else: - engine_req_counts_list = self._get_engine_counts() - stats_changed = False -+ counts_source = "current" -+ -+ snapshot_id += 1 -+ frontier_trace.log_dp_placement_record( -+ "coordinator_publish", -+ snapshot=snapshot_id, -+ counts=engine_req_counts_list, -+ counts_source=counts_source, -+ wave=current_wave, -+ engines_running=engines_running) - - to_publish = (engine_req_counts_list, current_wave, -- engines_running) -+ engines_running, snapshot_id) - publish_front.send(msgspec.msgpack.encode(to_publish)) - last_publish_time = int(time.time() * 1000) - continue -@@ -290,21 +306,35 @@ class DPCoordinatorProc: - stats = self.engines[eng_index].request_counts - stats_step = scheduler_stats.step_counter - stats_wave = scheduler_stats.current_wave -+ disposition = "applied_without_latch" - if (stats_wave > last_stats_wave - or stats_wave == last_stats_wave - and stats_step > last_stats_step): - if stats_changed: - last_step_counts = self._get_engine_counts( - do_copy=True) -+ disposition = "latched_previous_step" -+ else: -+ disposition = "advanced_without_latch" - last_stats_step = stats_step - last_stats_wave = stats_wave - elif stats_wave != last_stats_wave or ( - stats_step != last_stats_step): -+ disposition = "out_of_order" - logger.warning( - "Received stats for out-of-order " - "step (%d, %d) from engine %d (expected " - "> (%d, %d))", stats_wave, stats_step, - eng_index, last_stats_wave, last_stats_step) -+ frontier_trace.log_dp_placement_record( -+ "coordinator_receive", -+ engine=eng_index, -+ wave=stats_wave, -+ step=stats_step, -+ waiting=scheduler_stats.num_waiting_reqs, -+ running=scheduler_stats.num_running_reqs, -+ disposition=disposition, -+ latched_counts=last_step_counts) - stats[0] = scheduler_stats.num_waiting_reqs - stats[1] = scheduler_stats.num_running_reqs - stats_changed = True -@@ -335,7 +365,8 @@ class DPCoordinatorProc: - self._send_start_wave(publish_back, wave, eng_index) - - if wave_state_changed: -- message = (None, current_wave, engines_running) -+ message = (None, current_wave, engines_running, -+ snapshot_id) - publish_front.send(msgspec.msgpack.encode(message)) - - @staticmethod -diff --git a/vllm/v1/engine/core.py b/vllm/v1/engine/core.py -index bfc29fc78..c61236776 100644 ---- a/vllm/v1/engine/core.py -+++ b/vllm/v1/engine/core.py -@@ -45,6 +45,7 @@ from vllm.v1.engine.utils import (EngineHandshakeMetadata, EngineZmqAddresses, - from vllm.v1.executor.abstract import Executor - from vllm.v1.kv_cache_interface import KVCacheConfig - from vllm.v1.metrics.stats import SchedulerStats -+from vllm.v1 import frontier_trace - from vllm.v1.outputs import ModelRunnerOutput - from vllm.v1.request import Request, RequestStatus - from vllm.v1.serial_utils import MsgpackDecoder, MsgpackEncoder -@@ -156,6 +157,12 @@ class EngineCore: - self.batch_queue_size) - self.batch_queue = deque(maxlen=self.batch_queue_size) - -+ # Classification of the most recent iteration, written by the step -+ # methods and consumed by the data-parallel busy loop, which is where -+ # the wave, step counter and published counts are known. Stays None -+ # while Frontier placement logging is off. -+ self.frontier_iteration: Optional[dict[str, Any]] = None -+ - self.request_block_hasher: Optional[Callable[[Request], - list[BlockHash]]] = None - if (self.vllm_config.cache_config.enable_prefix_caching -@@ -305,9 +312,48 @@ class EngineCore: - engine_core_outputs = self.scheduler.update_from_output( - scheduler_output, model_output) # type: ignore - -+ self._record_frontier_iteration(scheduler_output, -+ applied_output=True, -+ queue_occupancy=0) -+ - return (engine_core_outputs, - scheduler_output.total_num_scheduled_tokens > 0) - -+ def _record_frontier_iteration(self, -+ scheduled_output: Optional[SchedulerOutput], -+ *, applied_output: bool, -+ queue_occupancy: int) -> None: -+ """Classify one engine iteration for Frontier placement analysis. -+ -+ Which branch an iteration takes is what decides whether it publishes -+ request counts on its own, so the branch is recorded rather than -+ inferred later from the counts. -+ """ -+ if not frontier_trace.is_dp_placement_logging_enabled(): -+ return -+ -+ if scheduled_output is None: -+ branch = "applied_without_scheduling" -+ elif applied_output: -+ branch = "applied_after_scheduling" -+ else: -+ branch = "scheduled_without_applying" -+ -+ self.frontier_iteration = { -+ "branch": -+ branch, -+ "applied_output": -+ applied_output, -+ "queue_occupancy": -+ queue_occupancy, -+ "scheduled_new_req_ids": -+ [data.req_id for data in scheduled_output.scheduled_new_reqs] -+ if scheduled_output is not None else [], -+ "num_scheduled_tokens": -+ scheduled_output.total_num_scheduled_tokens -+ if scheduled_output is not None else 0, -+ } -+ - def post_step(self, model_executed: bool) -> None: - if self.use_spec_decode and model_executed: - # Take the draft token ids. -@@ -339,17 +385,22 @@ class EngineCore: - assert len(batch_queue) < self.batch_queue_size - - model_executed = False -+ scheduled_output: Optional[SchedulerOutput] = None - if self.scheduler.has_requests(): -- scheduler_output = self.scheduler.schedule() -- future = self.model_executor.execute_model(scheduler_output) -+ scheduled_output = self.scheduler.schedule() -+ future = self.model_executor.execute_model(scheduled_output) - batch_queue.appendleft( -- (future, scheduler_output)) # type: ignore[arg-type] -+ (future, scheduled_output)) # type: ignore[arg-type] - -- model_executed = scheduler_output.total_num_scheduled_tokens > 0 -+ model_executed = scheduled_output.total_num_scheduled_tokens > 0 - if model_executed and len(batch_queue) < self.batch_queue_size \ - and not batch_queue[-1][0].done(): - # Don't block on next worker response unless the queue is full - # or there are no more requests to schedule. -+ self._record_frontier_iteration( -+ scheduled_output, -+ applied_output=False, -+ queue_occupancy=len(batch_queue)) - return None, True - - elif not batch_queue: -@@ -366,6 +417,10 @@ class EngineCore: - engine_core_outputs = self.scheduler.update_from_output( - scheduler_output, model_output) - -+ self._record_frontier_iteration(scheduled_output, -+ applied_output=True, -+ queue_occupancy=len(batch_queue)) -+ - return engine_core_outputs, model_executed - - def shutdown(self): -@@ -1072,9 +1127,10 @@ class DPEngineCoreProc(EngineCoreProc): - else: - super()._handle_client_request(request_type, request) - -- def _maybe_publish_request_counts(self): -+ def _maybe_publish_request_counts(self) -> bool: -+ """Returns whether this iteration published its request counts.""" - if not self.publish_dp_lb_stats: -- return -+ return False - - # Publish our request counts (if they've changed). - counts = self.scheduler.get_request_counts() -@@ -1085,6 +1141,31 @@ class DPEngineCoreProc(EngineCoreProc): - current_wave=self.current_wave) - self.output_queue.put_nowait( - (-1, EngineCoreOutputs(scheduler_stats=stats))) -+ return True -+ return False -+ -+ def _log_frontier_iteration(self, published: bool) -> None: -+ """Emit the iteration the step methods classified. -+ -+ `step_counter` is read before `_has_global_unfinished_reqs` advances -+ it, so it is the same value a published report carried, which makes -+ `(engine, wave, step)` the join for the whole placement chain. -+ """ -+ iteration = self.frontier_iteration -+ if iteration is None: -+ return -+ self.frontier_iteration = None -+ -+ num_running_reqs, num_waiting_reqs = self.scheduler.get_request_counts() -+ frontier_trace.log_dp_placement_record( -+ "engine_iteration", -+ engine=self.dp_rank, -+ wave=self.current_wave, -+ step=self.step_counter, -+ waiting=num_waiting_reqs, -+ running=num_running_reqs, -+ published=published, -+ **iteration) - - def run_busy_loop(self): - """Core busy loop of the EngineCore for data parallel case.""" -@@ -1096,7 +1177,7 @@ class DPEngineCoreProc(EngineCoreProc): - - # 2) Step the engine core. - executed = self._process_engine_step() -- self._maybe_publish_request_counts() -+ self._log_frontier_iteration(self._maybe_publish_request_counts()) - - local_unfinished_reqs = self.scheduler.has_unfinished_requests() - if not executed: -diff --git a/vllm/v1/engine/core_client.py b/vllm/v1/engine/core_client.py -index 605bedaf1..daf8a67ef 100644 ---- a/vllm/v1/engine/core_client.py -+++ b/vllm/v1/engine/core_client.py -@@ -29,6 +29,7 @@ from vllm.v1.engine import (EngineCoreOutputs, EngineCoreRequest, - EngineCoreRequestType, - ReconfigureDistributedRequest, ReconfigureRankType, - UtilityOutput) -+from vllm.v1 import frontier_trace - from vllm.v1.engine.coordinator import DPCoordinator - from vllm.v1.engine.core import EngineCore, EngineCoreProc - from vllm.v1.engine.exceptions import EngineDeadError -@@ -977,6 +978,10 @@ class DPAsyncMPClient(AsyncMPClient): - # List of [waiting, running] pair per engine. - # Used only by DPLBAsyncMPClient subclass. - self.lb_engines: list[list[int]] = [[0, 0] for _ in self.core_engines] -+ # Coordinator snapshot the counts above came from; 0 until the first -+ # one arrives, so a placement made from the initial zeros is visible -+ # as such. -+ self.lb_snapshot: int = 0 - - self.first_req_sock_addr = get_open_zmq_inproc_path() - self.first_req_send_socket = self.resources.first_req_send_socket = ( -@@ -1070,12 +1075,20 @@ class DPAsyncMPClient(AsyncMPClient): - continue - - # Update local load-balancing state. -- counts, wave, running = msgspec.msgpack.decode(buf) -+ counts, wave, running, snapshot = msgspec.msgpack.decode( -+ buf) - self.current_wave = wave - self.engines_running = running - if counts is not None: - sliced_counts = counts[count_slice] - self.lb_engines = sliced_counts -+ self.lb_snapshot = snapshot -+ frontier_trace.log_dp_placement_record( -+ "frontend_snapshot", -+ snapshot=snapshot, -+ counts=sliced_counts, -+ wave=wave, -+ engines_running=running) - logger.debug("Received counts: %s (%s)", sliced_counts, - count_slice) - -@@ -1147,6 +1160,15 @@ class DPLBAsyncMPClient(DPAsyncMPClient): - if score < min_score: - min_score = score - eng_index = idx -+ frontier_trace.log_dp_placement_record( -+ "frontend_route", -+ request_id=request.request_id, -+ engine=eng_index, -+ snapshot=self.lb_snapshot, -+ counts=[list(counts) for counts in current_counts], -+ score=min_score, -+ start_index=self.eng_start_index, -+ reservation=self.client_count) - # Increment local waiting count for better balancing between stats - # updates from the coordinator (which happen every 100ms). - current_counts[eng_index][0] += self.client_count -diff --git a/vllm/v1/frontier_trace.py b/vllm/v1/frontier_trace.py -index d9bddeea5..a727a86b9 100644 ---- a/vllm/v1/frontier_trace.py -+++ b/vllm/v1/frontier_trace.py -@@ -2,14 +2,18 @@ - # SPDX-FileCopyrightText: Copyright contributors to the vLLM project - """Runtime gate for Frontier trace logging.""" - -+import atexit - from contextlib import contextmanager - import json - import os -+import time - from typing import Any, Mapping - - _SKIP_WARMUP = os.environ.get("VLLM_FRONTIER_TRACE_SKIP_WARMUP", "0") == "1" - _TRACE_ACTIVE = not _SKIP_WARMUP - _PP_BOUNDARY_LOG_ENV_VAR = "VLLM_FRONTIER_PP_BOUNDARY_LOG_PATH" -+_DP_PLACEMENT_LOG_DIR_ENV_VAR = "VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR" -+_DP_PLACEMENT_FLUSH_EVERY = 1024 - _PP_BOUNDARY_REQUIRED_FIELDS = ( - "model_name", - "timestamp", -@@ -99,3 +103,67 @@ def disable_for_warmup(): - yield - finally: - _TRACE_ACTIVE = True -+ -+ -+# Data-parallel placement records. Every process that takes part in a placement -+# decision -- each engine core, the coordinator, each API server -- writes its -+# own file, so records never interleave and the reader can tell the roles -+# apart. They are buffered because an engine core writes one per iteration of -+# its busy loop; they reach disk every _DP_PLACEMENT_FLUSH_EVERY records and at -+# normal process exit. -+_dp_placement_records: list[dict[str, Any]] = [] -+_dp_placement_seq = 0 -+ -+ -+def get_dp_placement_log_dir() -> str: -+ return os.environ.get(_DP_PLACEMENT_LOG_DIR_ENV_VAR, "") -+ -+ -+def is_dp_placement_logging_enabled() -> bool: -+ return is_active() and bool(get_dp_placement_log_dir()) -+ -+ -+def log_dp_placement_record(kind: str, **fields: Any) -> None: -+ """Buffer one placement record of the given kind. -+ -+ `seq` orders the records one process wrote and is the tie-break when two -+ of them carry the same correlation id. Timestamps are for reading; the -+ join between processes is always a correlation id. -+ """ -+ if not is_dp_placement_logging_enabled(): -+ return -+ -+ global _dp_placement_seq -+ record: dict[str, Any] = { -+ "kind": kind, -+ "pid": os.getpid(), -+ "seq": _dp_placement_seq, -+ "monotonic": time.monotonic(), -+ } -+ record.update(fields) -+ _dp_placement_records.append(record) -+ _dp_placement_seq += 1 -+ -+ if len(_dp_placement_records) >= _DP_PLACEMENT_FLUSH_EVERY: -+ flush_dp_placement_records() -+ -+ -+def flush_dp_placement_records() -> None: -+ if not _dp_placement_records: -+ return -+ -+ log_dir = get_dp_placement_log_dir() -+ os.makedirs(log_dir, exist_ok=True) -+ log_path = os.path.join(log_dir, f"dp_placement_{os.getpid()}.jsonl") -+ try: -+ with open(log_path, "a", encoding="utf-8") as file: -+ for record in _dp_placement_records: -+ file.write(json.dumps(record) + "\n") -+ except OSError as exc: -+ raise RuntimeError( -+ f"Failed to write Frontier DP placement log file: {log_path}" -+ ) from exc -+ _dp_placement_records.clear() -+ -+ -+atexit.register(flush_dp_placement_records) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch deleted file mode 100644 index ec4805ba..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch +++ /dev/null @@ -1,24 +0,0 @@ ---- a/vllm/_custom_ops.py -+++ b/vllm/_custom_ops.py -@@ -1505,9 +1505,9 @@ - - def topk_softmax(topk_weights: torch.Tensor, topk_ids: torch.Tensor, - token_expert_indices: torch.Tensor, -- gating_output: torch.Tensor, renormalize: bool) -> None: -+ gating_output: torch.Tensor) -> None: - torch.ops._moe_C.topk_softmax(topk_weights, topk_ids, token_expert_indices, -- gating_output, renormalize) -+ gating_output) - - - def grouped_topk(scores: torch.Tensor, scores_with_bias: torch.Tensor, ---- a/vllm/model_executor/layers/fused_moe/fused_moe.py -+++ b/vllm/model_executor/layers/fused_moe/fused_moe.py -@@ -890,7 +890,6 @@ - topk_indices, - token_expert_indices, - gating_output, -- renormalize, - ) - if renormalize: - topk_weights = topk_weights / topk_weights.sum(dim=-1, keepdim=True) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml deleted file mode 100644 index 97d4d580..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/manifest.yaml +++ /dev/null @@ -1,155 +0,0 @@ -# Frontier calibration case: stage admission of attention-DP lanes under PP. -# Structural comparison (plan §4.7, D-8). No E2E latency gate: the Frontier -# side runs the dummy predictor, and the fix changes admission, not durations. - -case_id: stage_admission_case_001 -run_generation: 1 -created_at_utc: "2026-09-23" -requesting_user: i-fengyicheng -reviewer_identity: i-fengyicheng -auto_recycle: false - -purpose: >- - Check that the stage-admission rule of plan P1 makes Frontier's attention-DP - lanes behave like vLLM V1 data-parallel ranks at pipeline_parallel_size 2: - a rank with runnable work is not refused at its stage by another rank's - queued work, both ranks start in the same forward slot, and stage-0 forwards - pair one to one. Criterion C7 of plan.md. - -selected_checkout: - frontier_worktree: /data/ycfeng/Frontier/.worktrees/stage-admission-ordering - frontier_branch: fix/stage-admission-ordering - frontier_before_source: 1f694f7 # P0 set "base"; source identical to origin/main - frontier_after_commit: dac4e69 # P1 rule commit; P3 set "after" - frontier_after_r2_commit: a8e8d8a # R-10 rule refactor 1661bf1 plus harness; set "after-r2", G7 byte-identical to "after" - frontier_harness: tests/e2e/stage_admission_matrix.py (group G7) - -groundtruth_checkout_path: /data/ycfeng/Frontier/.real-engine/vLLM-BS -groundtruth_branch: feature/frontier-comparison-instrumentation -groundtruth_ref: refs/heads/feature/frontier-comparison-instrumentation -groundtruth_commit: 494b9f327036d4493034a9b37ebb343354884e01 -groundtruth_tree_dirty: false -groundtruth_remote_url: https://github.com/fwyc0573/vLLM-BS.git -groundtruth_remote_tip: ea95f571e20937c7c908c6d59ddd1cd6bf9268f1 -groundtruth_remote_tip_note: >- - The local commit is one commit ahead of the remote tip (494b9f327, "Trace the - data-parallel placement chain for Frontier calibration"), unpushed; it adds - only the DP placement records. Its diff is inputs/groundtruth_local_commit.diff, - identical to the parent task's dp_pp_case_001/g1_instrumentation.diff. -groundtruth_diff_artifact: inputs/groundtruth_local_commit.diff -groundtruth_diff_sha256: 84fc24db0e2411268a93f8be7ca5f8e4e5072ea09d86063cac2cfb98381feb2c -groundtruth_upstream_base: 01efc7ef7 (tag upstream-v0.10.2) -groundtruth_fork_changes: inputs/fork_changed_files.txt # 61 files under vllm/, 59 .py -groundtruth_dirty_patch_sha256: null -groundtruth_overlay_patch_applied: true # run sa-pp-20260923b; run sa-pp-20260923a had none -groundtruth_overlay_patch: inputs/groundtruth_overlay.patch -groundtruth_overlay_patch_sha256: 8d47678911b7b689bc644ea81bd9d2200ea296d773ee9c07a1c5a9bc9b3a9c81 -groundtruth_overlay_patch_reason: >- - Fork commit 1109c4f16 passes a fifth renormalize argument to - _moe_C::topk_softmax; the fork's own csrc (unchanged from upstream-v0.10.2) - and the v0.10.2 image declare four, so run a failed in the MoE profile_run. - The patch restores the upstream four-argument wrapper (vllm/_custom_ops.py) - and call (fused_moe.py vllm_topk_softmax). Numerics are unchanged: Python - renormalizes after the call. The checkout is not modified. -groundtruth_overlay: >- - The worker copies the image's installed vllm package and copies every - vllm/**/*.py of the checkout over it. It runs only when the files where image - and checkout differ are exactly the .py files of fork_changed_files.txt - (overlay_report.json in the run directory). -groundtruth_weight_mode: dummy -real_weight_download: false - -topology: - vllm: "data_parallel_size=2, pipeline_parallel_size=2, tensor_parallel_size=1; MoE adds enable_expert_parallel" - frontier: "attn_dp=2, num_pipeline_stages=2, attn_tp=1, one Replica; MoE adds moe_tp=1, moe_ep=2" - devices: 4x H800 (vLLM); h800 / h800_dgx (Frontier) - cluster_scheduler: round_robin (request i to lane i mod 2) - replica_scheduler: vllm_v1 - -models: - moe: data/config/models/Qwen3-30B-A3B-tiny.json - dense: data/config/models/Llama-3.2-1B-Instruct.json - -settings: - dtype: bfloat16 - prompt_tokens: 256 - output_tokens: 1 - max_num_batched_tokens: 256 - max_num_seqs: 4 - block_size: 16 - chunked_prefill: true - prefix_caching: false - graph_mode: eager - frontier_num_blocks: 1024 - vllm_gpu_memory_utilization: 0.5 - vllm_num_gpu_blocks: recorded in runs/vllm-instrumented//runs//summary.json - -workload: - request_id_namespace: "-q; burst = warmup | b-r" - request_id_encoding: "AsyncLLM.add_request request_id, unchanged into the engine (n=1)" - frontier_request_ids: "0..n-1, mapped to vLLM b-r-q by index" - lane_assignment: "index mod 2 (vLLM data_parallel_rank; Frontier round robin)" - warmup_request_ids: [warmup-q0, warmup-q1, warmup-q2, warmup-q3] - formal_request_ids: "b8-r{0,1,2}-q{0..7}, b16-r{0,1,2}-q{0..15}" - rounds_per_burst: 3 - idle_between_rounds: "engines paused (dp_engines_running false), then 1 s" - -modes: - groundtruth_instrumented: - env: "VLLM_FRONTIER_INSTRUMENTATION=1, VLLM_FRONTIER_PP_BOUNDARY_LOG_PATH, VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR, VLLM_WORKER_MULTIPROC_METHOD=spawn" - producer: tests/comparison/stage_admission_pp/run_vllm_worker.sh - artifact_path: runs/vllm-instrumented// (archive /mnt/codesign-exp/ycfeng/frontier/stage_admission_pp//) - status: COMPLETE - runs: - - run_tag: sa-pp-20260923a - rjob: exp-0923-022226-151935 - overlay_patch: none - result: "dense complete; MoE failed (_moe_C::topk_softmax 5 vs 4 arguments); job Failed" - - run_tag: sa-pp-20260923b - rjob: exp-0923-024146-345158 - overlay_patch: inputs/groundtruth_overlay.patch - result: "MoE and dense complete; job Succeeded; used by analysis/" - groundtruth_clean: - status: NOT_APPLICABLE # no E2E latency gate (D-8) - simulator_before: - source: /data/ycfeng/tmp/stage_admission_ordering/base/G7-* - status: COMPLETE # MoE n8/n16 admission_deadlock; dense n8/n16 success - simulator_after: - source: /data/ycfeng/tmp/stage_admission_ordering/after-r2/G7-* - status: COMPLETE # all four G7 cases success; set "after" (dac4e69) was used before R-10 and is byte-identical - -analysis: - producer: tests/comparison/stage_admission_pp/compare_lanes.py - outputs: [analysis/workflow_gap_table.csv, analysis/lane_metrics.json, analysis/workflow_gap_status.json, analysis/workflow_gap_summary.md] - gates_not_applicable: - e2e_gate: "structural comparison; Frontier uses the dummy predictor (D-8)" - moe_routing_distortion_gate: "no request-level parity claim; admission order does not depend on routing" - -decisions: - - decision_id: R-6 - question: Validate the admission fix against vLLM on a GPU worker? - answer: "按照已有plan执行上述修复（该修复需要和在gpu worker上运行的vllm进行合理的对比验证，确保修改的有效性）" - decided_at_utc: "2026-09-23" - - decision_id: D-8 - question: Comparison scope and mode. - answer: Structural M1-M5 in instrumented mode, prefill-only, existing model configs, no E2E gate (plan.md D-8). - decided_at_utc: "2026-09-23" - - decision_id: gpu-charged-group - question: GPU cluster. - answer: "codesign only (后续的gpu worker集群只允许使用 codesign（暂停对steptron_ci的使用，直至得到我允许）)" - - decision_id: topk-softmax-abi - question: How to run the MoE ground truth after the topk_softmax ABI failure of run a? - answer: "topk_softmax 统一修复为4 个参数的版本" - decided_at_utc: "2026-09-23" - - decision_id: D-9 - question: V5 on the dense shape, after the first analysis found vLLM's own dense co-execution spread wider than 0.10. - answer: "采纳你的推荐，继续" - outcome: V5 gates MoE only; dense M5 is reported with its start/end decomposition. - decided_at_utc: "2026-09-23" - -analysis_result: >- - 56 rows: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5, D-9), 4 HOLDS - (negative controls N1 and N4, separate rows since R-10); rerun against the - Frontier set after-r2. The first analysis, before D-9, had the two dense V5 - rows as MISMATCH; cause in analysis/workflow_gap_summary.md. -status: COMPLETE diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/COMPLETE b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/COMPLETE deleted file mode 100644 index f47e4716..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/COMPLETE +++ /dev/null @@ -1 +0,0 @@ -status=1 diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/overlay_report.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/overlay_report.json deleted file mode 100644 index a41bc82e..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/overlay_report.json +++ /dev/null @@ -1,130 +0,0 @@ -{ - "site_vllm": "/usr/local/lib/python3.12/dist-packages/vllm", - "checkout": "/data/ycfeng/Frontier/.real-engine/vLLM-BS", - "overlay": "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm", - "differing_py_files": [ - "vllm/_C.py", - "vllm/_custom_ops.py", - "vllm/_moe_C.py", - "vllm/attention/layer.py", - "vllm/benchmarks/throughput.py", - "vllm/compilation/compiler_interface.py", - "vllm/config/__init__.py", - "vllm/distributed/communication_op.py", - "vllm/distributed/kv_transfer/kv_connector/v1/base.py", - "vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/frontier_kv_transfer_logger.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", - "vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py", - "vllm/distributed/parallel_state.py", - "vllm/engine/arg_utils.py", - "vllm/engine/llm_engine.py", - "vllm/entrypoints/openai/frontier_request_metrics.py", - "vllm/entrypoints/openai/serving_chat.py", - "vllm/entrypoints/openai/serving_completion.py", - "vllm/entrypoints/openai/serving_engine.py", - "vllm/envs.py", - "vllm/model_executor/custom_op.py", - "vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", - "vllm/model_executor/layers/fused_moe/deepep_ll_prepare_finalize.py", - "vllm/model_executor/layers/fused_moe/fused_moe.py", - "vllm/model_executor/layers/fused_moe/layer.py", - "vllm/model_executor/layers/fused_moe/pplx_prepare_finalize.py", - "vllm/model_executor/layers/linear.py", - "vllm/model_executor/layers/vocab_parallel_embedding.py", - "vllm/model_executor/models/llama.py", - "vllm/model_executor/models/phimoe.py", - "vllm/model_executor/models/qwen3_moe.py", - "vllm/model_executor/models/qwen3_moe_mtp.py", - "vllm/model_executor/models/registry.py", - "vllm/request_generator/__init__.py", - "vllm/request_generator/config.py", - "vllm/request_generator/kv_sync.py", - "vllm/request_generator/prompt_generator.py", - "vllm/request_generator/vllm_request_generator.py", - "vllm/v1/attention/backends/flash_attn.py", - "vllm/v1/attention/backends/flashinfer.py", - "vllm/v1/attention/backends/mla/common.py", - "vllm/v1/attention/backends/mla/flashinfer_mla.py", - "vllm/v1/attention/backends/utils.py", - "vllm/v1/core/sched/scheduler.py", - "vllm/v1/engine/coordinator.py", - "vllm/v1/engine/core.py", - "vllm/v1/engine/core_client.py", - "vllm/v1/engine/output_processor.py", - "vllm/v1/engine/processor.py", - "vllm/v1/frontier_trace.py", - "vllm/v1/metrics/stats.py", - "vllm/v1/spec_decode/eagle.py", - "vllm/v1/utils.py", - "vllm/v1/worker/gpu_model_runner.py", - "vllm/v1/worker/gpu_worker.py", - "vllm/worker/model_runner.py", - "vllm/worker/worker.py" - ], - "expected_py_changes": [ - "vllm/_C.py", - "vllm/_custom_ops.py", - "vllm/_moe_C.py", - "vllm/attention/layer.py", - "vllm/benchmarks/throughput.py", - "vllm/compilation/compiler_interface.py", - "vllm/config/__init__.py", - "vllm/distributed/communication_op.py", - "vllm/distributed/kv_transfer/kv_connector/v1/base.py", - "vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/frontier_kv_transfer_logger.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", - "vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py", - "vllm/distributed/parallel_state.py", - "vllm/engine/arg_utils.py", - "vllm/engine/llm_engine.py", - "vllm/entrypoints/openai/frontier_request_metrics.py", - "vllm/entrypoints/openai/serving_chat.py", - "vllm/entrypoints/openai/serving_completion.py", - "vllm/entrypoints/openai/serving_engine.py", - "vllm/envs.py", - "vllm/model_executor/custom_op.py", - "vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", - "vllm/model_executor/layers/fused_moe/deepep_ll_prepare_finalize.py", - "vllm/model_executor/layers/fused_moe/fused_moe.py", - "vllm/model_executor/layers/fused_moe/layer.py", - "vllm/model_executor/layers/fused_moe/pplx_prepare_finalize.py", - "vllm/model_executor/layers/linear.py", - "vllm/model_executor/layers/vocab_parallel_embedding.py", - "vllm/model_executor/models/llama.py", - "vllm/model_executor/models/phimoe.py", - "vllm/model_executor/models/qwen3_moe.py", - "vllm/model_executor/models/qwen3_moe_mtp.py", - "vllm/model_executor/models/registry.py", - "vllm/request_generator/__init__.py", - "vllm/request_generator/config.py", - "vllm/request_generator/kv_sync.py", - "vllm/request_generator/prompt_generator.py", - "vllm/request_generator/vllm_request_generator.py", - "vllm/v1/attention/backends/flash_attn.py", - "vllm/v1/attention/backends/flashinfer.py", - "vllm/v1/attention/backends/mla/common.py", - "vllm/v1/attention/backends/mla/flashinfer_mla.py", - "vllm/v1/attention/backends/utils.py", - "vllm/v1/core/sched/scheduler.py", - "vllm/v1/engine/coordinator.py", - "vllm/v1/engine/core.py", - "vllm/v1/engine/core_client.py", - "vllm/v1/engine/output_processor.py", - "vllm/v1/engine/processor.py", - "vllm/v1/frontier_trace.py", - "vllm/v1/metrics/stats.py", - "vllm/v1/spec_decode/eagle.py", - "vllm/v1/utils.py", - "vllm/v1/worker/gpu_model_runner.py", - "vllm/v1/worker/gpu_worker.py", - "vllm/worker/model_runner.py", - "vllm/worker/worker.py" - ], - "unexpected": [], - "missing": [], - "accepted": true -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/replica_log.txt b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/replica_log.txt deleted file mode 100644 index 834e9102..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/replica_log.txt +++ /dev/null @@ -1,73 +0,0 @@ -# Replica log tail of RJob exp-0923-022226-151935 (codesign, H800 x4, creator i-fengyicheng). -# Platform init lines (node addresses, NCCL interface settings) are removed; workload output is verbatim. -{"python": "3.12.11", "torch": "2.8.0+cu128", "cuda_available": true, "device_count": 4, "devices": ["NVIDIA H800", "NVIDIA H800", "NVIDIA H800", "NVIDIA H800"]} -{"accepted": true, "unexpected": [], "missing": []} -/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. - import pynvml # type: ignore[import] -VLLM_IMPORT 0.10.2 /tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/__init__.py /tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/frontier_trace.py -SCENARIO_FAIL moe exit=1 -(EngineCore_DP0 pid=232) return self.collective_rpc("determine_available_memory") -(EngineCore_DP0 pid=232) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -(EngineCore_DP0 pid=232) File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/executor/multiproc_executor.py", line 257, in collective_rpc -(EngineCore_DP0 pid=232) result = result.result() -(EngineCore_DP0 pid=232) ^^^^^^^^^^^^^^^ -(EngineCore_DP0 pid=232) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result -(EngineCore_DP0 pid=232) return self.__get_result() -(EngineCore_DP0 pid=232) ^^^^^^^^^^^^^^^^^^^ -(EngineCore_DP0 pid=232) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result -(EngineCore_DP0 pid=232) raise self._exception -(EngineCore_DP0 pid=232) File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run -(EngineCore_DP0 pid=232) result = self.fn(*self.args, **self.kwargs) -(EngineCore_DP0 pid=232) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -(EngineCore_DP0 pid=232) File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/executor/multiproc_executor.py", line 243, in get_response -(EngineCore_DP0 pid=232) raise RuntimeError( -(EngineCore_DP0 pid=232) RuntimeError: Worker failed with error '_moe_C::topk_softmax() expected at most 4 argument(s) but received 5 argument(s). Declaration: _moe_C::topk_softmax(Tensor($0! -> ) topk_weights, Tensor($1! -> ) topk_indices, Tensor($2! -> ) token_expert_indices, Tensor gating_output) -> ()', please check the stack trace above for the root cause -Traceback (most recent call last): - File "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/tests/comparison/stage_admission_pp/vllm_burst_driver.py", line 221, in - sys.exit(main()) - ^^^^^^ - File "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/tests/comparison/stage_admission_pp/vllm_burst_driver.py", line 213, in main - summary = asyncio.run(run_bursts(args)) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run - return runner.run(main) - ^^^^^^^^^^^^^^^^ - File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run - return self._loop.run_until_complete(task) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/usr/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete - return future.result() - ^^^^^^^^^^^^^^^ - File "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/tests/comparison/stage_admission_pp/vllm_burst_driver.py", line 113, in run_bursts - engine = AsyncLLM.from_engine_args(engine_args) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/async_llm.py", line 240, in from_engine_args - return cls( - ^^^^ - File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/async_llm.py", line 136, in __init__ - self.engine_core = EngineCoreClient.make_async_mp_client( - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client - return DPLBAsyncMPClient(*client_args) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/core_client.py", line 1137, in __init__ - super().__init__(vllm_config, executor_class, log_stats, - File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/core_client.py", line 975, in __init__ - super().__init__(vllm_config, executor_class, log_stats, - File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/core_client.py", line 770, in __init__ - super().__init__( - File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/core_client.py", line 449, in __init__ - with launch_core_engines(vllm_config, executor_class, - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ - next(self.gen) - File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/utils.py", line 729, in launch_core_engines - wait_for_engine_startup( - File "/tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/engine/utils.py", line 782, in wait_for_engine_startup - raise RuntimeError("Engine core initialization failed. " -RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} -SCENARIO_PASS dense -dense DRIVER_DONE {"rounds": 7, "num_gpu_blocks": 304854} -152 /tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/pp_boundary.jsonl -WORKER_STATUS=1 RUN_TAG=sa-pp-20260923a -ExitCode= 1 diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_588.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_588.jsonl deleted file mode 100644 index 7ff91c38..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_588.jsonl +++ /dev/null @@ -1,22 +0,0 @@ -{"kind": "frontend_snapshot", "pid": 588, "seq": 0, "monotonic": 9119575.220499707, "snapshot": 2, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 1, "monotonic": 9119576.500151489, "snapshot": 3, "counts": [[1, 1], [1, 1]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 2, "monotonic": 9119576.66924366, "snapshot": 4, "counts": [[0, 1], [0, 1]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 3, "monotonic": 9119576.769698702, "snapshot": 5, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 4, "monotonic": 9119577.8353699, "snapshot": 6, "counts": [[0, 1], [0, 1]], "wave": 1, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 5, "monotonic": 9119577.935569072, "snapshot": 7, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 588, "seq": 6, "monotonic": 9119578.035744542, "snapshot": 8, "counts": [[0, 0], [0, 0]], "wave": 2, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 588, "seq": 7, "monotonic": 9119578.971015744, "snapshot": 9, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 8, "monotonic": 9119579.070546191, "snapshot": 10, "counts": [[0, 1], [0, 1]], "wave": 3, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 588, "seq": 9, "monotonic": 9119579.17074387, "snapshot": 11, "counts": [[0, 0], [0, 0]], "wave": 3, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 588, "seq": 10, "monotonic": 9119580.105585678, "snapshot": 12, "counts": [[0, 1], [0, 1]], "wave": 3, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 11, "monotonic": 9119580.205964977, "snapshot": 13, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 588, "seq": 12, "monotonic": 9119580.306053106, "snapshot": 14, "counts": [[0, 0], [0, 0]], "wave": 4, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 588, "seq": 13, "monotonic": 9119581.244764512, "snapshot": 15, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 14, "monotonic": 9119581.344573118, "snapshot": 16, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 15, "monotonic": 9119581.444366362, "snapshot": 17, "counts": [[0, 0], [0, 0]], "wave": 5, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 588, "seq": 16, "monotonic": 9119582.409289015, "snapshot": 18, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 17, "monotonic": 9119582.509353423, "snapshot": 19, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 18, "monotonic": 9119582.610055115, "snapshot": 20, "counts": [[0, 0], [0, 0]], "wave": 6, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 588, "seq": 19, "monotonic": 9119583.568792408, "snapshot": 21, "counts": [[0, 1], [0, 1]], "wave": 6, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 20, "monotonic": 9119583.669192385, "snapshot": 22, "counts": [[0, 1], [0, 1]], "wave": 6, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 588, "seq": 21, "monotonic": 9119583.768215796, "snapshot": 23, "counts": [[0, 0], [0, 0]], "wave": 7, "engines_running": false} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_663.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_663.jsonl deleted file mode 100644 index 6dbe9039..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_663.jsonl +++ /dev/null @@ -1,65 +0,0 @@ -{"kind": "engine_iteration", "pid": 663, "seq": 0, "monotonic": 9119576.449129984, "engine": 0, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 1, "monotonic": 9119576.663859723, "engine": 0, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 2, "monotonic": 9119576.66974478, "engine": 0, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 3, "monotonic": 9119576.673491668, "engine": 0, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 4, "monotonic": 9119576.675901318, "engine": 0, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 5, "monotonic": 9119577.823941316, "engine": 0, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 6, "monotonic": 9119577.834619695, "engine": 0, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 7, "monotonic": 9119577.838431131, "engine": 0, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 8, "monotonic": 9119577.846998611, "engine": 0, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 9, "monotonic": 9119577.851663468, "engine": 0, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 10, "monotonic": 9119577.856308192, "engine": 0, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 11, "monotonic": 9119577.859561805, "engine": 0, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 12, "monotonic": 9119577.862482356, "engine": 0, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 13, "monotonic": 9119578.960288996, "engine": 0, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 14, "monotonic": 9119578.970791968, "engine": 0, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 15, "monotonic": 9119578.973420218, "engine": 0, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 16, "monotonic": 9119578.982067386, "engine": 0, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 17, "monotonic": 9119578.986149205, "engine": 0, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 18, "monotonic": 9119578.990410643, "engine": 0, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 19, "monotonic": 9119578.992635172, "engine": 0, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 20, "monotonic": 9119578.994783333, "engine": 0, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 21, "monotonic": 9119580.09426199, "engine": 0, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 22, "monotonic": 9119580.104928317, "engine": 0, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 23, "monotonic": 9119580.108950997, "engine": 0, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 24, "monotonic": 9119580.11632106, "engine": 0, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 25, "monotonic": 9119580.122566212, "engine": 0, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 26, "monotonic": 9119580.127797948, "engine": 0, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 27, "monotonic": 9119580.131552352, "engine": 0, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 28, "monotonic": 9119580.135168187, "engine": 0, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 29, "monotonic": 9119581.232589915, "engine": 0, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 30, "monotonic": 9119581.244141446, "engine": 0, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 31, "monotonic": 9119581.24668991, "engine": 0, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 32, "monotonic": 9119581.255638912, "engine": 0, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 33, "monotonic": 9119581.262119388, "engine": 0, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 34, "monotonic": 9119581.268199256, "engine": 0, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q8"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 35, "monotonic": 9119581.2744289, "engine": 0, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q10"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 36, "monotonic": 9119581.280617448, "engine": 0, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q12"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 37, "monotonic": 9119581.286810076, "engine": 0, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q14"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 38, "monotonic": 9119581.29294334, "engine": 0, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 39, "monotonic": 9119581.296367176, "engine": 0, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 40, "monotonic": 9119581.299756235, "engine": 0, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 41, "monotonic": 9119582.398610272, "engine": 0, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 42, "monotonic": 9119582.40874644, "engine": 0, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 43, "monotonic": 9119582.41231597, "engine": 0, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 44, "monotonic": 9119582.420458922, "engine": 0, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 45, "monotonic": 9119582.425835129, "engine": 0, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 46, "monotonic": 9119582.431018058, "engine": 0, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q8"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 47, "monotonic": 9119582.436320836, "engine": 0, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q10"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 48, "monotonic": 9119582.441395594, "engine": 0, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q12"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 49, "monotonic": 9119582.446620272, "engine": 0, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q14"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 50, "monotonic": 9119582.451802012, "engine": 0, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 51, "monotonic": 9119582.45515298, "engine": 0, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 52, "monotonic": 9119582.457941312, "engine": 0, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 53, "monotonic": 9119583.557296367, "engine": 0, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 54, "monotonic": 9119583.568288304, "engine": 0, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 55, "monotonic": 9119583.571709411, "engine": 0, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 56, "monotonic": 9119583.57967443, "engine": 0, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 57, "monotonic": 9119583.584644731, "engine": 0, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 58, "monotonic": 9119583.590087384, "engine": 0, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q8"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 59, "monotonic": 9119583.595089003, "engine": 0, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q10"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 60, "monotonic": 9119583.600473017, "engine": 0, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q12"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 61, "monotonic": 9119583.60545546, "engine": 0, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q14"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 663, "seq": 62, "monotonic": 9119583.610792452, "engine": 0, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 63, "monotonic": 9119583.61387913, "engine": 0, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 663, "seq": 64, "monotonic": 9119583.616643798, "engine": 0, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_664.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_664.jsonl deleted file mode 100644 index bb20bb4f..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/dp_placement/dp_placement_664.jsonl +++ /dev/null @@ -1,65 +0,0 @@ -{"kind": "engine_iteration", "pid": 664, "seq": 0, "monotonic": 9119576.448517464, "engine": 1, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 1, "monotonic": 9119576.656539064, "engine": 1, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 2, "monotonic": 9119576.668673038, "engine": 1, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 3, "monotonic": 9119576.67321912, "engine": 1, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 4, "monotonic": 9119576.675677937, "engine": 1, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 5, "monotonic": 9119577.824109633, "engine": 1, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 6, "monotonic": 9119577.835726645, "engine": 1, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 7, "monotonic": 9119577.838386636, "engine": 1, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 8, "monotonic": 9119577.847060977, "engine": 1, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 9, "monotonic": 9119577.851408212, "engine": 1, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 10, "monotonic": 9119577.856307589, "engine": 1, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 11, "monotonic": 9119577.858678106, "engine": 1, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 12, "monotonic": 9119577.861769507, "engine": 1, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 13, "monotonic": 9119578.960459523, "engine": 1, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 14, "monotonic": 9119578.970415901, "engine": 1, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 15, "monotonic": 9119578.973142773, "engine": 1, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 16, "monotonic": 9119578.9821197, "engine": 1, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 17, "monotonic": 9119578.986046756, "engine": 1, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 18, "monotonic": 9119578.990198545, "engine": 1, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 19, "monotonic": 9119578.992546445, "engine": 1, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 20, "monotonic": 9119578.99476318, "engine": 1, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 21, "monotonic": 9119580.0946057, "engine": 1, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 22, "monotonic": 9119580.105009504, "engine": 1, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 23, "monotonic": 9119580.107590236, "engine": 1, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 24, "monotonic": 9119580.116323853, "engine": 1, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 25, "monotonic": 9119580.12049956, "engine": 1, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 26, "monotonic": 9119580.126651892, "engine": 1, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 27, "monotonic": 9119580.130101508, "engine": 1, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 28, "monotonic": 9119580.133673932, "engine": 1, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 29, "monotonic": 9119581.232942274, "engine": 1, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 30, "monotonic": 9119581.244137796, "engine": 1, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 31, "monotonic": 9119581.2480332, "engine": 1, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 32, "monotonic": 9119581.257721173, "engine": 1, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 33, "monotonic": 9119581.26389318, "engine": 1, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 34, "monotonic": 9119581.270159176, "engine": 1, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q9"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 35, "monotonic": 9119581.276280183, "engine": 1, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q11"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 36, "monotonic": 9119581.282644592, "engine": 1, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q13"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 37, "monotonic": 9119581.288699504, "engine": 1, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q15"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 38, "monotonic": 9119581.29402939, "engine": 1, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 39, "monotonic": 9119581.297625517, "engine": 1, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 40, "monotonic": 9119581.301283844, "engine": 1, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 41, "monotonic": 9119582.398801848, "engine": 1, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 42, "monotonic": 9119582.409782464, "engine": 1, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 43, "monotonic": 9119582.412839167, "engine": 1, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 44, "monotonic": 9119582.421456927, "engine": 1, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 45, "monotonic": 9119582.42656548, "engine": 1, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 46, "monotonic": 9119582.431824055, "engine": 1, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q9"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 47, "monotonic": 9119582.43702584, "engine": 1, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q11"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 48, "monotonic": 9119582.442215288, "engine": 1, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q13"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 49, "monotonic": 9119582.447442189, "engine": 1, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q15"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 50, "monotonic": 9119582.452716084, "engine": 1, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 51, "monotonic": 9119582.455679407, "engine": 1, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 52, "monotonic": 9119582.45858248, "engine": 1, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 53, "monotonic": 9119583.557630615, "engine": 1, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 54, "monotonic": 9119583.56901594, "engine": 1, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 55, "monotonic": 9119583.57212383, "engine": 1, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 56, "monotonic": 9119583.580448361, "engine": 1, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 57, "monotonic": 9119583.585726876, "engine": 1, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 58, "monotonic": 9119583.590783136, "engine": 1, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q9"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 59, "monotonic": 9119583.596036963, "engine": 1, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q11"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 60, "monotonic": 9119583.601235135, "engine": 1, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q13"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 61, "monotonic": 9119583.606379524, "engine": 1, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q15"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 664, "seq": 62, "monotonic": 9119583.611563925, "engine": 1, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 63, "monotonic": 9119583.614556208, "engine": 1, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 664, "seq": 64, "monotonic": 9119583.617356433, "engine": 1, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/model/config.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/model/config.json deleted file mode 100644 index e9b20534..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/model/config.json +++ /dev/null @@ -1,95 +0,0 @@ -{ - "vocab_size": 128256, - "max_position_embeddings": 131072, - "hidden_size": 2048, - "intermediate_size": 8192, - "num_hidden_layers": 16, - "num_attention_heads": 32, - "num_key_value_heads": 8, - "hidden_act": "silu", - "initializer_range": 0.02, - "rms_norm_eps": 1e-05, - "pretraining_tp": 1, - "use_cache": true, - "rope_theta": 500000.0, - "rope_scaling": { - "factor": 32.0, - "high_freq_factor": 4.0, - "low_freq_factor": 1.0, - "original_max_position_embeddings": 8192, - "rope_type": "llama3" - }, - "attention_bias": false, - "attention_dropout": 0.0, - "mlp_bias": false, - "head_dim": 64, - "return_dict": true, - "output_hidden_states": false, - "torchscript": false, - "dtype": "bfloat16", - "torch_dtype": "bfloat16", - "pruned_heads": {}, - "tie_word_embeddings": true, - "chunk_size_feed_forward": 0, - "is_encoder_decoder": false, - "is_decoder": false, - "cross_attention_hidden_size": null, - "add_cross_attention": false, - "tie_encoder_decoder": false, - "architectures": [ - "LlamaForCausalLM" - ], - "finetuning_task": null, - "id2label": { - "0": "LABEL_0", - "1": "LABEL_1" - }, - "label2id": { - "LABEL_0": 0, - "LABEL_1": 1 - }, - "task_specific_params": null, - "problem_type": null, - "tokenizer_class": null, - "prefix": null, - "bos_token_id": 128000, - "pad_token_id": null, - "eos_token_id": [ - 128001, - 128008, - 128009 - ], - "sep_token_id": null, - "decoder_start_token_id": null, - "max_length": 20, - "min_length": 0, - "do_sample": false, - "early_stopping": false, - "num_beams": 1, - "temperature": 1.0, - "top_k": 50, - "top_p": 1.0, - "typical_p": 1.0, - "repetition_penalty": 1.0, - "length_penalty": 1.0, - "no_repeat_ngram_size": 0, - "encoder_no_repeat_ngram_size": 0, - "bad_words_ids": null, - "num_return_sequences": 1, - "output_scores": false, - "return_dict_in_generate": false, - "forced_bos_token_id": null, - "forced_eos_token_id": null, - "remove_invalid_values": false, - "exponential_decay_length_penalty": null, - "suppress_tokens": null, - "begin_suppress_tokens": null, - "num_beam_groups": 1, - "diversity_penalty": 0.0, - "_name_or_path": "meta-llama/Llama-3.2-1B-Instruct", - "transformers_version": "4.57.3", - "model_type": "llama", - "tf_legacy_loss": false, - "use_bfloat16": false, - "output_attentions": false -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/pp_boundary.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/pp_boundary.jsonl deleted file mode 100644 index 1998efab..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/pp_boundary.jsonl +++ /dev/null @@ -1,152 +0,0 @@ -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119576.449235303, "preprocess_end_ts": 9119576.451109972, "forward_start_ts": 9119576.451299587, "forward_end_ts": 9119576.460063873, "timestamp": 1790101597.6622696, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119576.46019184, "send_end_ts": 9119576.629189717} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119576.44977724, "preprocess_end_ts": 9119576.451516641, "forward_start_ts": 9119576.451719085, "forward_end_ts": 9119576.46017466, "timestamp": 1790101597.6628628, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119576.460299674, "send_end_ts": 9119576.629782932} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119576.629035799, "preprocess_end_ts": 9119576.630523184, "forward_start_ts": 9119576.630714431, "forward_end_ts": 9119576.639729928, "timestamp": 1790101597.6889172, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119576.448764324, "recv_end_ts": 9119576.628623897, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119576.629958017, "preprocess_end_ts": 9119576.630545435, "forward_start_ts": 9119576.630566584, "forward_end_ts": 9119576.63488724, "timestamp": 1790101597.6897466, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119576.635018112, "send_end_ts": 9119576.65667815} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119576.629221069, "preprocess_end_ts": 9119576.631134983, "forward_start_ts": 9119576.631357692, "forward_end_ts": 9119576.64234354, "timestamp": 1790101597.696209, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119576.449436652, "recv_end_ts": 9119576.62873706, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119576.63039467, "preprocess_end_ts": 9119576.630852029, "forward_start_ts": 9119576.630868305, "forward_end_ts": 9119576.633887624, "timestamp": 1790101597.6971579, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119576.633995306, "send_end_ts": 9119576.664088145} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119576.656937769, "preprocess_end_ts": 9119576.65735066, "forward_start_ts": 9119576.657365354, "forward_end_ts": 9119576.667654537, "timestamp": 1790101597.701157, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119576.65626875, "recv_end_ts": 9119576.656732377, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119576.664351743, "preprocess_end_ts": 9119576.664894195, "forward_start_ts": 9119576.664911393, "forward_end_ts": 9119576.668773573, "timestamp": 1790101597.7023337, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119576.663628172, "recv_end_ts": 9119576.664111456, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.824357212, "preprocess_end_ts": 9119577.82498523, "forward_start_ts": 9119577.825001959, "forward_end_ts": 9119577.828014908, "timestamp": 1790101598.8617194, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.828126712, "send_end_ts": 9119577.828654032} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.824468488, "preprocess_end_ts": 9119577.825073266, "forward_start_ts": 9119577.825089, "forward_end_ts": 9119577.829188433, "timestamp": 1790101598.8629055, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.829313964, "send_end_ts": 9119577.82983897} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.828956548, "preprocess_end_ts": 9119577.829398211, "forward_start_ts": 9119577.829411317, "forward_end_ts": 9119577.833676076, "timestamp": 1790101598.867198, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.824065764, "recv_end_ts": 9119577.828752125, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.830214947, "preprocess_end_ts": 9119577.830757245, "forward_start_ts": 9119577.830772884, "forward_end_ts": 9119577.83463634, "timestamp": 1790101598.868191, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.824199883, "recv_end_ts": 9119577.829978593, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.838627815, "preprocess_end_ts": 9119577.838983875, "forward_start_ts": 9119577.838994114, "forward_end_ts": 9119577.841707371, "timestamp": 1790101598.8752441, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.84179205, "send_end_ts": 9119577.842179088} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.838771084, "preprocess_end_ts": 9119577.83919682, "forward_start_ts": 9119577.839209195, "forward_end_ts": 9119577.842601636, "timestamp": 1790101598.8762252, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.842700256, "send_end_ts": 9119577.843159849} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.842395412, "preprocess_end_ts": 9119577.842754915, "forward_start_ts": 9119577.84276472, "forward_end_ts": 9119577.84616774, "timestamp": 1790101598.879648, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.838463878, "recv_end_ts": 9119577.842247857, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.843345407, "preprocess_end_ts": 9119577.843700437, "forward_start_ts": 9119577.84371037, "forward_end_ts": 9119577.84624232, "timestamp": 1790101598.879721, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.838398213, "recv_end_ts": 9119577.84319912, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.842513002, "preprocess_end_ts": 9119577.842835452, "forward_start_ts": 9119577.842845012, "forward_end_ts": 9119577.84654935, "timestamp": 1790101598.8802576, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.84662892, "send_end_ts": 9119577.847191827} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.843563482, "preprocess_end_ts": 9119577.843991311, "forward_start_ts": 9119577.844012097, "forward_end_ts": 9119577.84732702, "timestamp": 1790101598.8808856, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.847418964, "send_end_ts": 9119577.847820364} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.847960591, "preprocess_end_ts": 9119577.84833606, "forward_start_ts": 9119577.84834514, "forward_end_ts": 9119577.85068873, "timestamp": 1790101598.8841507, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.84689201, "recv_end_ts": 9119577.847834667, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.84745461, "preprocess_end_ts": 9119577.84791852, "forward_start_ts": 9119577.84792845, "forward_end_ts": 9119577.850802308, "timestamp": 1790101598.8842638, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.84683005, "recv_end_ts": 9119577.847290784, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.847578784, "preprocess_end_ts": 9119577.847931173, "forward_start_ts": 9119577.847940467, "forward_end_ts": 9119577.851136187, "timestamp": 1790101598.884929, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.851210548, "send_end_ts": 9119577.851864755} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119577.84824756, "preprocess_end_ts": 9119577.84866038, "forward_start_ts": 9119577.848671071, "forward_end_ts": 9119577.852039874, "timestamp": 1790101598.885622, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119577.852133228, "send_end_ts": 9119577.852556845} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.852132866, "preprocess_end_ts": 9119577.852478208, "forward_start_ts": 9119577.85248706, "forward_end_ts": 9119577.855527874, "timestamp": 1790101598.8889954, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.851488229, "recv_end_ts": 9119577.851950396, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119577.852745475, "preprocess_end_ts": 9119577.853077188, "forward_start_ts": 9119577.853086252, "forward_end_ts": 9119577.855551347, "timestamp": 1790101598.8890097, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119577.85131053, "recv_end_ts": 9119577.85258692, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.960668135, "preprocess_end_ts": 9119578.96123012, "forward_start_ts": 9119578.961244408, "forward_end_ts": 9119578.964522295, "timestamp": 1790101599.9982448, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.964636814, "send_end_ts": 9119578.96517948} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.96082784, "preprocess_end_ts": 9119578.961398747, "forward_start_ts": 9119578.961414317, "forward_end_ts": 9119578.965519715, "timestamp": 1790101599.9992168, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.965648709, "send_end_ts": 9119578.96615019} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.966426106, "preprocess_end_ts": 9119578.966873564, "forward_start_ts": 9119578.966887629, "forward_end_ts": 9119578.969609275, "timestamp": 1790101600.003122, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.960561955, "recv_end_ts": 9119578.966229323, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.965645164, "preprocess_end_ts": 9119578.966221929, "forward_start_ts": 9119578.966246396, "forward_end_ts": 9119578.969896998, "timestamp": 1790101600.003438, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.960641623, "recv_end_ts": 9119578.965302303, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.973591277, "preprocess_end_ts": 9119578.9739243, "forward_start_ts": 9119578.973934824, "forward_end_ts": 9119578.976901937, "timestamp": 1790101600.0104132, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.976986956, "send_end_ts": 9119578.977347884} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.973924132, "preprocess_end_ts": 9119578.974346958, "forward_start_ts": 9119578.974358551, "forward_end_ts": 9119578.977764774, "timestamp": 1790101600.0114522, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.977866407, "send_end_ts": 9119578.97838593} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.977527997, "preprocess_end_ts": 9119578.97787376, "forward_start_ts": 9119578.977883596, "forward_end_ts": 9119578.981314112, "timestamp": 1790101600.0147886, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.97344376, "recv_end_ts": 9119578.977390269, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.978509312, "preprocess_end_ts": 9119578.978842791, "forward_start_ts": 9119578.978853147, "forward_end_ts": 9119578.981392836, "timestamp": 1790101600.0148735, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.973179871, "recv_end_ts": 9119578.978368731, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.977689132, "preprocess_end_ts": 9119578.978013683, "forward_start_ts": 9119578.978024937, "forward_end_ts": 9119578.981589071, "timestamp": 1790101600.0153449, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.981677303, "send_end_ts": 9119578.982279787} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.978735156, "preprocess_end_ts": 9119578.979071628, "forward_start_ts": 9119578.979081316, "forward_end_ts": 9119578.981540589, "timestamp": 1790101600.0155048, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.981624395, "send_end_ts": 9119578.982438985} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.982645798, "preprocess_end_ts": 9119578.982993113, "forward_start_ts": 9119578.983002448, "forward_end_ts": 9119578.985380031, "timestamp": 1790101600.0188494, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.982039647, "recv_end_ts": 9119578.98251816, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.982495245, "preprocess_end_ts": 9119578.982915264, "forward_start_ts": 9119578.982923692, "forward_end_ts": 9119578.985420316, "timestamp": 1790101600.0188775, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.981969379, "recv_end_ts": 9119578.982336044, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.982795704, "preprocess_end_ts": 9119578.983114818, "forward_start_ts": 9119578.983123522, "forward_end_ts": 9119578.985487437, "timestamp": 1790101600.0194476, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.985561136, "send_end_ts": 9119578.986381425} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119578.982634112, "preprocess_end_ts": 9119578.982959624, "forward_start_ts": 9119578.982968632, "forward_end_ts": 9119578.985553572, "timestamp": 1790101600.0194964, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119578.985630652, "send_end_ts": 9119578.98643162} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.986565378, "preprocess_end_ts": 9119578.98689252, "forward_start_ts": 9119578.986901361, "forward_end_ts": 9119578.98951606, "timestamp": 1790101600.022985, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.986011704, "recv_end_ts": 9119578.986432532, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119578.98668281, "preprocess_end_ts": 9119578.987024937, "forward_start_ts": 9119578.987034166, "forward_end_ts": 9119578.989679078, "timestamp": 1790101600.0231612, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119578.986048156, "recv_end_ts": 9119578.986526525, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.094688626, "preprocess_end_ts": 9119580.095366264, "forward_start_ts": 9119580.095383598, "forward_end_ts": 9119580.09878654, "timestamp": 1790101601.1325057, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.098893972, "send_end_ts": 9119580.099439714} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.09495793, "preprocess_end_ts": 9119580.09543242, "forward_start_ts": 9119580.095445976, "forward_end_ts": 9119580.099593991, "timestamp": 1790101601.1333818, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.099735184, "send_end_ts": 9119580.100314468} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.099875676, "preprocess_end_ts": 9119580.100399459, "forward_start_ts": 9119580.100419892, "forward_end_ts": 9119580.104041908, "timestamp": 1790101601.1375847, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.094411604, "recv_end_ts": 9119580.099614464, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.100623075, "preprocess_end_ts": 9119580.10111244, "forward_start_ts": 9119580.101126062, "forward_end_ts": 9119580.104044553, "timestamp": 1790101601.1376822, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.09472104, "recv_end_ts": 9119580.100411221, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.109105298, "preprocess_end_ts": 9119580.109411485, "forward_start_ts": 9119580.109421978, "forward_end_ts": 9119580.111969125, "timestamp": 1790101601.1455402, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.112062896, "send_end_ts": 9119580.112475628} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.109155536, "preprocess_end_ts": 9119580.10950957, "forward_start_ts": 9119580.10952064, "forward_end_ts": 9119580.112086548, "timestamp": 1790101601.145642, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.112181759, "send_end_ts": 9119580.11257712} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.112802109, "preprocess_end_ts": 9119580.113141969, "forward_start_ts": 9119580.113151772, "forward_end_ts": 9119580.11560526, "timestamp": 1790101601.1490958, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.107629674, "recv_end_ts": 9119580.1126576, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.11271294, "preprocess_end_ts": 9119580.113064012, "forward_start_ts": 9119580.113074807, "forward_end_ts": 9119580.115638377, "timestamp": 1790101601.149124, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.109004192, "recv_end_ts": 9119580.112562789, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.11292049, "preprocess_end_ts": 9119580.11325336, "forward_start_ts": 9119580.113263048, "forward_end_ts": 9119580.115688888, "timestamp": 1790101601.1496584, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.115762897, "send_end_ts": 9119580.11659234} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.112800252, "preprocess_end_ts": 9119580.113136455, "forward_start_ts": 9119580.113145532, "forward_end_ts": 9119580.115697565, "timestamp": 1790101601.1497052, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.115773346, "send_end_ts": 9119580.116639584} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.116801972, "preprocess_end_ts": 9119580.117155287, "forward_start_ts": 9119580.117163748, "forward_end_ts": 9119580.119808223, "timestamp": 1790101601.1532898, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.116262624, "recv_end_ts": 9119580.116673816, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.11686216, "preprocess_end_ts": 9119580.117208404, "forward_start_ts": 9119580.117217377, "forward_end_ts": 9119580.12174194, "timestamp": 1790101601.1552868, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.11630194, "recv_end_ts": 9119580.116723644, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.116933325, "preprocess_end_ts": 9119580.11725814, "forward_start_ts": 9119580.117267575, "forward_end_ts": 9119580.121900078, "timestamp": 1790101601.1555579, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.121995708, "send_end_ts": 9119580.122491708} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119580.117043357, "preprocess_end_ts": 9119580.117375923, "forward_start_ts": 9119580.117384301, "forward_end_ts": 9119580.119822728, "timestamp": 1790101601.1559808, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119580.11989524, "send_end_ts": 9119580.122915024} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.122579215, "preprocess_end_ts": 9119580.122919368, "forward_start_ts": 9119580.12292863, "forward_end_ts": 9119580.125977028, "timestamp": 1790101601.159441, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.12045625, "recv_end_ts": 9119580.122441912, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119580.123137092, "preprocess_end_ts": 9119580.123574857, "forward_start_ts": 9119580.123585664, "forward_end_ts": 9119580.127115352, "timestamp": 1790101601.16059, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119580.122521417, "recv_end_ts": 9119580.122969313, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.233396504, "preprocess_end_ts": 9119581.233994463, "forward_start_ts": 9119581.234011868, "forward_end_ts": 9119581.238167107, "timestamp": 1790101602.2719352, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.238298027, "send_end_ts": 9119581.238868516} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.233019358, "preprocess_end_ts": 9119581.233695393, "forward_start_ts": 9119581.233710907, "forward_end_ts": 9119581.238902632, "timestamp": 1790101602.272517, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.239007484, "send_end_ts": 9119581.239451487} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.239113316, "preprocess_end_ts": 9119581.239545468, "forward_start_ts": 9119581.239558164, "forward_end_ts": 9119581.243230795, "timestamp": 1790101602.2767441, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.233050244, "recv_end_ts": 9119581.238915678, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.239849389, "preprocess_end_ts": 9119581.240372729, "forward_start_ts": 9119581.240387648, "forward_end_ts": 9119581.243204897, "timestamp": 1790101602.2767386, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.232745023, "recv_end_ts": 9119581.239610076, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.24686108, "preprocess_end_ts": 9119581.247199334, "forward_start_ts": 9119581.247208878, "forward_end_ts": 9119581.251029195, "timestamp": 1790101602.284532, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.251113068, "send_end_ts": 9119581.25146761} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.24822918, "preprocess_end_ts": 9119581.248552652, "forward_start_ts": 9119581.248562204, "forward_end_ts": 9119581.251013158, "timestamp": 1790101602.284648, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.25109625, "send_end_ts": 9119581.251581362} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.251669185, "preprocess_end_ts": 9119581.252024984, "forward_start_ts": 9119581.252034616, "forward_end_ts": 9119581.254945168, "timestamp": 1790101602.2884212, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.246744012, "recv_end_ts": 9119581.251521748, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.251804125, "preprocess_end_ts": 9119581.252146717, "forward_start_ts": 9119581.252156204, "forward_end_ts": 9119581.254929967, "timestamp": 1790101602.2890077, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.255009815, "send_end_ts": 9119581.255942978} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.251821188, "preprocess_end_ts": 9119581.2523388, "forward_start_ts": 9119581.252351632, "forward_end_ts": 9119581.256925946, "timestamp": 1790101602.2904587, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.248077651, "recv_end_ts": 9119581.251637876, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.251998393, "preprocess_end_ts": 9119581.252419272, "forward_start_ts": 9119581.252429983, "forward_end_ts": 9119581.256907957, "timestamp": 1790101602.2910933, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.257015044, "send_end_ts": 9119581.258026555} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.25614549, "preprocess_end_ts": 9119581.256490285, "forward_start_ts": 9119581.256499529, "forward_end_ts": 9119581.261377309, "timestamp": 1790101602.294866, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.255611897, "recv_end_ts": 9119581.255996894, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.25627696, "preprocess_end_ts": 9119581.256595548, "forward_start_ts": 9119581.256605532, "forward_end_ts": 9119581.261548655, "timestamp": 1790101602.2954452, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.26162432, "send_end_ts": 9119581.2623796} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.258215534, "preprocess_end_ts": 9119581.258696891, "forward_start_ts": 9119581.25870836, "forward_end_ts": 9119581.263100836, "timestamp": 1790101602.2966354, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.257630471, "recv_end_ts": 9119581.25805199, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.258432096, "preprocess_end_ts": 9119581.258829989, "forward_start_ts": 9119581.258840077, "forward_end_ts": 9119581.26315874, "timestamp": 1790101602.297273, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.263248276, "send_end_ts": 9119581.264206264} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.262610223, "preprocess_end_ts": 9119581.262939585, "forward_start_ts": 9119581.262948113, "forward_end_ts": 9119581.267470272, "timestamp": 1790101602.3009524, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.262037065, "recv_end_ts": 9119581.262459122, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.262731668, "preprocess_end_ts": 9119581.263055056, "forward_start_ts": 9119581.26306337, "forward_end_ts": 9119581.267570898, "timestamp": 1790101602.3015566, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.267647356, "send_end_ts": 9119581.268490193} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.264412211, "preprocess_end_ts": 9119581.264882725, "forward_start_ts": 9119581.264894173, "forward_end_ts": 9119581.269408816, "timestamp": 1790101602.3029337, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.2637949, "recv_end_ts": 9119581.264242757, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.2646153, "preprocess_end_ts": 9119581.2650162, "forward_start_ts": 9119581.265026176, "forward_end_ts": 9119581.269391648, "timestamp": 1790101602.3035674, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.26949292, "send_end_ts": 9119581.270500356} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.268705528, "preprocess_end_ts": 9119581.269011945, "forward_start_ts": 9119581.269020028, "forward_end_ts": 9119581.2737324, "timestamp": 1790101602.3072135, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.268147953, "recv_end_ts": 9119581.26856769, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.268824603, "preprocess_end_ts": 9119581.26916552, "forward_start_ts": 9119581.269175928, "forward_end_ts": 9119581.273862423, "timestamp": 1790101602.3078134, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.273937115, "send_end_ts": 9119581.274747202} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.270700352, "preprocess_end_ts": 9119581.271185782, "forward_start_ts": 9119581.271197073, "forward_end_ts": 9119581.275499867, "timestamp": 1790101602.3090343, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.270090705, "recv_end_ts": 9119581.270530147, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.270897005, "preprocess_end_ts": 9119581.271298785, "forward_start_ts": 9119581.271308472, "forward_end_ts": 9119581.275674287, "timestamp": 1790101602.3096607, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.275766958, "send_end_ts": 9119581.276593788} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.274924587, "preprocess_end_ts": 9119581.2752397, "forward_start_ts": 9119581.275248451, "forward_end_ts": 9119581.27991012, "timestamp": 1790101602.3134055, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.274385016, "recv_end_ts": 9119581.274778528, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.2750879, "preprocess_end_ts": 9119581.27542442, "forward_start_ts": 9119581.275435092, "forward_end_ts": 9119581.279930985, "timestamp": 1790101602.3139422, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.280019982, "send_end_ts": 9119581.280877493} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.27679698, "preprocess_end_ts": 9119581.277281515, "forward_start_ts": 9119581.277292844, "forward_end_ts": 9119581.281861791, "timestamp": 1790101602.3153794, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.276213275, "recv_end_ts": 9119581.276628692, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.277001103, "preprocess_end_ts": 9119581.2773985, "forward_start_ts": 9119581.277407784, "forward_end_ts": 9119581.281844368, "timestamp": 1790101602.3159869, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.281945448, "send_end_ts": 9119581.282920448} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.28111526, "preprocess_end_ts": 9119581.28146688, "forward_start_ts": 9119581.281476619, "forward_end_ts": 9119581.286099674, "timestamp": 1790101602.3195717, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.280585507, "recv_end_ts": 9119581.280974768, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.281199312, "preprocess_end_ts": 9119581.281508658, "forward_start_ts": 9119581.281517176, "forward_end_ts": 9119581.286307205, "timestamp": 1790101602.3201282, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.286381297, "send_end_ts": 9119581.287063923} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.283120563, "preprocess_end_ts": 9119581.283600004, "forward_start_ts": 9119581.283611668, "forward_end_ts": 9119581.287884478, "timestamp": 1790101602.3214347, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.282539101, "recv_end_ts": 9119581.282952696, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119581.283317965, "preprocess_end_ts": 9119581.283716684, "forward_start_ts": 9119581.28372626, "forward_end_ts": 9119581.288230387, "timestamp": 1790101602.3220446, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119581.288325038, "send_end_ts": 9119581.288978308} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.287315471, "preprocess_end_ts": 9119581.287657458, "forward_start_ts": 9119581.28766616, "forward_end_ts": 9119581.292271994, "timestamp": 1790101602.3257442, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.28679424, "recv_end_ts": 9119581.28717838, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119581.289188562, "preprocess_end_ts": 9119581.289658416, "forward_start_ts": 9119581.289669557, "forward_end_ts": 9119581.293196363, "timestamp": 1790101602.3267112, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119581.288626568, "recv_end_ts": 9119581.289021444, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.399505384, "preprocess_end_ts": 9119582.400126928, "forward_start_ts": 9119582.400153896, "forward_end_ts": 9119582.403315112, "timestamp": 1790101603.4370353, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.403440015, "send_end_ts": 9119582.403964752} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.399045372, "preprocess_end_ts": 9119582.399619577, "forward_start_ts": 9119582.399633583, "forward_end_ts": 9119582.40350036, "timestamp": 1790101603.4372048, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.40361334, "send_end_ts": 9119582.404137751} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.404470395, "preprocess_end_ts": 9119582.404925624, "forward_start_ts": 9119582.404938199, "forward_end_ts": 9119582.407814419, "timestamp": 1790101603.4413486, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.398723489, "recv_end_ts": 9119582.4042588, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.404359484, "preprocess_end_ts": 9119582.4049385, "forward_start_ts": 9119582.404952949, "forward_end_ts": 9119582.408852275, "timestamp": 1790101603.442421, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.398926804, "recv_end_ts": 9119582.404123345, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.413005928, "preprocess_end_ts": 9119582.4133225, "forward_start_ts": 9119582.413331948, "forward_end_ts": 9119582.415712643, "timestamp": 1790101603.4492488, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.415795052, "send_end_ts": 9119582.41618442} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.412871776, "preprocess_end_ts": 9119582.413243042, "forward_start_ts": 9119582.413253022, "forward_end_ts": 9119582.416106895, "timestamp": 1790101603.4496534, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.41620316, "send_end_ts": 9119582.416588666} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.4167915, "preprocess_end_ts": 9119582.417173, "forward_start_ts": 9119582.417184316, "forward_end_ts": 9119582.419704515, "timestamp": 1790101603.453184, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.412371628, "recv_end_ts": 9119582.416646589, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.416910045, "preprocess_end_ts": 9119582.417237205, "forward_start_ts": 9119582.417245656, "forward_end_ts": 9119582.41993879, "timestamp": 1790101603.4537365, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.420030707, "send_end_ts": 9119582.42067046} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.416453078, "preprocess_end_ts": 9119582.416901791, "forward_start_ts": 9119582.416913465, "forward_end_ts": 9119582.420616964, "timestamp": 1790101603.454135, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.412896767, "recv_end_ts": 9119582.41628768, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.416508028, "preprocess_end_ts": 9119582.416822912, "forward_start_ts": 9119582.41683184, "forward_end_ts": 9119582.419869848, "timestamp": 1790101603.4547722, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.419945642, "send_end_ts": 9119582.42170742} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.42084617, "preprocess_end_ts": 9119582.421175722, "forward_start_ts": 9119582.421185223, "forward_end_ts": 9119582.424993424, "timestamp": 1790101603.4584696, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.420361483, "recv_end_ts": 9119582.420718808, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.421002967, "preprocess_end_ts": 9119582.421317281, "forward_start_ts": 9119582.421325749, "forward_end_ts": 9119582.42481764, "timestamp": 1790101603.4590611, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.424892709, "send_end_ts": 9119582.425996372} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.421963971, "preprocess_end_ts": 9119582.422395866, "forward_start_ts": 9119582.422406385, "forward_end_ts": 9119582.425856559, "timestamp": 1790101603.459334, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.421367256, "recv_end_ts": 9119582.421805905, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.42202926, "preprocess_end_ts": 9119582.422343152, "forward_start_ts": 9119582.422351869, "forward_end_ts": 9119582.424909133, "timestamp": 1790101603.4599416, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.42499226, "send_end_ts": 9119582.426876452} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.42624215, "preprocess_end_ts": 9119582.426581772, "forward_start_ts": 9119582.426591672, "forward_end_ts": 9119582.430140233, "timestamp": 1790101603.4636054, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.425643357, "recv_end_ts": 9119582.426081654, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.426323969, "preprocess_end_ts": 9119582.426648231, "forward_start_ts": 9119582.426657196, "forward_end_ts": 9119582.43019246, "timestamp": 1790101603.464215, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.430267744, "send_end_ts": 9119582.431150008} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.427180208, "preprocess_end_ts": 9119582.427611424, "forward_start_ts": 9119582.427621692, "forward_end_ts": 9119582.431075092, "timestamp": 1790101603.464572, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.426556783, "recv_end_ts": 9119582.427014092, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.427201988, "preprocess_end_ts": 9119582.427503897, "forward_start_ts": 9119582.4275129, "forward_end_ts": 9119582.429906465, "timestamp": 1790101603.465202, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.429982044, "send_end_ts": 9119582.432137204} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.431276957, "preprocess_end_ts": 9119582.431606408, "forward_start_ts": 9119582.431616047, "forward_end_ts": 9119582.435424551, "timestamp": 1790101603.468891, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.430785645, "recv_end_ts": 9119582.431143759, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.431484496, "preprocess_end_ts": 9119582.431882525, "forward_start_ts": 9119582.431892028, "forward_end_ts": 9119582.435344663, "timestamp": 1790101603.4694836, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.435420744, "send_end_ts": 9119582.436418865} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.432389896, "preprocess_end_ts": 9119582.432831071, "forward_start_ts": 9119582.432842351, "forward_end_ts": 9119582.43632091, "timestamp": 1790101603.469801, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.43179648, "recv_end_ts": 9119582.432225011, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.432463683, "preprocess_end_ts": 9119582.43278008, "forward_start_ts": 9119582.432788637, "forward_end_ts": 9119582.435299978, "timestamp": 1790101603.470434, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.435382178, "send_end_ts": 9119582.437368937} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.436651569, "preprocess_end_ts": 9119582.436987516, "forward_start_ts": 9119582.436997196, "forward_end_ts": 9119582.440632757, "timestamp": 1790101603.474107, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.436072463, "recv_end_ts": 9119582.43649928, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.43674938, "preprocess_end_ts": 9119582.43708408, "forward_start_ts": 9119582.437093578, "forward_end_ts": 9119582.440551614, "timestamp": 1790101603.4746928, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.440625504, "send_end_ts": 9119582.441628233} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.437632522, "preprocess_end_ts": 9119582.438060312, "forward_start_ts": 9119582.438070623, "forward_end_ts": 9119582.441493956, "timestamp": 1790101603.4749963, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.437028263, "recv_end_ts": 9119582.437467912, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.43771562, "preprocess_end_ts": 9119582.438037302, "forward_start_ts": 9119582.438046437, "forward_end_ts": 9119582.440517, "timestamp": 1790101603.4756887, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.440590328, "send_end_ts": 9119582.442623949} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.441851616, "preprocess_end_ts": 9119582.442183517, "forward_start_ts": 9119582.442192648, "forward_end_ts": 9119582.445875488, "timestamp": 1790101603.47935, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.441275446, "recv_end_ts": 9119582.44170566, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.441955976, "preprocess_end_ts": 9119582.442265827, "forward_start_ts": 9119582.442274155, "forward_end_ts": 9119582.445797782, "timestamp": 1790101603.4799547, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.445872858, "send_end_ts": 9119582.446887596} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.442881905, "preprocess_end_ts": 9119582.443313569, "forward_start_ts": 9119582.443323756, "forward_end_ts": 9119582.4466977, "timestamp": 1790101603.480205, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.44221971, "recv_end_ts": 9119582.442719487, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119582.442944948, "preprocess_end_ts": 9119582.44325064, "forward_start_ts": 9119582.4432588, "forward_end_ts": 9119582.445691545, "timestamp": 1790101603.480905, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119582.445766605, "send_end_ts": 9119582.447840024} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.447107095, "preprocess_end_ts": 9119582.447438892, "forward_start_ts": 9119582.447447576, "forward_end_ts": 9119582.45109568, "timestamp": 1790101603.4845707, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.4465232, "recv_end_ts": 9119582.446960269, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119582.448111612, "preprocess_end_ts": 9119582.448540188, "forward_start_ts": 9119582.44855094, "forward_end_ts": 9119582.45196748, "timestamp": 1790101603.4855008, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119582.447434956, "recv_end_ts": 9119582.447922956, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.558165964, "preprocess_end_ts": 9119583.558815712, "forward_start_ts": 9119583.558831848, "forward_end_ts": 9119583.562208481, "timestamp": 1790101604.5959175, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.56232964, "send_end_ts": 9119583.56285016} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.558093507, "preprocess_end_ts": 9119583.55883775, "forward_start_ts": 9119583.558868157, "forward_end_ts": 9119583.562416296, "timestamp": 1790101604.596084, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.5625461, "send_end_ts": 9119583.563014574} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.563554471, "preprocess_end_ts": 9119583.564202191, "forward_start_ts": 9119583.564220864, "forward_end_ts": 9119583.567211548, "timestamp": 1790101604.600776, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.557605112, "recv_end_ts": 9119583.563252116, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.56323242, "preprocess_end_ts": 9119583.563799536, "forward_start_ts": 9119583.563816488, "forward_end_ts": 9119583.568085903, "timestamp": 1790101604.601647, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.557727855, "recv_end_ts": 9119583.562990518, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.572272748, "preprocess_end_ts": 9119583.572581254, "forward_start_ts": 9119583.572590098, "forward_end_ts": 9119583.57498684, "timestamp": 1790101604.6084995, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.575065097, "send_end_ts": 9119583.575435093} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.571874056, "preprocess_end_ts": 9119583.57219586, "forward_start_ts": 9119583.57220512, "forward_end_ts": 9119583.575020477, "timestamp": 1790101604.6085463, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.575109651, "send_end_ts": 9119583.575481975} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.575726492, "preprocess_end_ts": 9119583.5761047, "forward_start_ts": 9119583.576115075, "forward_end_ts": 9119583.57891527, "timestamp": 1790101604.612406, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.571748208, "recv_end_ts": 9119583.575573377, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.575801192, "preprocess_end_ts": 9119583.576123495, "forward_start_ts": 9119583.576132122, "forward_end_ts": 9119583.578572733, "timestamp": 1790101604.6129484, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.578646213, "send_end_ts": 9119583.579884293} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.575709935, "preprocess_end_ts": 9119583.576208178, "forward_start_ts": 9119583.576221002, "forward_end_ts": 9119583.579639962, "timestamp": 1790101604.6131327, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.572167996, "recv_end_ts": 9119583.575538272, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.575764172, "preprocess_end_ts": 9119583.57608008, "forward_start_ts": 9119583.5760887, "forward_end_ts": 9119583.578583768, "timestamp": 1790101604.61376, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.578659184, "send_end_ts": 9119583.58069524} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.580092108, "preprocess_end_ts": 9119583.580472684, "forward_start_ts": 9119583.580483211, "forward_end_ts": 9119583.583912114, "timestamp": 1790101604.6173956, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.579599932, "recv_end_ts": 9119583.579955412, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.580210296, "preprocess_end_ts": 9119583.580518411, "forward_start_ts": 9119583.580526842, "forward_end_ts": 9119583.584042951, "timestamp": 1790101604.6179266, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.58412996, "send_end_ts": 9119583.584862104} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.580960056, "preprocess_end_ts": 9119583.58138511, "forward_start_ts": 9119583.58139578, "forward_end_ts": 9119583.584819203, "timestamp": 1790101604.618354, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.580370374, "recv_end_ts": 9119583.580805363, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.581018431, "preprocess_end_ts": 9119583.581334531, "forward_start_ts": 9119583.581343347, "forward_end_ts": 9119583.583715232, "timestamp": 1790101604.6189904, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.583787864, "send_end_ts": 9119583.585925717} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.58506961, "preprocess_end_ts": 9119583.585432064, "forward_start_ts": 9119583.58544058, "forward_end_ts": 9119583.58920126, "timestamp": 1790101604.6226776, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.584563052, "recv_end_ts": 9119583.58492705, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.585209263, "preprocess_end_ts": 9119583.585548878, "forward_start_ts": 9119583.585558156, "forward_end_ts": 9119583.58915847, "timestamp": 1790101604.6232069, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.589234628, "send_end_ts": 9119583.590142863} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.586181484, "preprocess_end_ts": 9119583.586610127, "forward_start_ts": 9119583.58662052, "forward_end_ts": 9119583.59006598, "timestamp": 1790101604.623529, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.585576976, "recv_end_ts": 9119583.58602296, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.586244408, "preprocess_end_ts": 9119583.586552983, "forward_start_ts": 9119583.586561283, "forward_end_ts": 9119583.589022089, "timestamp": 1790101604.6242101, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.589095414, "send_end_ts": 9119583.591145668} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.590339264, "preprocess_end_ts": 9119583.590705585, "forward_start_ts": 9119583.590714654, "forward_end_ts": 9119583.594391404, "timestamp": 1790101604.627862, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.589851815, "recv_end_ts": 9119583.59020824, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.590481328, "preprocess_end_ts": 9119583.59083188, "forward_start_ts": 9119583.59084208, "forward_end_ts": 9119583.594294608, "timestamp": 1790101604.6283815, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.59436937, "send_end_ts": 9119583.59531654} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.591388373, "preprocess_end_ts": 9119583.591828829, "forward_start_ts": 9119583.591839086, "forward_end_ts": 9119583.595257632, "timestamp": 1790101604.6287599, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.59074742, "recv_end_ts": 9119583.591225345, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.591467204, "preprocess_end_ts": 9119583.591777526, "forward_start_ts": 9119583.591786014, "forward_end_ts": 9119583.594190637, "timestamp": 1790101604.6294143, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.594262984, "send_end_ts": 9119583.596349882} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.5955398, "preprocess_end_ts": 9119583.5958819, "forward_start_ts": 9119583.595890991, "forward_end_ts": 9119583.599585062, "timestamp": 1790101604.6330583, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.595037963, "recv_end_ts": 9119583.595384393, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.595650123, "preprocess_end_ts": 9119583.5959664, "forward_start_ts": 9119583.595974859, "forward_end_ts": 9119583.599450566, "timestamp": 1790101604.6335962, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.599524356, "send_end_ts": 9119583.600531695} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.596603245, "preprocess_end_ts": 9119583.597033508, "forward_start_ts": 9119583.597043771, "forward_end_ts": 9119583.600491628, "timestamp": 1790101604.633969, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.595993277, "recv_end_ts": 9119583.596438153, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.596669847, "preprocess_end_ts": 9119583.596980136, "forward_start_ts": 9119583.59698864, "forward_end_ts": 9119583.59938621, "timestamp": 1790101604.6346073, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.599459851, "send_end_ts": 9119583.601542953} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.600735752, "preprocess_end_ts": 9119583.601096991, "forward_start_ts": 9119583.601106236, "forward_end_ts": 9119583.60471283, "timestamp": 1790101604.638188, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.60024349, "recv_end_ts": 9119583.600602657, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.600859625, "preprocess_end_ts": 9119583.601178572, "forward_start_ts": 9119583.601187274, "forward_end_ts": 9119583.604693076, "timestamp": 1790101604.6387181, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.6047763, "send_end_ts": 9119583.60565389} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.601779047, "preprocess_end_ts": 9119583.602213588, "forward_start_ts": 9119583.602223707, "forward_end_ts": 9119583.605615718, "timestamp": 1790101604.6391351, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.601194521, "recv_end_ts": 9119583.6016186, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9119583.601859516, "preprocess_end_ts": 9119583.602171978, "forward_start_ts": 9119583.60218029, "forward_end_ts": 9119583.60453762, "timestamp": 1790101604.6397457, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9119583.604610441, "send_end_ts": 9119583.606681332} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.60586302, "preprocess_end_ts": 9119583.606205292, "forward_start_ts": 9119583.606213644, "forward_end_ts": 9119583.609991904, "timestamp": 1790101604.643459, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.605360894, "recv_end_ts": 9119583.605715549, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9119583.606932731, "preprocess_end_ts": 9119583.607365588, "forward_start_ts": 9119583.607375856, "forward_end_ts": 9119583.610797776, "timestamp": 1790101604.6443014, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9119583.606361384, "recv_end_ts": 9119583.606772551, "send_start_ts": null, "send_end_ts": null} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/requests.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/requests.jsonl deleted file mode 100644 index be5aaf77..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/requests.jsonl +++ /dev/null @@ -1,76 +0,0 @@ -{"request_id": "warmup-q0", "burst": "warmup", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9119570.292903332, "finish_monotonic": 9119576.664363926, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "warmup-q1", "burst": "warmup", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9119570.2957572, "finish_monotonic": 9119576.664373389, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "warmup-q2", "burst": "warmup", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9119570.296333695, "finish_monotonic": 9119576.670321444, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "warmup-q3", "burst": "warmup", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9119570.296722282, "finish_monotonic": 9119576.670327768, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q0", "burst": "b8-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9119577.822749784, "finish_monotonic": 9119577.835280377, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q1", "burst": "b8-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9119577.823230114, "finish_monotonic": 9119577.83628296, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q2", "burst": "b8-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9119577.823472217, "finish_monotonic": 9119577.847771548, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q3", "burst": "b8-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9119577.823677383, "finish_monotonic": 9119577.847779376, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q4", "burst": "b8-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9119577.823869893, "finish_monotonic": 9119577.85206842, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q5", "burst": "b8-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9119577.824067151, "finish_monotonic": 9119577.852074748, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q6", "burst": "b8-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9119577.824303448, "finish_monotonic": 9119577.856742447, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q7", "burst": "b8-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9119577.824581333, "finish_monotonic": 9119577.856747039, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q0", "burst": "b8-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9119578.95893912, "finish_monotonic": 9119578.9712066, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q1", "burst": "b8-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9119578.959609669, "finish_monotonic": 9119578.971213512, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q2", "burst": "b8-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9119578.95999268, "finish_monotonic": 9119578.982687123, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q3", "burst": "b8-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9119578.960343532, "finish_monotonic": 9119578.982695224, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q4", "burst": "b8-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9119578.9606352, "finish_monotonic": 9119578.986493504, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q5", "burst": "b8-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9119578.960871626, "finish_monotonic": 9119578.986498725, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q6", "burst": "b8-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9119578.961075956, "finish_monotonic": 9119578.990688741, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q7", "burst": "b8-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9119578.961258136, "finish_monotonic": 9119578.990693668, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q0", "burst": "b8-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9119580.093087101, "finish_monotonic": 9119580.105471551, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q1", "burst": "b8-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9119580.093768604, "finish_monotonic": 9119580.105479572, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q2", "burst": "b8-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9119580.094143052, "finish_monotonic": 9119580.116958952, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q3", "burst": "b8-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9119580.094440961, "finish_monotonic": 9119580.116967266, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q4", "burst": "b8-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9119580.094725056, "finish_monotonic": 9119580.122895185, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q5", "burst": "b8-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9119580.09502366, "finish_monotonic": 9119580.122903796, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q6", "burst": "b8-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9119580.095331525, "finish_monotonic": 9119580.128065204, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q7", "burst": "b8-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9119580.095670745, "finish_monotonic": 9119580.12807052, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q0", "burst": "b16-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9119581.231372556, "finish_monotonic": 9119581.244669287, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q1", "burst": "b16-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9119581.232019192, "finish_monotonic": 9119581.244676992, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q2", "burst": "b16-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9119581.23240773, "finish_monotonic": 9119581.256066673, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q3", "burst": "b16-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9119581.232733017, "finish_monotonic": 9119581.258137325, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q4", "burst": "b16-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9119581.233046625, "finish_monotonic": 9119581.262572609, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q5", "burst": "b16-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9119581.233357828, "finish_monotonic": 9119581.264275284, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q6", "burst": "b16-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9119581.233661748, "finish_monotonic": 9119581.268603608, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q7", "burst": "b16-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9119581.233952317, "finish_monotonic": 9119581.2705439, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q8", "burst": "b16-r0", "round": 0, "index": 8, "rank": 0, "submit_monotonic": 9119581.234164078, "finish_monotonic": 9119581.274798244, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q9", "burst": "b16-r0", "round": 0, "index": 9, "rank": 1, "submit_monotonic": 9119581.234352224, "finish_monotonic": 9119581.276661308, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q10", "burst": "b16-r0", "round": 0, "index": 10, "rank": 0, "submit_monotonic": 9119581.234614696, "finish_monotonic": 9119581.280968891, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q11", "burst": "b16-r0", "round": 0, "index": 11, "rank": 1, "submit_monotonic": 9119581.234914288, "finish_monotonic": 9119581.283019535, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q12", "burst": "b16-r0", "round": 0, "index": 12, "rank": 0, "submit_monotonic": 9119581.235235328, "finish_monotonic": 9119581.28715656, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q13", "burst": "b16-r0", "round": 0, "index": 13, "rank": 1, "submit_monotonic": 9119581.235567328, "finish_monotonic": 9119581.288995408, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q14", "burst": "b16-r0", "round": 0, "index": 14, "rank": 0, "submit_monotonic": 9119581.235901883, "finish_monotonic": 9119581.293288851, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q15", "burst": "b16-r0", "round": 0, "index": 15, "rank": 1, "submit_monotonic": 9119581.23628505, "finish_monotonic": 9119581.294291746, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q0", "burst": "b16-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9119582.397575928, "finish_monotonic": 9119582.409182351, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q1", "burst": "b16-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9119582.398024382, "finish_monotonic": 9119582.410082297, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q2", "burst": "b16-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9119582.398251675, "finish_monotonic": 9119582.420878284, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q3", "burst": "b16-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9119582.398454148, "finish_monotonic": 9119582.421804003, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q4", "burst": "b16-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9119582.398645584, "finish_monotonic": 9119582.426290512, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q5", "burst": "b16-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9119582.398865044, "finish_monotonic": 9119582.4269211, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q6", "burst": "b16-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9119582.399150457, "finish_monotonic": 9119582.431440856, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q7", "burst": "b16-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9119582.399460008, "finish_monotonic": 9119582.432184016, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q8", "burst": "b16-r1", "round": 1, "index": 8, "rank": 0, "submit_monotonic": 9119582.399763105, "finish_monotonic": 9119582.436708044, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q9", "burst": "b16-r1", "round": 1, "index": 9, "rank": 1, "submit_monotonic": 9119582.400090864, "finish_monotonic": 9119582.437363334, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q10", "burst": "b16-r1", "round": 1, "index": 10, "rank": 0, "submit_monotonic": 9119582.400385031, "finish_monotonic": 9119582.441793997, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q11", "burst": "b16-r1", "round": 1, "index": 11, "rank": 1, "submit_monotonic": 9119582.400697349, "finish_monotonic": 9119582.442543592, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q12", "burst": "b16-r1", "round": 1, "index": 12, "rank": 0, "submit_monotonic": 9119582.40096812, "finish_monotonic": 9119582.446937904, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q13", "burst": "b16-r1", "round": 1, "index": 13, "rank": 1, "submit_monotonic": 9119582.40129884, "finish_monotonic": 9119582.4477446, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q14", "burst": "b16-r1", "round": 1, "index": 14, "rank": 0, "submit_monotonic": 9119582.401608504, "finish_monotonic": 9119582.452116895, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q15", "burst": "b16-r1", "round": 1, "index": 15, "rank": 1, "submit_monotonic": 9119582.401885726, "finish_monotonic": 9119582.452971255, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q0", "burst": "b16-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9119583.556243872, "finish_monotonic": 9119583.568614528, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q1", "burst": "b16-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9119583.556679232, "finish_monotonic": 9119583.569503242, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q2", "burst": "b16-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9119583.556910612, "finish_monotonic": 9119583.580152176, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q3", "burst": "b16-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9119583.557116171, "finish_monotonic": 9119583.58094232, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q4", "burst": "b16-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9119583.557302792, "finish_monotonic": 9119583.585108727, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q5", "burst": "b16-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9119583.55747951, "finish_monotonic": 9119583.58618874, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q6", "burst": "b16-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9119583.55769601, "finish_monotonic": 9119583.590519594, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q7", "burst": "b16-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9119583.557939816, "finish_monotonic": 9119583.591250531, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q8", "burst": "b16-r2", "round": 2, "index": 8, "rank": 0, "submit_monotonic": 9119583.558238704, "finish_monotonic": 9119583.595475748, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q9", "burst": "b16-r2", "round": 2, "index": 9, "rank": 1, "submit_monotonic": 9119583.558546081, "finish_monotonic": 9119583.596459309, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q10", "burst": "b16-r2", "round": 2, "index": 10, "rank": 0, "submit_monotonic": 9119583.55885946, "finish_monotonic": 9119583.600849977, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q11", "burst": "b16-r2", "round": 2, "index": 11, "rank": 1, "submit_monotonic": 9119583.55918352, "finish_monotonic": 9119583.601664387, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q12", "burst": "b16-r2", "round": 2, "index": 12, "rank": 0, "submit_monotonic": 9119583.5594886, "finish_monotonic": 9119583.605823291, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q13", "burst": "b16-r2", "round": 2, "index": 13, "rank": 1, "submit_monotonic": 9119583.559807884, "finish_monotonic": 9119583.606727397, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q14", "burst": "b16-r2", "round": 2, "index": 14, "rank": 0, "submit_monotonic": 9119583.5601061, "finish_monotonic": 9119583.611053113, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q15", "burst": "b16-r2", "round": 2, "index": 15, "rank": 1, "submit_monotonic": 9119583.560411915, "finish_monotonic": 9119583.611848287, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/summary.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/summary.json deleted file mode 100644 index b1c8e0d3..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/dense/summary.json +++ /dev/null @@ -1,186 +0,0 @@ -{ - "model_config": "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/data/config/models/Llama-3.2-1B-Instruct.json", - "num_gpu_blocks": 304854, - "block_size": 16, - "engine_args": { - "model": "/tmp/stage_admission_pp/sa-pp-20260923a/runs/dense/model", - "served_model_name": null, - "tokenizer": null, - "hf_config_path": null, - "runner": "auto", - "convert": "auto", - "task": null, - "skip_tokenizer_init": true, - "enable_prompt_embeds": false, - "tokenizer_mode": "auto", - "trust_remote_code": false, - "allowed_local_media_path": "", - "download_dir": null, - "safetensors_load_strategy": "lazy", - "load_format": "dummy", - "config_format": "auto", - "dtype": "bfloat16", - "kv_cache_dtype": "auto", - "seed": 0, - "max_model_len": 512, - "distributed_executor_backend": null, - "pipeline_parallel_size": 2, - "tensor_parallel_size": 1, - "decode_context_parallel_size": 1, - "data_parallel_size": 2, - "data_parallel_rank": null, - "data_parallel_start_rank": null, - "data_parallel_size_local": null, - "data_parallel_address": null, - "data_parallel_rpc_port": null, - "data_parallel_hybrid_lb": false, - "data_parallel_backend": "mp", - "enable_expert_parallel": false, - "enable_eplb": false, - "num_redundant_experts": 0, - "eplb_window_size": 1000, - "eplb_step_interval": 3000, - "eplb_log_balancedness": false, - "max_parallel_loading_workers": null, - "block_size": 16, - "enable_prefix_caching": false, - "prefix_caching_hash_algo": "sha256", - "disable_sliding_window": false, - "disable_cascade_attn": false, - "swap_space": 4, - "cpu_offload_gb": 0, - "gpu_memory_utilization": 0.5, - "kv_cache_memory_bytes": null, - "max_num_batched_tokens": 256, - "max_num_partial_prefills": 1, - "max_long_partial_prefills": 1, - "long_prefill_token_threshold": 0, - "max_num_seqs": 4, - "max_logprobs": 20, - "disable_log_stats": true, - "revision": null, - "code_revision": null, - "rope_theta": null, - "hf_token": null, - "tokenizer_revision": null, - "quantization": null, - "enforce_eager": true, - "max_seq_len_to_capture": 8192, - "disable_custom_all_reduce": false, - "interleave_mm_strings": false, - "mm_processor_kwargs": null, - "disable_mm_preprocessor_cache": false, - "mm_processor_cache_gb": 4, - "mm_encoder_tp_mode": "weights", - "io_processor_plugin": null, - "skip_mm_profiling": false, - "enable_lora": false, - "enable_lora_bias": false, - "max_loras": 1, - "max_lora_rank": 16, - "default_mm_loras": null, - "fully_sharded_loras": false, - "max_cpu_loras": null, - "lora_dtype": "auto", - "lora_extra_vocab_size": 256, - "ray_workers_use_nsight": false, - "num_gpu_blocks_override": null, - "num_lookahead_slots": 0, - "ignore_patterns": null, - "preemption_mode": null, - "scheduler_delay_factor": 0.0, - "enable_chunked_prefill": true, - "disable_chunked_mm_input": false, - "disable_hybrid_kv_cache_manager": false, - "guided_decoding_backend": "auto", - "guided_decoding_disable_fallback": false, - "guided_decoding_disable_any_whitespace": false, - "guided_decoding_disable_additional_properties": false, - "logits_processor_pattern": null, - "speculative_config": null, - "show_hidden_metrics_for_version": null, - "otlp_traces_endpoint": null, - "collect_detailed_traces": null, - "disable_async_output_proc": false, - "scheduling_policy": "fcfs", - "scheduler_cls": "vllm.v1.core.sched.scheduler.Scheduler", - "override_pooler_config": null, - "worker_cls": "auto", - "worker_extension_cls": "", - "kv_transfer_config": null, - "kv_events_config": null, - "generation_config": "auto", - "enable_sleep_mode": false, - "model_impl": "auto", - "override_attention_dtype": null, - "calculate_kv_scales": false, - "mamba_cache_dtype": "auto", - "mamba_ssm_cache_dtype": "auto", - "reasoning_parser": "", - "use_tqdm_on_load": true, - "pt_load_map_location": "cpu", - "enable_multimodal_encoder_data_parallel": false, - "logits_processors": null, - "async_scheduling": false, - "kv_sharing_fast_prefill": false, - "enable_log_requests": false - }, - "rounds": [ - { - "label": "warmup", - "round": 0, - "num_requests": 4, - "wall_minus_monotonic_before": 1780982021.0330582, - "wall_minus_monotonic_after": 1780982021.0330582, - "idle_wait_s": 1.1522713880985975 - }, - { - "label": "b8-r0", - "round": 0, - "num_requests": 8, - "wall_minus_monotonic_before": 1780982021.0330582, - "wall_minus_monotonic_after": 1780982021.033058, - "idle_wait_s": 1.1019583977758884 - }, - { - "label": "b8-r1", - "round": 1, - "num_requests": 8, - "wall_minus_monotonic_before": 1780982021.0330582, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.1021581627428532 - }, - { - "label": "b8-r2", - "round": 2, - "num_requests": 8, - "wall_minus_monotonic_before": 1780982021.0330584, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.1030645035207272 - }, - { - "label": "b16-r0", - "round": 0, - "num_requests": 16, - "wall_minus_monotonic_before": 1780982021.0330584, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.1031487435102463 - }, - { - "label": "b16-r1", - "round": 1, - "num_requests": 16, - "wall_minus_monotonic_before": 1780982021.0330584, - "wall_minus_monotonic_after": 1780982021.0330582, - "idle_wait_s": 1.1031373273581266 - }, - { - "label": "b16-r2", - "round": 2, - "num_requests": 16, - "wall_minus_monotonic_before": 1780982021.0330586, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.1024576723575592 - } - ] -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/moe/model/config.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/moe/model/config.json deleted file mode 100644 index 9b897034..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/runs/moe/model/config.json +++ /dev/null @@ -1,95 +0,0 @@ -{ - "_comment": "Qwen3-30B-A3B-tiny: Modified Qwen3-30B-A3B for MoE profiling testing (reduced layers and experts)", - "vocab_size": 151936, - "max_position_embeddings": 40960, - "hidden_size": 2048, - "intermediate_size": 6144, - "num_hidden_layers": 8, - "num_attention_heads": 32, - "use_sliding_window": false, - "sliding_window": null, - "num_key_value_heads": 4, - "hidden_act": "silu", - "initializer_range": 0.02, - "rms_norm_eps": 1e-06, - "use_cache": true, - "rope_theta": 1000000.0, - "rope_scaling": null, - "attention_bias": false, - "attention_dropout": 0.0, - "decoder_sparse_step": 1, - "moe_intermediate_size": 768, - "num_experts_per_tok": 8, - "num_experts": 16, - "norm_topk_prob": true, - "output_router_logits": false, - "router_aux_loss_coef": 0.001, - "mlp_only_layers": [], - "return_dict": true, - "output_hidden_states": false, - "torchscript": false, - "dtype": "bfloat16", - "pruned_heads": {}, - "tie_word_embeddings": false, - "chunk_size_feed_forward": 0, - "is_encoder_decoder": false, - "is_decoder": false, - "cross_attention_hidden_size": null, - "add_cross_attention": false, - "tie_encoder_decoder": false, - "architectures": [ - "Qwen3MoeForCausalLM" - ], - "finetuning_task": null, - "id2label": { - "0": "LABEL_0", - "1": "LABEL_1" - }, - "label2id": { - "LABEL_0": 0, - "LABEL_1": 1 - }, - "task_specific_params": null, - "problem_type": null, - "tokenizer_class": null, - "prefix": null, - "bos_token_id": 151643, - "pad_token_id": null, - "eos_token_id": 151645, - "sep_token_id": null, - "decoder_start_token_id": null, - "max_length": 20, - "min_length": 0, - "do_sample": false, - "early_stopping": false, - "num_beams": 1, - "temperature": 1.0, - "top_k": 50, - "top_p": 1.0, - "typical_p": 1.0, - "repetition_penalty": 1.0, - "length_penalty": 1.0, - "no_repeat_ngram_size": 0, - "encoder_no_repeat_ngram_size": 0, - "bad_words_ids": null, - "num_return_sequences": 1, - "output_scores": false, - "return_dict_in_generate": false, - "forced_bos_token_id": null, - "forced_eos_token_id": null, - "remove_invalid_values": false, - "exponential_decay_length_penalty": null, - "suppress_tokens": null, - "begin_suppress_tokens": null, - "num_beam_groups": 1, - "diversity_penalty": 0.0, - "_name_or_path": "Qwen/Qwen3-30B-A3B", - "transformers_version": "4.57.3", - "head_dim": 128, - "max_window_layers": 48, - "model_type": "qwen3_moe", - "tf_legacy_loss": false, - "use_bfloat16": false, - "output_attentions": false, - "use_qk_norm": true -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/vllm_import.txt b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/vllm_import.txt deleted file mode 100644 index 6aa03d4f..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/vllm_import.txt +++ /dev/null @@ -1 +0,0 @@ -VLLM_IMPORT 0.10.2 /tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/__init__.py /tmp/stage_admission_pp/sa-pp-20260923a/overlay/vllm/v1/frontier_trace.py diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/worker_env.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/worker_env.json deleted file mode 100644 index 00b7f124..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923a/worker_env.json +++ /dev/null @@ -1 +0,0 @@ -{"python": "3.12.11", "torch": "2.8.0+cu128", "cuda_available": true, "device_count": 4, "devices": ["NVIDIA H800", "NVIDIA H800", "NVIDIA H800", "NVIDIA H800"]} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/COMPLETE b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/COMPLETE deleted file mode 100644 index bbfcfc41..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/COMPLETE +++ /dev/null @@ -1 +0,0 @@ -status=0 diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/overlay_report.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/overlay_report.json deleted file mode 100644 index f3a11b19..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/overlay_report.json +++ /dev/null @@ -1,142 +0,0 @@ -{ - "site_vllm": "/usr/local/lib/python3.12/dist-packages/vllm", - "checkout": "/data/ycfeng/Frontier/.real-engine/vLLM-BS", - "overlay": "/tmp/stage_admission_pp/sa-pp-20260923b/overlay/vllm", - "differing_py_files": [ - "vllm/_C.py", - "vllm/_custom_ops.py", - "vllm/_moe_C.py", - "vllm/attention/layer.py", - "vllm/benchmarks/throughput.py", - "vllm/compilation/compiler_interface.py", - "vllm/config/__init__.py", - "vllm/distributed/communication_op.py", - "vllm/distributed/kv_transfer/kv_connector/v1/base.py", - "vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/frontier_kv_transfer_logger.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", - "vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py", - "vllm/distributed/parallel_state.py", - "vllm/engine/arg_utils.py", - "vllm/engine/llm_engine.py", - "vllm/entrypoints/openai/frontier_request_metrics.py", - "vllm/entrypoints/openai/serving_chat.py", - "vllm/entrypoints/openai/serving_completion.py", - "vllm/entrypoints/openai/serving_engine.py", - "vllm/envs.py", - "vllm/model_executor/custom_op.py", - "vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", - "vllm/model_executor/layers/fused_moe/deepep_ll_prepare_finalize.py", - "vllm/model_executor/layers/fused_moe/fused_moe.py", - "vllm/model_executor/layers/fused_moe/layer.py", - "vllm/model_executor/layers/fused_moe/pplx_prepare_finalize.py", - "vllm/model_executor/layers/linear.py", - "vllm/model_executor/layers/vocab_parallel_embedding.py", - "vllm/model_executor/models/llama.py", - "vllm/model_executor/models/phimoe.py", - "vllm/model_executor/models/qwen3_moe.py", - "vllm/model_executor/models/qwen3_moe_mtp.py", - "vllm/model_executor/models/registry.py", - "vllm/request_generator/__init__.py", - "vllm/request_generator/config.py", - "vllm/request_generator/kv_sync.py", - "vllm/request_generator/prompt_generator.py", - "vllm/request_generator/vllm_request_generator.py", - "vllm/v1/attention/backends/flash_attn.py", - "vllm/v1/attention/backends/flashinfer.py", - "vllm/v1/attention/backends/mla/common.py", - "vllm/v1/attention/backends/mla/flashinfer_mla.py", - "vllm/v1/attention/backends/utils.py", - "vllm/v1/core/sched/scheduler.py", - "vllm/v1/engine/coordinator.py", - "vllm/v1/engine/core.py", - "vllm/v1/engine/core_client.py", - "vllm/v1/engine/output_processor.py", - "vllm/v1/engine/processor.py", - "vllm/v1/frontier_trace.py", - "vllm/v1/metrics/stats.py", - "vllm/v1/spec_decode/eagle.py", - "vllm/v1/utils.py", - "vllm/v1/worker/gpu_model_runner.py", - "vllm/v1/worker/gpu_worker.py", - "vllm/worker/model_runner.py", - "vllm/worker/worker.py" - ], - "expected_py_changes": [ - "vllm/_C.py", - "vllm/_custom_ops.py", - "vllm/_moe_C.py", - "vllm/attention/layer.py", - "vllm/benchmarks/throughput.py", - "vllm/compilation/compiler_interface.py", - "vllm/config/__init__.py", - "vllm/distributed/communication_op.py", - "vllm/distributed/kv_transfer/kv_connector/v1/base.py", - "vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/frontier_kv_transfer_logger.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", - "vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", - "vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py", - "vllm/distributed/parallel_state.py", - "vllm/engine/arg_utils.py", - "vllm/engine/llm_engine.py", - "vllm/entrypoints/openai/frontier_request_metrics.py", - "vllm/entrypoints/openai/serving_chat.py", - "vllm/entrypoints/openai/serving_completion.py", - "vllm/entrypoints/openai/serving_engine.py", - "vllm/envs.py", - "vllm/model_executor/custom_op.py", - "vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", - "vllm/model_executor/layers/fused_moe/deepep_ll_prepare_finalize.py", - "vllm/model_executor/layers/fused_moe/fused_moe.py", - "vllm/model_executor/layers/fused_moe/layer.py", - "vllm/model_executor/layers/fused_moe/pplx_prepare_finalize.py", - "vllm/model_executor/layers/linear.py", - "vllm/model_executor/layers/vocab_parallel_embedding.py", - "vllm/model_executor/models/llama.py", - "vllm/model_executor/models/phimoe.py", - "vllm/model_executor/models/qwen3_moe.py", - "vllm/model_executor/models/qwen3_moe_mtp.py", - "vllm/model_executor/models/registry.py", - "vllm/request_generator/__init__.py", - "vllm/request_generator/config.py", - "vllm/request_generator/kv_sync.py", - "vllm/request_generator/prompt_generator.py", - "vllm/request_generator/vllm_request_generator.py", - "vllm/v1/attention/backends/flash_attn.py", - "vllm/v1/attention/backends/flashinfer.py", - "vllm/v1/attention/backends/mla/common.py", - "vllm/v1/attention/backends/mla/flashinfer_mla.py", - "vllm/v1/attention/backends/utils.py", - "vllm/v1/core/sched/scheduler.py", - "vllm/v1/engine/coordinator.py", - "vllm/v1/engine/core.py", - "vllm/v1/engine/core_client.py", - "vllm/v1/engine/output_processor.py", - "vllm/v1/engine/processor.py", - "vllm/v1/frontier_trace.py", - "vllm/v1/metrics/stats.py", - "vllm/v1/spec_decode/eagle.py", - "vllm/v1/utils.py", - "vllm/v1/worker/gpu_model_runner.py", - "vllm/v1/worker/gpu_worker.py", - "vllm/worker/model_runner.py", - "vllm/worker/worker.py" - ], - "unexpected": [], - "missing": [], - "accepted": true, - "patch": { - "path": "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch", - "sha256": "8d47678911b7b689bc644ea81bd9d2200ea296d773ee9c07a1c5a9bc9b3a9c81", - "files": [ - "vllm/_custom_ops.py", - "vllm/model_executor/layers/fused_moe/fused_moe.py" - ], - "equal_to_image_after_patch": { - "vllm/_custom_ops.py": true, - "vllm/model_executor/layers/fused_moe/fused_moe.py": false - } - } -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/replica_log.txt b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/replica_log.txt deleted file mode 100644 index 32857af2..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/replica_log.txt +++ /dev/null @@ -1,14 +0,0 @@ -# Replica log tail of RJob exp-0923-024146-345158 (codesign, H800 x4, creator i-fengyicheng). -# Platform init lines (node addresses, NCCL interface settings) are removed; workload output is verbatim. -{"python": "3.12.11", "torch": "2.8.0+cu128", "cuda_available": true, "device_count": 4, "devices": ["NVIDIA H800", "NVIDIA H800", "NVIDIA H800", "NVIDIA H800"]} -{"accepted": true, "unexpected": [], "missing": []} -/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. - import pynvml # type: ignore[import] -VLLM_IMPORT 0.10.2 /tmp/stage_admission_pp/sa-pp-20260923b/overlay/vllm/__init__.py /tmp/stage_admission_pp/sa-pp-20260923b/overlay/vllm/v1/frontier_trace.py -SCENARIO_PASS moe -SCENARIO_PASS dense -moe DRIVER_DONE {"rounds": 7, "num_gpu_blocks": 600666} -152 /tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/pp_boundary.jsonl -dense DRIVER_DONE {"rounds": 7, "num_gpu_blocks": 304854} -152 /tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/pp_boundary.jsonl -WORKER_STATUS=0 RUN_TAG=sa-pp-20260923b diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_844.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_844.jsonl deleted file mode 100644 index c04d76a6..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_844.jsonl +++ /dev/null @@ -1,22 +0,0 @@ -{"kind": "frontend_snapshot", "pid": 844, "seq": 0, "monotonic": 9120624.62208326, "snapshot": 2, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 1, "monotonic": 9120626.021095267, "snapshot": 3, "counts": [[1, 1], [1, 1]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 2, "monotonic": 9120626.18612816, "snapshot": 4, "counts": [[0, 1], [0, 1]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 3, "monotonic": 9120626.285689717, "snapshot": 5, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 4, "monotonic": 9120627.350581912, "snapshot": 6, "counts": [[0, 1], [0, 1]], "wave": 1, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 5, "monotonic": 9120627.45029127, "snapshot": 7, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 844, "seq": 6, "monotonic": 9120627.550534572, "snapshot": 8, "counts": [[0, 0], [0, 0]], "wave": 2, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 844, "seq": 7, "monotonic": 9120628.489587313, "snapshot": 9, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 8, "monotonic": 9120628.589672543, "snapshot": 10, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 9, "monotonic": 9120628.689554924, "snapshot": 11, "counts": [[0, 0], [0, 0]], "wave": 3, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 844, "seq": 10, "monotonic": 9120629.62920076, "snapshot": 12, "counts": [[0, 1], [0, 1]], "wave": 3, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 11, "monotonic": 9120629.729838043, "snapshot": 13, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 844, "seq": 12, "monotonic": 9120629.83007456, "snapshot": 14, "counts": [[0, 0], [0, 0]], "wave": 4, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 844, "seq": 13, "monotonic": 9120630.766499234, "snapshot": 15, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 14, "monotonic": 9120630.866801092, "snapshot": 16, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 844, "seq": 15, "monotonic": 9120630.967000738, "snapshot": 17, "counts": [[0, 0], [0, 0]], "wave": 5, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 844, "seq": 16, "monotonic": 9120631.86492882, "snapshot": 18, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 17, "monotonic": 9120631.965198906, "snapshot": 19, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 18, "monotonic": 9120632.065105148, "snapshot": 20, "counts": [[0, 0], [0, 0]], "wave": 6, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 844, "seq": 19, "monotonic": 9120633.018878696, "snapshot": 21, "counts": [[0, 1], [0, 1]], "wave": 6, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 844, "seq": 20, "monotonic": 9120633.118211748, "snapshot": 22, "counts": [[0, 1], [0, 1]], "wave": 7, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 844, "seq": 21, "monotonic": 9120633.218477592, "snapshot": 23, "counts": [[0, 0], [0, 0]], "wave": 7, "engines_running": false} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_919.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_919.jsonl deleted file mode 100644 index d5cba488..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_919.jsonl +++ /dev/null @@ -1,65 +0,0 @@ -{"kind": "engine_iteration", "pid": 919, "seq": 0, "monotonic": 9120625.96791757, "engine": 0, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 1, "monotonic": 9120626.180983996, "engine": 0, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 2, "monotonic": 9120626.18561802, "engine": 0, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 3, "monotonic": 9120626.189165356, "engine": 0, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 4, "monotonic": 9120626.191278495, "engine": 0, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 5, "monotonic": 9120627.338840736, "engine": 0, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 6, "monotonic": 9120627.35003926, "engine": 0, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 7, "monotonic": 9120627.352656804, "engine": 0, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 8, "monotonic": 9120627.362038797, "engine": 0, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 9, "monotonic": 9120627.368717212, "engine": 0, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 10, "monotonic": 9120627.37527868, "engine": 0, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 11, "monotonic": 9120627.378601518, "engine": 0, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 12, "monotonic": 9120627.380811714, "engine": 0, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 13, "monotonic": 9120628.47956062, "engine": 0, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 14, "monotonic": 9120628.4902294, "engine": 0, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 15, "monotonic": 9120628.493471997, "engine": 0, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 16, "monotonic": 9120628.502409011, "engine": 0, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 17, "monotonic": 9120628.507723143, "engine": 0, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 18, "monotonic": 9120628.513031827, "engine": 0, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 19, "monotonic": 9120628.516112473, "engine": 0, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 20, "monotonic": 9120628.519086625, "engine": 0, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 21, "monotonic": 9120629.616881263, "engine": 0, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 22, "monotonic": 9120629.62966153, "engine": 0, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 23, "monotonic": 9120629.632997084, "engine": 0, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 24, "monotonic": 9120629.642491208, "engine": 0, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 25, "monotonic": 9120629.647711592, "engine": 0, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 26, "monotonic": 9120629.652978007, "engine": 0, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 27, "monotonic": 9120629.656049391, "engine": 0, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 28, "monotonic": 9120629.658983247, "engine": 0, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 29, "monotonic": 9120630.756324528, "engine": 0, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 30, "monotonic": 9120630.765886173, "engine": 0, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 31, "monotonic": 9120630.7685595, "engine": 0, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 32, "monotonic": 9120630.776217248, "engine": 0, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 33, "monotonic": 9120630.780365208, "engine": 0, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 34, "monotonic": 9120630.784431065, "engine": 0, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q8"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 35, "monotonic": 9120630.788409028, "engine": 0, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q10"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 36, "monotonic": 9120630.79243354, "engine": 0, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q12"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 37, "monotonic": 9120630.796628807, "engine": 0, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q14"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 38, "monotonic": 9120630.800747246, "engine": 0, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 39, "monotonic": 9120630.803301813, "engine": 0, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 40, "monotonic": 9120630.80553382, "engine": 0, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 41, "monotonic": 9120631.854623005, "engine": 0, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 42, "monotonic": 9120631.864230804, "engine": 0, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 43, "monotonic": 9120631.866910912, "engine": 0, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 44, "monotonic": 9120631.87423912, "engine": 0, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 45, "monotonic": 9120631.87832478, "engine": 0, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 46, "monotonic": 9120631.882377015, "engine": 0, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q8"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 47, "monotonic": 9120631.888180677, "engine": 0, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q10"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 48, "monotonic": 9120631.89380424, "engine": 0, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q12"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 49, "monotonic": 9120631.899139255, "engine": 0, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q14"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 50, "monotonic": 9120631.90448134, "engine": 0, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 51, "monotonic": 9120631.906846043, "engine": 0, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 52, "monotonic": 9120631.910034545, "engine": 0, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 53, "monotonic": 9120633.008425832, "engine": 0, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 54, "monotonic": 9120633.019671384, "engine": 0, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 55, "monotonic": 9120633.022307867, "engine": 0, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 56, "monotonic": 9120633.029821588, "engine": 0, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 57, "monotonic": 9120633.03408219, "engine": 0, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 58, "monotonic": 9120633.038265277, "engine": 0, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q8"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 59, "monotonic": 9120633.042499, "engine": 0, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q10"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 60, "monotonic": 9120633.049114825, "engine": 0, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q12"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 61, "monotonic": 9120633.053355124, "engine": 0, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q14"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 919, "seq": 62, "monotonic": 9120633.057866, "engine": 0, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 63, "monotonic": 9120633.060146462, "engine": 0, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 919, "seq": 64, "monotonic": 9120633.063857228, "engine": 0, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_920.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_920.jsonl deleted file mode 100644 index 8318fd1b..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/dp_placement/dp_placement_920.jsonl +++ /dev/null @@ -1,65 +0,0 @@ -{"kind": "engine_iteration", "pid": 920, "seq": 0, "monotonic": 9120625.970102616, "engine": 1, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 1, "monotonic": 9120626.180614032, "engine": 1, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 2, "monotonic": 9120626.185660796, "engine": 1, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 3, "monotonic": 9120626.18911583, "engine": 1, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 4, "monotonic": 9120626.19126469, "engine": 1, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 5, "monotonic": 9120627.338931331, "engine": 1, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 6, "monotonic": 9120627.350064572, "engine": 1, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 7, "monotonic": 9120627.354388159, "engine": 1, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 8, "monotonic": 9120627.364628045, "engine": 1, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 9, "monotonic": 9120627.371056518, "engine": 1, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 10, "monotonic": 9120627.376390046, "engine": 1, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 11, "monotonic": 9120627.378577605, "engine": 1, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 12, "monotonic": 9120627.380741952, "engine": 1, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 13, "monotonic": 9120628.479725206, "engine": 1, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 14, "monotonic": 9120628.489064472, "engine": 1, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 15, "monotonic": 9120628.492970873, "engine": 1, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 16, "monotonic": 9120628.501319665, "engine": 1, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 17, "monotonic": 9120628.506610187, "engine": 1, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 18, "monotonic": 9120628.511955587, "engine": 1, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 19, "monotonic": 9120628.51537628, "engine": 1, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 20, "monotonic": 9120628.518301548, "engine": 1, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 21, "monotonic": 9120629.617205078, "engine": 1, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 22, "monotonic": 9120629.628686544, "engine": 1, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 23, "monotonic": 9120629.632389316, "engine": 1, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 24, "monotonic": 9120629.641484171, "engine": 1, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 25, "monotonic": 9120629.646681543, "engine": 1, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 26, "monotonic": 9120629.651882049, "engine": 1, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 27, "monotonic": 9120629.65544816, "engine": 1, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 28, "monotonic": 9120629.658302251, "engine": 1, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 29, "monotonic": 9120630.756586129, "engine": 1, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 30, "monotonic": 9120630.765814532, "engine": 1, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 31, "monotonic": 9120630.768750964, "engine": 1, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 32, "monotonic": 9120630.7762622, "engine": 1, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 33, "monotonic": 9120630.78039026, "engine": 1, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 34, "monotonic": 9120630.78437535, "engine": 1, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q9"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 35, "monotonic": 9120630.788444983, "engine": 1, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q11"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 36, "monotonic": 9120630.7924497, "engine": 1, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q13"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 37, "monotonic": 9120630.796446582, "engine": 1, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q15"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 38, "monotonic": 9120630.800872909, "engine": 1, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 39, "monotonic": 9120630.80312862, "engine": 1, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 40, "monotonic": 9120630.8055922, "engine": 1, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 41, "monotonic": 9120631.854982505, "engine": 1, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 42, "monotonic": 9120631.864391916, "engine": 1, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 43, "monotonic": 9120631.867039468, "engine": 1, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 44, "monotonic": 9120631.874338027, "engine": 1, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 45, "monotonic": 9120631.87846945, "engine": 1, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 46, "monotonic": 9120631.882465012, "engine": 1, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q9"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 47, "monotonic": 9120631.886601208, "engine": 1, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q11"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 48, "monotonic": 9120631.892491497, "engine": 1, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q13"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 49, "monotonic": 9120631.897972204, "engine": 1, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q15"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 50, "monotonic": 9120631.903372144, "engine": 1, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 51, "monotonic": 9120631.90681644, "engine": 1, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 52, "monotonic": 9120631.90912707, "engine": 1, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 53, "monotonic": 9120633.008795032, "engine": 1, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 54, "monotonic": 9120633.018359303, "engine": 1, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 55, "monotonic": 9120633.02237447, "engine": 1, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 56, "monotonic": 9120633.029993236, "engine": 1, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 57, "monotonic": 9120633.034136882, "engine": 1, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 58, "monotonic": 9120633.038214054, "engine": 1, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q9"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 59, "monotonic": 9120633.044925317, "engine": 1, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q11"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 60, "monotonic": 9120633.049206803, "engine": 1, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q13"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 61, "monotonic": 9120633.053342188, "engine": 1, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q15"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 920, "seq": 62, "monotonic": 9120633.057768637, "engine": 1, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 63, "monotonic": 9120633.061733505, "engine": 1, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 920, "seq": 64, "monotonic": 9120633.064055149, "engine": 1, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/model/config.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/model/config.json deleted file mode 100644 index e9b20534..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/model/config.json +++ /dev/null @@ -1,95 +0,0 @@ -{ - "vocab_size": 128256, - "max_position_embeddings": 131072, - "hidden_size": 2048, - "intermediate_size": 8192, - "num_hidden_layers": 16, - "num_attention_heads": 32, - "num_key_value_heads": 8, - "hidden_act": "silu", - "initializer_range": 0.02, - "rms_norm_eps": 1e-05, - "pretraining_tp": 1, - "use_cache": true, - "rope_theta": 500000.0, - "rope_scaling": { - "factor": 32.0, - "high_freq_factor": 4.0, - "low_freq_factor": 1.0, - "original_max_position_embeddings": 8192, - "rope_type": "llama3" - }, - "attention_bias": false, - "attention_dropout": 0.0, - "mlp_bias": false, - "head_dim": 64, - "return_dict": true, - "output_hidden_states": false, - "torchscript": false, - "dtype": "bfloat16", - "torch_dtype": "bfloat16", - "pruned_heads": {}, - "tie_word_embeddings": true, - "chunk_size_feed_forward": 0, - "is_encoder_decoder": false, - "is_decoder": false, - "cross_attention_hidden_size": null, - "add_cross_attention": false, - "tie_encoder_decoder": false, - "architectures": [ - "LlamaForCausalLM" - ], - "finetuning_task": null, - "id2label": { - "0": "LABEL_0", - "1": "LABEL_1" - }, - "label2id": { - "LABEL_0": 0, - "LABEL_1": 1 - }, - "task_specific_params": null, - "problem_type": null, - "tokenizer_class": null, - "prefix": null, - "bos_token_id": 128000, - "pad_token_id": null, - "eos_token_id": [ - 128001, - 128008, - 128009 - ], - "sep_token_id": null, - "decoder_start_token_id": null, - "max_length": 20, - "min_length": 0, - "do_sample": false, - "early_stopping": false, - "num_beams": 1, - "temperature": 1.0, - "top_k": 50, - "top_p": 1.0, - "typical_p": 1.0, - "repetition_penalty": 1.0, - "length_penalty": 1.0, - "no_repeat_ngram_size": 0, - "encoder_no_repeat_ngram_size": 0, - "bad_words_ids": null, - "num_return_sequences": 1, - "output_scores": false, - "return_dict_in_generate": false, - "forced_bos_token_id": null, - "forced_eos_token_id": null, - "remove_invalid_values": false, - "exponential_decay_length_penalty": null, - "suppress_tokens": null, - "begin_suppress_tokens": null, - "num_beam_groups": 1, - "diversity_penalty": 0.0, - "_name_or_path": "meta-llama/Llama-3.2-1B-Instruct", - "transformers_version": "4.57.3", - "model_type": "llama", - "tf_legacy_loss": false, - "use_bfloat16": false, - "output_attentions": false -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/pp_boundary.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/pp_boundary.jsonl deleted file mode 100644 index 0b955342..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/pp_boundary.jsonl +++ /dev/null @@ -1,152 +0,0 @@ -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120625.968609303, "preprocess_end_ts": 9120625.97053415, "forward_start_ts": 9120625.970763369, "forward_end_ts": 9120625.980689062, "timestamp": 1790102647.1819882, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120625.980833208, "send_end_ts": 9120626.148909405} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120625.970645007, "preprocess_end_ts": 9120625.972136268, "forward_start_ts": 9120625.972334806, "forward_end_ts": 9120625.980693843, "timestamp": 1790102647.186707, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120625.98082598, "send_end_ts": 9120626.153632233} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120626.154382523, "preprocess_end_ts": 9120626.155663108, "forward_start_ts": 9120626.155793374, "forward_end_ts": 9120626.16366869, "timestamp": 1790102647.2130804, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120625.970227623, "recv_end_ts": 9120626.154076628, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120626.149046848, "preprocess_end_ts": 9120626.15055543, "forward_start_ts": 9120626.150740927, "forward_end_ts": 9120626.164248995, "timestamp": 1790102647.2134495, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120625.968175467, "recv_end_ts": 9120626.148630315, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120626.154215973, "preprocess_end_ts": 9120626.154697996, "forward_start_ts": 9120626.154713957, "forward_end_ts": 9120626.157916266, "timestamp": 1790102647.2138479, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120626.15802846, "send_end_ts": 9120626.180778168} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120626.149568712, "preprocess_end_ts": 9120626.150059056, "forward_start_ts": 9120626.150077468, "forward_end_ts": 9120626.158014348, "timestamp": 1790102647.2142947, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120626.158127574, "send_end_ts": 9120626.181224626} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120626.181134202, "preprocess_end_ts": 9120626.181580093, "forward_start_ts": 9120626.1815955, "forward_end_ts": 9120626.184787473, "timestamp": 1790102647.2182894, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120626.180380125, "recv_end_ts": 9120626.18090347, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120626.181630112, "preprocess_end_ts": 9120626.182067346, "forward_start_ts": 9120626.18208268, "forward_end_ts": 9120626.184812859, "timestamp": 1790102647.2183256, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120626.18079424, "recv_end_ts": 9120626.181401048, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.339258784, "preprocess_end_ts": 9120627.339764776, "forward_start_ts": 9120627.339779196, "forward_end_ts": 9120627.342864996, "timestamp": 1790102648.376574, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.342970168, "send_end_ts": 9120627.343508072} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.339179536, "preprocess_end_ts": 9120627.33978918, "forward_start_ts": 9120627.339804886, "forward_end_ts": 9120627.342916146, "timestamp": 1790102648.3766828, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.343029773, "send_end_ts": 9120627.343614984} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.344056806, "preprocess_end_ts": 9120627.344689254, "forward_start_ts": 9120627.344704311, "forward_end_ts": 9120627.348888468, "timestamp": 1790102648.3826468, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.33892894, "recv_end_ts": 9120627.343743356, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.344097933, "preprocess_end_ts": 9120627.344858024, "forward_start_ts": 9120627.344889121, "forward_end_ts": 9120627.349071456, "timestamp": 1790102648.3826966, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.339267079, "recv_end_ts": 9120627.343679167, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.354537774, "preprocess_end_ts": 9120627.354850387, "forward_start_ts": 9120627.35486039, "forward_end_ts": 9120627.357288308, "timestamp": 1790102648.390863, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.357378667, "send_end_ts": 9120627.357797865} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.354125634, "preprocess_end_ts": 9120627.354576396, "forward_start_ts": 9120627.354588164, "forward_end_ts": 9120627.357361272, "timestamp": 1790102648.390979, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.357461236, "send_end_ts": 9120627.357911903} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.358045336, "preprocess_end_ts": 9120627.358420543, "forward_start_ts": 9120627.358431606, "forward_end_ts": 9120627.361312632, "timestamp": 1790102648.3947983, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.35269102, "recv_end_ts": 9120627.35790194, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.358344212, "preprocess_end_ts": 9120627.358774984, "forward_start_ts": 9120627.358787036, "forward_end_ts": 9120627.36326022, "timestamp": 1790102648.396899, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.363368012, "send_end_ts": 9120627.363832705} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.35815126, "preprocess_end_ts": 9120627.35879615, "forward_start_ts": 9120627.358808013, "forward_end_ts": 9120627.36377378, "timestamp": 1790102648.3973336, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.35445616, "recv_end_ts": 9120627.35796816, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.358132698, "preprocess_end_ts": 9120627.358457213, "forward_start_ts": 9120627.358466452, "forward_end_ts": 9120627.361300139, "timestamp": 1790102648.3980007, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.361378863, "send_end_ts": 9120627.364935782} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.36394886, "preprocess_end_ts": 9120627.364315215, "forward_start_ts": 9120627.364325054, "forward_end_ts": 9120627.368024694, "timestamp": 1790102648.4014964, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.361983083, "recv_end_ts": 9120627.36381411, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.365165556, "preprocess_end_ts": 9120627.365513904, "forward_start_ts": 9120627.365523864, "forward_end_ts": 9120627.370255912, "timestamp": 1790102648.4037986, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.364590388, "recv_end_ts": 9120627.365023067, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.364283687, "preprocess_end_ts": 9120627.364721015, "forward_start_ts": 9120627.364732655, "forward_end_ts": 9120627.37016959, "timestamp": 1790102648.403803, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.370273894, "send_end_ts": 9120627.370736608} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120627.36527259, "preprocess_end_ts": 9120627.365592506, "forward_start_ts": 9120627.365601173, "forward_end_ts": 9120627.368131137, "timestamp": 1790102648.404424, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120627.368207466, "send_end_ts": 9120627.371359697} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.370907487, "preprocess_end_ts": 9120627.371254314, "forward_start_ts": 9120627.37126446, "forward_end_ts": 9120627.374588244, "timestamp": 1790102648.4080584, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.368684333, "recv_end_ts": 9120627.370726237, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120627.3716548, "preprocess_end_ts": 9120627.37212815, "forward_start_ts": 9120627.3721393, "forward_end_ts": 9120627.375675013, "timestamp": 1790102648.4091454, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120627.371039772, "recv_end_ts": 9120627.371491624, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.480072675, "preprocess_end_ts": 9120628.480590845, "forward_start_ts": 9120628.48060594, "forward_end_ts": 9120628.48369596, "timestamp": 1790102649.5173385, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.483807502, "send_end_ts": 9120628.48427284} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.479916755, "preprocess_end_ts": 9120628.480465902, "forward_start_ts": 9120628.480479784, "forward_end_ts": 9120628.48361477, "timestamp": 1790102649.5173392, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.483726276, "send_end_ts": 9120628.484273072} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.484656258, "preprocess_end_ts": 9120628.485159677, "forward_start_ts": 9120628.485177867, "forward_end_ts": 9120628.488222428, "timestamp": 1790102649.5217633, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.47985394, "recv_end_ts": 9120628.484396309, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.484680768, "preprocess_end_ts": 9120628.48525294, "forward_start_ts": 9120628.485268185, "forward_end_ts": 9120628.489254864, "timestamp": 1790102649.522823, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.479656564, "recv_end_ts": 9120628.484431809, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.49364094, "preprocess_end_ts": 9120628.494016288, "forward_start_ts": 9120628.494027358, "forward_end_ts": 9120628.496669484, "timestamp": 1790102649.5301967, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.496759748, "send_end_ts": 9120628.497131933} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.493183425, "preprocess_end_ts": 9120628.493563408, "forward_start_ts": 9120628.493574992, "forward_end_ts": 9120628.49667986, "timestamp": 1790102649.5302074, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.496780356, "send_end_ts": 9120628.497143047} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.497340364, "preprocess_end_ts": 9120628.49771906, "forward_start_ts": 9120628.497730436, "forward_end_ts": 9120628.500560196, "timestamp": 1790102649.5340424, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.492983911, "recv_end_ts": 9120628.4971987, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.497483142, "preprocess_end_ts": 9120628.497825736, "forward_start_ts": 9120628.49783498, "forward_end_ts": 9120628.500394216, "timestamp": 1790102649.5346537, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.500477197, "send_end_ts": 9120628.501588022} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.497430827, "preprocess_end_ts": 9120628.497908764, "forward_start_ts": 9120628.49792198, "forward_end_ts": 9120628.501545314, "timestamp": 1790102649.5350845, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.4935184, "recv_end_ts": 9120628.497251464, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.497464197, "preprocess_end_ts": 9120628.497778434, "forward_start_ts": 9120628.49778684, "forward_end_ts": 9120628.500323832, "timestamp": 1790102649.535703, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.500402601, "send_end_ts": 9120628.502637304} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.50173274, "preprocess_end_ts": 9120628.502162267, "forward_start_ts": 9120628.502172364, "forward_end_ts": 9120628.505890612, "timestamp": 1790102649.5393727, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.50123235, "recv_end_ts": 9120628.501591649, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.501935309, "preprocess_end_ts": 9120628.502257, "forward_start_ts": 9120628.502265956, "forward_end_ts": 9120628.505892256, "timestamp": 1790102649.5399516, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.505970024, "send_end_ts": 9120628.506887212} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.502906872, "preprocess_end_ts": 9120628.503386345, "forward_start_ts": 9120628.503397947, "forward_end_ts": 9120628.506884232, "timestamp": 1790102649.5404081, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.502334353, "recv_end_ts": 9120628.502734972, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120628.50298448, "preprocess_end_ts": 9120628.503298933, "forward_start_ts": 9120628.503308216, "forward_end_ts": 9120628.50581078, "timestamp": 1790102649.5410109, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120628.505886668, "send_end_ts": 9120628.507945262} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.50709637, "preprocess_end_ts": 9120628.507446185, "forward_start_ts": 9120628.507455656, "forward_end_ts": 9120628.511246327, "timestamp": 1790102649.5447104, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.506540043, "recv_end_ts": 9120628.50692816, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120628.508232096, "preprocess_end_ts": 9120628.508691614, "forward_start_ts": 9120628.508702967, "forward_end_ts": 9120628.51224555, "timestamp": 1790102649.545758, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120628.507640127, "recv_end_ts": 9120628.508053012, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.617235024, "preprocess_end_ts": 9120629.617765486, "forward_start_ts": 9120629.617780456, "forward_end_ts": 9120629.62183598, "timestamp": 1790102650.6555102, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.62194472, "send_end_ts": 9120629.62244408} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.617889555, "preprocess_end_ts": 9120629.618670886, "forward_start_ts": 9120629.61870918, "forward_end_ts": 9120629.623213852, "timestamp": 1790102650.6569853, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.623365313, "send_end_ts": 9120629.62391574} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.624280203, "preprocess_end_ts": 9120629.624772113, "forward_start_ts": 9120629.624787886, "forward_end_ts": 9120629.627719384, "timestamp": 1790102650.6612575, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.617380893, "recv_end_ts": 9120629.624033524, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.622827092, "preprocess_end_ts": 9120629.623382833, "forward_start_ts": 9120629.623396477, "forward_end_ts": 9120629.628656372, "timestamp": 1790102650.6622326, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.616972368, "recv_end_ts": 9120629.62258668, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.633139703, "preprocess_end_ts": 9120629.633528728, "forward_start_ts": 9120629.63354104, "forward_end_ts": 9120629.636019705, "timestamp": 1790102650.6695786, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.63611123, "send_end_ts": 9120629.63651369} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.632777745, "preprocess_end_ts": 9120629.633213265, "forward_start_ts": 9120629.633225307, "forward_end_ts": 9120629.637058595, "timestamp": 1790102650.670677, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.637166584, "send_end_ts": 9120629.637611376} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.637793612, "preprocess_end_ts": 9120629.63816856, "forward_start_ts": 9120629.638179317, "forward_end_ts": 9120629.640670473, "timestamp": 1790102650.6741621, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.632422255, "recv_end_ts": 9120629.63764468, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.636803849, "preprocess_end_ts": 9120629.637284296, "forward_start_ts": 9120629.6372973, "forward_end_ts": 9120629.641628912, "timestamp": 1790102650.6751661, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.63300281, "recv_end_ts": 9120629.63662434, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.638032388, "preprocess_end_ts": 9120629.638448147, "forward_start_ts": 9120629.63845889, "forward_end_ts": 9120629.641865157, "timestamp": 1790102650.6754727, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.641960869, "send_end_ts": 9120629.642407075} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.636854624, "preprocess_end_ts": 9120629.637174541, "forward_start_ts": 9120629.637183553, "forward_end_ts": 9120629.64094268, "timestamp": 1790102650.67577, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.641025716, "send_end_ts": 9120629.642704451} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.642587366, "preprocess_end_ts": 9120629.643015442, "forward_start_ts": 9120629.64302696, "forward_end_ts": 9120629.645904802, "timestamp": 1790102650.679381, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.641346332, "recv_end_ts": 9120629.642447297, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.642962169, "preprocess_end_ts": 9120629.643422045, "forward_start_ts": 9120629.64343342, "forward_end_ts": 9120629.646858996, "timestamp": 1790102650.6803827, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.64240166, "recv_end_ts": 9120629.642798074, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.642824026, "preprocess_end_ts": 9120629.643243168, "forward_start_ts": 9120629.643254532, "forward_end_ts": 9120629.646820208, "timestamp": 1790102650.6804247, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.646915665, "send_end_ts": 9120629.647358976} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120629.643039608, "preprocess_end_ts": 9120629.643353136, "forward_start_ts": 9120629.643361958, "forward_end_ts": 9120629.645853216, "timestamp": 1790102650.6810222, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120629.645927992, "send_end_ts": 9120629.647957291} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.647541158, "preprocess_end_ts": 9120629.647900688, "forward_start_ts": 9120629.64791027, "forward_end_ts": 9120629.651127145, "timestamp": 1790102650.6845968, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.64656526, "recv_end_ts": 9120629.647406936, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120629.648230966, "preprocess_end_ts": 9120629.648682158, "forward_start_ts": 9120629.648692459, "forward_end_ts": 9120629.652173012, "timestamp": 1790102650.6856806, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120629.647617482, "recv_end_ts": 9120629.648054553, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.756728144, "preprocess_end_ts": 9120630.757250639, "forward_start_ts": 9120630.75726482, "forward_end_ts": 9120630.760731801, "timestamp": 1790102651.794396, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.760846628, "send_end_ts": 9120630.761330284} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.757091507, "preprocess_end_ts": 9120630.757696709, "forward_start_ts": 9120630.757711885, "forward_end_ts": 9120630.76082835, "timestamp": 1790102651.7944815, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.760945473, "send_end_ts": 9120630.761415496} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.761716532, "preprocess_end_ts": 9120630.762154792, "forward_start_ts": 9120630.762168357, "forward_end_ts": 9120630.764944864, "timestamp": 1790102651.7984564, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.756694091, "recv_end_ts": 9120630.761507204, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.761652008, "preprocess_end_ts": 9120630.762110276, "forward_start_ts": 9120630.762123177, "forward_end_ts": 9120630.765052913, "timestamp": 1790102651.7985692, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.756420648, "recv_end_ts": 9120630.761439895, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.76871565, "preprocess_end_ts": 9120630.769057296, "forward_start_ts": 9120630.769067828, "forward_end_ts": 9120630.771755356, "timestamp": 1790102651.8052824, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.77184165, "send_end_ts": 9120630.772216788} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.768905576, "preprocess_end_ts": 9120630.769261034, "forward_start_ts": 9120630.7692735, "forward_end_ts": 9120630.771849427, "timestamp": 1790102651.8054075, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.771936612, "send_end_ts": 9120630.772342429} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.772571083, "preprocess_end_ts": 9120630.772931213, "forward_start_ts": 9120630.77294206, "forward_end_ts": 9120630.775472678, "timestamp": 1790102651.8089557, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.768785909, "recv_end_ts": 9120630.772411592, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.772432297, "preprocess_end_ts": 9120630.772792043, "forward_start_ts": 9120630.772801967, "forward_end_ts": 9120630.775467785, "timestamp": 1790102651.8089523, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.76860636, "recv_end_ts": 9120630.772278393, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.772544274, "preprocess_end_ts": 9120630.772870028, "forward_start_ts": 9120630.772879288, "forward_end_ts": 9120630.775547277, "timestamp": 1790102651.8095398, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.775630755, "send_end_ts": 9120630.776473988} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.772706363, "preprocess_end_ts": 9120630.77307342, "forward_start_ts": 9120630.773083623, "forward_end_ts": 9120630.7756091, "timestamp": 1790102651.809545, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.775689287, "send_end_ts": 9120630.776480244} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.776726035, "preprocess_end_ts": 9120630.777065463, "forward_start_ts": 9120630.777075876, "forward_end_ts": 9120630.779607631, "timestamp": 1790102651.8130927, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.77616563, "recv_end_ts": 9120630.776593987, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.776674774, "preprocess_end_ts": 9120630.777029077, "forward_start_ts": 9120630.777038317, "forward_end_ts": 9120630.779634157, "timestamp": 1790102651.8131168, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.776156606, "recv_end_ts": 9120630.776536943, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.77682408, "preprocess_end_ts": 9120630.777144736, "forward_start_ts": 9120630.777153444, "forward_end_ts": 9120630.779752117, "timestamp": 1790102651.81366, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.779828325, "send_end_ts": 9120630.780594528} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.776808364, "preprocess_end_ts": 9120630.777129512, "forward_start_ts": 9120630.777138796, "forward_end_ts": 9120630.779630387, "timestamp": 1790102651.8136709, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.779704709, "send_end_ts": 9120630.78060522} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.780836565, "preprocess_end_ts": 9120630.781162376, "forward_start_ts": 9120630.78117098, "forward_end_ts": 9120630.78364742, "timestamp": 1790102651.8171222, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.780266693, "recv_end_ts": 9120630.780703856, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.780801186, "preprocess_end_ts": 9120630.781138828, "forward_start_ts": 9120630.781147344, "forward_end_ts": 9120630.783756781, "timestamp": 1790102651.8172321, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.780290967, "recv_end_ts": 9120630.780661805, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.780920591, "preprocess_end_ts": 9120630.781235833, "forward_start_ts": 9120630.781244667, "forward_end_ts": 9120630.783854818, "timestamp": 1790102651.8177342, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.783930436, "send_end_ts": 9120630.78466948} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.78094024, "preprocess_end_ts": 9120630.781254206, "forward_start_ts": 9120630.781264195, "forward_end_ts": 9120630.783766752, "timestamp": 1790102651.8177555, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.783840356, "send_end_ts": 9120630.784690293} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.78488378, "preprocess_end_ts": 9120630.785230393, "forward_start_ts": 9120630.785238812, "forward_end_ts": 9120630.787740665, "timestamp": 1790102651.8211982, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.784309767, "recv_end_ts": 9120630.784752062, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.784940084, "preprocess_end_ts": 9120630.78528827, "forward_start_ts": 9120630.785297353, "forward_end_ts": 9120630.787742209, "timestamp": 1790102651.8212004, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.784412878, "recv_end_ts": 9120630.78479814, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.785016892, "preprocess_end_ts": 9120630.785338355, "forward_start_ts": 9120630.785348654, "forward_end_ts": 9120630.787786445, "timestamp": 1790102651.8217356, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.787860107, "send_end_ts": 9120630.788669506} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.78500206, "preprocess_end_ts": 9120630.785324523, "forward_start_ts": 9120630.78533306, "forward_end_ts": 9120630.787884472, "timestamp": 1790102651.821771, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.787958836, "send_end_ts": 9120630.788706576} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.788912522, "preprocess_end_ts": 9120630.789256187, "forward_start_ts": 9120630.789264748, "forward_end_ts": 9120630.791709485, "timestamp": 1790102651.8251615, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.788366668, "recv_end_ts": 9120630.78875686, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.788930383, "preprocess_end_ts": 9120630.789263789, "forward_start_ts": 9120630.789272215, "forward_end_ts": 9120630.791742334, "timestamp": 1790102651.8252108, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.78837028, "recv_end_ts": 9120630.78879814, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.78899092, "preprocess_end_ts": 9120630.789308038, "forward_start_ts": 9120630.789316673, "forward_end_ts": 9120630.791789923, "timestamp": 1790102651.82569, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.791863224, "send_end_ts": 9120630.792625263} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.789033458, "preprocess_end_ts": 9120630.789348353, "forward_start_ts": 9120630.78935704, "forward_end_ts": 9120630.791831916, "timestamp": 1790102651.8257744, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.79190594, "send_end_ts": 9120630.792709809} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.792934729, "preprocess_end_ts": 9120630.793263268, "forward_start_ts": 9120630.793271432, "forward_end_ts": 9120630.795742974, "timestamp": 1790102651.82921, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.792379472, "recv_end_ts": 9120630.792802677, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.792865507, "preprocess_end_ts": 9120630.793207586, "forward_start_ts": 9120630.793215957, "forward_end_ts": 9120630.795875024, "timestamp": 1790102651.8293643, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.792329932, "recv_end_ts": 9120630.792715462, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.793027883, "preprocess_end_ts": 9120630.793340776, "forward_start_ts": 9120630.79334936, "forward_end_ts": 9120630.795806972, "timestamp": 1790102651.8297832, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.795879817, "send_end_ts": 9120630.79671858} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120630.79295564, "preprocess_end_ts": 9120630.793271327, "forward_start_ts": 9120630.793279773, "forward_end_ts": 9120630.795758668, "timestamp": 1790102651.8299458, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120630.795833847, "send_end_ts": 9120630.796881536} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.797101848, "preprocess_end_ts": 9120630.797448287, "forward_start_ts": 9120630.797457129, "forward_end_ts": 9120630.799931336, "timestamp": 1790102651.8334818, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.796561232, "recv_end_ts": 9120630.796963716, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120630.796984555, "preprocess_end_ts": 9120630.797322156, "forward_start_ts": 9120630.797330884, "forward_end_ts": 9120630.800157882, "timestamp": 1790102651.8336236, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120630.79638398, "recv_end_ts": 9120630.796833448, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.855282338, "preprocess_end_ts": 9120631.855767962, "forward_start_ts": 9120631.855781684, "forward_end_ts": 9120631.858707948, "timestamp": 1790102652.8924625, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.858814804, "send_end_ts": 9120631.859396908} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.854982182, "preprocess_end_ts": 9120631.855524568, "forward_start_ts": 9120631.855538204, "forward_end_ts": 9120631.858833825, "timestamp": 1790102652.892465, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.858942743, "send_end_ts": 9120631.859399797} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.85971038, "preprocess_end_ts": 9120631.86018892, "forward_start_ts": 9120631.86020302, "forward_end_ts": 9120631.86336512, "timestamp": 1790102652.896879, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.85471212, "recv_end_ts": 9120631.859493257, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.859894043, "preprocess_end_ts": 9120631.860471463, "forward_start_ts": 9120631.86049034, "forward_end_ts": 9120631.863395637, "timestamp": 1790102652.896951, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.855249437, "recv_end_ts": 9120631.85961009, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.867067054, "preprocess_end_ts": 9120631.8673978, "forward_start_ts": 9120631.867408087, "forward_end_ts": 9120631.869902749, "timestamp": 1790102652.9034307, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.86998835, "send_end_ts": 9120631.870366186} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.867175924, "preprocess_end_ts": 9120631.867486104, "forward_start_ts": 9120631.867495667, "forward_end_ts": 9120631.869951831, "timestamp": 1790102652.9034617, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.870038316, "send_end_ts": 9120631.87039729} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.870574372, "preprocess_end_ts": 9120631.870930824, "forward_start_ts": 9120631.87094088, "forward_end_ts": 9120631.873476159, "timestamp": 1790102652.9069595, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.866951348, "recv_end_ts": 9120631.87042774, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.870622411, "preprocess_end_ts": 9120631.87098984, "forward_start_ts": 9120631.871000435, "forward_end_ts": 9120631.873544944, "timestamp": 1790102652.9070256, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.86708166, "recv_end_ts": 9120631.870477917, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.8706924, "preprocess_end_ts": 9120631.871004255, "forward_start_ts": 9120631.87101292, "forward_end_ts": 9120631.873535637, "timestamp": 1790102652.9075143, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.873615613, "send_end_ts": 9120631.874449918} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.87072502, "preprocess_end_ts": 9120631.871036137, "forward_start_ts": 9120631.871044472, "forward_end_ts": 9120631.873580053, "timestamp": 1790102652.9075842, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.873659866, "send_end_ts": 9120631.874519236} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.874674369, "preprocess_end_ts": 9120631.875019744, "forward_start_ts": 9120631.87502963, "forward_end_ts": 9120631.877606152, "timestamp": 1790102652.9110744, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.874144344, "recv_end_ts": 9120631.874530405, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.874781609, "preprocess_end_ts": 9120631.875126224, "forward_start_ts": 9120631.875135522, "forward_end_ts": 9120631.877671083, "timestamp": 1790102652.9111435, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.874217808, "recv_end_ts": 9120631.87465144, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.874851827, "preprocess_end_ts": 9120631.875212194, "forward_start_ts": 9120631.875220904, "forward_end_ts": 9120631.87776462, "timestamp": 1790102652.9116647, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.877839718, "send_end_ts": 9120631.878598472} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.874852149, "preprocess_end_ts": 9120631.875159081, "forward_start_ts": 9120631.875167888, "forward_end_ts": 9120631.877867987, "timestamp": 1790102652.9117095, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.877956215, "send_end_ts": 9120631.87864424} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.878800401, "preprocess_end_ts": 9120631.879158728, "forward_start_ts": 9120631.879168024, "forward_end_ts": 9120631.881685087, "timestamp": 1790102652.915156, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.878245916, "recv_end_ts": 9120631.878645755, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.878901208, "preprocess_end_ts": 9120631.879240762, "forward_start_ts": 9120631.87924943, "forward_end_ts": 9120631.881735718, "timestamp": 1790102652.9152062, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.878322909, "recv_end_ts": 9120631.8787669, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.878932904, "preprocess_end_ts": 9120631.879247816, "forward_start_ts": 9120631.879257228, "forward_end_ts": 9120631.881905263, "timestamp": 1790102652.915742, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.881980268, "send_end_ts": 9120631.882677028} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.878995804, "preprocess_end_ts": 9120631.87943838, "forward_start_ts": 9120631.879449736, "forward_end_ts": 9120631.881915934, "timestamp": 1790102652.9157922, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.881994404, "send_end_ts": 9120631.882726965} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.882963836, "preprocess_end_ts": 9120631.883310096, "forward_start_ts": 9120631.883319356, "forward_end_ts": 9120631.885828119, "timestamp": 1790102652.9193022, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.882381609, "recv_end_ts": 9120631.882829992, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.883066133, "preprocess_end_ts": 9120631.883382568, "forward_start_ts": 9120631.883393092, "forward_end_ts": 9120631.88592888, "timestamp": 1790102652.9198632, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.88600756, "send_end_ts": 9120631.886799075} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.8829099, "preprocess_end_ts": 9120631.883262, "forward_start_ts": 9120631.88327184, "forward_end_ts": 9120631.887270372, "timestamp": 1790102652.920824, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.882331923, "recv_end_ts": 9120631.882742755, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.88300686, "preprocess_end_ts": 9120631.88332234, "forward_start_ts": 9120631.883331155, "forward_end_ts": 9120631.8858272, "timestamp": 1790102652.9214623, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.885903863, "send_end_ts": 9120631.888396887} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.88706456, "preprocess_end_ts": 9120631.887404244, "forward_start_ts": 9120631.887412826, "forward_end_ts": 9120631.891734196, "timestamp": 1790102652.9252076, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.886480557, "recv_end_ts": 9120631.886926131, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.887137188, "preprocess_end_ts": 9120631.88746548, "forward_start_ts": 9120631.887475893, "forward_end_ts": 9120631.891716296, "timestamp": 1790102652.9257696, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.891795672, "send_end_ts": 9120631.892704656} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.888714574, "preprocess_end_ts": 9120631.889211528, "forward_start_ts": 9120631.889223566, "forward_end_ts": 9120631.892954983, "timestamp": 1790102652.9264822, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.888097525, "recv_end_ts": 9120631.888519298, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.888762549, "preprocess_end_ts": 9120631.889092516, "forward_start_ts": 9120631.889102204, "forward_end_ts": 9120631.891640492, "timestamp": 1790102652.92707, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.891717687, "send_end_ts": 9120631.894005127} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.892962996, "preprocess_end_ts": 9120631.893312791, "forward_start_ts": 9120631.893321082, "forward_end_ts": 9120631.897233916, "timestamp": 1790102652.930702, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.892387312, "recv_end_ts": 9120631.892826892, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.89305972, "preprocess_end_ts": 9120631.893376531, "forward_start_ts": 9120631.893385472, "forward_end_ts": 9120631.897246273, "timestamp": 1790102652.9312675, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.897326227, "send_end_ts": 9120631.898203159} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.894286577, "preprocess_end_ts": 9120631.894758344, "forward_start_ts": 9120631.894769112, "forward_end_ts": 9120631.898311712, "timestamp": 1790102652.9318323, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.893723847, "recv_end_ts": 9120631.894104771, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120631.894357976, "preprocess_end_ts": 9120631.894676585, "forward_start_ts": 9120631.8946855, "forward_end_ts": 9120631.897187008, "timestamp": 1790102652.9324157, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120631.897261884, "send_end_ts": 9120631.899351345} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.898422357, "preprocess_end_ts": 9120631.89876118, "forward_start_ts": 9120631.898769883, "forward_end_ts": 9120631.902630253, "timestamp": 1790102652.9360933, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.897881148, "recv_end_ts": 9120631.898289489, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120631.899629433, "preprocess_end_ts": 9120631.900082408, "forward_start_ts": 9120631.900092972, "forward_end_ts": 9120631.903661069, "timestamp": 1790102652.9371703, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120631.89906192, "recv_end_ts": 9120631.899461376, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.00912462, "preprocess_end_ts": 9120633.009628447, "forward_start_ts": 9120633.009642484, "forward_end_ts": 9120633.012897952, "timestamp": 1790102654.046612, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.013004847, "send_end_ts": 9120633.013546484} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.009092316, "preprocess_end_ts": 9120633.009735657, "forward_start_ts": 9120633.0097653, "forward_end_ts": 9120633.013069568, "timestamp": 1790102654.0467336, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.01319092, "send_end_ts": 9120633.013664292} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.013865676, "preprocess_end_ts": 9120633.014310487, "forward_start_ts": 9120633.014323035, "forward_end_ts": 9120633.017461952, "timestamp": 1790102654.050983, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.008865193, "recv_end_ts": 9120633.013653051, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.014084011, "preprocess_end_ts": 9120633.014665373, "forward_start_ts": 9120633.014681084, "forward_end_ts": 9120633.01861633, "timestamp": 1790102654.0522833, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.008523732, "recv_end_ts": 9120633.013835423, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.022484371, "preprocess_end_ts": 9120633.022825, "forward_start_ts": 9120633.022836223, "forward_end_ts": 9120633.02539309, "timestamp": 1790102654.0589721, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.025478587, "send_end_ts": 9120633.025907433} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.022576105, "preprocess_end_ts": 9120633.022932714, "forward_start_ts": 9120633.0229432, "forward_end_ts": 9120633.025504356, "timestamp": 1790102654.0590165, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.02559088, "send_end_ts": 9120633.025951508} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.026155168, "preprocess_end_ts": 9120633.026519824, "forward_start_ts": 9120633.0265308, "forward_end_ts": 9120633.0290985, "timestamp": 1790102654.0625827, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.022371592, "recv_end_ts": 9120633.02599124, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.026150355, "preprocess_end_ts": 9120633.026512844, "forward_start_ts": 9120633.026522849, "forward_end_ts": 9120633.02926104, "timestamp": 1790102654.0627458, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.022381136, "recv_end_ts": 9120633.026005764, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.026241688, "preprocess_end_ts": 9120633.026547017, "forward_start_ts": 9120633.02655548, "forward_end_ts": 9120633.029101554, "timestamp": 1790102654.0631752, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.029178979, "send_end_ts": 9120633.030110529} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.026288347, "preprocess_end_ts": 9120633.026598904, "forward_start_ts": 9120633.026607355, "forward_end_ts": 9120633.029218858, "timestamp": 1790102654.0632923, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.029298743, "send_end_ts": 9120633.030227378} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.030323096, "preprocess_end_ts": 9120633.030673208, "forward_start_ts": 9120633.030682083, "forward_end_ts": 9120633.033343269, "timestamp": 1790102654.0668237, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.029770184, "recv_end_ts": 9120633.03016674, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.030455012, "preprocess_end_ts": 9120633.030802865, "forward_start_ts": 9120633.030814027, "forward_end_ts": 9120633.033471644, "timestamp": 1790102654.0669456, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.029932916, "recv_end_ts": 9120633.030308735, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.030438948, "preprocess_end_ts": 9120633.030747175, "forward_start_ts": 9120633.03075576, "forward_end_ts": 9120633.03336456, "timestamp": 1790102654.0674243, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.033438902, "send_end_ts": 9120633.034359409} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.030559953, "preprocess_end_ts": 9120633.03087182, "forward_start_ts": 9120633.030880343, "forward_end_ts": 9120633.03336346, "timestamp": 1790102654.0674803, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.033438088, "send_end_ts": 9120633.034415359} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.03465004, "preprocess_end_ts": 9120633.03498366, "forward_start_ts": 9120633.03499256, "forward_end_ts": 9120633.037458057, "timestamp": 1790102654.0709217, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.03410994, "recv_end_ts": 9120633.034518484, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.03459124, "preprocess_end_ts": 9120633.034921931, "forward_start_ts": 9120633.034930164, "forward_end_ts": 9120633.037582994, "timestamp": 1790102654.0710557, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.034022192, "recv_end_ts": 9120633.034451906, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.034739578, "preprocess_end_ts": 9120633.035048965, "forward_start_ts": 9120633.035057332, "forward_end_ts": 9120633.037575962, "timestamp": 1790102654.07146, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.037650784, "send_end_ts": 9120633.038394252} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.034679515, "preprocess_end_ts": 9120633.034986053, "forward_start_ts": 9120633.034994485, "forward_end_ts": 9120633.037597232, "timestamp": 1790102654.0717065, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.037673317, "send_end_ts": 9120633.038641788} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.038858248, "preprocess_end_ts": 9120633.039213931, "forward_start_ts": 9120633.039223202, "forward_end_ts": 9120633.041816534, "timestamp": 1790102654.0752876, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.038227772, "recv_end_ts": 9120633.038721519, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.038574662, "preprocess_end_ts": 9120633.0389408, "forward_start_ts": 9120633.038949773, "forward_end_ts": 9120633.044092435, "timestamp": 1790102654.0776374, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.03808926, "recv_end_ts": 9120633.03842598, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.038994944, "preprocess_end_ts": 9120633.039339826, "forward_start_ts": 9120633.039349278, "forward_end_ts": 9120633.044071345, "timestamp": 1790102654.0777137, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.044175034, "send_end_ts": 9120633.04464744} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.038739506, "preprocess_end_ts": 9120633.039062949, "forward_start_ts": 9120633.039071944, "forward_end_ts": 9120633.041868236, "timestamp": 1790102654.078329, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.041944608, "send_end_ts": 9120633.04526319} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.044770325, "preprocess_end_ts": 9120633.045113716, "forward_start_ts": 9120633.045122268, "forward_end_ts": 9120633.048448041, "timestamp": 1790102654.0819178, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.042455737, "recv_end_ts": 9120633.044629203, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.045538375, "preprocess_end_ts": 9120633.045897225, "forward_start_ts": 9120633.045906844, "forward_end_ts": 9120633.04848914, "timestamp": 1790102654.0819652, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.044870075, "recv_end_ts": 9120633.045367245, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.045021882, "preprocess_end_ts": 9120633.045462469, "forward_start_ts": 9120633.045473741, "forward_end_ts": 9120633.048544453, "timestamp": 1790102654.0824502, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.04862099, "send_end_ts": 9120633.0493857} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.045603164, "preprocess_end_ts": 9120633.045917977, "forward_start_ts": 9120633.04592652, "forward_end_ts": 9120633.04844103, "timestamp": 1790102654.0825758, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.048513452, "send_end_ts": 9120633.049511343} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.049668692, "preprocess_end_ts": 9120633.050129907, "forward_start_ts": 9120633.050141344, "forward_end_ts": 9120633.052645463, "timestamp": 1790102654.0861177, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.049130924, "recv_end_ts": 9120633.049530424, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.049608896, "preprocess_end_ts": 9120633.049943777, "forward_start_ts": 9120633.049952548, "forward_end_ts": 9120633.052688764, "timestamp": 1790102654.086164, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.049085557, "recv_end_ts": 9120633.049453532, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.049826369, "preprocess_end_ts": 9120633.05013533, "forward_start_ts": 9120633.050143477, "forward_end_ts": 9120633.052648343, "timestamp": 1790102654.086692, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.052720623, "send_end_ts": 9120633.053626252} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120633.049700836, "preprocess_end_ts": 9120633.050009804, "forward_start_ts": 9120633.050018644, "forward_end_ts": 9120633.052618377, "timestamp": 1790102654.0867481, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120633.052692603, "send_end_ts": 9120633.053681875} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.053981254, "preprocess_end_ts": 9120633.054455565, "forward_start_ts": 9120633.054466892, "forward_end_ts": 9120633.057019856, "timestamp": 1790102654.0904927, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.05328541, "recv_end_ts": 9120633.0537842, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120633.053851355, "preprocess_end_ts": 9120633.054199465, "forward_start_ts": 9120633.054208232, "forward_end_ts": 9120633.057175398, "timestamp": 1790102654.0906513, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120633.05333501, "recv_end_ts": 9120633.053714588, "send_start_ts": null, "send_end_ts": null} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/requests.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/requests.jsonl deleted file mode 100644 index a32e0ab6..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/requests.jsonl +++ /dev/null @@ -1,76 +0,0 @@ -{"request_id": "warmup-q0", "burst": "warmup", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120619.693064611, "finish_monotonic": 9120626.1815256, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "warmup-q1", "burst": "warmup", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120619.695883848, "finish_monotonic": 9120626.181534523, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "warmup-q2", "burst": "warmup", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120619.69638608, "finish_monotonic": 9120626.185938066, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "warmup-q3", "burst": "warmup", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120619.696744837, "finish_monotonic": 9120626.186357344, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q0", "burst": "b8-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120627.33764526, "finish_monotonic": 9120627.350517169, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q1", "burst": "b8-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120627.338142376, "finish_monotonic": 9120627.35052416, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q2", "burst": "b8-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120627.338400908, "finish_monotonic": 9120627.3626592, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q3", "burst": "b8-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120627.3387301, "finish_monotonic": 9120627.365025744, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q4", "burst": "b8-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9120627.33905247, "finish_monotonic": 9120627.369012823, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q5", "burst": "b8-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9120627.339336585, "finish_monotonic": 9120627.37133308, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q6", "burst": "b8-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9120627.339580867, "finish_monotonic": 9120627.375533052, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q7", "burst": "b8-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9120627.339853652, "finish_monotonic": 9120627.376606883, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q0", "burst": "b8-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9120628.478456767, "finish_monotonic": 9120628.490733864, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q1", "burst": "b8-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9120628.478942016, "finish_monotonic": 9120628.490742048, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q2", "burst": "b8-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9120628.479174543, "finish_monotonic": 9120628.502861716, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q3", "burst": "b8-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9120628.479384871, "finish_monotonic": 9120628.502868343, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q4", "burst": "b8-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9120628.4796049, "finish_monotonic": 9120628.508141924, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q5", "burst": "b8-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9120628.479847856, "finish_monotonic": 9120628.508147068, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q6", "burst": "b8-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9120628.480164504, "finish_monotonic": 9120628.513450736, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q7", "burst": "b8-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9120628.480518768, "finish_monotonic": 9120628.51345778, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q0", "burst": "b8-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9120629.615764981, "finish_monotonic": 9120629.630154692, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q1", "burst": "b8-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9120629.616310723, "finish_monotonic": 9120629.6301628, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q2", "burst": "b8-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9120629.616553552, "finish_monotonic": 9120629.642988376, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q3", "burst": "b8-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9120629.616755284, "finish_monotonic": 9120629.642997924, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q4", "burst": "b8-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9120629.61694975, "finish_monotonic": 9120629.648127552, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q5", "burst": "b8-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9120629.617157232, "finish_monotonic": 9120629.648135507, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q6", "burst": "b8-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9120629.617386969, "finish_monotonic": 9120629.653310588, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q7", "burst": "b8-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9120629.617586398, "finish_monotonic": 9120629.653318169, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q0", "burst": "b16-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120630.755274627, "finish_monotonic": 9120630.766395576, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q1", "burst": "b16-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120630.755743297, "finish_monotonic": 9120630.766404217, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q2", "burst": "b16-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120630.7559694, "finish_monotonic": 9120630.776710898, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q3", "burst": "b16-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120630.756184224, "finish_monotonic": 9120630.777303033, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q4", "burst": "b16-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9120630.756377677, "finish_monotonic": 9120630.780823816, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q5", "burst": "b16-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9120630.756564396, "finish_monotonic": 9120630.780830188, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q6", "burst": "b16-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9120630.756747931, "finish_monotonic": 9120630.78486779, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q7", "burst": "b16-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9120630.756918237, "finish_monotonic": 9120630.784872321, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q8", "burst": "b16-r0", "round": 0, "index": 8, "rank": 0, "submit_monotonic": 9120630.757167717, "finish_monotonic": 9120630.788809026, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q9", "burst": "b16-r0", "round": 0, "index": 9, "rank": 1, "submit_monotonic": 9120630.757407872, "finish_monotonic": 9120630.788894637, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q10", "burst": "b16-r0", "round": 0, "index": 10, "rank": 0, "submit_monotonic": 9120630.75773546, "finish_monotonic": 9120630.792928468, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q11", "burst": "b16-r0", "round": 0, "index": 11, "rank": 1, "submit_monotonic": 9120630.758017642, "finish_monotonic": 9120630.7929341, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q12", "burst": "b16-r0", "round": 0, "index": 12, "rank": 0, "submit_monotonic": 9120630.758321553, "finish_monotonic": 9120630.79972709, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q13", "burst": "b16-r0", "round": 0, "index": 13, "rank": 1, "submit_monotonic": 9120630.758614428, "finish_monotonic": 9120630.799734142, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q14", "burst": "b16-r0", "round": 0, "index": 14, "rank": 0, "submit_monotonic": 9120630.75890563, "finish_monotonic": 9120630.801104853, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q15", "burst": "b16-r0", "round": 0, "index": 15, "rank": 1, "submit_monotonic": 9120630.759184089, "finish_monotonic": 9120630.801354105, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q0", "burst": "b16-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9120631.853388796, "finish_monotonic": 9120631.864818912, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q1", "burst": "b16-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9120631.854072178, "finish_monotonic": 9120631.864826111, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q2", "burst": "b16-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9120631.854456872, "finish_monotonic": 9120631.87491754, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q3", "burst": "b16-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9120631.854810825, "finish_monotonic": 9120631.874924596, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q4", "burst": "b16-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9120631.855116468, "finish_monotonic": 9120631.878823621, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q5", "burst": "b16-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9120631.85540741, "finish_monotonic": 9120631.878926156, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q6", "burst": "b16-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9120631.855626736, "finish_monotonic": 9120631.883009087, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q7", "burst": "b16-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9120631.855811985, "finish_monotonic": 9120631.88301812, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q8", "burst": "b16-r1", "round": 1, "index": 8, "rank": 0, "submit_monotonic": 9120631.855988791, "finish_monotonic": 9120631.888617085, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q9", "burst": "b16-r1", "round": 1, "index": 9, "rank": 1, "submit_monotonic": 9120631.856176008, "finish_monotonic": 9120631.888627496, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q10", "burst": "b16-r1", "round": 1, "index": 10, "rank": 0, "submit_monotonic": 9120631.856351316, "finish_monotonic": 9120631.894228024, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q11", "burst": "b16-r1", "round": 1, "index": 11, "rank": 1, "submit_monotonic": 9120631.856521145, "finish_monotonic": 9120631.894245336, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q12", "burst": "b16-r1", "round": 1, "index": 12, "rank": 0, "submit_monotonic": 9120631.856704928, "finish_monotonic": 9120631.899613872, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q13", "burst": "b16-r1", "round": 1, "index": 13, "rank": 1, "submit_monotonic": 9120631.856886072, "finish_monotonic": 9120631.899623835, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q14", "burst": "b16-r1", "round": 1, "index": 14, "rank": 0, "submit_monotonic": 9120631.857065404, "finish_monotonic": 9120631.904759064, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q15", "burst": "b16-r1", "round": 1, "index": 15, "rank": 1, "submit_monotonic": 9120631.857236683, "finish_monotonic": 9120631.90476597, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q0", "burst": "b16-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9120633.007311763, "finish_monotonic": 9120633.019998591, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q1", "burst": "b16-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9120633.007788422, "finish_monotonic": 9120633.02001751, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q2", "burst": "b16-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9120633.008083, "finish_monotonic": 9120633.03034932, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q3", "burst": "b16-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9120633.00830188, "finish_monotonic": 9120633.030445123, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q4", "burst": "b16-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9120633.008509327, "finish_monotonic": 9120633.034537788, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q5", "burst": "b16-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9120633.008716583, "finish_monotonic": 9120633.034543687, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q6", "burst": "b16-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9120633.00893061, "finish_monotonic": 9120633.038714863, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q7", "burst": "b16-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9120633.009198193, "finish_monotonic": 9120633.03872045, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q8", "burst": "b16-r2", "round": 2, "index": 8, "rank": 0, "submit_monotonic": 9120633.009473508, "finish_monotonic": 9120633.042995991, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q9", "burst": "b16-r2", "round": 2, "index": 9, "rank": 1, "submit_monotonic": 9120633.00972221, "finish_monotonic": 9120633.045476284, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q10", "burst": "b16-r2", "round": 2, "index": 10, "rank": 0, "submit_monotonic": 9120633.009998403, "finish_monotonic": 9120633.049505532, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q11", "burst": "b16-r2", "round": 2, "index": 11, "rank": 1, "submit_monotonic": 9120633.010321893, "finish_monotonic": 9120633.049663324, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q12", "burst": "b16-r2", "round": 2, "index": 12, "rank": 0, "submit_monotonic": 9120633.0106315, "finish_monotonic": 9120633.053712623, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q13", "burst": "b16-r2", "round": 2, "index": 13, "rank": 1, "submit_monotonic": 9120633.010876952, "finish_monotonic": 9120633.053716576, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q14", "burst": "b16-r2", "round": 2, "index": 14, "rank": 0, "submit_monotonic": 9120633.011186015, "finish_monotonic": 9120633.058281144, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q15", "burst": "b16-r2", "round": 2, "index": 15, "rank": 1, "submit_monotonic": 9120633.011494512, "finish_monotonic": 9120633.058289863, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/summary.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/summary.json deleted file mode 100644 index e03d6e86..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/dense/summary.json +++ /dev/null @@ -1,186 +0,0 @@ -{ - "model_config": "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/data/config/models/Llama-3.2-1B-Instruct.json", - "num_gpu_blocks": 304854, - "block_size": 16, - "engine_args": { - "model": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/dense/model", - "served_model_name": null, - "tokenizer": null, - "hf_config_path": null, - "runner": "auto", - "convert": "auto", - "task": null, - "skip_tokenizer_init": true, - "enable_prompt_embeds": false, - "tokenizer_mode": "auto", - "trust_remote_code": false, - "allowed_local_media_path": "", - "download_dir": null, - "safetensors_load_strategy": "lazy", - "load_format": "dummy", - "config_format": "auto", - "dtype": "bfloat16", - "kv_cache_dtype": "auto", - "seed": 0, - "max_model_len": 512, - "distributed_executor_backend": null, - "pipeline_parallel_size": 2, - "tensor_parallel_size": 1, - "decode_context_parallel_size": 1, - "data_parallel_size": 2, - "data_parallel_rank": null, - "data_parallel_start_rank": null, - "data_parallel_size_local": null, - "data_parallel_address": null, - "data_parallel_rpc_port": null, - "data_parallel_hybrid_lb": false, - "data_parallel_backend": "mp", - "enable_expert_parallel": false, - "enable_eplb": false, - "num_redundant_experts": 0, - "eplb_window_size": 1000, - "eplb_step_interval": 3000, - "eplb_log_balancedness": false, - "max_parallel_loading_workers": null, - "block_size": 16, - "enable_prefix_caching": false, - "prefix_caching_hash_algo": "sha256", - "disable_sliding_window": false, - "disable_cascade_attn": false, - "swap_space": 4, - "cpu_offload_gb": 0, - "gpu_memory_utilization": 0.5, - "kv_cache_memory_bytes": null, - "max_num_batched_tokens": 256, - "max_num_partial_prefills": 1, - "max_long_partial_prefills": 1, - "long_prefill_token_threshold": 0, - "max_num_seqs": 4, - "max_logprobs": 20, - "disable_log_stats": true, - "revision": null, - "code_revision": null, - "rope_theta": null, - "hf_token": null, - "tokenizer_revision": null, - "quantization": null, - "enforce_eager": true, - "max_seq_len_to_capture": 8192, - "disable_custom_all_reduce": false, - "interleave_mm_strings": false, - "mm_processor_kwargs": null, - "disable_mm_preprocessor_cache": false, - "mm_processor_cache_gb": 4, - "mm_encoder_tp_mode": "weights", - "io_processor_plugin": null, - "skip_mm_profiling": false, - "enable_lora": false, - "enable_lora_bias": false, - "max_loras": 1, - "max_lora_rank": 16, - "default_mm_loras": null, - "fully_sharded_loras": false, - "max_cpu_loras": null, - "lora_dtype": "auto", - "lora_extra_vocab_size": 256, - "ray_workers_use_nsight": false, - "num_gpu_blocks_override": null, - "num_lookahead_slots": 0, - "ignore_patterns": null, - "preemption_mode": null, - "scheduler_delay_factor": 0.0, - "enable_chunked_prefill": true, - "disable_chunked_mm_input": false, - "disable_hybrid_kv_cache_manager": false, - "guided_decoding_backend": "auto", - "guided_decoding_disable_fallback": false, - "guided_decoding_disable_any_whitespace": false, - "guided_decoding_disable_additional_properties": false, - "logits_processor_pattern": null, - "speculative_config": null, - "show_hidden_metrics_for_version": null, - "otlp_traces_endpoint": null, - "collect_detailed_traces": null, - "disable_async_output_proc": false, - "scheduling_policy": "fcfs", - "scheduler_cls": "vllm.v1.core.sched.scheduler.Scheduler", - "override_pooler_config": null, - "worker_cls": "auto", - "worker_extension_cls": "", - "kv_transfer_config": null, - "kv_events_config": null, - "generation_config": "auto", - "enable_sleep_mode": false, - "model_impl": "auto", - "override_attention_dtype": null, - "calculate_kv_scales": false, - "mamba_cache_dtype": "auto", - "mamba_ssm_cache_dtype": "auto", - "reasoning_parser": "", - "use_tqdm_on_load": true, - "pt_load_map_location": "cpu", - "enable_multimodal_encoder_data_parallel": false, - "logits_processors": null, - "async_scheduling": false, - "kv_sharing_fast_prefill": false, - "enable_log_requests": false - }, - "rounds": [ - { - "label": "warmup", - "round": 0, - "num_requests": 4, - "wall_minus_monotonic_before": 1780982021.0330577, - "wall_minus_monotonic_after": 1780982021.0330582, - "idle_wait_s": 1.1511429883539677 - }, - { - "label": "b8-r0", - "round": 0, - "num_requests": 8, - "wall_minus_monotonic_before": 1780982021.0330582, - "wall_minus_monotonic_after": 1780982021.0330582, - "idle_wait_s": 1.1017107591032982 - }, - { - "label": "b8-r1", - "round": 1, - "num_requests": 8, - "wall_minus_monotonic_before": 1780982021.0330584, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.1021708622574806 - }, - { - "label": "b8-r2", - "round": 2, - "num_requests": 8, - "wall_minus_monotonic_before": 1780982021.0330582, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.101815516129136 - }, - { - "label": "b16-r0", - "round": 0, - "num_requests": 16, - "wall_minus_monotonic_before": 1780982021.0330582, - "wall_minus_monotonic_after": 1780982021.0330582, - "idle_wait_s": 1.0517608132213354 - }, - { - "label": "b16-r1", - "round": 1, - "num_requests": 16, - "wall_minus_monotonic_before": 1780982021.0330582, - "wall_minus_monotonic_after": 1780982021.0330582, - "idle_wait_s": 1.1024115402251482 - }, - { - "label": "b16-r2", - "round": 2, - "num_requests": 16, - "wall_minus_monotonic_before": 1780982021.0330584, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.05128800496459 - } - ] -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_157.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_157.jsonl deleted file mode 100644 index 596b4858..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_157.jsonl +++ /dev/null @@ -1,22 +0,0 @@ -{"kind": "frontend_snapshot", "pid": 157, "seq": 0, "monotonic": 9120574.796476487, "snapshot": 2, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 1, "monotonic": 9120576.119036447, "snapshot": 3, "counts": [[1, 1], [1, 1]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 2, "monotonic": 9120576.294565408, "snapshot": 4, "counts": [[0, 1], [0, 1]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 3, "monotonic": 9120576.395176036, "snapshot": 5, "counts": [[0, 0], [0, 0]], "wave": 0, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 4, "monotonic": 9120577.567102993, "snapshot": 6, "counts": [[0, 1], [0, 1]], "wave": 1, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 5, "monotonic": 9120577.666602153, "snapshot": 7, "counts": [[0, 1], [0, 1]], "wave": 1, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 6, "monotonic": 9120577.76681236, "snapshot": 8, "counts": [[0, 0], [0, 0]], "wave": 1, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 7, "monotonic": 9120578.830534104, "snapshot": 9, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 8, "monotonic": 9120578.930689773, "snapshot": 10, "counts": [[0, 1], [0, 1]], "wave": 2, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 9, "monotonic": 9120579.030739984, "snapshot": 11, "counts": [[0, 0], [0, 0]], "wave": 2, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 10, "monotonic": 9120580.093352964, "snapshot": 12, "counts": [[0, 1], [0, 1]], "wave": 3, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 11, "monotonic": 9120580.192300623, "snapshot": 13, "counts": [[0, 1], [0, 1]], "wave": 3, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 12, "monotonic": 9120580.292534402, "snapshot": 14, "counts": [[0, 0], [0, 0]], "wave": 3, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 13, "monotonic": 9120581.361761319, "snapshot": 15, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 14, "monotonic": 9120581.46223673, "snapshot": 16, "counts": [[0, 1], [0, 1]], "wave": 4, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 15, "monotonic": 9120581.599077467, "snapshot": 17, "counts": [[0, 0], [0, 0]], "wave": 5, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 157, "seq": 16, "monotonic": 9120582.59408684, "snapshot": 18, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 17, "monotonic": 9120582.694060272, "snapshot": 19, "counts": [[0, 1], [0, 1]], "wave": 5, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 18, "monotonic": 9120582.833616564, "snapshot": 20, "counts": [[0, 0], [0, 0]], "wave": 6, "engines_running": false} -{"kind": "frontend_snapshot", "pid": 157, "seq": 19, "monotonic": 9120583.839656929, "snapshot": 21, "counts": [[0, 1], [0, 1]], "wave": 6, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 20, "monotonic": 9120583.939854259, "snapshot": 22, "counts": [[0, 1], [0, 1]], "wave": 6, "engines_running": true} -{"kind": "frontend_snapshot", "pid": 157, "seq": 21, "monotonic": 9120584.040139776, "snapshot": 23, "counts": [[0, 0], [0, 0]], "wave": 6, "engines_running": true} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_232.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_232.jsonl deleted file mode 100644 index 41c8042c..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_232.jsonl +++ /dev/null @@ -1,65 +0,0 @@ -{"kind": "engine_iteration", "pid": 232, "seq": 0, "monotonic": 9120576.068102015, "engine": 0, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 1, "monotonic": 9120576.285050265, "engine": 0, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 2, "monotonic": 9120576.293990524, "engine": 0, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 3, "monotonic": 9120576.302651672, "engine": 0, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 4, "monotonic": 9120576.309567677, "engine": 0, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 5, "monotonic": 9120577.548942052, "engine": 0, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 6, "monotonic": 9120577.56650086, "engine": 0, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 7, "monotonic": 9120577.57367626, "engine": 0, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 8, "monotonic": 9120577.588618556, "engine": 0, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 9, "monotonic": 9120577.597501721, "engine": 0, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 10, "monotonic": 9120577.606690852, "engine": 0, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 11, "monotonic": 9120577.613504106, "engine": 0, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 12, "monotonic": 9120577.620328449, "engine": 0, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 13, "monotonic": 9120578.811790982, "engine": 0, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 14, "monotonic": 9120578.829836285, "engine": 0, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 15, "monotonic": 9120578.837146323, "engine": 0, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 16, "monotonic": 9120578.851954928, "engine": 0, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 17, "monotonic": 9120578.860749573, "engine": 0, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 18, "monotonic": 9120578.869652415, "engine": 0, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 19, "monotonic": 9120578.876468524, "engine": 0, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 20, "monotonic": 9120578.883193335, "engine": 0, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 21, "monotonic": 9120580.0741665, "engine": 0, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 22, "monotonic": 9120580.092560664, "engine": 0, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 23, "monotonic": 9120580.099539263, "engine": 0, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 24, "monotonic": 9120580.119030692, "engine": 0, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 25, "monotonic": 9120580.129233615, "engine": 0, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 26, "monotonic": 9120580.138761727, "engine": 0, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 27, "monotonic": 9120580.145595407, "engine": 0, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 28, "monotonic": 9120580.152127512, "engine": 0, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 29, "monotonic": 9120581.343008349, "engine": 0, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 30, "monotonic": 9120581.36114861, "engine": 0, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 31, "monotonic": 9120581.366927866, "engine": 0, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 32, "monotonic": 9120581.379694436, "engine": 0, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 33, "monotonic": 9120581.386357982, "engine": 0, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 34, "monotonic": 9120581.39295362, "engine": 0, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q8"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 35, "monotonic": 9120581.401882611, "engine": 0, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q10"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 36, "monotonic": 9120581.409054143, "engine": 0, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q12"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 37, "monotonic": 9120581.415643968, "engine": 0, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q14"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 38, "monotonic": 9120581.422455283, "engine": 0, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 39, "monotonic": 9120581.427875658, "engine": 0, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 40, "monotonic": 9120581.432896864, "engine": 0, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 41, "monotonic": 9120582.576324183, "engine": 0, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 42, "monotonic": 9120582.593326459, "engine": 0, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 43, "monotonic": 9120582.598984815, "engine": 0, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 44, "monotonic": 9120582.615470257, "engine": 0, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 45, "monotonic": 9120582.62383548, "engine": 0, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 46, "monotonic": 9120582.632251784, "engine": 0, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q8"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 47, "monotonic": 9120582.64085373, "engine": 0, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q10"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 48, "monotonic": 9120582.649460929, "engine": 0, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q12"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 49, "monotonic": 9120582.657748772, "engine": 0, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q14"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 50, "monotonic": 9120582.667302229, "engine": 0, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 51, "monotonic": 9120582.672339192, "engine": 0, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 52, "monotonic": 9120582.677481571, "engine": 0, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 53, "monotonic": 9120583.821056273, "engine": 0, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q0"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 54, "monotonic": 9120583.83902434, "engine": 0, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 55, "monotonic": 9120583.844306864, "engine": 0, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q2"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 56, "monotonic": 9120583.860620424, "engine": 0, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q4"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 57, "monotonic": 9120583.869644647, "engine": 0, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q6"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 58, "monotonic": 9120583.878017789, "engine": 0, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q8"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 59, "monotonic": 9120583.886359224, "engine": 0, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q10"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 60, "monotonic": 9120583.894910123, "engine": 0, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q12"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 61, "monotonic": 9120583.903448792, "engine": 0, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q14"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 232, "seq": 62, "monotonic": 9120583.911947114, "engine": 0, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 63, "monotonic": 9120583.918588044, "engine": 0, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 232, "seq": 64, "monotonic": 9120583.925187727, "engine": 0, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_233.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_233.jsonl deleted file mode 100644 index c7e40079..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/dp_placement/dp_placement_233.jsonl +++ /dev/null @@ -1,65 +0,0 @@ -{"kind": "engine_iteration", "pid": 233, "seq": 0, "monotonic": 9120576.066573633, "engine": 1, "wave": 0, "step": 0, "waiting": 1, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 1, "monotonic": 9120576.28477715, "engine": 1, "wave": 0, "step": 1, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["warmup-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 2, "monotonic": 9120576.294229764, "engine": 1, "wave": 0, "step": 2, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 3, "monotonic": 9120576.302748293, "engine": 1, "wave": 0, "step": 3, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 4, "monotonic": 9120576.30969718, "engine": 1, "wave": 0, "step": 4, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 5, "monotonic": 9120577.549137808, "engine": 1, "wave": 1, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 6, "monotonic": 9120577.566702828, "engine": 1, "wave": 1, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 7, "monotonic": 9120577.573778512, "engine": 1, "wave": 1, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 8, "monotonic": 9120577.588767529, "engine": 1, "wave": 1, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 9, "monotonic": 9120577.597639225, "engine": 1, "wave": 1, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r0-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 10, "monotonic": 9120577.606814198, "engine": 1, "wave": 1, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 11, "monotonic": 9120577.613607325, "engine": 1, "wave": 1, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 12, "monotonic": 9120577.62049405, "engine": 1, "wave": 1, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 13, "monotonic": 9120578.812228687, "engine": 1, "wave": 2, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 14, "monotonic": 9120578.829954091, "engine": 1, "wave": 2, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 15, "monotonic": 9120578.837080324, "engine": 1, "wave": 2, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 16, "monotonic": 9120578.851976847, "engine": 1, "wave": 2, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 17, "monotonic": 9120578.860928042, "engine": 1, "wave": 2, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r1-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 18, "monotonic": 9120578.869871182, "engine": 1, "wave": 2, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 19, "monotonic": 9120578.876708878, "engine": 1, "wave": 2, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 20, "monotonic": 9120578.883417822, "engine": 1, "wave": 2, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 21, "monotonic": 9120580.074600577, "engine": 1, "wave": 3, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 22, "monotonic": 9120580.09266292, "engine": 1, "wave": 3, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 23, "monotonic": 9120580.099672109, "engine": 1, "wave": 3, "step": 2, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 24, "monotonic": 9120580.119215572, "engine": 1, "wave": 3, "step": 3, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 25, "monotonic": 9120580.129176484, "engine": 1, "wave": 3, "step": 4, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b8-r2-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 26, "monotonic": 9120580.138748785, "engine": 1, "wave": 3, "step": 5, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 27, "monotonic": 9120580.145636436, "engine": 1, "wave": 3, "step": 6, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 28, "monotonic": 9120580.152310977, "engine": 1, "wave": 3, "step": 7, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 29, "monotonic": 9120581.343315285, "engine": 1, "wave": 4, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 30, "monotonic": 9120581.361277947, "engine": 1, "wave": 4, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 31, "monotonic": 9120581.367144477, "engine": 1, "wave": 4, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 32, "monotonic": 9120581.379747193, "engine": 1, "wave": 4, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 33, "monotonic": 9120581.386451261, "engine": 1, "wave": 4, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 34, "monotonic": 9120581.392984452, "engine": 1, "wave": 4, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q9"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 35, "monotonic": 9120581.401938003, "engine": 1, "wave": 4, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q11"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 36, "monotonic": 9120581.409002103, "engine": 1, "wave": 4, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q13"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 37, "monotonic": 9120581.415707905, "engine": 1, "wave": 4, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r0-q15"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 38, "monotonic": 9120581.422552591, "engine": 1, "wave": 4, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 39, "monotonic": 9120581.427866183, "engine": 1, "wave": 4, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 40, "monotonic": 9120581.432943664, "engine": 1, "wave": 4, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 41, "monotonic": 9120582.57642316, "engine": 1, "wave": 5, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 42, "monotonic": 9120582.593373183, "engine": 1, "wave": 5, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 43, "monotonic": 9120582.598950867, "engine": 1, "wave": 5, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 44, "monotonic": 9120582.615441436, "engine": 1, "wave": 5, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 45, "monotonic": 9120582.62390378, "engine": 1, "wave": 5, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 46, "monotonic": 9120582.632348128, "engine": 1, "wave": 5, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q9"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 47, "monotonic": 9120582.640770746, "engine": 1, "wave": 5, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q11"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 48, "monotonic": 9120582.649319585, "engine": 1, "wave": 5, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q13"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 49, "monotonic": 9120582.657647757, "engine": 1, "wave": 5, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r1-q15"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 50, "monotonic": 9120582.667399248, "engine": 1, "wave": 5, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 51, "monotonic": 9120582.672459846, "engine": 1, "wave": 5, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 52, "monotonic": 9120582.677348372, "engine": 1, "wave": 5, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 53, "monotonic": 9120583.821293969, "engine": 1, "wave": 6, "step": 0, "waiting": 0, "running": 1, "published": true, "branch": "scheduled_without_applying", "applied_output": false, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q1"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 54, "monotonic": 9120583.839159656, "engine": 1, "wave": 6, "step": 1, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 55, "monotonic": 9120583.844510851, "engine": 1, "wave": 6, "step": 2, "waiting": 6, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q3"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 56, "monotonic": 9120583.860668816, "engine": 1, "wave": 6, "step": 3, "waiting": 5, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q5"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 57, "monotonic": 9120583.86950854, "engine": 1, "wave": 6, "step": 4, "waiting": 4, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q7"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 58, "monotonic": 9120583.878138375, "engine": 1, "wave": 6, "step": 5, "waiting": 3, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q9"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 59, "monotonic": 9120583.886439929, "engine": 1, "wave": 6, "step": 6, "waiting": 2, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q11"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 60, "monotonic": 9120583.894956497, "engine": 1, "wave": 6, "step": 7, "waiting": 1, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q13"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 61, "monotonic": 9120583.903405605, "engine": 1, "wave": 6, "step": 8, "waiting": 0, "running": 1, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": ["b16-r2-q15"], "num_scheduled_tokens": 256} -{"kind": "engine_iteration", "pid": 233, "seq": 62, "monotonic": 9120583.911890052, "engine": 1, "wave": 6, "step": 9, "waiting": 0, "running": 0, "published": true, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 63, "monotonic": 9120583.918713111, "engine": 1, "wave": 6, "step": 10, "waiting": 0, "running": 0, "published": false, "branch": "applied_after_scheduling", "applied_output": true, "queue_occupancy": 1, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} -{"kind": "engine_iteration", "pid": 233, "seq": 64, "monotonic": 9120583.925296413, "engine": 1, "wave": 6, "step": 11, "waiting": 0, "running": 0, "published": false, "branch": "applied_without_scheduling", "applied_output": true, "queue_occupancy": 0, "scheduled_new_req_ids": [], "num_scheduled_tokens": 0} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/model/config.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/model/config.json deleted file mode 100644 index 9b897034..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/model/config.json +++ /dev/null @@ -1,95 +0,0 @@ -{ - "_comment": "Qwen3-30B-A3B-tiny: Modified Qwen3-30B-A3B for MoE profiling testing (reduced layers and experts)", - "vocab_size": 151936, - "max_position_embeddings": 40960, - "hidden_size": 2048, - "intermediate_size": 6144, - "num_hidden_layers": 8, - "num_attention_heads": 32, - "use_sliding_window": false, - "sliding_window": null, - "num_key_value_heads": 4, - "hidden_act": "silu", - "initializer_range": 0.02, - "rms_norm_eps": 1e-06, - "use_cache": true, - "rope_theta": 1000000.0, - "rope_scaling": null, - "attention_bias": false, - "attention_dropout": 0.0, - "decoder_sparse_step": 1, - "moe_intermediate_size": 768, - "num_experts_per_tok": 8, - "num_experts": 16, - "norm_topk_prob": true, - "output_router_logits": false, - "router_aux_loss_coef": 0.001, - "mlp_only_layers": [], - "return_dict": true, - "output_hidden_states": false, - "torchscript": false, - "dtype": "bfloat16", - "pruned_heads": {}, - "tie_word_embeddings": false, - "chunk_size_feed_forward": 0, - "is_encoder_decoder": false, - "is_decoder": false, - "cross_attention_hidden_size": null, - "add_cross_attention": false, - "tie_encoder_decoder": false, - "architectures": [ - "Qwen3MoeForCausalLM" - ], - "finetuning_task": null, - "id2label": { - "0": "LABEL_0", - "1": "LABEL_1" - }, - "label2id": { - "LABEL_0": 0, - "LABEL_1": 1 - }, - "task_specific_params": null, - "problem_type": null, - "tokenizer_class": null, - "prefix": null, - "bos_token_id": 151643, - "pad_token_id": null, - "eos_token_id": 151645, - "sep_token_id": null, - "decoder_start_token_id": null, - "max_length": 20, - "min_length": 0, - "do_sample": false, - "early_stopping": false, - "num_beams": 1, - "temperature": 1.0, - "top_k": 50, - "top_p": 1.0, - "typical_p": 1.0, - "repetition_penalty": 1.0, - "length_penalty": 1.0, - "no_repeat_ngram_size": 0, - "encoder_no_repeat_ngram_size": 0, - "bad_words_ids": null, - "num_return_sequences": 1, - "output_scores": false, - "return_dict_in_generate": false, - "forced_bos_token_id": null, - "forced_eos_token_id": null, - "remove_invalid_values": false, - "exponential_decay_length_penalty": null, - "suppress_tokens": null, - "begin_suppress_tokens": null, - "num_beam_groups": 1, - "diversity_penalty": 0.0, - "_name_or_path": "Qwen/Qwen3-30B-A3B", - "transformers_version": "4.57.3", - "head_dim": 128, - "max_window_layers": 48, - "model_type": "qwen3_moe", - "tf_legacy_loss": false, - "use_bfloat16": false, - "output_attentions": false, - "use_qk_norm": true -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/pp_boundary.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/pp_boundary.jsonl deleted file mode 100644 index c99eea4d..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/pp_boundary.jsonl +++ /dev/null @@ -1,152 +0,0 @@ -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120576.06912624, "preprocess_end_ts": 9120576.08141513, "forward_start_ts": 9120576.081611509, "forward_end_ts": 9120576.09329179, "timestamp": 1790102597.2836564, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120576.093549645, "send_end_ts": 9120576.25057326} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120576.067281932, "preprocess_end_ts": 9120576.078339692, "forward_start_ts": 9120576.078541664, "forward_end_ts": 9120576.093035089, "timestamp": 1790102597.2917824, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120576.093350155, "send_end_ts": 9120576.258700026} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120576.258467624, "preprocess_end_ts": 9120576.271626204, "forward_start_ts": 9120576.271777231, "forward_end_ts": 9120576.283202868, "timestamp": 1790102597.3172412, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120576.06675498, "recv_end_ts": 9120576.25815104, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 0, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120576.251813464, "preprocess_end_ts": 9120576.266894476, "forward_start_ts": 9120576.267087674, "forward_end_ts": 9120576.283423033, "timestamp": 1790102597.3174233, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120576.068382105, "recv_end_ts": 9120576.251342716, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120576.25950386, "preprocess_end_ts": 9120576.260119552, "forward_start_ts": 9120576.26014196, "forward_end_ts": 9120576.268356636, "timestamp": 1790102597.3179822, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120576.269240877, "send_end_ts": 9120576.284913171} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120576.251260724, "preprocess_end_ts": 9120576.251849292, "forward_start_ts": 9120576.251871116, "forward_end_ts": 9120576.2692128, "timestamp": 1790102597.3182423, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120576.269356001, "send_end_ts": 9120576.28517143} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120576.285398638, "preprocess_end_ts": 9120576.28580193, "forward_start_ts": 9120576.285816776, "forward_end_ts": 9120576.291115416, "timestamp": 1790102597.326646, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120576.284717713, "recv_end_ts": 9120576.28519154, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 1, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["warmup-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120576.285317533, "preprocess_end_ts": 9120576.285914203, "forward_start_ts": 9120576.285933616, "forward_end_ts": 9120576.293196166, "timestamp": 1790102597.3267689, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120576.284539368, "recv_end_ts": 9120576.28504768, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.549571201, "preprocess_end_ts": 9120577.55009034, "forward_start_ts": 9120577.550104069, "forward_end_ts": 9120577.55616991, "timestamp": 1790102598.5898101, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.55628498, "send_end_ts": 9120577.556742346} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.549495123, "preprocess_end_ts": 9120577.55013278, "forward_start_ts": 9120577.55014999, "forward_end_ts": 9120577.55611502, "timestamp": 1790102598.589807, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.556241311, "send_end_ts": 9120577.556741873} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.55712839, "preprocess_end_ts": 9120577.557572547, "forward_start_ts": 9120577.557585603, "forward_end_ts": 9120577.563604295, "timestamp": 1790102598.5991333, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.549080389, "recv_end_ts": 9120577.55690697, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 2, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.557268629, "preprocess_end_ts": 9120577.557913676, "forward_start_ts": 9120577.557930704, "forward_end_ts": 9120577.565687869, "timestamp": 1790102598.599265, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.549293175, "recv_end_ts": 9120577.556970704, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.573831668, "preprocess_end_ts": 9120577.574154207, "forward_start_ts": 9120577.574164923, "forward_end_ts": 9120577.579463609, "timestamp": 1790102598.6130702, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.579588724, "send_end_ts": 9120577.579996692} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.573992858, "preprocess_end_ts": 9120577.574350936, "forward_start_ts": 9120577.574361622, "forward_end_ts": 9120577.579558125, "timestamp": 1790102598.6131058, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.579642821, "send_end_ts": 9120577.580039855} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.580262372, "preprocess_end_ts": 9120577.580620212, "forward_start_ts": 9120577.580630852, "forward_end_ts": 9120577.585889503, "timestamp": 1790102598.6212263, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.57371594, "recv_end_ts": 9120577.580101142, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 3, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.580354996, "preprocess_end_ts": 9120577.580813946, "forward_start_ts": 9120577.580826651, "forward_end_ts": 9120577.587808212, "timestamp": 1790102598.6213608, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.573786428, "recv_end_ts": 9120577.580179015, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.580370976, "preprocess_end_ts": 9120577.580712583, "forward_start_ts": 9120577.580723124, "forward_end_ts": 9120577.58584904, "timestamp": 1790102598.6218507, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.585956562, "send_end_ts": 9120577.588786367} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.580392718, "preprocess_end_ts": 9120577.58073172, "forward_start_ts": 9120577.580740644, "forward_end_ts": 9120577.585923836, "timestamp": 1790102598.6219738, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.586004073, "send_end_ts": 9120577.588908568} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.589121113, "preprocess_end_ts": 9120577.589516431, "forward_start_ts": 9120577.589526488, "forward_end_ts": 9120577.594744284, "timestamp": 1790102598.6301506, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.588429969, "recv_end_ts": 9120577.58895459, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 4, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.589128325, "preprocess_end_ts": 9120577.589584732, "forward_start_ts": 9120577.58959636, "forward_end_ts": 9120577.596727813, "timestamp": 1790102598.630274, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.588598276, "recv_end_ts": 9120577.588972116, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.589193283, "preprocess_end_ts": 9120577.589508563, "forward_start_ts": 9120577.589517279, "forward_end_ts": 9120577.594660422, "timestamp": 1790102598.6308415, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.594835784, "send_end_ts": 9120577.597777065} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120577.589247884, "preprocess_end_ts": 9120577.589569096, "forward_start_ts": 9120577.589578193, "forward_end_ts": 9120577.594807643, "timestamp": 1790102598.630937, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120577.594884492, "send_end_ts": 9120577.597870873} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.598051693, "preprocess_end_ts": 9120577.59839024, "forward_start_ts": 9120577.598399421, "forward_end_ts": 9120577.603558876, "timestamp": 1790102598.6393318, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.597388487, "recv_end_ts": 9120577.597895443, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 5, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120577.59808894, "preprocess_end_ts": 9120577.5985329, "forward_start_ts": 9120577.598543633, "forward_end_ts": 9120577.605920406, "timestamp": 1790102598.6394594, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120577.597512793, "recv_end_ts": 9120577.597917935, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.812697122, "preprocess_end_ts": 9120578.813320901, "forward_start_ts": 9120578.813337032, "forward_end_ts": 9120578.819544006, "timestamp": 1790102599.853156, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.819649665, "send_end_ts": 9120578.82009076} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.812466595, "preprocess_end_ts": 9120578.813126571, "forward_start_ts": 9120578.813147675, "forward_end_ts": 9120578.819582999, "timestamp": 1790102599.8532457, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.819706036, "send_end_ts": 9120578.82017994} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.820711149, "preprocess_end_ts": 9120578.821310576, "forward_start_ts": 9120578.821343388, "forward_end_ts": 9120578.827582017, "timestamp": 1790102599.8624983, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.812198428, "recv_end_ts": 9120578.82035111, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 6, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.820530899, "preprocess_end_ts": 9120578.821104528, "forward_start_ts": 9120578.821120372, "forward_end_ts": 9120578.829012496, "timestamp": 1790102599.8625872, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.812379096, "recv_end_ts": 9120578.820289545, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.83731334, "preprocess_end_ts": 9120578.837618664, "forward_start_ts": 9120578.83762973, "forward_end_ts": 9120578.842705376, "timestamp": 1790102599.8762634, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.842833433, "send_end_ts": 9120578.843199044} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.837259185, "preprocess_end_ts": 9120578.837583508, "forward_start_ts": 9120578.837593429, "forward_end_ts": 9120578.84280425, "timestamp": 1790102599.8763123, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.842886532, "send_end_ts": 9120578.843248315} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.843512638, "preprocess_end_ts": 9120578.843970332, "forward_start_ts": 9120578.843982011, "forward_end_ts": 9120578.851015653, "timestamp": 1790102599.8845687, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.837185057, "recv_end_ts": 9120578.84332666, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 7, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.8435567, "preprocess_end_ts": 9120578.844025772, "forward_start_ts": 9120578.844038067, "forward_end_ts": 9120578.851041503, "timestamp": 1790102599.8845901, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.837132009, "recv_end_ts": 9120578.843378464, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.843535822, "preprocess_end_ts": 9120578.84385769, "forward_start_ts": 9120578.843867462, "forward_end_ts": 9120578.848940914, "timestamp": 1790102599.8852592, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.849080913, "send_end_ts": 9120578.852194704} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.843580656, "preprocess_end_ts": 9120578.843910933, "forward_start_ts": 9120578.843920348, "forward_end_ts": 9120578.849048804, "timestamp": 1790102599.885258, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.849123884, "send_end_ts": 9120578.85219327} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.852500973, "preprocess_end_ts": 9120578.85295746, "forward_start_ts": 9120578.852968963, "forward_end_ts": 9120578.858358746, "timestamp": 1790102599.8935063, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.851837687, "recv_end_ts": 9120578.85232474, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 8, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.852441452, "preprocess_end_ts": 9120578.85289666, "forward_start_ts": 9120578.852908885, "forward_end_ts": 9120578.86007354, "timestamp": 1790102599.8936193, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.85182794, "recv_end_ts": 9120578.85227706, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.852539623, "preprocess_end_ts": 9120578.852858113, "forward_start_ts": 9120578.852866942, "forward_end_ts": 9120578.857989244, "timestamp": 1790102599.8940766, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.858322257, "send_end_ts": 9120578.861012325} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120578.852627993, "preprocess_end_ts": 9120578.85298128, "forward_start_ts": 9120578.852990692, "forward_end_ts": 9120578.8582934, "timestamp": 1790102599.8943467, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120578.858367149, "send_end_ts": 9120578.861281207} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.861280115, "preprocess_end_ts": 9120578.861627065, "forward_start_ts": 9120578.861636132, "forward_end_ts": 9120578.86709386, "timestamp": 1790102599.9024386, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.860707372, "recv_end_ts": 9120578.861127583, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 9, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120578.8614549, "preprocess_end_ts": 9120578.86190434, "forward_start_ts": 9120578.861915732, "forward_end_ts": 9120578.868989771, "timestamp": 1790102599.9025333, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120578.860852864, "recv_end_ts": 9120578.861284569, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.074596796, "preprocess_end_ts": 9120580.075139388, "forward_start_ts": 9120580.075154155, "forward_end_ts": 9120580.081509704, "timestamp": 1790102601.1157746, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.082217831, "send_end_ts": 9120580.082709515} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.075100802, "preprocess_end_ts": 9120580.075675512, "forward_start_ts": 9120580.075690024, "forward_end_ts": 9120580.082179496, "timestamp": 1790102601.1159906, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.082398407, "send_end_ts": 9120580.082924727} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.083210835, "preprocess_end_ts": 9120580.083756076, "forward_start_ts": 9120580.083771244, "forward_end_ts": 9120580.08955698, "timestamp": 1790102601.1250973, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.074461987, "recv_end_ts": 9120580.082938297, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 10, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.083296858, "preprocess_end_ts": 9120580.083885018, "forward_start_ts": 9120580.083901592, "forward_end_ts": 9120580.09166343, "timestamp": 1790102601.125262, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.074712954, "recv_end_ts": 9120580.083052542, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.100881828, "preprocess_end_ts": 9120580.101360453, "forward_start_ts": 9120580.101374147, "forward_end_ts": 9120580.109860908, "timestamp": 1790102601.1435313, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.109983295, "send_end_ts": 9120580.11046472} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.100881828, "preprocess_end_ts": 9120580.101360446, "forward_start_ts": 9120580.101374116, "forward_end_ts": 9120580.109896155, "timestamp": 1790102601.1435313, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.110006873, "send_end_ts": 9120580.110464765} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.110640671, "preprocess_end_ts": 9120580.111033125, "forward_start_ts": 9120580.111044472, "forward_end_ts": 9120580.11642792, "timestamp": 1790102601.1517575, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.099583574, "recv_end_ts": 9120580.110478189, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 11, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.1107548, "preprocess_end_ts": 9120580.111248905, "forward_start_ts": 9120580.11126213, "forward_end_ts": 9120580.11832074, "timestamp": 1790102601.151885, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.099723669, "recv_end_ts": 9120580.110559914, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.110935118, "preprocess_end_ts": 9120580.111394485, "forward_start_ts": 9120580.111406164, "forward_end_ts": 9120580.119771758, "timestamp": 1790102601.153413, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.119873416, "send_end_ts": 9120580.120346544} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.110935053, "preprocess_end_ts": 9120580.111394396, "forward_start_ts": 9120580.111406185, "forward_end_ts": 9120580.119771767, "timestamp": 1790102601.153413, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.119873255, "send_end_ts": 9120580.120346576} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.120589023, "preprocess_end_ts": 9120580.12105302, "forward_start_ts": 9120580.12106498, "forward_end_ts": 9120580.128097888, "timestamp": 1790102601.1617668, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.119136358, "recv_end_ts": 9120580.12041668, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 12, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.120535705, "preprocess_end_ts": 9120580.120907266, "forward_start_ts": 9120580.120917823, "forward_end_ts": 9120580.128306828, "timestamp": 1790102601.161878, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.118960636, "recv_end_ts": 9120580.120388944, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.120777884, "preprocess_end_ts": 9120580.121225763, "forward_start_ts": 9120580.121237168, "forward_end_ts": 9120580.12957196, "timestamp": 1790102601.1631815, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.129670313, "send_end_ts": 9120580.130115215} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120580.120777883, "preprocess_end_ts": 9120580.121225769, "forward_start_ts": 9120580.121237092, "forward_end_ts": 9120580.129571958, "timestamp": 1790102601.1631815, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120580.12967029, "send_end_ts": 9120580.130115697} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.130371591, "preprocess_end_ts": 9120580.130897123, "forward_start_ts": 9120580.130908811, "forward_end_ts": 9120580.137896962, "timestamp": 1790102601.1714482, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.129156964, "recv_end_ts": 9120580.130191972, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 13, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b8-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120580.130379148, "preprocess_end_ts": 9120580.130824532, "forward_start_ts": 9120580.13083534, "forward_end_ts": 9120580.137938311, "timestamp": 1790102601.1714747, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120580.12900894, "recv_end_ts": 9120580.13020648, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.343732748, "preprocess_end_ts": 9120581.344221508, "forward_start_ts": 9120581.344235841, "forward_end_ts": 9120581.35028834, "timestamp": 1790102602.3839042, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.350392705, "send_end_ts": 9120581.350838909} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.343441563, "preprocess_end_ts": 9120581.34409297, "forward_start_ts": 9120581.344110794, "forward_end_ts": 9120581.350297496, "timestamp": 1790102602.3839097, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.350408033, "send_end_ts": 9120581.350844346} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.351336911, "preprocess_end_ts": 9120581.351899972, "forward_start_ts": 9120581.351921292, "forward_end_ts": 9120581.35798718, "timestamp": 1790102602.3936594, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.343240311, "recv_end_ts": 9120581.351054829, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 14, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.351373885, "preprocess_end_ts": 9120581.351970855, "forward_start_ts": 9120581.35198758, "forward_end_ts": 9120581.360200204, "timestamp": 1790102602.393783, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.343579505, "recv_end_ts": 9120581.351054737, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.367097467, "preprocess_end_ts": 9120581.367441077, "forward_start_ts": 9120581.367452266, "forward_end_ts": 9120581.372749446, "timestamp": 1790102602.4062917, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.37283898, "send_end_ts": 9120581.373226961} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.36734962, "preprocess_end_ts": 9120581.367687093, "forward_start_ts": 9120581.367698174, "forward_end_ts": 9120581.37280371, "timestamp": 1790102602.4063165, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.372886997, "send_end_ts": 9120581.373252112} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.373437852, "preprocess_end_ts": 9120581.373789055, "forward_start_ts": 9120581.373799719, "forward_end_ts": 9120581.378876328, "timestamp": 1790102602.41244, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.366961967, "recv_end_ts": 9120581.37329134, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 15, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.373489227, "preprocess_end_ts": 9120581.373844985, "forward_start_ts": 9120581.373855025, "forward_end_ts": 9120581.378997438, "timestamp": 1790102602.4124928, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.367181359, "recv_end_ts": 9120581.373334873, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.37357514, "preprocess_end_ts": 9120581.373903705, "forward_start_ts": 9120581.373913374, "forward_end_ts": 9120581.379154231, "timestamp": 1790102602.413001, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.379235808, "send_end_ts": 9120581.379936537} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.373586047, "preprocess_end_ts": 9120581.373905532, "forward_start_ts": 9120581.373914763, "forward_end_ts": 9120581.37915932, "timestamp": 1790102602.4131227, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.379233835, "send_end_ts": 9120581.380057868} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.380174685, "preprocess_end_ts": 9120581.380519679, "forward_start_ts": 9120581.380529135, "forward_end_ts": 9120581.38564956, "timestamp": 1790102602.419137, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.379627496, "recv_end_ts": 9120581.380037796, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 16, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.380258484, "preprocess_end_ts": 9120581.380620003, "forward_start_ts": 9120581.38062962, "forward_end_ts": 9120581.3856865, "timestamp": 1790102602.4191911, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.379681192, "recv_end_ts": 9120581.38012081, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.380323807, "preprocess_end_ts": 9120581.380649032, "forward_start_ts": 9120581.380658597, "forward_end_ts": 9120581.385759557, "timestamp": 1790102602.4196768, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.38585794, "send_end_ts": 9120581.386612376} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.380388295, "preprocess_end_ts": 9120581.380705131, "forward_start_ts": 9120581.380713452, "forward_end_ts": 9120581.385826873, "timestamp": 1790102602.419787, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.385898268, "send_end_ts": 9120581.386721654} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.386851596, "preprocess_end_ts": 9120581.38718201, "forward_start_ts": 9120581.387190906, "forward_end_ts": 9120581.392265217, "timestamp": 1790102602.42575, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.386318544, "recv_end_ts": 9120581.386711184, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 17, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.38691968, "preprocess_end_ts": 9120581.387254061, "forward_start_ts": 9120581.387263125, "forward_end_ts": 9120581.392278733, "timestamp": 1790102602.4257653, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.386369605, "recv_end_ts": 9120581.386778977, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.38695468, "preprocess_end_ts": 9120581.38727, "forward_start_ts": 9120581.387279348, "forward_end_ts": 9120581.392546862, "timestamp": 1790102602.426297, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.39261858, "send_end_ts": 9120581.393232038} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.38705204, "preprocess_end_ts": 9120581.387368888, "forward_start_ts": 9120581.387377515, "forward_end_ts": 9120581.392576084, "timestamp": 1790102602.426374, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.392646756, "send_end_ts": 9120581.393308524} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.39347056, "preprocess_end_ts": 9120581.393800288, "forward_start_ts": 9120581.393808791, "forward_end_ts": 9120581.398831991, "timestamp": 1790102602.434644, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.392939206, "recv_end_ts": 9120581.393334216, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 18, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.39349049, "preprocess_end_ts": 9120581.393821685, "forward_start_ts": 9120581.393830648, "forward_end_ts": 9120581.401207073, "timestamp": 1790102602.434714, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.3929638, "recv_end_ts": 9120581.393355276, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.3935938, "preprocess_end_ts": 9120581.393916002, "forward_start_ts": 9120581.39392472, "forward_end_ts": 9120581.398994233, "timestamp": 1790102602.4352026, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.399140792, "send_end_ts": 9120581.402138196} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.393645296, "preprocess_end_ts": 9120581.393961037, "forward_start_ts": 9120581.393969523, "forward_end_ts": 9120581.39911096, "timestamp": 1790102602.4353406, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.399182292, "send_end_ts": 9120581.402275685} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.402473414, "preprocess_end_ts": 9120581.402806476, "forward_start_ts": 9120581.402815038, "forward_end_ts": 9120581.408110779, "timestamp": 1790102602.4417818, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.401888557, "recv_end_ts": 9120581.402333334, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 19, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.402364124, "preprocess_end_ts": 9120581.402693668, "forward_start_ts": 9120581.40270258, "forward_end_ts": 9120581.408346636, "timestamp": 1790102602.441835, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.401829578, "recv_end_ts": 9120581.40222036, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.402505113, "preprocess_end_ts": 9120581.402833944, "forward_start_ts": 9120581.402842766, "forward_end_ts": 9120581.408678753, "timestamp": 1790102602.442369, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.408749774, "send_end_ts": 9120581.40930469} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.4026157, "preprocess_end_ts": 9120581.40293946, "forward_start_ts": 9120581.402949404, "forward_end_ts": 9120581.40806486, "timestamp": 1790102602.4423795, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.408707408, "send_end_ts": 9120581.409314485} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.40956396, "preprocess_end_ts": 9120581.409890601, "forward_start_ts": 9120581.40989973, "forward_end_ts": 9120581.414940232, "timestamp": 1790102602.4484506, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.409017527, "recv_end_ts": 9120581.409412911, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 20, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.40952547, "preprocess_end_ts": 9120581.40986102, "forward_start_ts": 9120581.409869485, "forward_end_ts": 9120581.415017493, "timestamp": 1790102602.4485135, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.40897754, "recv_end_ts": 9120581.40938676, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.409655448, "preprocess_end_ts": 9120581.409974145, "forward_start_ts": 9120581.409983275, "forward_end_ts": 9120581.41501236, "timestamp": 1790102602.4489937, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.415114151, "send_end_ts": 9120581.415927997} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120581.409677612, "preprocess_end_ts": 9120581.40999598, "forward_start_ts": 9120581.410004305, "forward_end_ts": 9120581.415084803, "timestamp": 1790102602.449092, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120581.41515612, "send_end_ts": 9120581.41602733} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.416166749, "preprocess_end_ts": 9120581.416485762, "forward_start_ts": 9120581.416494308, "forward_end_ts": 9120581.421585016, "timestamp": 1790102602.4552574, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.415618919, "recv_end_ts": 9120581.416028755, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 21, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r0-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120581.41623896, "preprocess_end_ts": 9120581.416565629, "forward_start_ts": 9120581.416574404, "forward_end_ts": 9120581.421821417, "timestamp": 1790102602.4553607, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120581.415697642, "recv_end_ts": 9120581.416099109, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.576923266, "preprocess_end_ts": 9120582.577481993, "forward_start_ts": 9120582.577497356, "forward_end_ts": 9120582.583782172, "timestamp": 1790102603.6189995, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.58547205, "send_end_ts": 9120582.585934443} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.576914553, "preprocess_end_ts": 9120582.577515438, "forward_start_ts": 9120582.57753102, "forward_end_ts": 9120582.585449988, "timestamp": 1790102603.6191525, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.58558134, "send_end_ts": 9120582.58608655} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.586269628, "preprocess_end_ts": 9120582.586710135, "forward_start_ts": 9120582.586722473, "forward_end_ts": 9120582.592280349, "timestamp": 1790102603.6259403, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.576488996, "recv_end_ts": 9120582.586054899, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 22, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.586409232, "preprocess_end_ts": 9120582.586853774, "forward_start_ts": 9120582.586867034, "forward_end_ts": 9120582.592480345, "timestamp": 1790102603.626008, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.576604463, "recv_end_ts": 9120582.58619712, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.599168986, "preprocess_end_ts": 9120582.599562293, "forward_start_ts": 9120582.599573622, "forward_end_ts": 9120582.605908327, "timestamp": 1790102603.6415508, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.608063184, "send_end_ts": 9120582.608485227} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.600425657, "preprocess_end_ts": 9120582.600854224, "forward_start_ts": 9120582.60086626, "forward_end_ts": 9120582.60805398, "timestamp": 1790102603.6417098, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.608168932, "send_end_ts": 9120582.60864294} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.608761417, "preprocess_end_ts": 9120582.60915175, "forward_start_ts": 9120582.609163187, "forward_end_ts": 9120582.614419485, "timestamp": 1790102603.6480691, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.599062407, "recv_end_ts": 9120582.608579684, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 23, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.608863711, "preprocess_end_ts": 9120582.609250143, "forward_start_ts": 9120582.609261172, "forward_end_ts": 9120582.614605544, "timestamp": 1790102603.6481156, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.59895246, "recv_end_ts": 9120582.608695596, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.608876169, "preprocess_end_ts": 9120582.60922144, "forward_start_ts": 9120582.609231612, "forward_end_ts": 9120582.614570571, "timestamp": 1790102603.650187, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.616727713, "send_end_ts": 9120582.617122507} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.609102508, "preprocess_end_ts": 9120582.609549047, "forward_start_ts": 9120582.609561363, "forward_end_ts": 9120582.61670548, "timestamp": 1790102603.650329, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.61681186, "send_end_ts": 9120582.61726228} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.617367228, "preprocess_end_ts": 9120582.617727084, "forward_start_ts": 9120582.617737848, "forward_end_ts": 9120582.622994997, "timestamp": 1790102603.6565177, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.615312329, "recv_end_ts": 9120582.617211921, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 24, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.61746808, "preprocess_end_ts": 9120582.617837433, "forward_start_ts": 9120582.617848467, "forward_end_ts": 9120582.623067038, "timestamp": 1790102603.6565673, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.615323745, "recv_end_ts": 9120582.617316635, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.617499232, "preprocess_end_ts": 9120582.617834708, "forward_start_ts": 9120582.617844084, "forward_end_ts": 9120582.623157457, "timestamp": 1790102603.6586914, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.625230229, "send_end_ts": 9120582.62562664} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.617709432, "preprocess_end_ts": 9120582.618145688, "forward_start_ts": 9120582.618157173, "forward_end_ts": 9120582.625210725, "timestamp": 1790102603.6588135, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.625308787, "send_end_ts": 9120582.625747615} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.625877107, "preprocess_end_ts": 9120582.626224307, "forward_start_ts": 9120582.626234315, "forward_end_ts": 9120582.631413292, "timestamp": 1790102603.664929, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.623729687, "recv_end_ts": 9120582.62571517, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 25, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.625945903, "preprocess_end_ts": 9120582.626318509, "forward_start_ts": 9120582.62632906, "forward_end_ts": 9120582.63149246, "timestamp": 1790102603.6649878, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.623764066, "recv_end_ts": 9120582.62579982, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.625981929, "preprocess_end_ts": 9120582.626311673, "forward_start_ts": 9120582.626321288, "forward_end_ts": 9120582.63160772, "timestamp": 1790102603.6671853, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.633735837, "send_end_ts": 9120582.634120768} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.626183568, "preprocess_end_ts": 9120582.62661848, "forward_start_ts": 9120582.626630077, "forward_end_ts": 9120582.6337102, "timestamp": 1790102603.6673136, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.633808337, "send_end_ts": 9120582.634248195} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.634447675, "preprocess_end_ts": 9120582.63480125, "forward_start_ts": 9120582.634811662, "forward_end_ts": 9120582.639880544, "timestamp": 1790102603.6734767, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.63219121, "recv_end_ts": 9120582.634303868, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 26, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.63435824, "preprocess_end_ts": 9120582.63472431, "forward_start_ts": 9120582.634734532, "forward_end_ts": 9120582.640016876, "timestamp": 1790102603.6735172, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.63213902, "recv_end_ts": 9120582.634207767, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.634462172, "preprocess_end_ts": 9120582.63478484, "forward_start_ts": 9120582.634794144, "forward_end_ts": 9120582.640136449, "timestamp": 1790102603.6756687, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.642227378, "send_end_ts": 9120582.642603584} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.634682417, "preprocess_end_ts": 9120582.635108972, "forward_start_ts": 9120582.635119941, "forward_end_ts": 9120582.64221128, "timestamp": 1790102603.6758075, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.64230869, "send_end_ts": 9120582.642742233} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.642930498, "preprocess_end_ts": 9120582.64326802, "forward_start_ts": 9120582.6432767, "forward_end_ts": 9120582.648492845, "timestamp": 1790102603.6820428, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.640663384, "recv_end_ts": 9120582.642789513, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 27, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.642838014, "preprocess_end_ts": 9120582.64318108, "forward_start_ts": 9120582.64319026, "forward_end_ts": 9120582.648611909, "timestamp": 1790102603.6821089, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.640733434, "recv_end_ts": 9120582.642695213, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.642951105, "preprocess_end_ts": 9120582.64327012, "forward_start_ts": 9120582.643279066, "forward_end_ts": 9120582.648580177, "timestamp": 1790102603.6842473, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.650807079, "send_end_ts": 9120582.651182815} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.64317249, "preprocess_end_ts": 9120582.643593661, "forward_start_ts": 9120582.643604832, "forward_end_ts": 9120582.650789414, "timestamp": 1790102603.6843874, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.650882209, "send_end_ts": 9120582.65132139} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.651511695, "preprocess_end_ts": 9120582.651844472, "forward_start_ts": 9120582.651853641, "forward_end_ts": 9120582.656878868, "timestamp": 1790102603.690367, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.649217356, "recv_end_ts": 9120582.6513709, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 28, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.65140676, "preprocess_end_ts": 9120582.651739815, "forward_start_ts": 9120582.651748355, "forward_end_ts": 9120582.656938508, "timestamp": 1790102603.6904323, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.64931462, "recv_end_ts": 9120582.651264912, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.651514748, "preprocess_end_ts": 9120582.651831552, "forward_start_ts": 9120582.651840601, "forward_end_ts": 9120582.65721628, "timestamp": 1790102603.6933012, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.65985598, "send_end_ts": 9120582.660236675} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120582.651739929, "preprocess_end_ts": 9120582.652174477, "forward_start_ts": 9120582.65218502, "forward_end_ts": 9120582.659841204, "timestamp": 1790102603.6934776, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120582.659936965, "send_end_ts": 9120582.660411868} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.66045627, "preprocess_end_ts": 9120582.660783246, "forward_start_ts": 9120582.660792032, "forward_end_ts": 9120582.666339248, "timestamp": 1790102603.7000058, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.657644669, "recv_end_ts": 9120582.66031364, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 29, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r1-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120582.660600586, "preprocess_end_ts": 9120582.660947189, "forward_start_ts": 9120582.660958167, "forward_end_ts": 9120582.666573424, "timestamp": 1790102603.7000759, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120582.657538248, "recv_end_ts": 9120582.660460543, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.821541695, "preprocess_end_ts": 9120583.822123745, "forward_start_ts": 9120583.822139362, "forward_end_ts": 9120583.828476297, "timestamp": 1790102604.8637803, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.830264147, "send_end_ts": 9120583.830715194} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.82172466, "preprocess_end_ts": 9120583.822294204, "forward_start_ts": 9120583.82230907, "forward_end_ts": 9120583.830244573, "timestamp": 1790102604.8639398, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.830370191, "send_end_ts": 9120583.83087368} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q0"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.831101157, "preprocess_end_ts": 9120583.831546476, "forward_start_ts": 9120583.831559015, "forward_end_ts": 9120583.837595148, "timestamp": 1790102604.8716521, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.821289523, "recv_end_ts": 9120583.830886656, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 30, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q1"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.831336629, "preprocess_end_ts": 9120583.83196254, "forward_start_ts": 9120583.831993213, "forward_end_ts": 9120583.838179354, "timestamp": 1790102604.8717523, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.821598507, "recv_end_ts": 9120583.83101284, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.844497563, "preprocess_end_ts": 9120583.844813723, "forward_start_ts": 9120583.844823988, "forward_end_ts": 9120583.851622906, "timestamp": 1790102604.8871188, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.853661515, "send_end_ts": 9120583.854054332} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.84607402, "preprocess_end_ts": 9120583.84649125, "forward_start_ts": 9120583.846503396, "forward_end_ts": 9120583.853645235, "timestamp": 1790102604.887264, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.853750424, "send_end_ts": 9120583.854198288} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q2"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.854285724, "preprocess_end_ts": 9120583.854634887, "forward_start_ts": 9120583.854645245, "forward_end_ts": 9120583.859757915, "timestamp": 1790102604.8933077, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.844375689, "recv_end_ts": 9120583.854136454, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 31, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q3"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.854376545, "preprocess_end_ts": 9120583.854727993, "forward_start_ts": 9120583.854738573, "forward_end_ts": 9120583.859866332, "timestamp": 1790102604.8933706, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.844526729, "recv_end_ts": 9120583.854231462, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.85440786, "preprocess_end_ts": 9120583.854735203, "forward_start_ts": 9120583.854744412, "forward_end_ts": 9120583.860074464, "timestamp": 1790102604.895916, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.862467133, "send_end_ts": 9120583.86285158} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.854629343, "preprocess_end_ts": 9120583.855060486, "forward_start_ts": 9120583.855072329, "forward_end_ts": 9120583.862451376, "timestamp": 1790102604.896046, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.862548715, "send_end_ts": 9120583.86298042} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q5"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.863156855, "preprocess_end_ts": 9120583.863527492, "forward_start_ts": 9120583.863537077, "forward_end_ts": 9120583.868671449, "timestamp": 1790102604.9022155, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.860549659, "recv_end_ts": 9120583.863022398, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 32, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q4"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.863093857, "preprocess_end_ts": 9120583.863435747, "forward_start_ts": 9120583.863445677, "forward_end_ts": 9120583.868781999, "timestamp": 1790102604.902278, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.860511964, "recv_end_ts": 9120583.862957884, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.863200076, "preprocess_end_ts": 9120583.863517843, "forward_start_ts": 9120583.863527136, "forward_end_ts": 9120583.868867451, "timestamp": 1790102604.9044716, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.871042717, "send_end_ts": 9120583.871406928} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.86340592, "preprocess_end_ts": 9120583.863826012, "forward_start_ts": 9120583.863836592, "forward_end_ts": 9120583.871030165, "timestamp": 1790102604.9046118, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.871124, "send_end_ts": 9120583.87154668} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q7"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.871732265, "preprocess_end_ts": 9120583.872077122, "forward_start_ts": 9120583.872086916, "forward_end_ts": 9120583.87720402, "timestamp": 1790102604.9106915, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.869396372, "recv_end_ts": 9120583.871590275, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 33, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q6"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.87163014, "preprocess_end_ts": 9120583.871973757, "forward_start_ts": 9120583.871982671, "forward_end_ts": 9120583.877214683, "timestamp": 1790102604.910703, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.869494393, "recv_end_ts": 9120583.871493684, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.871741671, "preprocess_end_ts": 9120583.87206858, "forward_start_ts": 9120583.872077484, "forward_end_ts": 9120583.877404189, "timestamp": 1790102604.91298, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.879556447, "send_end_ts": 9120583.879915643} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.871962866, "preprocess_end_ts": 9120583.872391593, "forward_start_ts": 9120583.872401793, "forward_end_ts": 9120583.879540468, "timestamp": 1790102604.9131334, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.879632792, "send_end_ts": 9120583.880067756} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q8"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.880156256, "preprocess_end_ts": 9120583.880481124, "forward_start_ts": 9120583.880489904, "forward_end_ts": 9120583.88560634, "timestamp": 1790102604.9191005, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.877898745, "recv_end_ts": 9120583.880015442, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 34, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q9"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.880242497, "preprocess_end_ts": 9120583.88057212, "forward_start_ts": 9120583.880580816, "forward_end_ts": 9120583.885655653, "timestamp": 1790102604.9191403, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.877860919, "recv_end_ts": 9120583.880106712, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.880259756, "preprocess_end_ts": 9120583.88057252, "forward_start_ts": 9120583.880582193, "forward_end_ts": 9120583.885980275, "timestamp": 1790102604.9214914, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.8880694, "send_end_ts": 9120583.888426863} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.880488636, "preprocess_end_ts": 9120583.880906789, "forward_start_ts": 9120583.880917268, "forward_end_ts": 9120583.88806401, "timestamp": 1790102604.921652, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.888157409, "send_end_ts": 9120583.888586646} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q10"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.888661517, "preprocess_end_ts": 9120583.888987511, "forward_start_ts": 9120583.8889958, "forward_end_ts": 9120583.894107789, "timestamp": 1790102604.9276292, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.886290956, "recv_end_ts": 9120583.88852129, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 35, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q11"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.888765827, "preprocess_end_ts": 9120583.889099708, "forward_start_ts": 9120583.889109552, "forward_end_ts": 9120583.894193923, "timestamp": 1790102604.9276834, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.88630878, "recv_end_ts": 9120583.888627024, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.888755564, "preprocess_end_ts": 9120583.889065951, "forward_start_ts": 9120583.889074624, "forward_end_ts": 9120583.894493733, "timestamp": 1790102604.9300442, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.896602778, "send_end_ts": 9120583.896979593} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.889009971, "preprocess_end_ts": 9120583.88942914, "forward_start_ts": 9120583.88943924, "forward_end_ts": 9120583.896590112, "timestamp": 1790102604.9301734, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.89668142, "send_end_ts": 9120583.897108141} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q13"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.897287816, "preprocess_end_ts": 9120583.89762028, "forward_start_ts": 9120583.897630377, "forward_end_ts": 9120583.90267408, "timestamp": 1790102604.936159, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.894849937, "recv_end_ts": 9120583.897149628, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 36, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q12"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.897259109, "preprocess_end_ts": 9120583.89759168, "forward_start_ts": 9120583.897600047, "forward_end_ts": 9120583.902698973, "timestamp": 1790102604.936189, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.894804897, "recv_end_ts": 9120583.897117823, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.897304306, "preprocess_end_ts": 9120583.897619875, "forward_start_ts": 9120583.897628644, "forward_end_ts": 9120583.902977347, "timestamp": 1790102604.9385567, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.905150736, "send_end_ts": 9120583.905492732} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 0, "pp_world_size": 2, "is_first_rank": true, "is_last_rank": false, "preprocess_start_ts": 9120583.897520913, "preprocess_end_ts": 9120583.89793452, "forward_start_ts": 9120583.897944763, "forward_end_ts": 9120583.905138481, "timestamp": 1790102604.9387238, "activation_bytes_per_rank": 2097152, "recv_start_ts": null, "recv_end_ts": null, "send_start_ts": 9120583.905231489, "send_end_ts": 9120583.905658696} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q14"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.905695103, "preprocess_end_ts": 9120583.906016463, "forward_start_ts": 9120583.906024937, "forward_end_ts": 9120583.911124228, "timestamp": 1790102604.9446628, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.903391425, "recv_end_ts": 9120583.905563416, "send_start_ts": null, "send_end_ts": null} -{"model_name": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", "batch_id": 37, "batch_size": 1, "tensor_parallel_degree": 1, "num_prefill_tokens": 256, "num_decode_tokens": 0, "request_ids": ["b16-r2-q15"], "pp_rank": 1, "pp_world_size": 2, "is_first_rank": false, "is_last_rank": true, "preprocess_start_ts": 9120583.905837096, "preprocess_end_ts": 9120583.90618202, "forward_start_ts": 9120583.906192401, "forward_end_ts": 9120583.911191944, "timestamp": 1790102604.9446793, "activation_bytes_per_rank": 2097152, "recv_start_ts": 9120583.903329002, "recv_end_ts": 9120583.905698072, "send_start_ts": null, "send_end_ts": null} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/requests.jsonl b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/requests.jsonl deleted file mode 100644 index 0a6f40d0..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/requests.jsonl +++ /dev/null @@ -1,76 +0,0 @@ -{"request_id": "warmup-q0", "burst": "warmup", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120569.869602332, "finish_monotonic": 9120576.285742093, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "warmup-q1", "burst": "warmup", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120569.872307455, "finish_monotonic": 9120576.285754615, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "warmup-q2", "burst": "warmup", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120569.872824574, "finish_monotonic": 9120576.294476116, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "warmup-q3", "burst": "warmup", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120569.873199455, "finish_monotonic": 9120576.294771325, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q0", "burst": "b8-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120577.547693444, "finish_monotonic": 9120577.567003388, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q1", "burst": "b8-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120577.548223112, "finish_monotonic": 9120577.56724617, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q2", "burst": "b8-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120577.54847057, "finish_monotonic": 9120577.58936014, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q3", "burst": "b8-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120577.548694393, "finish_monotonic": 9120577.589365425, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q4", "burst": "b8-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9120577.548904004, "finish_monotonic": 9120577.598108912, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q5", "burst": "b8-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9120577.54911402, "finish_monotonic": 9120577.598117024, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q6", "burst": "b8-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9120577.549561707, "finish_monotonic": 9120577.60704437, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r0-q7", "burst": "b8-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9120577.54983673, "finish_monotonic": 9120577.607169196, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q0", "burst": "b8-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9120578.810612688, "finish_monotonic": 9120578.830455095, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q1", "burst": "b8-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9120578.811313707, "finish_monotonic": 9120578.830462607, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q2", "burst": "b8-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9120578.81171338, "finish_monotonic": 9120578.852622977, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q3", "burst": "b8-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9120578.812109131, "finish_monotonic": 9120578.852632107, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q4", "burst": "b8-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9120578.812452069, "finish_monotonic": 9120578.861199869, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q5", "burst": "b8-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9120578.812766358, "finish_monotonic": 9120578.861310275, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q6", "burst": "b8-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9120578.813098883, "finish_monotonic": 9120578.869919628, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r1-q7", "burst": "b8-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9120578.813434564, "finish_monotonic": 9120578.870209333, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q0", "burst": "b8-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9120580.07300282, "finish_monotonic": 9120580.0932383, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q1", "burst": "b8-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9120580.073717771, "finish_monotonic": 9120580.093249517, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q2", "burst": "b8-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9120580.074123846, "finish_monotonic": 9120580.119620783, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q3", "burst": "b8-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9120580.07445321, "finish_monotonic": 9120580.119723072, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q4", "burst": "b8-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9120580.074752178, "finish_monotonic": 9120580.129679734, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q5", "burst": "b8-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9120580.074970948, "finish_monotonic": 9120580.12968696, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q6", "burst": "b8-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9120580.075345183, "finish_monotonic": 9120580.139080467, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b8-r2-q7", "burst": "b8-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9120580.075559553, "finish_monotonic": 9120580.139085893, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q0", "burst": "b16-r0", "round": 0, "index": 0, "rank": 0, "submit_monotonic": 9120581.341815123, "finish_monotonic": 9120581.361618357, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q1", "burst": "b16-r0", "round": 0, "index": 1, "rank": 1, "submit_monotonic": 9120581.342365772, "finish_monotonic": 9120581.36186946, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q2", "burst": "b16-r0", "round": 0, "index": 2, "rank": 0, "submit_monotonic": 9120581.342633724, "finish_monotonic": 9120581.380279476, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q3", "burst": "b16-r0", "round": 0, "index": 3, "rank": 1, "submit_monotonic": 9120581.342859657, "finish_monotonic": 9120581.380283996, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q4", "burst": "b16-r0", "round": 0, "index": 4, "rank": 0, "submit_monotonic": 9120581.343072204, "finish_monotonic": 9120581.386880303, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q5", "burst": "b16-r0", "round": 0, "index": 5, "rank": 1, "submit_monotonic": 9120581.3432637, "finish_monotonic": 9120581.386887154, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q6", "burst": "b16-r0", "round": 0, "index": 6, "rank": 0, "submit_monotonic": 9120581.343454594, "finish_monotonic": 9120581.393417716, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q7", "burst": "b16-r0", "round": 0, "index": 7, "rank": 1, "submit_monotonic": 9120581.343649356, "finish_monotonic": 9120581.393422365, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q8", "burst": "b16-r0", "round": 0, "index": 8, "rank": 0, "submit_monotonic": 9120581.343870228, "finish_monotonic": 9120581.402437443, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q9", "burst": "b16-r0", "round": 0, "index": 9, "rank": 1, "submit_monotonic": 9120581.344084216, "finish_monotonic": 9120581.40244758, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q10", "burst": "b16-r0", "round": 0, "index": 10, "rank": 0, "submit_monotonic": 9120581.344304737, "finish_monotonic": 9120581.409558967, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q11", "burst": "b16-r0", "round": 0, "index": 11, "rank": 1, "submit_monotonic": 9120581.34456862, "finish_monotonic": 9120581.409563392, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q12", "burst": "b16-r0", "round": 0, "index": 12, "rank": 0, "submit_monotonic": 9120581.344810119, "finish_monotonic": 9120581.416050447, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q13", "burst": "b16-r0", "round": 0, "index": 13, "rank": 1, "submit_monotonic": 9120581.345005594, "finish_monotonic": 9120581.416054724, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q14", "burst": "b16-r0", "round": 0, "index": 14, "rank": 0, "submit_monotonic": 9120581.345214237, "finish_monotonic": 9120581.42277076, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r0-q15", "burst": "b16-r0", "round": 0, "index": 15, "rank": 1, "submit_monotonic": 9120581.345402837, "finish_monotonic": 9120581.422895985, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q0", "burst": "b16-r1", "round": 1, "index": 0, "rank": 0, "submit_monotonic": 9120582.575048696, "finish_monotonic": 9120582.593977828, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q1", "burst": "b16-r1", "round": 1, "index": 1, "rank": 1, "submit_monotonic": 9120582.575551683, "finish_monotonic": 9120582.593985632, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q2", "burst": "b16-r1", "round": 1, "index": 2, "rank": 0, "submit_monotonic": 9120582.575873053, "finish_monotonic": 9120582.616082978, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q3", "burst": "b16-r1", "round": 1, "index": 3, "rank": 1, "submit_monotonic": 9120582.57612324, "finish_monotonic": 9120582.616088783, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q4", "burst": "b16-r1", "round": 1, "index": 4, "rank": 0, "submit_monotonic": 9120582.576336274, "finish_monotonic": 9120582.624345195, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q5", "burst": "b16-r1", "round": 1, "index": 5, "rank": 1, "submit_monotonic": 9120582.576600255, "finish_monotonic": 9120582.624350388, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q6", "burst": "b16-r1", "round": 1, "index": 6, "rank": 0, "submit_monotonic": 9120582.576868795, "finish_monotonic": 9120582.6327847, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q7", "burst": "b16-r1", "round": 1, "index": 7, "rank": 1, "submit_monotonic": 9120582.577115236, "finish_monotonic": 9120582.63279038, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q8", "burst": "b16-r1", "round": 1, "index": 8, "rank": 0, "submit_monotonic": 9120582.57734158, "finish_monotonic": 9120582.641252786, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q9", "burst": "b16-r1", "round": 1, "index": 9, "rank": 1, "submit_monotonic": 9120582.577593436, "finish_monotonic": 9120582.64126444, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q10", "burst": "b16-r1", "round": 1, "index": 10, "rank": 0, "submit_monotonic": 9120582.577844236, "finish_monotonic": 9120582.649838036, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q11", "burst": "b16-r1", "round": 1, "index": 11, "rank": 1, "submit_monotonic": 9120582.578143867, "finish_monotonic": 9120582.649843188, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q12", "burst": "b16-r1", "round": 1, "index": 12, "rank": 0, "submit_monotonic": 9120582.578371843, "finish_monotonic": 9120582.658064105, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q13", "burst": "b16-r1", "round": 1, "index": 13, "rank": 1, "submit_monotonic": 9120582.578631695, "finish_monotonic": 9120582.658071209, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q14", "burst": "b16-r1", "round": 1, "index": 14, "rank": 0, "submit_monotonic": 9120582.578840362, "finish_monotonic": 9120582.667599283, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r1-q15", "burst": "b16-r1", "round": 1, "index": 15, "rank": 1, "submit_monotonic": 9120582.579079762, "finish_monotonic": 9120582.667743009, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q0", "burst": "b16-r2", "round": 2, "index": 0, "rank": 0, "submit_monotonic": 9120583.81993872, "finish_monotonic": 9120583.839589516, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q1", "burst": "b16-r2", "round": 2, "index": 1, "rank": 1, "submit_monotonic": 9120583.82045228, "finish_monotonic": 9120583.839597132, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q2", "burst": "b16-r2", "round": 2, "index": 2, "rank": 0, "submit_monotonic": 9120583.820687912, "finish_monotonic": 9120583.86100596, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q3", "burst": "b16-r2", "round": 2, "index": 3, "rank": 1, "submit_monotonic": 9120583.820890976, "finish_monotonic": 9120583.861143965, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q4", "burst": "b16-r2", "round": 2, "index": 4, "rank": 0, "submit_monotonic": 9120583.821091052, "finish_monotonic": 9120583.869980576, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q5", "burst": "b16-r2", "round": 2, "index": 5, "rank": 1, "submit_monotonic": 9120583.821283503, "finish_monotonic": 9120583.869986843, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q6", "burst": "b16-r2", "round": 2, "index": 6, "rank": 0, "submit_monotonic": 9120583.82154948, "finish_monotonic": 9120583.878445651, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q7", "burst": "b16-r2", "round": 2, "index": 7, "rank": 1, "submit_monotonic": 9120583.82179376, "finish_monotonic": 9120583.878554093, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q8", "burst": "b16-r2", "round": 2, "index": 8, "rank": 0, "submit_monotonic": 9120583.822033968, "finish_monotonic": 9120583.886686856, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q9", "burst": "b16-r2", "round": 2, "index": 9, "rank": 1, "submit_monotonic": 9120583.822236164, "finish_monotonic": 9120583.886816649, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q10", "burst": "b16-r2", "round": 2, "index": 10, "rank": 0, "submit_monotonic": 9120583.822420392, "finish_monotonic": 9120583.8952481, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q11", "burst": "b16-r2", "round": 2, "index": 11, "rank": 1, "submit_monotonic": 9120583.82260614, "finish_monotonic": 9120583.895373551, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q12", "burst": "b16-r2", "round": 2, "index": 12, "rank": 0, "submit_monotonic": 9120583.822795508, "finish_monotonic": 9120583.903785357, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q13", "burst": "b16-r2", "round": 2, "index": 13, "rank": 1, "submit_monotonic": 9120583.82299076, "finish_monotonic": 9120583.903789992, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q14", "burst": "b16-r2", "round": 2, "index": 14, "rank": 0, "submit_monotonic": 9120583.82317668, "finish_monotonic": 9120583.912484672, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} -{"request_id": "b16-r2-q15", "burst": "b16-r2", "round": 2, "index": 15, "rank": 1, "submit_monotonic": 9120583.823371433, "finish_monotonic": 9120583.912491629, "num_prompt_tokens": 256, "num_output_tokens": 1, "finish_reason": "length"} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/summary.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/summary.json deleted file mode 100644 index 637f420b..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/runs/moe/summary.json +++ /dev/null @@ -1,186 +0,0 @@ -{ - "model_config": "/data/ycfeng/Frontier/.worktrees/stage-admission-ordering/data/config/models/Qwen3-30B-A3B-tiny.json", - "num_gpu_blocks": 600666, - "block_size": 16, - "engine_args": { - "model": "/tmp/stage_admission_pp/sa-pp-20260923b/runs/moe/model", - "served_model_name": null, - "tokenizer": null, - "hf_config_path": null, - "runner": "auto", - "convert": "auto", - "task": null, - "skip_tokenizer_init": true, - "enable_prompt_embeds": false, - "tokenizer_mode": "auto", - "trust_remote_code": false, - "allowed_local_media_path": "", - "download_dir": null, - "safetensors_load_strategy": "lazy", - "load_format": "dummy", - "config_format": "auto", - "dtype": "bfloat16", - "kv_cache_dtype": "auto", - "seed": 0, - "max_model_len": 512, - "distributed_executor_backend": null, - "pipeline_parallel_size": 2, - "tensor_parallel_size": 1, - "decode_context_parallel_size": 1, - "data_parallel_size": 2, - "data_parallel_rank": null, - "data_parallel_start_rank": null, - "data_parallel_size_local": null, - "data_parallel_address": null, - "data_parallel_rpc_port": null, - "data_parallel_hybrid_lb": false, - "data_parallel_backend": "mp", - "enable_expert_parallel": true, - "enable_eplb": false, - "num_redundant_experts": 0, - "eplb_window_size": 1000, - "eplb_step_interval": 3000, - "eplb_log_balancedness": false, - "max_parallel_loading_workers": null, - "block_size": 16, - "enable_prefix_caching": false, - "prefix_caching_hash_algo": "sha256", - "disable_sliding_window": false, - "disable_cascade_attn": false, - "swap_space": 4, - "cpu_offload_gb": 0, - "gpu_memory_utilization": 0.5, - "kv_cache_memory_bytes": null, - "max_num_batched_tokens": 256, - "max_num_partial_prefills": 1, - "max_long_partial_prefills": 1, - "long_prefill_token_threshold": 0, - "max_num_seqs": 4, - "max_logprobs": 20, - "disable_log_stats": true, - "revision": null, - "code_revision": null, - "rope_theta": null, - "hf_token": null, - "tokenizer_revision": null, - "quantization": null, - "enforce_eager": true, - "max_seq_len_to_capture": 8192, - "disable_custom_all_reduce": false, - "interleave_mm_strings": false, - "mm_processor_kwargs": null, - "disable_mm_preprocessor_cache": false, - "mm_processor_cache_gb": 4, - "mm_encoder_tp_mode": "weights", - "io_processor_plugin": null, - "skip_mm_profiling": false, - "enable_lora": false, - "enable_lora_bias": false, - "max_loras": 1, - "max_lora_rank": 16, - "default_mm_loras": null, - "fully_sharded_loras": false, - "max_cpu_loras": null, - "lora_dtype": "auto", - "lora_extra_vocab_size": 256, - "ray_workers_use_nsight": false, - "num_gpu_blocks_override": null, - "num_lookahead_slots": 0, - "ignore_patterns": null, - "preemption_mode": null, - "scheduler_delay_factor": 0.0, - "enable_chunked_prefill": true, - "disable_chunked_mm_input": false, - "disable_hybrid_kv_cache_manager": false, - "guided_decoding_backend": "auto", - "guided_decoding_disable_fallback": false, - "guided_decoding_disable_any_whitespace": false, - "guided_decoding_disable_additional_properties": false, - "logits_processor_pattern": null, - "speculative_config": null, - "show_hidden_metrics_for_version": null, - "otlp_traces_endpoint": null, - "collect_detailed_traces": null, - "disable_async_output_proc": false, - "scheduling_policy": "fcfs", - "scheduler_cls": "vllm.v1.core.sched.scheduler.Scheduler", - "override_pooler_config": null, - "worker_cls": "auto", - "worker_extension_cls": "", - "kv_transfer_config": null, - "kv_events_config": null, - "generation_config": "auto", - "enable_sleep_mode": false, - "model_impl": "auto", - "override_attention_dtype": null, - "calculate_kv_scales": false, - "mamba_cache_dtype": "auto", - "mamba_ssm_cache_dtype": "auto", - "reasoning_parser": "", - "use_tqdm_on_load": true, - "pt_load_map_location": "cpu", - "enable_multimodal_encoder_data_parallel": false, - "logits_processors": null, - "async_scheduling": false, - "kv_sharing_fast_prefill": false, - "enable_log_requests": false - }, - "rounds": [ - { - "label": "warmup", - "round": 0, - "num_requests": 4, - "wall_minus_monotonic_before": 1780982021.0330575, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.2527571953833103 - }, - { - "label": "b8-r0", - "round": 0, - "num_requests": 8, - "wall_minus_monotonic_before": 1780982021.0330584, - "wall_minus_monotonic_after": 1780982021.0330582, - "idle_wait_s": 1.203191703185439 - }, - { - "label": "b8-r1", - "round": 1, - "num_requests": 8, - "wall_minus_monotonic_before": 1780982021.0330582, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.2025362998247147 - }, - { - "label": "b8-r2", - "round": 2, - "num_requests": 8, - "wall_minus_monotonic_before": 1780982021.0330584, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.202582010999322 - }, - { - "label": "b16-r0", - "round": 0, - "num_requests": 16, - "wall_minus_monotonic_before": 1780982021.0330582, - "wall_minus_monotonic_after": 1780982021.0330586, - "idle_wait_s": 1.1520026791840792 - }, - { - "label": "b16-r1", - "round": 1, - "num_requests": 16, - "wall_minus_monotonic_before": 1780982021.0330584, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.1520404387265444 - }, - { - "label": "b16-r2", - "round": 2, - "num_requests": 16, - "wall_minus_monotonic_before": 1780982021.0330582, - "wall_minus_monotonic_after": 1780982021.0330584, - "idle_wait_s": 1.152070851996541 - } - ] -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/vllm_import.txt b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/vllm_import.txt deleted file mode 100644 index cd9a481f..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/vllm_import.txt +++ /dev/null @@ -1 +0,0 @@ -VLLM_IMPORT 0.10.2 /tmp/stage_admission_pp/sa-pp-20260923b/overlay/vllm/__init__.py /tmp/stage_admission_pp/sa-pp-20260923b/overlay/vllm/v1/frontier_trace.py diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/worker_env.json b/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/worker_env.json deleted file mode 100644 index 00b7f124..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b/worker_env.json +++ /dev/null @@ -1 +0,0 @@ -{"python": "3.12.11", "torch": "2.8.0+cu128", "cuda_available": true, "device_count": 4, "devices": ["NVIDIA H800", "NVIDIA H800", "NVIDIA H800", "NVIDIA H800"]} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/design.md b/task_memory/task_2026-09-22_stage_admission_ordering/design.md deleted file mode 100644 index 300af11d..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/design.md +++ /dev/null @@ -1,330 +0,0 @@ -# Stage admission ordering under pipeline parallelism — Design - -## Modification History - -| Date | Change | -| --- | --- | -| 2026-09-23 | R-10: added the round-2 notes on the capacity-1 contract (R2-12), FIFO meaning on shared-lane contexts (R2-13) and the deferred EP-only queue variant. | -| 2026-09-23 | Applied the round-1 plan review (`review.md`). Changes: the admission-loop anchor now points to the `MONOLITHIC`/`PREFILL` path; the drain condition is stated as a queued-ticket arrangement, not a shape; the shape table is marked author-reported until P0; added where queued EP waves exist; `remove(ticket)` made explicit; option A's stall trace labelled an unverified hypothesis; the capacity-1 section rewritten as a caller-level condition; the queue bound narrowed; added the mixed-phase scope boundary; the dense "lanes serialized" label withdrawn as unmeasured and replaced by the admission sequence read from source. | -| 2026-09-22 | Created: defect restated from source on `origin/main` `1f694f7`, what the FIFO guarantees today, four options, recommended rule with its invariants, fidelity expectation. For review before implementation. | - -All line references are to `origin/main` at `1f694f7`, checked out in -`/data/ycfeng/Frontier/.worktrees/stage-admission-ordering`. - -## The defect, restated from source - -One `StageExecutionContext` owns each physical `(replica, stage)`. All -attention-DP lanes of that stage share it. Each lane has its own -`ReplicaStageScheduler` with its own batch heap and its own `_is_busy` flag, so -**a lane consumes at most one ticket at a time**. - -Admission is a two-step handshake: - -1. `ReplicaStageScheduler.add_batch` (`replica_stage_schduler.py:145-162`) mints - a `StageAdmissionTicket` at batch **arrival** through - `StageExecutionContext.enqueue_full_stage`, which appends it to the shared - `_ready_fifo` (`stage_execution_context.py:188`). -2. `pop_batch_if_not_busy` (`:286-359`) returns at once when its lane is busy; - otherwise it takes the lane's own heap head and, unless the context already - `owns` that ticket, asks `try_acquire` (`:329`). `try_acquire` (`:322-343`) - refuses when an EP wave is active, when active full-stage owners already - fill `full_stage_capacity`, when the forward group is sealed, and finally - when the ticket **is not the FIFO head**: - - ```python - if not self._ready_fifo or self._ready_fifo[0] != ticket: - return False - self._ready_fifo.popleft() - ``` - -`full_stage_capacity` is `attn_dp` for `MONOLITHIC`, `PREFILL` and `DECODE` -and 1 otherwise (`stage_contexts.py:59-81`), so up to one ticket per lane may -be active at once: that is how the lanes of one forward co-own the stage. - -`BaseReplicaScheduler.on_schedule` admits batches while -`num_running_batches < num_stages`. The co-location reproduction runs the -`MONOLITHIC`/`PREFILL` branch (`base_replica_scheduler.py:1037-1054`, loop at -`:1039`); the unified `DECODE` branch (`:893`) has the same bound. At -`num_pipeline_stages > 1` a lane can therefore hold **several queued tickets** -at one stage while consuming one. - -The drain needs a specific arrangement, not merely `attn_dp > 1` and `PP > 1`: - -- a lane's queued ticket is at the FIFO head while that lane is busy with an - active ticket of an open (unsealed) forward group, and -- another lane with queued work presents a ticket behind it and is refused, - while the busy lane waits in a sync room for that refused lane. - -With too little queued work the same shape completes (the 3-request row -below). - -### Observed state at the drain (MoE `attn_dp=2, moe_ep=2, PP=2`, 4 requests) - -Author-run on 2026-09-22 with the session scripts. P0 republishes it from the -published case inputs (`plan.md` §4, group R0) and stores the state report. - -Both lanes admitted two batches each. Lane 1 scheduled first. - -| Where | State | -| --- | --- | -| Stage `(0,0)` context | `capacity=2 sealed=False group=0`; active `{seq0: batch 0 (lane 1)}`; FIFO `[seq1: batch 1 (lane 1), seq2: batch 2 (lane 0), seq3: batch 3 (lane 0)]` | -| Lane 1 stage 0 | `busy=True`, heap `[(batch 1, global_id 3, seq1)]` | -| Lane 0 stage 0 | `busy=False`, heap `[(batch 2, global_id 0, seq2), (batch 3, global_id 2, seq3)]` | -| Prefill sync room step 0, layer 0, `pre_moe` | `lanes_present=[1]`, waiting for lane 0 | -| Event queue | empty | - -The wait is circular: - -- Lane 1 holds `seq0`, has bound forward group 0 and sits in the sync room - until lane 0 joins. -- Lane 0 presents `seq2`; the FIFO head is `seq1`, which belongs to lane 1, so - `try_acquire` refuses. -- Lane 1 cannot consume `seq1` because it is busy. -- The room does not stand in an idle batch for lane 0: `_can_supply_idle_lane` - (`sync_entry.py:8-14`) returns `False` when the lane has queued work and the - group is unsealed, precisely because such a lane is expected to join. - -The last point is the crispest statement of the defect: **the sync room's -"this lane can still join" predicate and the context's `try_acquire` disagree -about the same lane.** The room is right about the model; the context's FIFO -position test is the part that has no counterpart in the system being -simulated. - -Two orderings also disagree with each other. The lane heap orders by -`global_id = counter * lane_count + lane_id` (`batch_ids.py:19`), which puts -lane 0's batch 2 (`global_id 0`) ahead of lane 1's batch 0 (`global_id 1`), -while the ticket FIFO orders by arrival, which puts lane 1 first. Within one -lane the two agree; across lanes they need not. - -### Why dense completes and MoE does not, and why PP=1 is not expected to drain - -- Dense never calls `bind_forward_group` (the call at - `replica_stage_schduler.py:347-356` is MoE-only), so it never seals and has no - sync room. A refused lane is re-woken at the next release - (`batch_stage_end_event.py:139-158`, `stage_wakeup.py:8-43`), so dense - finishes, but with lost overlap. From source, with every request at `t=0`: - the lane that schedules first mints two tickets before the other lane mints - any; the other lane is then refused while the first lane's first batch holds - stage 0, although capacity is free, and is admitted only at that release. - The first draft called this "lanes serialized". That was not measured: the - earlier runs checked completion only. P0 measures the loss with the `plan.md` §4.5 - ledger metric. The loss is the softer form of the same defect. -- At `num_pipeline_stages = 1` a lane admits its next batch only after the - previous one leaves the only stage, so it never holds a queued ticket while - busy, and the first bullet of the drain arrangement cannot form. - -Author-reported shapes (2026-09-22; default `astra_sim_analytical` backend, -Poisson `qps=1e6`, prefill 16 / decode 3; logs -`/data/ycfeng/tmp/w10_repro/case_*.log`). They remain author-reported -evidence until P0 reruns them as group R0 from the published inputs. - -| Shape (origin/main, fresh process each) | Requests | Result | -| --- | --- | --- | -| MoE `attn_dp=2, moe_ep=2, PP=2` | 3 | completes | -| MoE `attn_dp=2, moe_ep=2, PP=2` | 4, 6 | **drained** | -| MoE `attn_dp=4, moe_ep=4, PP=2` | 8 | **drained** | -| MoE `attn_dp=2, moe_ep=2, PP=1` | 6, 12 | completes | -| MoE `attn_dp=4, moe_ep=4, PP=1` | 8, 12 | completes | -| MoE `attn_dp=1, PP=2` / `PP=3` | 6 | completes | -| Dense `attn_dp=2, PP=2`, `attn_dp=4, PP=2` | 6, 8 | completes (overlap not measured) | -| Dense `attn_dp=2, PP=1`, `attn_dp=4, PP=1`, `attn_dp=1, PP=2` | 6, 8, 6 | completes | -| MoE `attn_dp=2, moe_ep=2, PP=3` | 6 | rejected at construction by the Replica-pod node-size rule (6 devices against node size 4; parent task W9-02) | - -In this table the drain first appears at 4 requests, where both lanes first -hold more than one batch. - -## What the FIFO position test guarantees today - -Read from the unit tests that pin it: - -| Test | Guarantee | -| --- | --- | -| `test_admission_fifo_cannot_skip_an_earlier_ready_wave` | Two queued EP waves are admitted in queue order. | -| `test_ep_wave_owns_stage_before_dense_can_start` | A full-stage ticket queued after an EP wave waits for it. | -| `test_started_group_blocks_new_lane_through_ep_restore_and_partial_release` | A lane arriving after a group started waits until every owner releases (seal), not FIFO. | -| `test_next_group_queue_does_not_block_current_group_idle_participation` | A lane whose queued work is refused by the **seal** is stood in as idle. | -| Comment at `replica_stage_schduler.py:301-303` | The lane must admit the same heap head it inspected (a bypass fix unrelated to cross-lane order). | - -None of them require that two full-stage tickets from **different lanes** be -admitted in arrival order. That ordering is the one piece with no stated -purpose, and it is the one that fails. This is a reading of the tests, not a -test result; C4 in the plan settles it. - -### Where queued EP waves exist - -`enqueue_ep_wave` has one caller, the `DECODE_FFN` M2N group path -(`round_robin_cluster_scheduler.py:1052-1057`). On `MONOLITHIC`, `PREFILL` and -`DECODE` contexts, `EP_WAVE` appears only as an active-scope transition of -owners already admitted (`transition_active_scope`, -`replace_full_stage_owners_with_ep_wave`, -`replace_ep_wave_with_full_stage_owners`, driven by -`forward_step_admission.py`); it never enters the FIFO there. So the FIFO of a -shared-lane context holds only full-stage tickets, and a mixed FIFO of -full-stage tickets and EP waves exists only on `DECODE_FFN` contexts, whose -capacity is 1. - -## Options - -| Option | Rule | Verdict | -| --- | --- | --- | -| A. Skip busy owners | The ticket carries its lane; an earlier queued full-stage ticket blocks admission only while its lane holds no active ticket. | **Rejected on design grounds.** It adds lane identity to tickets and makes one lane's admission depend on a peer lane's *acquisition*. Acquisition emits no retry; only `BatchStageEndEvent` wakes siblings (`batch_stage_end_event.py:148-158`), so A would need a new wake path on acquisition. The first draft also sketched a specific second-cohort stall. That trace is an **unverified hypothesis**. It did not account for the releasing lane's own retry, which is emitted before sibling retries (`:139-146`). It did not account for retries from several same-time releases, and the reviewer notes that prefill participants can release at one shared predicted time. And the DES orders equal-time events by `(time, id, event_type)` (`base_event.py:63-64`, `simulator.py:1268`), not by `BaseEvent.__lt__` (`:66-70`), which compares type before id. B does not depend on that trace, so it is not pursued. | -| B. Order only exclusive operations | A full-stage ticket is refused only when an **EP wave** is queued ahead of it; earlier full-stage tickets never block it. EP waves keep the strict head rule. Capacity, seal and EP-active checks unchanged. | **Recommended (adopted as D-1).** No new field, no interface change, one predicate. Every remaining refusal (capacity, seal, EP active, EP wave ahead) is cleared by a release, which already wakes siblings, so no new wait state exists. The room predicate and the context now agree. | -| C. Mint the ticket at the admission attempt instead of arrival | A busy lane never holds a queued ticket. | Rejected. Changes ordering semantics for every path and breaks the stale-drop logic, which relies on the ticket attached at arrival (`_discard_stale_ticket`, `_drop_queued_lanes_for_ticket`, sibling tickets in `DECODE_FFN`). | -| D. Stand in an idle lane when a lane is blocked by admission order | Change `_can_supply_idle_lane`. | Rejected. Lane 0 has real work for this forward; modelling it as absent skips that work into a later forward. An error-suppressing fallback in the sense of the working gates. | - -## Recommended rule - -In `StageExecutionContext.try_acquire`, replace the head test for full-stage -tickets with: - -> A full-stage ticket may be admitted when no EP wave is queued ahead of it. -> An EP wave may be admitted only as the FIFO head. - -Sketch (final wording at implementation; the existing scope, capacity and seal -checks above it are unchanged, and the EP-wave line is today's line): - -```python -if ticket.scope == EP_WAVE: - if not self._ready_fifo or self._ready_fifo[0] != ticket: - return False -else: - for queued in self._ready_fifo: - if queued == ticket: - break - if queued.scope == EP_WAVE: - return False -self._ready_fifo.remove(ticket) -``` - -- The admitted ticket is removed with `remove(ticket)`, not `popleft()`: once a - non-head ticket can be admitted, `popleft()` would dequeue a different - ticket. `cancel` already removes a queued ticket the same way - (`stage_execution_context.py:456`). -- The single caller reaches `try_acquire` only with a queued ticket: it checks - `owns` first (`replica_stage_schduler.py:328`), and `_validate_ticket` - rejects a ticket that is neither queued nor active. No branch is added for - other states. -- The FIFO stays one deque so that an EP wave still sees every ticket ahead of - it. -- Queue length: on the shared-lane contexts this change targets, the FIFO holds - at most `attn_dp × num_pipeline_stages` tickets, because each lane runs at - most `num_pipeline_stages` batches (`base_replica_scheduler.py:893,1039`). - `DECODE_FFN` contexts are fed by M2N groups and have no such bound; the scan - there is linear in the queue, as `cancel`'s `remove` already is. No index or - second queue is added. -- On shared-lane contexts no EP wave is ever queued (previous section), so the - loop only walks to the ticket; its EP clause acts on `DECODE_FFN`. - -Files touched: `stage_execution_context.py` (rule and the two docstrings that -describe admission as "FIFO-head"), no other source file. `_can_supply_idle_lane` -is left as is; it becomes consistent rather than changed. - -### Invariants after the change - -1. At most one active full-stage ticket per lane per stage (unchanged; from - `_is_busy`). -2. Within one lane, batches enter a stage in heap order (unchanged; the lane - presents only its heap head). -3. Exclusive operations (EP waves) are admitted in queue order and never - overtaken by full-stage work queued behind them; an EP wave still waits for - every earlier queued ticket and every active owner (unchanged; pinned by the - two EP tests and extended by P2(a)). -4. A lane with queued work is admitted at its next attempt when capacity is - free, the group is unsealed, no EP wave is active and no EP wave is queued - ahead of its ticket (new; this is the property the sync room already - assumes). -5. Every refusal is cleared by a release event, which wakes idle non-empty - sibling lanes (unchanged mechanism, now sufficient). - -Invariant 4 removes the admission-order refusal. It is not a whole-run -liveness proof; see the scope boundary below. - -### Where behaviour is expected to stay unchanged, and why - -The rule is not a no-op at the context API. With an idle capacity-1 context -and FIFO `[full0, full1]`, `try_acquire(full1)` is refused today and admitted -under B. Capacity prevents two simultaneous owners but does not preserve -arrival order when nothing is active. Unchanged behaviour is therefore a claim -about the callers, under this condition: - -> For a scheduler that presents only its heap head, B and today's rule make the -> same decision whenever no full-stage ticket of **another** scheduler is -> queued ahead of the presented ticket, and each scheduler's heap order agrees -> with FIFO order among its own full-stage tickets. Then only EP waves can be -> ahead of the presented ticket, and both rules refuse exactly when something -> is ahead. - -| Context | Why the condition is expected to hold | Evidence planned | -| --- | --- | --- | -| `DECODE_FFN` (capacity 1) | Every `DenseFFNBatchGroup` gets `global_id = _batch_group_creation_counter` (`round_robin_cluster_scheduler.py:1097,1118`) and one full-stage ticket (`:1138`), and is queued on the one full-stage scheduler of its replica (`:1100`), so its heap and FIFO both follow the group counter. EP child batches hold no full-stage ticket; the group shares one `EP_WAVE` ticket (`:1052-1057`). Shared EP sibling tickets are not multiple full-stage owners. | P2(a′) control with two successive dense FFN groups and a neighbouring EP group through the real full-stage scheduler; G6 byte comparison. | -| `DECODE_ATTN` (capacity 1) | `attn_dp=1` with `replica_local_id=None` (AGENTS.md): one scheduler per stage, so no other scheduler's ticket can be ahead. | G6 byte comparison. | -| Shared-lane contexts, `PP = 1` | A lane never holds a queued ticket while busy. A cross-lane inversion needs one release to wake two or more idle siblings whose tickets are queued in the opposite order to the wake order: wake-ups follow lane-key order (`stage_wakeup.py:30-32`), and today's rule refuses the first sibling woken. The releasing lane has no queued ticket at `PP=1` and is excluded, so this needs `attn_dp ≥ 3`. | G1, G3 and G4 `PP=1` byte comparison. `attn_dp=2` is expected unchanged. `attn_dp=4` is expected, not guaranteed, unchanged, and a difference stops the work for diagnosis (plan P3). | -| `attn_dp = 1`, any PP | One lane, so FIFO order equals heap order. | G5 byte comparison. | - -No capacity-1 or `PP=1` special case is added: no supported caller has been -shown to need arbitrary cross-lane full-stage FIFO order. An unexpected -difference in any of these classes stops the work and is reported. - -### Round-2 notes (R-10) - -- Capacity-1 contract (R2-12). At the context API this is a contract change, - not an unaffected path: an idle capacity-1 context used to admit full-stage - tickets in enqueue order and now admits whichever one its lane stage - scheduler presents, unless an EP wave is queued ahead. Order among - full-stage tickets now comes from the callers. Measured on the callers: - every capacity-1 context in the matrix is byte-identical. That covers the - 10 PD-AF release recipes (`DECODE_ATTN`, `DECODE_FFN`, `PREFILL`) and the 4 - PD-AF recipes with `PREFILL_PP=2` (G11), offline and online. -- FIFO meaning on shared-lane contexts (R2-13). `enqueue_ep_wave` has one - caller, the `DECODE_FFN` M2N path (`round_robin_cluster_scheduler.py:1052`). - So the FIFO of a `MONOLITHIC`, `PREFILL` or unified `DECODE` context only - ever holds full-stage tickets, and under B it no longer orders admission - there. `queued_tickets` and `admission_seq` record enqueue order only, and - the class docstring says so. -- Variant considered and deferred: queue only EP waves, and keep queued - full-stage tickets as an unordered set. The data structure would then - match the rule. But it changes `queued_tickets`, `is_queued` and `cancel`, - and the drain diagnostics that read FIFO order, with no change in - behaviour. It is not pursued unless a caller needs the queue to express - admission order. - -## Scope boundary: mixed-phase forwards - -This branch is based on `main`, where prefill and decode source lanes still -enter separate synchronization paths. The shared forward across mixed prefill -and decode lanes is PR 35 W3 (`65ed8a7`), not on `main`. Fixing admission does -not fix that. A shape that deadlocked at admission may, once admitted, reach a -mixed-phase cohort and fail another way. Such a failure is recorded and -diagnosed separately; it is not repaired by widening this one-file change. - -Consequences for verification: - -- The C1 witnesses are phase-controlled. All requests arrive at `t=0` with - equal prompt lengths, and the primary group is prefill-only - (`decode_tokens=1`). A `MONOLITHIC` request of that shape completes at the - prefill boundary, which grants its one decode token - (`request.py:1286-1293,1379-1384`), so no decode batch forms. -- Composition with PR 35 is validated in the parent task after this branch is - merged forward, before Step 9 is declared unblocked. - -## Fidelity expectation, stated before measuring - -| Scenario class | Acceptance path (plan §4.2) | Expected after the change | -| --- | --- | --- | -| Declared `PP = 1` scenarios (release examples, synthetic `PP=1` cells) | U | Byte-identical metrics files. `attn_dp=4` cells carry the caveat in the table above. | -| PD-AF `DECODE_ATTN` / `DECODE_FFN` (capacity 1) in the declared recipes | U | Byte-identical, for the caller-level reason above. | -| `attn_dp = 1`, any PP | U | Byte-identical. | -| MoE `attn_dp > 1`, `PP > 1`, cells P0 classifies as admission deadlock | L | Completes, with request and token conservation. | -| MoE `attn_dp > 1`, `PP > 1`, cells that complete on base | T | Byte-identical, or a difference explained with the stage ledger. | -| Dense `attn_dp > 1`, `PP > 1` | T | Completes before and after. A lane refused only by FIFO position is admitted at once, so lane overlap increases and makespan and per-request latencies may **change**. This is the same defect's softer symptom and was accepted as a fidelity fix (D-2). | - -Any outcome outside its row stops the work. - -## What this is not - -- Not a change to `full_stage_capacity`, the seal, the EP wave protocol, the - sync rooms, or the wake-up helper. -- Not a new flag or configuration field. -- Not a fix for mixed-phase forwards on `main` (PR 35 W3). -- Not the Step 9 report key (D9-2 in the parent plan); that design resumes once - this lands and the `attn_dp=2, PP=2` shape runs. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/base_negative_controls.log b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/base_negative_controls.log deleted file mode 100644 index c44f32a2..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/base_negative_controls.log +++ /dev/null @@ -1,59 +0,0 @@ -F.F.FFFFF [100%] -=================================== FAILURES =================================== -__ test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave __ -tests/unit/test_stage_execution_context.py:111: in test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave - assert context.try_acquire(full1) is True -E AssertionError: assert False is True -E + where False = try_acquire(StageAdmissionTicket(replica_id=0, stage_id=0, admission_seq=1, operation_id='full1', scope='FULL_STAGE_WORLD', participant_ep_ids=())) -E + where try_acquire = .try_acquire -_____ test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket _____ -tests/unit/test_stage_execution_context.py:141: in test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket - assert context.try_acquire(full1) is True -E AssertionError: assert False is True -E + where False = try_acquire(StageAdmissionTicket(replica_id=0, stage_id=0, admission_seq=1, operation_id='full1', scope='FULL_STAGE_WORLD', participant_ep_ids=())) -E + where try_acquire = .try_acquire -________ test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0] ________ -tests/unit/test_shared_forward_group_admission.py:71: in test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket - assert stages[other_lane].pop_batch_if_not_busy() is other_now -E assert None is batch id = 10\ndecode_attn_original_replica_id=None, decode_attn_original_replica_local_id=None\nnum req = 1, [8]\n------...8\nnum_prefill_tokens=16\nnum_decode_tokens=4\nnum_processed_tokens=0\ncurrent_decode_token_index=1\ncompleted_layer_count=0 -E + where None = pop_batch_if_not_busy() -E + where pop_batch_if_not_busy = .pop_batch_if_not_busy -________ test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[1] ________ -tests/unit/test_shared_forward_group_admission.py:71: in test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket - assert stages[other_lane].pop_batch_if_not_busy() is other_now -E assert None is batch id = 14\ndecode_attn_original_replica_id=None, decode_attn_original_replica_local_id=None\nnum req = 1, [12]\n-----...2\nnum_prefill_tokens=16\nnum_decode_tokens=4\nnum_processed_tokens=0\ncurrent_decode_token_index=1\ncompleted_layer_count=0 -E + where None = pop_batch_if_not_busy() -E + where pop_batch_if_not_busy = .pop_batch_if_not_busy -_____ test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4-expected0] ______ -tests/integration/test_stage_admission_pipeline_lanes.py:41: in test_moe_lanes_complete_every_request - metrics_dir = run_case(tmp_path, case_id) -tests/integration/test_stage_admission_pipeline_lanes.py:31: in run_case - assert outcome["outcome"] == SUCCESS, outcome -E AssertionError: {'exception': 'Sequential simulation ended with non-empty scheduler state', 'outcome': 'admission_deadlock', 'simulation_time': 0.007, 'wall_end': 1790101999.9712346, ...} -E assert 'admission_deadlock' == 'success' -E -E - success -E + admission_deadlock -_____ test_moe_lanes_complete_every_request[G3a-moe-dp4-pp2-n8-expected1] ______ -tests/integration/test_stage_admission_pipeline_lanes.py:41: in test_moe_lanes_complete_every_request - metrics_dir = run_case(tmp_path, case_id) -tests/integration/test_stage_admission_pipeline_lanes.py:31: in run_case - assert outcome["outcome"] == SUCCESS, outcome -E AssertionError: {'exception': 'Sequential simulation ended with non-empty scheduler state', 'outcome': 'admission_deadlock', 'simulation_time': 0.007, 'wall_end': 1790102001.4043047, ...} -E assert 'admission_deadlock' == 'success' -E -E - success -E + admission_deadlock -_______________ test_dense_lanes_start_in_the_same_first_forward _______________ -tests/integration/test_stage_admission_pipeline_lanes.py:60: in test_dense_lanes_start_in_the_same_first_forward - assert first_start[0] == first_start[1] -E assert 0.05 == 0.0 -=========================== short test summary info ============================ -FAILED tests/unit/test_stage_execution_context.py::test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave -FAILED tests/unit/test_stage_execution_context.py::test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket -FAILED tests/unit/test_shared_forward_group_admission.py::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0] -FAILED tests/unit/test_shared_forward_group_admission.py::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[1] -FAILED tests/integration/test_stage_admission_pipeline_lanes.py::test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4-expected0] -FAILED tests/integration/test_stage_admission_pipeline_lanes.py::test_moe_lanes_complete_every_request[G3a-moe-dp4-pp2-n8-expected1] -FAILED tests/integration/test_stage_admission_pipeline_lanes.py::test_dense_lanes_start_in_the_same_first_forward -7 failed, 2 passed in 6.74s diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py deleted file mode 100644 index f7102aaf..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/decompose_co_execution.py +++ /dev/null @@ -1,76 +0,0 @@ -"""Split vLLM stage-0 non-overlap into start and end offsets of paired forwards. - -Pairs are the i-th lane-0 and i-th lane-1 stage-0 forwards of a round; the -output states whether this equals the M3 pairing of ``compare_lanes``. For an -overlapping pair, union minus overlap equals |start difference| + |end -difference|, so the non-overlap splits into a start part and an end part. A -disjoint pair has no such split; it is counted and its non-overlap is -reported whole. - -Two counterfactual co-execution fractions are reported: - -* ``M5_equal_durations``: every lane-1 forward keeps its start and takes the - duration of its lane-0 partner, as with the dummy predictor's equal - durations. -* ``M5_barrier_aligned``: both forwards of a pair start at the later of the - two starts and keep their own ends. vLLM 0.10.2 without CUDA graphs runs - the per-forward DP metadata all-reduce inside ``set_forward_context``, after - ``forward_start_ts``, so neither rank computes before the later one arrives. - The traces carry no timestamp after that exchange, so this value is derived - from where the all-reduce sits, not measured. It is ``None`` when a round - has a disjoint pair. - -Usage: python decompose_co_execution.py [ ...] -""" -import json -import statistics -import sys -from pathlib import Path - -from tests.comparison.stage_admission_pp.compare_lanes import lane_metrics, vllm_forwards -from tests.e2e.stage_admission_matrix import interval_overlap - -run_dir = Path(sys.argv[1]) -summary = {} -for model in sys.argv[2:]: - runs = vllm_forwards(run_dir / "runs" / model) - for (burst, round_index), run in sorted(runs.items()): - stage0 = [f for f in run["forwards"] if f["stage"] == 0] - lanes = {lane: sorted((f for f in stage0 if f["lane"] == lane), key=lambda f: f["start"]) for lane in (0, 1)} - pairs = list(zip(lanes[0], lanes[1])) - start_part = end_part = disjoint_part = 0.0 - disjoint = 0 - equal_duration, barrier_aligned = [], [] - for first, second in pairs: - later_start = max(first["start"], second["start"]) - if min(first["end"], second["end"]) > later_start: - start_part += abs(first["start"] - second["start"]) - end_part += abs(first["end"] - second["end"]) - else: - disjoint += 1 - disjoint_part += (first["end"] - first["start"]) + (second["end"] - second["start"]) - equal_duration.append((first["start"], first["end"], 0)) - equal_duration.append((second["start"], second["start"] + first["end"] - first["start"], 1)) - barrier_aligned.append((later_start, first["end"], 0)) - barrier_aligned.append((later_start, second["end"], 1)) - observed = interval_overlap([(f["start"], f["end"], f["lane"]) for f in stage0]) - equal = interval_overlap(equal_duration) - aligned = interval_overlap(barrier_aligned) - m3 = lane_metrics(run["forwards"])["stage0"]["M3_pairing"] - durations = [f["end"] - f["start"] for f in stage0] - summary[f"{model}/n{burst}/r{round_index}"] = { - "pairs": len(pairs), - "unpaired_forwards": abs(len(lanes[0]) - len(lanes[1])), - "pairing_equals_M3": m3 == [[list(a["indices"]), list(b["indices"])] for a, b in pairs], - "disjoint_pairs": disjoint, - "M5_observed": round(observed["multi_lane_busy_time"] / observed["busy_time"], 4), - "M5_equal_durations": round(equal["multi_lane_busy_time"] / equal["busy_time"], 4), - "M5_barrier_aligned": (round(aligned["multi_lane_busy_time"] / aligned["busy_time"], 4) - if not disjoint else None), - "non_overlap_ms_from_start_offsets": round(1e3 * start_part, 3), - "non_overlap_ms_from_end_offsets": round(1e3 * end_part, 3), - "non_overlap_ms_of_disjoint_pairs": round(1e3 * disjoint_part, 3), - "stage0_duration_ms_median": round(1e3 * statistics.median(durations), 3), - "stage0_duration_ms_cv": round(statistics.pstdev(durations) / statistics.mean(durations), 3), - } -print(json.dumps(summary, indent=1)) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py deleted file mode 100644 index be36bc2c..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/explain_t_path.py +++ /dev/null @@ -1,71 +0,0 @@ -"""Explain every path-T difference of the P3 comparison from the stage ledger. - -For each T case whose metrics differ, check that every (stage, lane) runs the -same ordered batches with the same component durations before and after, so -the difference is start times only; report the §4.5 metric as absolute and as -a fraction of stage busy time. - -Usage: python explain_t_path.py - -The before set is always ``base``. -""" -import json -import sys -from collections import defaultdict -from pathlib import Path - -root, after_set = Path(sys.argv[1]), sys.argv[2] -compare_path, output = Path(sys.argv[3]), Path(sys.argv[4]) - - -def lane_rows(set_name, case_id): - ledger = next((root / set_name / case_id / "metrics").rglob("frontier_stage_batch_ledger.jsonl")) - rows = defaultdict(list) - for line in ledger.read_text().splitlines(): - row = json.loads(line) - if row["execution_scope"] == "ATTN_DP_LANE": - rows[(row["stage_id"], row["replica_local_id"])].append(row) - for key in rows: - rows[key].sort(key=lambda row: row["stage_start_ts"]) - return rows - - -def signature(row): - return (tuple(row["request_ids"]), round(row["stage_end_ts"] - row["stage_start_ts"], 9), - json.dumps(row["execution_time"], sort_keys=True)) - - -def fraction(metric): - return {stage: round(m["multi_lane_busy_time"] / m["busy_time"], 4) for stage, m in metric.items()} - - -report = [] -for row in json.load(open(compare_path)): - if row["path"] != "T" or row["verdict"] == "PASS": - continue - before, after = lane_rows("base", row["case_id"]), lane_rows(after_set, row["case_id"]) - same_work = before.keys() == after.keys() and all( - [signature(r) for r in before[key]] == [signature(r) for r in after[key]] for key in before - ) - first_start = {side: {lane: rows[(0, lane)][0]["stage_start_ts"] for (stage, lane) in rows if stage == 0} - for side, rows in (("before", before), ("after", after))} - report.append({ - "case_id": row["case_id"], "verdict": row["verdict"], - "same_batches_and_component_durations": same_work, - "differing_files": row.get("differing_files"), - "witness_increase": row.get("witness_increase"), - "multi_lane_busy_time": {side: {s: m["multi_lane_busy_time"] for s, m in row[f"lane_metric_{side}"].items()} - for side in ("before", "after")}, - "co_execution_fraction": {side: fraction(row[f"lane_metric_{side}"]) for side in ("before", "after")}, - "peak_lanes": {side: {s: m["peak_lanes"] for s, m in row[f"lane_metric_{side}"].items()} - for side in ("before", "after")}, - "first_stage0_start": first_start, - }) -output.write_text(json.dumps(report, indent=1, sort_keys=True)) -for item in report: - print(item["case_id"], item["verdict"], "same_work=", item["same_batches_and_component_durations"], - "frac", item["co_execution_fraction"]["before"].get("MONOLITHIC/0/0"), "->", - item["co_execution_fraction"]["after"].get("MONOLITHIC/0/0"), - "peak", item["peak_lanes"]["before"].get("MONOLITHIC/0/0"), "->", - item["peak_lanes"]["after"].get("MONOLITHIC/0/0"), - "starts", item["first_stage0_start"]["before"], "->", item["first_stage0_start"]["after"]) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_integration_compare.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_integration_compare.json deleted file mode 100644 index 61d33632..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_integration_compare.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "regressions": [], - "new_failures": [], - "now_passing": [], - "skip_changes": [], - "only_before": [], - "only_after": [ - "tests.integration.test_stage_admission_pipeline_lanes::test_dense_lanes_start_in_the_same_first_forward", - "tests.integration.test_stage_admission_pipeline_lanes::test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4-expected0]", - "tests.integration.test_stage_admission_pipeline_lanes::test_moe_lanes_complete_every_request[G3a-moe-dp4-pp2-n8-expected1]" - ] -} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_unit_compare.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_unit_compare.json deleted file mode 100644 index 6615fe2f..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/g2_unit_compare.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "regressions": [], - "new_failures": [], - "now_passing": [], - "skip_changes": [], - "only_before": [], - "only_after": [ - "tests.unit.test_mixed_layer_decode_ffn_scheduling::test_decode_ffn_dense_groups_keep_counter_order_around_a_queued_ep_wave", - "tests.unit.test_shared_forward_group_admission::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0]", - "tests.unit.test_shared_forward_group_admission::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[1]", - "tests.unit.test_stage_execution_context::test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave", - "tests.unit.test_stage_execution_context::test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket", - "tests.unit.test_stage_execution_context::test_queued_ep_wave_orders_full_stage_work_on_both_sides" - ] -} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/p3_t_path_explanation.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/p3_t_path_explanation.json deleted file mode 100644 index 6a7451c5..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/p3_t_path_explanation.json +++ /dev/null @@ -1,436 +0,0 @@ -[ - { - "case_id": "G4-dense-dp2-pp2-n4", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.7143, - "MONOLITHIC/0/1": 0.7143 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.05, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.3, - "MONOLITHIC/0/1": 0.3 - }, - "before": { - "MONOLITHIC/0/0": 0.25, - "MONOLITHIC/0/1": 0.24999999999999997 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - }, - { - "case_id": "G4-dense-dp2-pp2-n8", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.8182, - "MONOLITHIC/0/1": 0.8182 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.05, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.49999999999999994, - "MONOLITHIC/0/1": 0.49999999999999994 - }, - "before": { - "MONOLITHIC/0/0": 0.44999999999999996, - "MONOLITHIC/0/1": 0.4499999999999999 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": true - }, - { - "case_id": "G4-dense-dp2-pp3-n4", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0, - "MONOLITHIC/0/2": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.3333, - "MONOLITHIC/0/1": 0.3333, - "MONOLITHIC/0/2": 0.3333 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.036000000000000004, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.21600000000000003, - "MONOLITHIC/0/1": 0.21600000000000008, - "MONOLITHIC/0/2": 0.21600000000000005 - }, - "before": { - "MONOLITHIC/0/0": 0.10800000000000004, - "MONOLITHIC/0/1": 0.10800000000000004, - "MONOLITHIC/0/2": 0.10800000000000001 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - }, - { - "case_id": "G4-dense-dp2-pp3-n8", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0, - "MONOLITHIC/0/2": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.375, - "MONOLITHIC/0/1": 0.375, - "MONOLITHIC/0/2": 0.375 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.07200000000000001, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.39600000000000013, - "MONOLITHIC/0/1": 0.39600000000000013, - "MONOLITHIC/0/2": 0.3960000000000002 - }, - "before": { - "MONOLITHIC/0/0": 0.21600000000000008, - "MONOLITHIC/0/1": 0.21600000000000014, - "MONOLITHIC/0/2": 0.2160000000000001 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": true - }, - { - "case_id": "G4-dense-dp4-pp2-n8", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.8462, - "MONOLITHIC/0/1": 0.8462 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0, - "2": 0.0, - "3": 0.0 - }, - "before": { - "0": 0.15000000000000002, - "1": 0.0, - "2": 0.05, - "3": 0.1 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.3, - "MONOLITHIC/0/1": 0.3 - }, - "before": { - "MONOLITHIC/0/0": 0.55, - "MONOLITHIC/0/1": 0.55 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 4, - "MONOLITHIC/0/1": 4 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "STOP", - "witness_increase": false - }, - { - "case_id": "G4-dense-dp4-pp3-n8", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0, - "MONOLITHIC/0/2": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.8462, - "MONOLITHIC/0/1": 0.8462, - "MONOLITHIC/0/2": 0.8462 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0, - "2": 0.0, - "3": 0.0 - }, - "before": { - "0": 0.10800000000000001, - "1": 0.0, - "2": 0.036000000000000004, - "3": 0.07200000000000001 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.21600000000000003, - "MONOLITHIC/0/1": 0.21600000000000008, - "MONOLITHIC/0/2": 0.21600000000000005 - }, - "before": { - "MONOLITHIC/0/0": 0.39600000000000013, - "MONOLITHIC/0/1": 0.3960000000000002, - "MONOLITHIC/0/2": 0.39600000000000024 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 4, - "MONOLITHIC/0/1": 4, - "MONOLITHIC/0/2": 4 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "STOP", - "witness_increase": false - }, - { - "case_id": "G7-dense-dp2-pp2-n8", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.6, - "MONOLITHIC/0/1": 0.6 - } - }, - "differing_files": [ - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/frontier_stage_batch_ledger.jsonl", - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/request_metrics.csv", - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.12000000000000001, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.48000000000000004, - "MONOLITHIC/0/1": 0.4800000000000001 - }, - "before": { - "MONOLITHIC/0/0": 0.36000000000000004, - "MONOLITHIC/0/1": 0.3600000000000001 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - }, - { - "case_id": "G7-dense-dp2-pp2-n16", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.7778, - "MONOLITHIC/0/1": 0.7778 - } - }, - "differing_files": [ - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/frontier_stage_batch_ledger.jsonl", - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/request_metrics.csv", - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.12000000000000001, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.9600000000000001, - "MONOLITHIC/0/1": 0.9600000000000001 - }, - "before": { - "MONOLITHIC/0/0": 0.8400000000000001, - "MONOLITHIC/0/1": 0.8400000000000001 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - } -] \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_integration_compare.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_integration_compare.json deleted file mode 100644 index 61d33632..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_integration_compare.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "regressions": [], - "new_failures": [], - "now_passing": [], - "skip_changes": [], - "only_before": [], - "only_after": [ - "tests.integration.test_stage_admission_pipeline_lanes::test_dense_lanes_start_in_the_same_first_forward", - "tests.integration.test_stage_admission_pipeline_lanes::test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4-expected0]", - "tests.integration.test_stage_admission_pipeline_lanes::test_moe_lanes_complete_every_request[G3a-moe-dp4-pp2-n8-expected1]" - ] -} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_unit_compare.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_unit_compare.json deleted file mode 100644 index 26aaaae2..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_g2_unit_compare.json +++ /dev/null @@ -1,25 +0,0 @@ -{ - "regressions": [], - "new_failures": [], - "now_passing": [], - "skip_changes": [], - "only_before": [], - "only_after": [ - "tests.unit.test_mixed_layer_decode_ffn_scheduling::test_decode_ffn_dense_groups_keep_counter_order_around_a_queued_ep_wave", - "tests.unit.test_shared_forward_group_admission::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0]", - "tests.unit.test_shared_forward_group_admission::test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[1]", - "tests.unit.test_stage_admission_pp_tools::test_apply_patch_keeps_every_section_of_a_repeated_file", - "tests.unit.test_stage_admission_pp_tools::test_apply_patch_reads_a_trimmed_context_line", - "tests.unit.test_stage_admission_pp_tools::test_apply_patch_rejects_an_unknown_hunk_line", - "tests.unit.test_stage_admission_pp_tools::test_fixed_base_loses_the_controls_without_a_mismatch", - "tests.unit.test_stage_admission_pp_tools::test_ideal_after_revision_matches_and_base_controls_hold", - "tests.unit.test_stage_admission_pp_tools::test_missing_placement_log_fails_the_comparison", - "tests.unit.test_stage_admission_pp_tools::test_overlay_acceptance_compares_file_sets[False-True]", - "tests.unit.test_stage_admission_pp_tools::test_overlay_acceptance_compares_file_sets[True-False]", - "tests.unit.test_stage_admission_pp_tools::test_vllm_round_with_a_late_lane_is_reported", - "tests.unit.test_stage_execution_context::test_active_full_stage_ticket_is_refused_without_changing_the_stage", - "tests.unit.test_stage_execution_context::test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave", - "tests.unit.test_stage_execution_context::test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket", - "tests.unit.test_stage_execution_context::test_queued_ep_wave_orders_full_stage_work_on_both_sides" - ] -} diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_t_path_explanation.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_t_path_explanation.json deleted file mode 100644 index 91379ba2..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_t_path_explanation.json +++ /dev/null @@ -1,660 +0,0 @@ -[ - { - "case_id": "G4-dense-dp2-pp2-n4", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.7143, - "MONOLITHIC/0/1": 0.7143 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n4/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.05, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.3, - "MONOLITHIC/0/1": 0.3 - }, - "before": { - "MONOLITHIC/0/0": 0.25, - "MONOLITHIC/0/1": 0.24999999999999997 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - }, - { - "case_id": "G4-dense-dp2-pp2-n8", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.8182, - "MONOLITHIC/0/1": 0.8182 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp2_n8/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.05, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.49999999999999994, - "MONOLITHIC/0/1": 0.49999999999999994 - }, - "before": { - "MONOLITHIC/0/0": 0.44999999999999996, - "MONOLITHIC/0/1": 0.4499999999999999 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": true - }, - { - "case_id": "G4-dense-dp2-pp3-n4", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0, - "MONOLITHIC/0/2": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.3333, - "MONOLITHIC/0/1": 0.3333, - "MONOLITHIC/0/2": 0.3333 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n4/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.036000000000000004, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.21600000000000003, - "MONOLITHIC/0/1": 0.21600000000000008, - "MONOLITHIC/0/2": 0.21600000000000005 - }, - "before": { - "MONOLITHIC/0/0": 0.10800000000000004, - "MONOLITHIC/0/1": 0.10800000000000004, - "MONOLITHIC/0/2": 0.10800000000000001 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - }, - { - "case_id": "G4-dense-dp2-pp3-n8", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0, - "MONOLITHIC/0/2": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.375, - "MONOLITHIC/0/1": 0.375, - "MONOLITHIC/0/2": 0.375 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp2_pp3_n8/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.07200000000000001, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.39600000000000013, - "MONOLITHIC/0/1": 0.39600000000000013, - "MONOLITHIC/0/2": 0.3960000000000002 - }, - "before": { - "MONOLITHIC/0/0": 0.21600000000000008, - "MONOLITHIC/0/1": 0.21600000000000014, - "MONOLITHIC/0/2": 0.2160000000000001 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": true - }, - { - "case_id": "G4-dense-dp4-pp2-n8", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.8462, - "MONOLITHIC/0/1": 0.8462 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp4_pp2_n8/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0, - "2": 0.0, - "3": 0.0 - }, - "before": { - "0": 0.15000000000000002, - "1": 0.0, - "2": 0.05, - "3": 0.1 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.3, - "MONOLITHIC/0/1": 0.3 - }, - "before": { - "MONOLITHIC/0/0": 0.55, - "MONOLITHIC/0/1": 0.55 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 4, - "MONOLITHIC/0/1": 4 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": true - }, - { - "case_id": "G4-dense-dp4-pp3-n8", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0, - "MONOLITHIC/0/2": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.8462, - "MONOLITHIC/0/1": 0.8462, - "MONOLITHIC/0/2": 0.8462 - } - }, - "differing_files": [ - "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/request_metrics.csv", - "stage_admission_dense/offline_batch/g4_dense_dp4_pp3_n8/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0, - "2": 0.0, - "3": 0.0 - }, - "before": { - "0": 0.10800000000000001, - "1": 0.0, - "2": 0.036000000000000004, - "3": 0.07200000000000001 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.21600000000000003, - "MONOLITHIC/0/1": 0.21600000000000008, - "MONOLITHIC/0/2": 0.21600000000000005 - }, - "before": { - "MONOLITHIC/0/0": 0.39600000000000013, - "MONOLITHIC/0/1": 0.3960000000000002, - "MONOLITHIC/0/2": 0.39600000000000024 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 4, - "MONOLITHIC/0/1": 4, - "MONOLITHIC/0/2": 4 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": true - }, - { - "case_id": "G7-dense-dp2-pp2-n8", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.6, - "MONOLITHIC/0/1": 0.6 - } - }, - "differing_files": [ - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/frontier_stage_batch_ledger.jsonl", - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/request_metrics.csv", - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n8/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.12000000000000001, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.48000000000000004, - "MONOLITHIC/0/1": 0.4800000000000001 - }, - "before": { - "MONOLITHIC/0/0": 0.36000000000000004, - "MONOLITHIC/0/1": 0.3600000000000001 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - }, - { - "case_id": "G7-dense-dp2-pp2-n16", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.7778, - "MONOLITHIC/0/1": 0.7778 - } - }, - "differing_files": [ - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/frontier_stage_batch_ledger.jsonl", - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/request_metrics.csv", - "llama_3_2_1b_instruct/offline_batch/g7_dense_dp2_pp2_n16/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.12000000000000001, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.9600000000000001, - "MONOLITHIC/0/1": 0.9600000000000001 - }, - "before": { - "MONOLITHIC/0/0": 0.8400000000000001, - "MONOLITHIC/0/1": 0.8400000000000001 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - }, - { - "case_id": "G10-dense-dp2-pp2-n8-burst", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.8182, - "MONOLITHIC/0/1": 0.8182 - } - }, - "differing_files": [ - "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_burst/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_burst/request_metrics.csv", - "stage_admission_dense/online_serving/g10_dense_dp2_pp2_n8_burst/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.05, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.49999999999999994, - "MONOLITHIC/0/1": 0.49999999999999994 - }, - "before": { - "MONOLITHIC/0/0": 0.44999999999999996, - "MONOLITHIC/0/1": 0.4499999999999999 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - }, - { - "case_id": "G10-dense-dp2-pp3-n8-burst", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0, - "MONOLITHIC/0/2": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.375, - "MONOLITHIC/0/1": 0.375, - "MONOLITHIC/0/2": 0.375 - } - }, - "differing_files": [ - "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8_burst/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8_burst/request_metrics.csv", - "stage_admission_dense/online_serving/g10_dense_dp2_pp3_n8_burst/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0 - }, - "before": { - "0": 0.07200000000000001, - "1": 0.0 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.39600000000000013, - "MONOLITHIC/0/1": 0.39600000000000013, - "MONOLITHIC/0/2": 0.3960000000000002 - }, - "before": { - "MONOLITHIC/0/0": 0.21600000000000008, - "MONOLITHIC/0/1": 0.21600000000000014, - "MONOLITHIC/0/2": 0.2160000000000001 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - }, - { - "case_id": "G10-dense-dp4-pp2-n8-burst", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.8462, - "MONOLITHIC/0/1": 0.8462 - } - }, - "differing_files": [ - "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8_burst/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8_burst/request_metrics.csv", - "stage_admission_dense/online_serving/g10_dense_dp4_pp2_n8_burst/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0, - "2": 0.0, - "3": 0.0 - }, - "before": { - "0": 0.15000000000000002, - "1": 0.0, - "2": 0.05, - "3": 0.1 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.3, - "MONOLITHIC/0/1": 0.3 - }, - "before": { - "MONOLITHIC/0/0": 0.55, - "MONOLITHIC/0/1": 0.55 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 4, - "MONOLITHIC/0/1": 4 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - }, - { - "case_id": "G10-dense-dp4-pp3-n8-burst", - "co_execution_fraction": { - "after": { - "MONOLITHIC/0/0": 1.0, - "MONOLITHIC/0/1": 1.0, - "MONOLITHIC/0/2": 1.0 - }, - "before": { - "MONOLITHIC/0/0": 0.8462, - "MONOLITHIC/0/1": 0.8462, - "MONOLITHIC/0/2": 0.8462 - } - }, - "differing_files": [ - "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8_burst/frontier_stage_batch_ledger.jsonl", - "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8_burst/request_metrics.csv", - "stage_admission_dense/online_serving/g10_dense_dp4_pp3_n8_burst/system_metrics.json" - ], - "first_stage0_start": { - "after": { - "0": 0.0, - "1": 0.0, - "2": 0.0, - "3": 0.0 - }, - "before": { - "0": 0.10800000000000001, - "1": 0.0, - "2": 0.036000000000000004, - "3": 0.07200000000000001 - } - }, - "multi_lane_busy_time": { - "after": { - "MONOLITHIC/0/0": 0.21600000000000003, - "MONOLITHIC/0/1": 0.21600000000000008, - "MONOLITHIC/0/2": 0.21600000000000005 - }, - "before": { - "MONOLITHIC/0/0": 0.39600000000000013, - "MONOLITHIC/0/1": 0.3960000000000002, - "MONOLITHIC/0/2": 0.39600000000000024 - } - }, - "peak_lanes": { - "after": { - "MONOLITHIC/0/0": 4, - "MONOLITHIC/0/1": 4, - "MONOLITHIC/0/2": 4 - }, - "before": { - "MONOLITHIC/0/0": 2, - "MONOLITHIC/0/1": 2, - "MONOLITHIC/0/2": 2 - } - }, - "same_batches_and_component_durations": true, - "verdict": "EXPLAIN", - "witness_increase": null - } -] \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_tool_tests_on_ecff89a.txt b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_tool_tests_on_ecff89a.txt deleted file mode 100644 index 0e1d48c0..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/r2_tool_tests_on_ecff89a.txt +++ /dev/null @@ -1,8 +0,0 @@ -FAILED tests/unit/test_stage_admission_pp_tools.py::test_ideal_after_revision_matches_and_base_controls_hold -FAILED tests/unit/test_stage_admission_pp_tools.py::test_fixed_base_loses_the_controls_without_a_mismatch -FAILED tests/unit/test_stage_admission_pp_tools.py::test_missing_placement_log_fails_the_comparison -FAILED tests/unit/test_stage_admission_pp_tools.py::test_overlay_acceptance_compares_file_sets[False-True] -FAILED tests/unit/test_stage_admission_pp_tools.py::test_apply_patch_keeps_every_section_of_a_repeated_file -FAILED tests/unit/test_stage_admission_pp_tools.py::test_apply_patch_reads_a_trimmed_context_line -FAILED tests/unit/test_stage_admission_pp_tools.py::test_apply_patch_rejects_an_unknown_hunk_line -7 failed, 2 passed in 0.32s diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_dense_dp1_pp2.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_dense_dp1_pp2.json deleted file mode 100644 index 06c42922..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_dense_dp1_pp2.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "shape": { - "is_moe": false, - "attn_dp": 1, - "moe_ep": 1, - "stages": 2 - }, - "completed": 6, - "requests": 6 -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp1.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp1.json deleted file mode 100644 index 1bec72f1..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp1.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "shape": { - "is_moe": true, - "attn_dp": 2, - "moe_ep": 2, - "stages": 1 - }, - "completed": 6, - "requests": 6 -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp2.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp2.json deleted file mode 100644 index 3f52dce5..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp2.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "shape": { - "is_moe": true, - "attn_dp": 2, - "moe_ep": 2, - "stages": 2 - }, - "completed": 6, - "requests": 6 -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp3.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp3.json deleted file mode 100644 index 67b94dac..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/after_moe_dp2_pp3.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "shape": { - "is_moe": true, - "attn_dp": 2, - "moe_ep": 2, - "stages": 3 - }, - "error": "ValueError('collective-sim physical topology requires cluster_total_devices 6 to be divisible by node size 4')" -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/base_moe_dp2_pp2.json b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/base_moe_dp2_pp2.json deleted file mode 100644 index a1c86beb..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/base_moe_dp2_pp2.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "shape": { - "is_moe": true, - "attn_dp": 2, - "moe_ep": 2, - "stages": 2 - }, - "error": "RuntimeError('Sequential simulation ended with non-empty scheduler state: ...')" -} \ No newline at end of file diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_completion.py b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_completion.py deleted file mode 100644 index 67b8b468..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_completion.py +++ /dev/null @@ -1,33 +0,0 @@ -"""C6: Step 9 boundary probe shapes on a branch without the PR 35 batch-end seam. - -Uses probe_main.build_config unchanged and reports completion only. -""" -import json -import sys -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parent)) -from probe_main import build_config # noqa: E402 - -SHAPES = { - "moe_dp2_pp1": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=1), - "moe_dp2_pp2": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=2), - "moe_dp2_pp3": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=3), - "dense_dp1_pp2": dict(is_moe=False, attn_dp=1, moe_ep=1, stages=2), -} - -root, label = Path(sys.argv[1]), sys.argv[2] -case_root = root / label -case_root.mkdir(parents=True, exist_ok=True) -from frontier.simulator import Simulator # noqa: E402 - -result = {"shape": SHAPES[label]} -try: - simulator = Simulator(build_config(case_root, **SHAPES[label])) - simulator.run() - requests = list(simulator._all_requests) - result.update(completed=sum(1 for r in requests if r.completed), requests=len(requests)) -except Exception as exc: - result.update(error=repr(exc)[:800]) -(case_root / "result.json").write_text(json.dumps(result, indent=1)) -print(label, json.dumps(result)) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py b/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py deleted file mode 100644 index 87743805..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/evidence/step9_probe/probe_main.py +++ /dev/null @@ -1,178 +0,0 @@ -"""P1(b): record Frontier's admission and completion boundaries under PP. - -No source change. The probe wraps `_get_next_batch` (one call per admission, -after `_running_requests` has grown) and the inert `on_replica_batch_end` seam, -and reads the candidate report key -- the Replica's next forward id held by -`ForwardSyncState` -- at each boundary. - -Run with ``PYTHONPATH`` set to the Frontier tree under test. -""" -from __future__ import annotations - -import json -import sys -from pathlib import Path - - -def build_config(root: Path, *, is_moe: bool, attn_dp: int, moe_ep: int, stages: int): - from frontier.config import ( - BaseModelConfig, ClusterConfig, FixedRequestLengthGeneratorConfig, - MetricsConfig, PoissonRequestIntervalGeneratorConfig, - RandomForrestExecutionTimePredictorConfig, ReplicaConfig, - RoundRobinClusterSchedulerConfig, SimulationConfig, - SyntheticRequestGeneratorConfig, VllmV1SchedulerConfig, - ) - from frontier.types import ActivationType, NormType - - model = BaseModelConfig( - num_layers=6, num_q_heads=4, num_kv_heads=2, embedding_dim=256, - mlp_hidden_dim=64, max_position_embeddings=4096, use_gated_mlp=True, - use_bias=False, use_qkv_bias=False, activation=ActivationType.SILU, - norm=NormType.RMS_NORM, post_attn_norm=True, vocab_size=1024, - is_moe=is_moe, num_experts=8 if is_moe else 0, - num_experts_per_tok=2 if is_moe else 0, torch_dtype="bfloat16", - ) - model._model_name = f"w9_probe_{'moe' if is_moe else 'dense'}" - original = BaseModelConfig.create_from_name - BaseModelConfig.create_from_name = classmethod( - lambda cls, name: model if name == model._model_name else original(name) - ) - moe_fields = dict( - moe_tensor_parallel_size=1, moe_expert_parallel_size=moe_ep, - total_expert_num=8, router_topk=2, - ) if is_moe else {} - replica = ReplicaConfig( - model_name=model._model_name, device="a100", - network_device="a100_pairwise_nvlink", num_pipeline_stages=stages, - attn_tensor_parallel_size=1, attn_dp=attn_dp, - memory_margin_fraction=0.1, **moe_fields, - ) - cluster = ClusterConfig( - replica_config=replica, - replica_scheduler_config=VllmV1SchedulerConfig( - num_blocks=128, block_size=16, batch_size_cap=4, - max_tokens_in_batch=16, enable_chunked_prefill=True, - ), - cluster_scheduler_config=RoundRobinClusterSchedulerConfig(), - execution_time_predictor_config=RandomForrestExecutionTimePredictorConfig( - enable_dummy_mode=True - ), - ) - return SimulationConfig( - simulation_mode="offline", sys_arch="co-location", - enable_parallel_clusters=False, decode_cuda_graph_mode="none", - cluster_config=cluster, - metrics_config=MetricsConfig( - output_dir=str(root / "metrics"), cache_dir=str(root / "cache"), - run_id="w9_probe", write_metrics=False, store_request_metrics=False, - store_batch_metrics=False, store_operation_metrics=False, - store_utilization_metrics=False, store_plots=False, - enable_chrome_trace=False, write_json_trace=False, - ), - request_generator_config=SyntheticRequestGeneratorConfig( - num_requests=6, - length_generator_config=FixedRequestLengthGeneratorConfig( - prefill_tokens=16, decode_tokens=3 - ), - interval_generator_config=PoissonRequestIntervalGeneratorConfig(qps=1e6), - ), - ) - - -def run(root: Path, *, is_moe: bool, attn_dp: int, moe_ep: int, stages: int): - from frontier.scheduler.cluster_scheduler.base_cluster_scheduler import ( - BaseClusterScheduler, - ) - from frontier.scheduler.replica_scheduler.vllm_v1_engine_replica_scheduler import ( - VLLMv1EngineReplicaScheduler, - ) - from frontier.scheduler.utils.forward_sync_state import ForwardSyncState - from frontier.simulator import Simulator - - events: list[dict] = [] - schedulers: dict = {} - - def next_forward_id(cluster_scheduler, replica_id): - state = cluster_scheduler._forward_sync_state - return int(state._next_step_id_by_replica.get(replica_id, 0)) - - original_next_batch = VLLMv1EngineReplicaScheduler._get_next_batch - - def observed_next_batch(self, is_micro_batch=False): - batch = original_next_batch(self, is_micro_batch=is_micro_batch) - if batch is not None: - schedulers[(self._replica_id, self._replica_local_id)] = self - events.append({ - "kind": "admit", - "time": round(float(self._current_schedule_time), 9), - "lane": self._replica_local_id, - "batch": batch.id, - "provisional": batch._forward_cohort_provisional_id, - "running_batches_before": self._num_running_batches, - "stages": self._num_stages, - "load": list(self.get_request_load()), - "candidate_key": next_forward_id(self._cluster_scheduler, self._replica_id), - }) - return batch - - original_batch_end = BaseClusterScheduler.on_replica_batch_end - - def observed_batch_end(self, time, replica_id, replica_local_id, batch): - result = original_batch_end(self, time, replica_id, replica_local_id, batch) - lane = self.get_replica_scheduler(replica_id, replica_local_id) - events.append({ - "kind": "complete", - "time": round(float(time), 9), - "lane": replica_local_id, - "batch": batch.id, - "provisional": batch._forward_cohort_provisional_id, - "resolved": ForwardSyncState.get_step_id(batch), - "running_batches_after": lane.num_running_batches, - "load": list(lane.get_request_load()), - "candidate_key": next_forward_id(self, replica_id), - }) - return result - - VLLMv1EngineReplicaScheduler._get_next_batch = observed_next_batch - BaseClusterScheduler.on_replica_batch_end = observed_batch_end - try: - config = build_config(root, is_moe=is_moe, attn_dp=attn_dp, - moe_ep=moe_ep, stages=stages) - simulator = Simulator(config) - simulator.run() - requests = list(simulator._all_requests) - finally: - VLLMv1EngineReplicaScheduler._get_next_batch = original_next_batch - BaseClusterScheduler.on_replica_batch_end = original_batch_end - return { - "completed": sum(1 for r in requests if r.completed), - "requests": len(requests), - "events": events, - } - - -if __name__ == "__main__": - root = Path(sys.argv[1]) - summary = {} - for label, shape in { - "moe_dp2_pp1": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=1), - "moe_dp2_pp2": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=2), - "moe_dp2_pp3": dict(is_moe=True, attn_dp=2, moe_ep=2, stages=3), - "dense_dp1_pp2": dict(is_moe=False, attn_dp=1, moe_ep=1, stages=2), - }.items(): - case_root = root / label - case_root.mkdir(parents=True, exist_ok=True) - result = run(case_root, **shape) - summary[label] = result - events = result["events"] - print(f"\n=== {label}: {result['completed']}/{result['requests']} completed, " - f"{len(events)} boundaries") - print(f"{'time':>9} {'kind':>8} {'lane':>4} {'batch':>5} {'prov':>4} " - f"{'resolved':>8} {'load':>8} {'key':>4} {'slots':>6}") - for e in events[:28]: - slots = (f"{e['running_batches_before']}/{e['stages']}" if e["kind"] == "admit" - else f"{e['running_batches_after']}") - print(f"{e['time']:>9.5f} {e['kind']:>8} {str(e['lane']):>4} {e['batch']:>5} " - f"{e['provisional']:>4} {str(e.get('resolved', '')):>8} " - f"{str(tuple(e['load'])):>8} {e['candidate_key']:>4} {slots:>6}") - (root / "frontier_boundaries.json").write_text(json.dumps(summary, indent=1)) diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/plan.md b/task_memory/task_2026-09-22_stage_admission_ordering/plan.md deleted file mode 100644 index 6a1a9334..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/plan.md +++ /dev/null @@ -1,421 +0,0 @@ -# Stage admission ordering under pipeline parallelism — Plan - -## Modification History - -| Date | Change | -| --- | --- | -| 2026-09-23 | R-10 executed: §7 group table amended with the online burst cells (lane-0 placement on main), D-9 (b) rationale restated (R2-02). | -| 2026-09-23 | R-10: §7 added, the round-2 remediation (R2 findings), new groups G8–G11 and the pre-merge step P6. | -| 2026-09-23 | R-8 / D-9: the C3 witness condition uses the co-execution fraction, and V5 gates MoE only (dense is reported). Adopted after the P3/P5 stops, before P4. | -| 2026-09-23 | R-7: the MoE retry job runs the ground truth with one recorded overlay patch (four-argument `topk_softmax`); §4.7 notes it. | -| 2026-09-23 | R-6: execution started. Added package P5 (vLLM comparison on a GPU worker), criterion C7, the vLLM-aligned group G7, §4.7 and D-8. | -| 2026-09-23 | Applied the round-1 plan review (`review.md`). Changes: C1 now targets confirmed admission-deadlock witnesses from a phase-controlled group; C3 uses the stage ledger and overlap duration; C4 takes the reviewer's wording; P0 lists its artifacts and outcome classes; P2 covers both sides of the EP boundary, the `DECODE_FFN` dense-group control, a second admission round and per-fixture base expectations, and the dense fixture asserts a same-start condition that discriminates on the base; P3 has three acceptance paths. The matrix now publishes a concrete case list on the analytical backend and keeps `attn_dp=2, PP=3`. Added D-6 and D-7. Not executed. | -| 2026-09-23 | D-1..D-5 adopted by the user; D-3 executed now (push + draft PR for remote review); D-5 adjusted so the records travel with the branch. Work packages P0–P4 unblocked. | -| 2026-09-22 | Created for user review. Scope, acceptance criteria, work packages P0–P4 with dependencies, verification matrix, decisions D-1..D-4. No source change yet. | - -Diagnosis, options and the recommended rule are in `design.md`. Review -dispositions are in `review.md`. This file is the executable plan. - -## 1. Scope - -Fix the pre-existing stage admission deadlock (W9-01 in the parent task) on its -own branch: - -- Branch `fix/stage-admission-ordering`, base `origin/main` `1f694f7`, worktree - `/data/ycfeng/Frontier/.worktrees/stage-admission-ordering`. -- Source change confined to - `frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`: - the full-stage admission predicate in `try_acquire`, removal of the admitted - ticket with `remove(ticket)`, and the two docstrings that describe admission - as "FIFO-head". -- Validation against vLLM on a GPU worker (R-6, package P5): tests and - comparison scripts only, no source change. -- Out of scope: `full_stage_capacity`, the forward-group seal, EP wave - protocol, sync rooms, wake-up helper, any configuration field, the Step 9 - report key, and mixed-phase forward failures on `main` (PR 35 W3; see - `design.md` "Scope boundary"). - -## 2. Acceptance criteria - -| Id | Criterion | Settled by | -| --- | --- | --- | -| C1 | Repaired liveness (path L). Every G3a case that P0 classifies as `admission_deadlock` completes after P1, with request count, prefill tokens and decode tokens conserved. G3a is the phase-controlled, prefill-only group. P0 must find at least one such case for each of `(attn_dp, PP)` ∈ {2, 4} × {2, 3}; if a pair has none, stop and report before P1, because the case list does not exercise the defect there. | P0 classification; P2(c); P3 path L. | -| C2 | Unchanged controls (path U). Every run-to-run-stable metrics file is byte-identical before and after for G1 (all 30 release recipes, including the 10 PD-AF recipes), every `PP = 1` cell of G3a, G3b and G4, and every G5 cell. | P0 vs P3 `sha256sums.txt`. | -| C3 | Timing change (path T). Base-successful cases with `attn_dp > 1` and `PP > 1` (all G4 `PP > 1` cells, and G3a/G3b cells P0 classifies as `success`) are either byte-identical, or their difference is explained with the stage-ledger metric of §4.5 plus batch membership and component durations. The designated contention witnesses (§4.2, marked W) show a strictly larger co-execution fraction `multi_lane_busy_time / busy_time` (§4.5, summed over the witness's stages) after P1 (D-9). In every case: no lane overlaps itself, and `peak_lanes ≤ attn_dp`. | P3, §4.5 metric from `frontier_stage_batch_ledger.jsonl`. | -| C4 | Existing passing tests must remain passing without assertion changes. Existing failures, collection errors, and skips must be compared against a fresh run of the exact base revision in the same environment. Any new failure or required change to an ordering assertion stops implementation for review. | G2, §4.6. | -| C5 | The predicate is one readable condition, and the module and method docstrings state the ordering contract as implemented. The change adds no flag, config field, `getattr` fallback, lane field on tickets, acquisition wake-up, PP-specific branch, second queue or capacity-1 special case. | Review of the diff against the quality gates. | -| C7 | vLLM comparison (path V, R-6). On the vLLM-aligned shapes of §4.7, the P1 revision matches vLLM on completion, per-lane batch sequences, stage-0 lane pairing, first-forward co-start and stage-0 co-execution, and the base revision fails the negative controls stated there. | P5, §4.7. | -| C6 | Informational. The Step 9 boundary probe on MoE `attn_dp=2, moe_ep=2, PP=2` runs to completion on this branch, or its remaining failure is classified. Composition with PR 35 (W3 mixed-phase forward) is validated in the parent task after merge-forward, before Step 9 is declared unblocked. | P3 probe rerun; parent-task follow-up (§6). | - -## 3. Work packages - -```text -P0 evidence and baseline (no source change) - -> P1 rule change - -> {P2 tests, P3 rerun and comparison} - -> P4 records, commit, push - -P5a vLLM driver, extraction and comparison scripts (independent of P1) - -> P5b GPU ground-truth run - -> P5c comparison against P0 (base) and P3 (after) G7 outputs - -> P4 -``` - -P5a and P5b run in parallel with P0–P3; P5c needs the G7 outputs of P0 and -P3. - -| Package | Content | Acceptance | -| --- | --- | --- | -| P0 Evidence and baseline | (1) Move the reproduction into `tests/e2e/stage_admission_matrix.py`, following the `tests/e2e/moe_ep_non_dummy_matrix.py` precedent. The module holds: the §4.1 fixture builder; the §4.2 case table; a runner with one child process per case, because `IS_MOE` is process-global; the outcome classifier and state-report writer of §4.3, which replace the session scripts `drain_state.py`/`drain_lanes.py`; and the §4.5 ledger metric. Outputs go to `resolve_scratch_root()/stage_admission_ordering/base//` (`tests/scratch_root.py`). (2) Confirm that the branch source equals `1f694f7` (`git diff --stat 1f694f7 -- . ':!task_memory' ':!.gitignore'` is empty). (3) Run R0, G1, G3a, G3b, G4, G5 and the G2 suites. (4) Write the §4.3 artifacts for every case. (5) Rerun two success cases (one G1 recipe, one G4 `PP=2` cell); an unstable file is named and excluded from C2, with the reason. (6) Record the classification table in the test report. | Every case has `case.json`, `run.json` and its class artifact. R0 reproduces the author-reported table, or each difference is explained. No `other_failure`. C1's per-pair witness condition holds. Every successful `attn_dp>1` case has `ATTN_DP_LANE` ledger rows for each of its lanes, otherwise §4.5 cannot be computed and P0 stops. | -| P1 Rule | Implement the `design.md` rule. Full-stage tickets are refused only by an EP wave queued ahead; EP waves keep the strict head rule; the admitted ticket leaves the FIFO by `remove(ticket)`. Update the `StageExecutionContext` class docstring ("A complete operation first enters the ready FIFO, then the owner admits it atomically") and the `try_acquire` docstring ("Acquire the FIFO-head ticket if this stage is currently idle") to state the implemented contract. Queued full-stage work may pass other full-stage work but not an earlier queued EP wave. Queued EP waves keep FIFO admission. Active layer-to-layer scope transitions remain a separate mechanism. | The diff touches one source file. `python -m pytest tests/unit/test_stage_execution_context.py tests/unit/test_shared_forward_group_admission.py -q` passes with no assertion change. | -| P2 Tests | **(a)** Contract tests in `tests/unit/test_stage_execution_context.py`. *Bypass* and *EP boundary* use a capacity-2 context (`ep_size=2`) with FIFO `full0, full1, wave0, full2`. *Bypass*: `full1` acquires before `full0`, and afterwards `queued_tickets == (full0, wave0, full2)`; `full2` is then refused although capacity remains, because `wave0` is ahead; `full0` acquires. *EP boundary* (acquisitions in head order, so it also runs on the base): `full0` and `full1` acquire; after `full1` releases, `full2` is still refused, because `wave0` is ahead; `wave0` is refused while `full0` is active and acquires once it releases; `full2` is refused while `wave0` is active and acquires after `wave0` releases. *Capacity 1*: on an idle context with FIFO `[full0, full1]`, `try_acquire(full1)` succeeds, pinning the API-level change stated in `design.md`. The two existing EP-order tests stay unchanged. **(a′)** A `DECODE_FFN` control in `tests/unit/test_mixed_layer_decode_ffn_scheduling.py` with its mixed-layer fixture. It materializes two successive `DenseFFNBatchGroup`s and a neighbouring EP group on one target replica and stage, through `_schedule_dense_ffn_from_m2n_group` and the real full-stage `ReplicaStageScheduler`. It asserts that FIFO order and heap order both follow the group counter, that the dense groups are admitted in counter order, and that neither dense group crosses an EP wave queued ahead of it. **(b)** A scheduler-level test in `tests/unit/test_shared_forward_group_admission.py` using its `make_stage`/`make_batch` helpers, parametrized over which lane enqueues first. It rebuilds the drain state: the first lane is active with a second ticket queued, and the other lane has two queued tickets. It asserts that the other lane's `pop_batch_if_not_busy` returns its heap head and binds the same forward group. It then continues through promotion to an EP wave and restoration to full-stage owners (`replace_full_stage_owners_with_ep_wave`, `replace_ep_wave_with_full_stage_owners`), release of both owners and `on_stage_end` of both lanes, and it asserts that both lanes admit their next queued batch into a later forward group, leaving the FIFO empty. **(c)** Simulator-level tests in `tests/integration/test_stage_admission_pipeline_lanes.py`, importing the §4.1 builder from `tests.e2e.stage_admission_matrix`, one child process per case. MoE witnesses `G3a-moe-dp2-pp2-n4` and `G3a-moe-dp4-pp2-n8` assert completion and conservation. The dense fixture `G4-dense-dp2-pp2-n8` asserts completion, and that the first stage-0 ledger rows of both lanes start at the same simulated time, because every request arrives at `t=0` and capacity admits both lanes into the first forward. A bare `multi_lane_busy_time > 0` would not discriminate: from source, the base already overlaps the lanes after the first release. Expected values are written from the scenario, not copied from a run. | Expected on `1f694f7`: (a) *bypass* fails at its first assertion, *capacity 1* fails, and *EP boundary* passes; (a′) passes; (b) fails at the other lane's first admission; (c) each MoE witness fails through the documented `admission_deadlock` signature, and the dense fixture completes but fails only its same-start assertion, because the second lane's first row starts at the first lane's first stage-0 end. After P1 all of them pass. The base failures are recorded as negative controls. | -| P3 Rerun | Rerun every P0 case on the P1 revision into `.../after//`, then apply the acceptance path of each case (§4.2). **U**: hashes identical. **L**: the case completes with conservation; there is no base metrics hash to compare. **T**: hashes identical, or the difference is explained by the §4.5 metric (before and after), batch membership and component durations; W cases must show a strict increase. Stop and report, adjusting nothing, on any of these: a U difference (including `attn_dp=4, PP=1`); an L case that fails in any other way, such as a mixed-phase failure in G3b; a T difference that the ledger does not explain; a class change outside these paths; a failure of the self-overlap or `peak_lanes ≤ attn_dp` checks. Rerun the Step 9 boundary probe for C6. | C1–C4 and C6 tables in `test_report__stage_admission_ordering.md`. | -| P5 vLLM comparison | **(a)** Scripts under `tests/comparison/stage_admission_pp/`: a burst driver that runs inside the vLLM image, an extractor that turns the vLLM trace files into per-forward lane rows, and a comparison that computes the §4.7 metrics for vLLM and for Frontier ledgers. They are checked on the CPU host against synthetic inputs before any GPU job. **(b)** One GPU job (§4.7 "GPU job"), recorded in the case directory `calibration/stage_admission_case_001/`. **(c)** Comparison of the vLLM rows with the base (P0) and after (P3) runs of G7, written as a workflow-gap table with one `MATCH`/`MISMATCH` row per metric and burst. | C7 per §4.7. A `MISMATCH` on the after revision stops before P4 and is reported with its Frontier owner; nothing is adjusted to make it match. | -| P4 Records | Test report, `progress.md`, `summary.md`. Commit P0's harness, P1 and P2 as code commits (harness separately from the rule, so the rule commit stays one file plus its tests), and the records as a docs commit. Push the branch and update the draft PR body with the C1–C3 tables. Note in the parent task (`issues.md` W9-01) the branch and commits. | Pushed and verified. | - -## 4. Verification matrix - -Environment: `/data/ycfeng/envs/frontier-py310/bin/python` (version recorded -in `run.json`), `PYTHONPATH` = the worktree, `WANDB_DISABLED=true`, -`VIDUR_DISABLE_WANDB=1`. Each Simulator run is a fresh process. Before P1 the -source tree is the base: P0 step (2) confirms that it equals `1f694f7`. - -### 4.1 Common fixture for the synthetic groups (R0, G3a, G3b, G4, G5) - -| Field | Value | -| --- | --- | -| Model | `num_layers=6` (divisible by PP 1, 2, 3), `num_q_heads=4`, `num_kv_heads=2`, `embedding_dim=256`, `mlp_hidden_dim=64`, `max_position_embeddings=4096`, `use_gated_mlp=True`, `use_bias=False`, `use_qkv_bias=False`, SiLU, RMSNorm, `post_attn_norm=True`, `vocab_size=1024`, `torch_dtype="bfloat16"`. MoE: `is_moe=True`, `num_experts=8`, `num_experts_per_tok=2`. Dense: `is_moe=False`. Injected by monkeypatching `BaseModelConfig.create_from_name`, as `tests/integration/test_pr33_nondummy_acceptance.py:170` does. | -| Replica | `device="a100"`, `network_device="a100_pairwise_nvlink"` (4 devices per node), `attn_tensor_parallel_size=1`, `attn_dp` and `num_pipeline_stages` per case, `memory_margin_fraction=0.1`. MoE: `moe_tensor_parallel_size=1`, `moe_expert_parallel_size=attn_dp`, `total_expert_num=8`, `router_topk=2`. | -| Replica scheduler | `VllmV1SchedulerConfig(num_blocks=128, block_size=16, batch_size_cap=4, max_tokens_in_batch=16, enable_chunked_prefill=True)`. With 16-token prompts each prefill batch carries one request. | -| Cluster scheduler, predictor | `RoundRobinClusterSchedulerConfig()`; `RandomForrestExecutionTimePredictorConfig(enable_dummy_mode=True)`. | -| Simulation | `simulation_mode="offline"`, `sys_arch="co-location"`, `enable_parallel_clusters=False`, `decode_cuda_graph_mode="none"`. | -| CC backend | `ClusterConfig.cc_backend_config = AnalyticalCCBackendConfig()` for G3a, G3b, G4 and G5, the public examples' choice (D-6). `analytical` applies no Replica-pod node-size rule, so `attn_dp=2, PP=3` (6 devices) is constructible; P0 confirms this, and a rejection is classified, not substituted. R0 leaves the default (`astra_sim_analytical`, `config.py:2494`) to reproduce the recorded evidence. | -| Arrivals | G3a, G3b, G4, G5: `StaticRequestIntervalGeneratorConfig()` (every request at `t=0`). R0: `PoissonRequestIntervalGeneratorConfig(qps=1e6)`. | -| Lengths | `FixedRequestLengthGeneratorConfig`. Profile **PF** (prefill-only): `prefill_tokens=16, decode_tokens=1`. Profile **PD**: `prefill_tokens=16, decode_tokens=3`. | -| Metrics | `write_metrics=True` (required: the stage ledger is written by `plot()`, which runs only with `write_metrics`), `store_request_metrics=True`, `store_plots=False`, `enable_chrome_trace=False`, `write_json_trace=False`; `store_frontier_stage_batch_ledger` left at its default `True`; `output_dir` = the case directory. | - -G1 recipes run unchanged with `PYTHON_BIN`, `METRICS_OUTPUT_DIR` and `RUN_ID` -overridden per case (each recipe reads these variables). - -### 4.2 Case list - -Case id: `--dp-pp-n`. "Base -hypothesis" is the expectation before P0: `PP=1` succeeds; at `PP>1`, -`n ≥ 2·attn_dp` (every lane holds two or more batches at stage 0) -deadlocks for MoE; otherwise the case succeeds. **P0's classification, not the -hypothesis, assigns each case its path.** Class `admission_deadlock` → path -L; class `success` at `attn_dp>1, PP>1` → path T; `PP=1` or `attn_dp=1` → -path U; any other class in P0 is handled by §4.3. - -| Group | Cases | Profile | Count | Base hypothesis | Path | -| --- | --- | --- | --- | --- | --- | -| R0 record | The 16 shapes of the `design.md` table: 15 author-run logs plus the MoE `dp2-pp3` rejection | PD, Poisson | 16 | As recorded in `design.md` | Evidence only. After P1: informational, classified the same way. | -| G1 release | The 30 `examples/architecture/{co-location,pdd,pd-af-disagg}/{offline,online}/*.sh` recipes (all `PP=1`) | recipe | 30 | success | U | -| G3a MoE, phase-controlled | `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}` × `n ∈ {4, 8, 12}` | PF | 18 | `PP=1`: success. `PP>1`: deadlock, except `dp4-n4`, which is success (one batch per lane). | U / L / T | -| G3b MoE, standard lengths | `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}` × `n ∈ {4, 8}` | PD | 12 | as G3a | U / L / T. A mixed-phase failure after P1 stops and is reported (scope boundary). | -| G4 dense | `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}` × `n ∈ {4, 8}` | PD | 12 | success; at `PP>1` the second lane's first admission waits for the first lane's first stage-0 release | `PP=1`: U. `PP>1`: T. Contention witnesses **W**: `n=8` at `PP ∈ {2,3}`, `attn_dp ∈ {2,4}` (4 cases). | -| G5 single lane | `attn_dp=1`, `PP ∈ {1,2,3}`, MoE (`moe_ep=1`) and dense, `n=6` | PD | 6 | success | U | -| G6 PD-AF | The 10 PD-AF recipes inside G1, plus `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`, `test_decode_ep_wave_materialization.py` and `test_prefill_ep_wave_materialization.py` | — | (in G1) | success; tests pass | U, C4 | -| G2 suites | `tests/unit` and `tests/integration` | — | 2 | base identities from P0 | C4 | -| G7 vLLM-aligned | §4.7 shapes: MoE `Qwen3-30B-A3B-tiny` and dense `Llama-3.2-1B-Instruct`, `attn_dp=2, PP=2`, `n ∈ {8, 16}` | V (256/1) | 4 | MoE: deadlock; dense: success with a delayed second lane | V (§4.7), plus L or T by P0 class | - -Simulator runs: 30 + 18 + 12 + 12 + 6 + 4 = 82, plus 16 R0 record runs and the -two pytest suites. This satisfies the AGENTS.md gate of at least 50 concrete -settings. - -### 4.3 Outcome classes and artifacts - -| Class | Definition | -| --- | --- | -| `success` | `Simulator.run()` returns and every request is completed. | -| `admission_deadlock` | The run ends with "Sequential simulation ended with non-empty scheduler state", and the state report read from the live simulator objects shows this signature: on some stage context, active full-stage owners are fewer than capacity, no EP wave is active, and the bound forward group is unsealed; the FIFO head is a full-stage ticket whose lane is busy; another lane is idle, with a non-empty heap whose head ticket is queued behind that head; and a sync room of the bound group lists the busy lane and waits for the idle one. | -| `configuration_rejection` | A `ValueError` from configuration or topology validation, such as the Replica-pod node-size rule, wherever it surfaces. | -| `other_failure` | Anything else, including a drain without the signature. In P0 this stops the work before P1. | - -Every case directory holds: - -- `case.json`: the case id, group, every fixture field of §4.1, and the - resolved `SimulationConfig` as JSON. -- `run.json`: the command line; interpreter path and `python -VV`; a digest of - `pip freeze`; `git rev-parse HEAD`; whether `git status --porcelain` is clean - outside `task_memory/`; start and end wall time; outcome class; exception - type and message. -- One class artifact: - - `success`: `sha256sums.txt` over every file in the metrics directory. - - `admission_deadlock`: `state_report.json`, with per context the capacity, - sealed flag, bound group, active owners, and the FIFO mapped ticket → lane - → batch id → `global_id`; per lane the busy flag and heap; the sync rooms - with the lanes present; and the simulated time. No metrics hash. - - The two failure classes: `error.txt` with the traceback. - -`cases.jsonl` indexes the cases, one line each. Large outputs stay under the -scratch root; the test report keeps the classification table and the metrics. - -### 4.4 Acceptance paths - -| Path | Applies to | Pass condition | -| --- | --- | --- | -| U unchanged | G1, G5, every `PP=1` cell, G6 recipes | Identical `sha256sums.txt` (run-to-run-unstable files excluded by P0 with a reason). | -| L repaired liveness | Cases P0 classifies as `admission_deadlock` | `success` after P1; completed requests = generated requests; the sums of prefill and decode tokens over `request_metrics.csv` equal the generated lengths. | -| T timing | Cases P0 classifies as `success` with `attn_dp>1, PP>1` | Identical hashes, or a ledger-explained difference (§4.5). W cases: strictly larger co-execution fraction (D-9). | - -### 4.5 Lane-overlap metric (C3) - -Source: `frontier_stage_batch_ledger.jsonl` in the case's metrics directory. -Its rows carry `cluster_type`, `replica_id`, `stage_id`, `execution_scope`, -`replica_local_id`, `stage_start_ts` and `stage_end_ts` -(`metrics_store.py:4401-4422`). No production metrics change. - -For each physical stage `(cluster_type, replica_id, stage_id)`, take the rows -with `execution_scope == "ATTN_DP_LANE"` as half-open intervals -`[stage_start_ts, stage_end_ts)`, keyed by `replica_local_id`. - -| Output | Definition | -| --- | --- | -| `multi_lane_busy_time` | Total simulated time during which at least two distinct lanes have an open interval. Touching endpoints overlap for zero time; zero-length rows contribute nothing. | -| `busy_time` | Total simulated time during which at least one lane has an open interval. | -| Co-execution fraction | `multi_lane_busy_time / busy_time`; over several stages, the sums of both. | -| `peak_lanes` | The largest number of distinct lanes open at one instant. | -| `makespan` | The largest `stage_end_ts` in the ledger. | -| Checks | No lane's intervals overlap one another. `peak_lanes ≤ attn_dp`. | - -A count of overlapping intervals is not used, because it depends on how -intervals are partitioned. For a T difference, the report pairs the metric -before and after with the batch membership (`request_ids` per row) and the -component durations (`execution_time`) of the rows that moved; a changed -aggregate latency alone does not establish the cause. - -### 4.6 Test-identity comparison (G2, C4) - -For `tests/unit` and `tests/integration` separately, run -`python -m pytest -q -p no:cacheprovider --continue-on-collection-errors --junitxml=/.xml` -on the base source (P0) and on P1, in the same environment. Compare the node -id → outcome maps (passed, failed, error, skipped) and the collection errors -per module. Pass: every base-passed node id still passes; no node id newly -fails or errors; collection errors and skips are unchanged. A base failure that -now passes is reported, not treated as a stop. No failure count from another -checkpoint is used. - -### 4.7 vLLM comparison (P5, C7) - -**Why vLLM is a valid reference for this rule.** In vLLM 0.10.2 (the -`vLLM-BS` checkout below): - -| Fact | Source | -| --- | --- | -| With `data_parallel_size > 1`, every DP rank is a `DPEngineCoreProc` with its own scheduler and its own PP workers on its own GPUs. Queued work on one rank cannot refuse admission to another rank's stage. | `vllm/v1/engine/core.py:773-779` | -| At PP > 1 each rank keeps up to PP batches in flight through `step_with_batch_queue`. | `core.py:152-158`, `core.py:364-420` | -| DP ranks at one PP stage meet once per forward in the `DPMetadata` token-count all-reduce, and a rank without runnable work executes a dummy batch, so stage forwards pair one to one across ranks. MoE layers add EP collectives inside the forward. | `vllm/forward_context.py:84,216`; `core.py:1185-1193` | -| A request can be pinned to a DP rank with `data_parallel_rank`. | `vllm/v1/engine/async_llm.py:275`; `vllm/v1/engine/core_client.py:1148` | - -Frontier's attention-DP lanes model those ranks (`AGENTS.md` "vLLM Parallel -Semantics and Frontier Mapping"). The fix claims that a lane with runnable work -is no longer refused at a shared stage by another lane's queued work, so the -comparison measures exactly the behaviours that claim predicts. - -**Shapes.** vLLM DP=2, PP=2, TP=1 on 4×H800; Frontier `attn_dp=2`, -`num_pipeline_stages=2`, `attn_tensor_parallel_size=1`, one Replica, -`RoundRobinClusterSchedulerConfig`, `vllm_v1` replica scheduler. - -| Item | MoE | Dense | -| --- | --- | --- | -| Model config (vLLM `config.json` and Frontier `model_name`) | `data/config/models/Qwen3-30B-A3B-tiny.json`: 8 layers, 16 experts, top-8 | `data/config/models/Llama-3.2-1B-Instruct.json`: 16 layers | -| Expert parallelism | vLLM `enable_expert_parallel` (EP=2 per stage); Frontier `moe_tensor_parallel_size=1`, `moe_expert_parallel_size=2` | — | -| Frontier device | `h800` / `h800_dgx`, dummy predictor, analytical CC backend | same | - -**Workload.** Prompt 256 tokens (distinct token ids per request), one output -token (`max_tokens=1`, `ignore_eos`), so every request completes at its prefill -boundary, as in G3a. Token budget 256 and at most 4 sequences per batch on both -sides, so every batch holds one request. Bursts of `n ∈ {8, 16}` requests; -request `i` goes to lane `i mod 2` (vLLM `data_parallel_rank`; Frontier's -round robin produces the same assignment, `round_robin_cluster_scheduler.py:381-387`, -checked from the ledger). vLLM runs each burst three times after four warmup -requests (two per rank), with the engines idle between rounds; Frontier runs each -burst once (deterministic). Common settings: eager mode, chunked prefill on, -prefix caching off, block size 16, FCFS. Frontier `num_blocks=1024`; vLLM's -`num_gpu_blocks` is recorded. Neither side is expected to preempt; a -preemption on either side is a `MISMATCH`. - -**Evidence.** vLLM runs in the instrumented mode of the calibration contract -(`VLLM_FRONTIER_INSTRUMENTATION=1`, which synchronizes after each forward), -with `VLLM_FRONTIER_PP_BOUNDARY_LOG_PATH` (one row per real forward: request -ids, `pp_rank`, monotonic `forward_start_ts` and `send_start_ts`) and -`VLLM_FRONTIER_DP_PLACEMENT_LOG_DIR` (engine iterations). The driver records the -rank, submit and finish times of every request. No clean E2E run is made: the -Frontier side uses the dummy predictor, so latency is not compared. Frontier -evidence is the stage ledger of the G7 cases. - -**Interval per forward.** vLLM stage 0: `[forward_start_ts, send_start_ts]` -(`send_start_ts` follows the post-forward synchronize). vLLM stage 1: -`[forward_start_ts, timestamp − offset]`, with `offset = time.time() − -time.monotonic()` sampled by the driver before and after each round; stage-1 -metrics are informational. Frontier: `[stage_start_ts, stage_end_ts)` of -`ATTN_DP_LANE` rows. Dummy forwards are not logged by vLLM and have no ledger -row in Frontier; both sides compare real forwards only. - -**Metrics** (same definitions on both sides, per burst and round): - -| Id | Metric | -| --- | --- | -| M1 | Completed formal requests / submitted. | -| M2 | Per `(lane, stage)`, the ordered list of request-id tuples of its forwards. | -| M3 | Stage-0 pairing: for each lane-0 forward, the lane-1 forward with the largest overlap (or none). | -| M4 | First-forward co-start: `|start(lane 0) − start(lane 1)|` of each lane's first stage-0 forward, divided by the median stage-0 forward duration of that run. | -| M5 | Stage-0 co-execution: `multi_lane_busy_time / union_busy_time` over stage-0 intervals (§4.5 definitions). | - -**Pass conditions (C7).** - -| Id | Condition | -| --- | --- | -| V1 | vLLM completes every formal request of every round; after-revision G7 cases complete. MoE negative control: P0 classifies the G7 MoE cases as `admission_deadlock`. If P0 finds them `success`, the aligned shape does not exercise the defect: record it, and V1 rests on the G3a witnesses only. | -| V2 | After-revision M2 equals vLLM M2 in every round. vLLM rounds that disagree with one another are reported, with the cause. | -| V3 | After-revision M3 equals vLLM M3 in every round. A vLLM round in which a dummy forward shifts the pairing is named and reported, not dropped. | -| V4 | vLLM M4 < 0.5 in every round and after-revision M4 < 0.5: both lanes start in the same forward slot. Dense negative control: base M4 ≥ 0.5. | -| V5 | MoE: `|M5(after) − mean M5(vLLM)| ≤ 0.10`. The 0.10 bound reuses the calibration contract's tolerance; it is applied to a fraction, not to latency. Dense: M5 is reported with its start/end decomposition, not gated (D-9). | - -The comparison writes `analysis/workflow_gap_table.csv` (one row per metric, -burst and round, with `MATCH`/`MISMATCH`, values, source and Frontier owner), -`workflow_gap_summary.md` and `workflow_gap_status.json` in the case directory. - -**GPU job.** StepMind Python `RJobBackend`, `i-fengyicheng` personal auth, -`charged_group="codesign"`, `positive_tags=["H800"]`, `gpu=4, cpu=16, -mem_gb=128`, image `artifactory.stepfun-inc.com/docker-public/vllm/vllm-openai:v0.10.2`, -`code_mount_point=/data/ycfeng/Frontier` (covers this worktree and the vLLM -checkout), cloud volume mounted with outputs under -`/mnt/codesign-exp/ycfeng/frontier/stage_admission_pp//`, and the extracted -evidence copied to the case directory. The worker repairs the libcuda loader -path (runbook §10), builds the overlay (the image's installed `vllm` package with -every `vllm/**/*.py` of the checkout copied over it, keeping the image's compiled -extensions), and verifies before the workload that the files where image and -checkout differ are exactly the fork's changes over its upstream base -`01efc7ef7`. Ground truth: `/data/ycfeng/Frontier/.real-engine/vLLM-BS`, -branch `feature/frontier-comparison-instrumentation`, commit `494b9f327`, -clean. Budget: at most two jobs of at most one hour (one run, one retry after -an environment failure). -The retry (R-7) applies `calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch` -to the accepted overlay: the fork passes a fifth `renormalize` argument to -`_moe_C::topk_softmax`, which its own csrc and the image declare with four. The launcher stays alive for the job; no resubmission -while queued. - -Adopted by the user on 2026-09-23 ("采纳你d1-d5的推荐决策"): - -| Id | Question | Recommendation | Outcome | -| --- | --- | --- | --- | -| D-1 | Adopt option B from `design.md` (full-stage tickets are ordered only behind EP waves) rather than option A (lane-aware skip) or C/D. | B. A adds lane identity and a dependency on peer acquisition that no wake covers; C and D are rejected on the working gates. | Adopted. | -| D-2 | Accept that dense `attn_dp>1, PP>1` timelines change (lanes overlap instead of serializing). | Accept as a fidelity fix; the serialization is the same defect. | Adopted. | -| D-3 | Authorize pushing `fix/stage-admission-ordering` and opening a draft PR against `main`. | Grant at P4; until then everything stays local. | Adopted and brought forward: the user reviews on the remote, so the branch is pushed and a draft PR opened with the plan itself (2026-09-23). Code commits follow per package. | -| D-4 | Baseline for byte comparison is `origin/main` `1f694f7`. PR 35 will merge this branch later instead of carrying the fix itself. | Confirm. | Adopted. | -| D-5 | `task_memory/` is ignored by `.gitignore` on `main` (line 171), so these records are local to the worktree unless force-added. Keep them local and archive the outcome in the parent task, or track them on this branch as PR 35 does? | Keep local; copy `summary.md` and the test report into the parent task at P4. | Adopted with one adjustment required by D-3: remote review needs the records on the branch, so `.gitignore` gets the same narrow exception PR 34/35 use (`task_memory/*` plus `!task_memory/task_2026-09-22_stage_admission_ordering/`). The copy into the parent task at P4 stands. | - -Adopted from the round-1 review on 2026-09-23, under the user's instruction -"采纳高价值和必要决策" (`review.md`): - -| Id | Decision | Reason | -| --- | --- | --- | -| D-6 | The synthetic groups select `AnalyticalCCBackendConfig` explicitly and keep `attn_dp=2, PP=3`, instead of substituting `attn_dp=4, PP=3`. | The change is admission-only. The node-size rule belongs to the `collective_sim`/`astra_sim_analytical` backends, and substituting the shape would leave part of C1 untested. | -| D-7 | C1 witnesses come from the phase-controlled prefill-only group G3a. Mixed-phase failures are out of scope: stop, report, diagnose separately. Composition with PR 35 is checked in the parent task. | `main` lacks PR 35 W3; this keeps the admission repair separable from the mixed-phase lifecycle. | - -Adopted under R-6 on 2026-09-23 (execution request; routine choices made from -evidence and recorded here for review): - -| Id | Decision | Reason | -| --- | --- | --- | -| D-8 | The vLLM comparison is structural (M1–M5), in vLLM's instrumented mode, with a prefill-only workload on one MoE and one dense model already in `data/config/models/`. No E2E latency gate. | The fix changes admission, not durations; the Frontier side runs the dummy predictor. Prefill-only keeps the comparison inside the D-7 scope boundary. Both models exist on both sides without new assets. | - -Adopted after the P3 and P5 stops on 2026-09-23 ("采纳你的推荐，继续"; evidence in -`test_report_2026-09-23_stage_admission_ordering.md` §4.3, §5.3): - -| Id | Decision | Reason | -| --- | --- | --- | -| D-9 | (a) A contention witness passes when its co-execution fraction strictly increases; the self-overlap and `peak_lanes` checks are unchanged. (b) V5 gates the MoE shape only; for the dense shape M5 is reported with its start/end decomposition. | (a) At `attn_dp=4` the fix makes all four lanes co-execute and shortens the busy period, so absolute `multi_lane_busy_time` falls (0.55 → 0.30) while overlap becomes complete; the fraction measures overlap independently of that compression. (b) Restated under R-10 (R2-02). vLLM's dense ranks meet once per forward, in the DP metadata all-reduce that runs after `forward_start_ts`. Their stage-0 non-overlap has two sources, neither of them admission: the rank that arrives first records its wait for the other as busy time (start offsets), and per-rank durations vary (end offsets, CV 0.10–0.29). Observed dense M5 is 0.54–0.93 across rounds, wider than 0.10. With both starts of each pair set to the later one (derived from the all-reduce position, not measured), dense M5 is 0.66–0.98 and MoE 0.988–0.994; the dense residual is the end offsets. The dummy predictor models neither source; admission is covered by V1–V4. MoE ranks stay aligned by in-forward EP collectives (M5 0.93–0.98). | - -## 6. Dependencies and risks - -- The reproduction scripts still live in the session scratchpad - (`w10/probe_main.py`, `w10/repro_main.py`, `w10/drain_state.py`, - `w10/drain_lanes.py`); P0 replaces them with `tests/e2e/stage_admission_matrix.py`. -- Risk: the `PP=1, attn_dp=4` cells could differ through the wake-order - inversion described in `design.md`. That is a stop-and-report condition, - not an automatic acceptance. -- Risk: a `DECODE_FFN` or `DECODE_ATTN` path that queues full-stage tickets from - two schedulers on one context would see admission order change. `design.md` - gives the caller-level reason this is not expected; P2(a′) and G6 measure it. - If either differs, stop and report before adjusting anything. -- Risk: after P1, a G3b case may reach a mixed-phase cohort and fail (scope - boundary). Stop and report; it is not repaired on this branch. -- Risk (P5): at a burst start, a vLLM rank may run a dummy forward before its - first request is visible, which shifts the stage-0 pairing by one forward. - V3 names such a round; the driver submits every request of a burst before - yielding to the event loop to make it unlikely. -- Risk (P5): vLLM stage-0 intervals start before the per-forward DP - all-reduce, so a rank that arrives early records its wait as busy time. M4 - uses start times only, and V5 has the stated 0.10 bound. Observed in P5: - on the dense shape the vLLM ranks' own M5 varies by more than 0.10 between - rounds, so dense V5 is reported, not gated (D-9). -- The parent task's Step 9 resumes only after this branch is merged into `main` - and merged forward into `fix/issue26-correctness-pr`. The parent task then - reruns G3b on that branch, where W3 is present, as the composition check - (C6). - -## 7. Round-2 remediation (R-10) - -Findings are in `review.md` Round 2. R2-06 got no answer on timing, so only -its pre-merge step is recorded (P6 below). - -| Finding | Change | Acceptance | -| --- | --- | --- | -| R2-01, R2-11 | `try_acquire` has one branch per scope. The EP wave leaves the FIFO by `popleft`. A full-stage ticket is found in one pass and deleted by index. A ticket that is not in the FIFO, because it is already active, is refused, as on the base. | A new unit test: re-acquiring an active full-stage ticket returns `False` and leaves the context unchanged. Existing contract tests pass without assertion changes. Every case of set `after` is byte-identical in set `after-r2`. | -| R2-12, R2-13 | Documentation. The class docstring names where full-stage order comes from. The PR body states the capacity-1 contract change. `design.md` records the EP-only-queue variant as considered and deferred. | Review. | -| R2-02 | The D-9 (b) rationale now names both sources. (1) Start offsets: `forward_start_ts` is taken before the per-forward DP all-reduce, so the rank that reaches it first records its wait as busy time. (2) End offsets: per-rank duration variance. `evidence/decompose_co_execution.py` states its identity for overlapping pairs only and counts disjoint pairs. It also reports M5 with both starts of each pair set to the later one, which is when the all-reduce releases both ranks. This is derived from the barrier; the traces carry no post-exchange timestamp. Dense V5 stays reported. | Rerun on runs a and b; values in the test report. No GPU job. | -| R2-04 | `vllm_placement` is `ok` only with no misplaced and no unseen request. | Unit test. | -| R2-05 | The negative controls become their own rows, `N1` (MoE base `admission_deadlock`) and `N4` (dense base M4 ≥ 0.5), each `HOLDS` or `LOST`. `workflow_gap_status.json` gains `negative_control_holds`. V1 and V4 compare vLLM with the after revision only. | Unit test: a base equal to the after revision leaves every V row `MATCH` and every N row `LOST`. The rerun on run b gives PASS with the controls holding. | -| R2-07 | Each matrix child runs in its own session under `--case-timeout` (seconds, default 600). On timeout the whole session is killed and the case is recorded as `other_failure`. | A case made to exceed a small timeout is recorded, and the set completes. | -| R2-08 | The shared `work/` path stays: files that embed the output path must compare byte for byte. `run` instead takes an exclusive lock on the matrix root, and the module docstring says sets run one at a time. | A second concurrent `run` fails at once with the lock message. | -| R2-09, R2-10 | Overlay acceptance compares file sets. `apply_patch` reads an empty hunk line as a trimmed context line, fails on any other unknown line, and accumulates hunks when one file appears in several sections. | Unit tests. | -| R2-14, R2-15 | `probe_main.py` drops its hard-coded path and imports from `PYTHONPATH`. `synthetic_check.py` is replaced by `tests/unit/test_stage_admission_pp_tools.py`. For #35, the C6 shapes are rerun as matrix group R0: `R0-moe-dp2-pp{1,2,3}-n6` and `R0-dense-dp1-pp2-n6` are the probe's configuration, apart from metrics flags and the model name. | Unit tests pass; no absolute path left in the evidence scripts. | -| R2-03 | New groups G8–G11 (below). `build_config` gains `sys_arch`, `simulation_mode` and a Poisson rate. The state report and deadlock signature are read per cluster type. Recipe cases take environment overrides. | Paths U/L/T of §4.4 on the new cells. | - -New groups (R2-03). PDD rejects dense `attn_dp > 1` at configuration -(`config.py` `_validate_replica_config`). PD-AF takes one `attn_dp` for every -role, and `DECODE_ATTN` requires 1, so neither can reach a multi-lane context. -PD-AF is therefore covered by `PP > 1` controls on its capacity-1 `PREFILL` -contexts, which exercise the R2-12 contract change. - -| Group | Cases | Profile, arrivals | Count | Path | -| --- | --- | --- | --- | --- | -| G8 PDD offline | MoE `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}`, `n=8`; dense `attn_dp=1, PP=2`, `n=8` | PD, static | 7 | U / L / T | -| G9 PDD online | the G8 shapes at Poisson 20/s; MoE `attn_dp ∈ {2,4}` × `PP ∈ {2,3}` as `-burst` cells | PD, Poisson or burst | 11 | U / L / T | -| G10 co-location online | MoE (PF) and dense (PD), `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}`, `n=8`, each at Poisson 20/s and as a `-burst` cell; plus MoE and dense `dp2-pp2` at 5/s and 80/s | PF / PD, Poisson or burst | 28 | U / L / T | -| G11 PD-AF `PP > 1` | the dense and MoE PD-AF recipes, offline and online, with `PREFILL_PP=2` | recipe | 4 | U | - -Amendment found while running (R-10). On `main`, `_schedule_batch_mode` -(MONOLITHIC and PREFILL) numbers DP lanes from 0 within each scheduling call, -so online Poisson arrivals, one per call, all land on lane 0; unified DECODE -rotates across calls. This is PR 35's W2 defect, fixed on that branch. The -Poisson cells therefore exercise one lane of a multi-lane context here. The -`-burst` cells (online mode, all requests at `t=0`, one call) reach every -lane and carry the online L/T coverage. After PR 35 merges `main` forward, -its composition check reruns G9 and G10 with lane rotation in place. - -Base runs for G8–G11 use the base rule: the one `frontier/` file differing -from `1f694f7` is swapped in the worktree for the run, and `run.json` records -the modified tree. Set `after-r2` then runs every case on the R2 revision. - -A T difference with the same batches is explained as in §4.4. Where online -arrivals let earlier admission change later batch composition, the -explanation must name the first ledger row that differs and show the base -refusal before it. Otherwise stop and report. - -Verification order: - -```text -R2-01/R2-11 rule refactor and unit test - -> {matrix harness (R2-03, R2-07, R2-08), tools and unit tests (R2-04, R2-05, R2-09, R2-10, R2-15), evidence scripts (R2-02, R2-14)} - -> base runs of G8–G11 -> after-r2 set -> compare (base, after-r2) and identity (after, after-r2) - -> compare_lanes and decomposition reruns -> G2 suites -> records, commits, push, PR body -``` - -**P6 pre-merge step (R2-06, not executed).** Before PR 36 merges, a last -commit drops the `!task_memory/task_2026-09-22_stage_admission_ordering/` -exception and untracks the directory. The archive copy stays in the parent -task. This deletes tracked records, so it runs only on the owner's go-ahead. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md b/task_memory/task_2026-09-22_stage_admission_ordering/progress.md deleted file mode 100644 index d230fb9b..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/progress.md +++ /dev/null @@ -1,107 +0,0 @@ -# Stage admission ordering under pipeline parallelism — Progress - -## Modification History - -| Date | Change | -| --- | --- | -| 2026-09-23 | R-10 executed: commits `1661bf1`, `a8e8d8a`, `e35242f` and records; all checks pass; R2-06 recorded as P6. | -| 2026-09-23 | R-10: round-2 remediation started (plan §7). | -| 2026-09-23 | Round-2 code review posted to PR 36 (15 inline comments, `review.md`); fixes deferred by the owner. | -| 2026-09-23 | P4 completed: branch pushed at `4bcd616`, PR 36 body updated (still draft), W9-01 resolution recorded in the parent task (`4c2d573`). | -| 2026-09-23 | R-8 / D-9 adopted; both comparisons rerun and pass (`aeeca93`); P4 in progress. | -| 2026-09-23 | P0–P3 and P5 executed. Rule committed (`dac4e69`). Two plan stop conditions reached (C3 witness metric, C7 V5 dense); P4 push held for the user. | -| 2026-09-23 | R-6 received: execution started; P5 (vLLM comparison) added to the plan. | -| 2026-09-23 | Round-1 plan review verified against source and applied to `design.md`, `plan.md` and `requirements.md`; `review.md` created; resume prompt updated for round 2. Not executed. | -| 2026-09-23 | Decisions adopted; records pushed for remote review. | -| 2026-09-22 | Created. Worktree and branch created; defect reproduced on `origin/main`; plan and design written for review. No source change. | - -## State - -| Item | State | Evidence | -| --- | --- | --- | -| Worktree `.worktrees/stage-admission-ordering` on `fix/stage-admission-ordering` @ `1f694f7` | completed | `git worktree list` | -| Reproduction on `origin/main`, 15 author-run shapes | completed; P0 republishes them as group R0 from published inputs | `design.md` shape table; logs under `/data/ycfeng/tmp/w10_repro/case_*.log` | -| Drain state dump (FIFO, active owners, lane heaps, sync room) | completed | `design.md` "Observed state at the drain" | -| Root-cause diagnosis and option analysis | completed | `design.md` | -| Plan for review | completed; round-1 review applied | `plan.md`, `review.md` | -| Records published for remote review (`.gitignore` exception, docs commit, push, draft PR) | completed 2026-09-23 | commit and PR recorded below | -| Round-1 plan review (10 findings) verified and applied | completed 2026-09-23 | `review.md`; plan D-6, D-7 | -| P0 evidence and baseline | completed 2026-09-23 | 98 cases classified as designed; rerun hashes stable; see "Execution" | -| P1 rule | completed | `dac4e69` (with P2 tests) | -| P2 tests and base negative controls | completed | `evidence/base_negative_controls.log`; all base outcomes as planned | -| P3 rerun and comparison | completed; the first comparison stopped on the C3 witness rule, passes under D-9 | `test_report_2026-09-23_stage_admission_ordering.md` §4 | -| P5 vLLM comparison | completed; the first analysis stopped on dense V5, passes under D-9 | case `calibration/stage_admission_case_001/`, report §5 | -| P4 records, commit, push | completed 2026-09-23 | "Execution" P4 rows | -| Round-2 code review (R2-01..R2-15) | posted; fixes pending the owner's decision | `review.md` Round 2; https://github.com/NetX-lab/Frontier/pull/36#pullrequestreview-5286523149 | - -## Commands run (2026-09-22) - -```bash -git -C /data/ycfeng/Frontier worktree add -b fix/stage-admission-ordering \ - /data/ycfeng/Frontier/.worktrees/stage-admission-ordering origin/main - -# fresh process per shape; scripts in the session scratchpad w10/ -python repro_main.py {moe|dense} -python drain_state.py 4 # context FIFO / active owners -python drain_lanes.py # lane heaps and sync room -``` - -Interpreter `/data/ycfeng/envs/frontier-py310/bin/python`, `PYTHONPATH` = the -worktree, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`. - -## Publication (2026-09-23) - -| Item | Value | -| --- | --- | -| Branch | `fix/stage-admission-ordering` on `NetX-lab/Frontier`, base `main` `1f694f7` | -| Records commit | `62d25b9` (plan, design, requirements, progress, `.gitignore` exception) | -| Draft PR | https://github.com/NetX-lab/Frontier/pull/36 | -| Reviewer resume prompt | `review_prompt.md` in this directory | -| Round-1 review applied | the commit that adds `review.md` (`git log -- task_memory/task_2026-09-22_stage_admission_ordering/review.md`) | - -## Execution (2026-09-23) - -| Step | Command / action | Evidence | Result | -| --- | --- | --- | --- | -| P0 base set | `python -m tests.e2e.stage_admission_matrix run --set base --jobs 8` at `a054d87` | `/data/ycfeng/tmp/stage_admission_ordering/base/` | G1 30 success; G3a 10 deadlock + 8 success; G3b 6 + 6; G4 12 success; G5 6 success; G7 MoE 2 deadlock, dense 2 success; R0 as `design.md` | -| P0 rerun stability | set `base-rerun`, 4 cases | same | hashes identical; no file excluded | -| P0 pytest | unit and integration with `--junitxml` | `/data/ycfeng/tmp/stage_admission_ordering/base-pytest/` | unit 84 failed / 3644 passed / 49 skipped / 10 errors; integration 11 / 21 skipped / 5 errors | -| P1 | `try_acquire` rule and docstrings | `dac4e69` | 34 passed on the two contract files, no assertion change | -| P2 | tests (a), (a′), (b), (c) | `dac4e69` | all pass after P1 | -| P2 base controls | new tests in a `git archive 799ccb4` export | `evidence/base_negative_controls.log` | 7 failed, 2 passed, each at the planned assertion | -| P3 after set | `run --set after --jobs 8` at `dac4e69` | `/data/ycfeng/tmp/stage_admission_ordering/after/` | 97 success, 1 configuration_rejection (R0 dp2-pp3) | -| P3 compare | `compare --before base --after after` | `compare_base_after.json` | U 50 PASS; L 18 PASS; T 6 PASS, 6 EXPLAIN, 2 STOP (dp4 witnesses) | -| P3 explain | `evidence/explain_t_path.py` | `evidence/p3_t_path_explanation.json` | all 8 differing T cases: same batches and component durations, start times only | -| P3 G2 | unit and integration after-runs | `evidence/g2_*_compare.json` | no regression, no new failure, skips and errors unchanged | -| C6 probe | `evidence/step9_probe/probe_completion.py` | `evidence/step9_probe/` | MoE dp2-ep2-pp2 6/6 (base drains); pp3 W9-02 rejection | -| P5b run a | RJob `exp-0923-022226-151935` | `runs/vllm-instrumented/sa-pp-20260923a/` | dense complete; MoE failed on `_moe_C::topk_softmax` 5 vs 4 args | -| Decision | user: "topk_softmax 统一修复为4 个参数的版本" | `requirements.md` R-7 | recorded overlay patch, checkout unchanged | -| Overlay patch support | `vllm_burst_driver.py overlay --patch`, worker `OVERLAY_PATCH` | `a1b9819`; CPU dry run against an `upstream-v0.10.2` export | accepted; `_custom_ops.py` equals upstream after patch; second application fails loudly | -| P5b run b | RJob `exp-0923-024146-345158` | `runs/vllm-instrumented/sa-pp-20260923b/` | MoE and dense complete, status 0 | -| P5c | `compare_lanes --vllm-run …/sa-pp-20260923b` | `calibration/stage_admission_case_001/analysis/` | 50/52 MATCH; V5 dense n8/n16 MISMATCH (vLLM 0.706/0.865 vs 1.0) | -| D-9 rules | witness by co-execution fraction; V5 gated on MoE only | `aeeca93` | — | -| P3 compare rerun | `compare --before base --after after --output …/compare_base_after_d9.json` | scratch root | U 50 PASS; L 18 PASS; T 6 PASS, 8 EXPLAIN; no STOP | -| P5c rerun | `compare_lanes --vllm-run …/sa-pp-20260923b` | `analysis/` | status PASS: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL | -| Checks after D-9 | P5a synthetic check; P2(c) integration test | — | synthetic planted round still caught by V3/V4; 3 passed | -| P4 push | `git push origin fix/stage-admission-ordering` | remote head `4bcd616` | records `df7868e`, `fc34341`; `4bcd616` force-adds `evidence/base_negative_controls.log`, which the repository-wide `*.log` rule had kept out of the tree | -| P4 PR body | REST `PATCH repos/NetX-lab/Frontier/pulls/36` (`gh pr edit` fails on the retired Projects classic query) | PR 36 | body carries the rule, commits, C1–C4/C7 and C3 tables, R-7/D-9 and open items; body read back identical; still draft | -| P4 parent note | parent `issues.md` W9-01 Resolution, `progress.md`, case manifest decision `W9-01-scope`; `summary.md` and the test report copied to `w9_01_stage_admission_ordering/` (D-5) | `fix/issue26-correctness-pr` `4c2d573`, pushed | PR 35 still draft | - -## Round-2 remediation (R-10, plan §7) - -| Step | Command / action | Evidence | Result | -| --- | --- | --- | --- | -| Records | `requirements.md` R-10, `plan.md` §7 | — | recorded | -| R2-01/R2-11/R2-13 | `try_acquire` one branch per scope; active ticket refused; docstring | `1661bf1` | 181 passed on the three context unit files; the new test's scenario: base `False`, `dac4e69` `ValueError`, now `False` | -| R2-03/R2-07/R2-08 | matrix: `sys_arch`, `simulation_mode`, Poisson rate, recipe env; groups G8–G11; cluster-keyed drain report; `--case-timeout`; set lock | `a8e8d8a` | probe set `r2-probe`: PDD first rejected for missing role replica counts, fixed by one Replica per role; online Poisson cells found on lane 0 only (PR 35 W2), burst cells added; lock and 2 s timeout checked | -| Base for G8–G11 | rule file swapped to `1f694f7` at `a8e8d8a`, `run --set base --group G8 … G11 --jobs 16`, then `git checkout` of the file | scratch `base/` | 12 `admission_deadlock` (G8 4, G9 burst 4, G10 MoE burst 4), 38 success | -| After set | `run --set after-r2 --jobs 16` at `a8e8d8a`, clean | scratch `after-r2/` | 147 success, 1 configuration rejection (R0 dp2-pp3, W9-02) | -| Identity | `after` vs `after-r2` | scratch `identity_after_after-r2.json` | 98/98 identical | -| Compare | `compare --before base --after after-r2` | scratch `compare_base_after-r2.json` | 120 PASS, 12 EXPLAIN, 16 informational, 0 STOP | -| Explain | `explain_t_path.py after-r2 …` | `evidence/r2_t_path_explanation.json` | 12 EXPLAIN: same batches and durations, start times only | -| Tools | compare_lanes N1/N4 rows, placement; driver overlay and patch; tool unit tests; `synthetic_check.py` removed; decomposition and probe scripts | `e35242f` | 9 passed; 7 fail on the `ecff89a` tools | -| Decomposition | `decompose_co_execution.py` on runs a and b | `analysis/co_execution_decomposition_*.json` | no disjoint pair; aligned M5 dense 0.66–0.98, MoE 0.988–0.994 | -| C7 rerun | `compare_lanes --after after-r2` on run b | `analysis/` | PASS, 56 rows, controls hold | -| C6 probe | `probe_completion.py` with `PYTHONPATH` only | scratch `r2/c6_probe/` | 6/6 for both shapes | -| G2 | unit and integration, `--junitxml` | `evidence/r2_g2_*_compare.json` | 0 regressions, 0 new failures, 0 skip changes | -| Records | test report §8, `summary.md`, `design.md`, `review.md`, manifest, workflow-gap summary, plan §7 amendment and D-9 (b) | this commit | — | diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md b/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md deleted file mode 100644 index 6c253e02..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/requirements.md +++ /dev/null @@ -1,103 +0,0 @@ -# Stage admission ordering under pipeline parallelism — Requirements - -## Modification History - -| Date | Change | -| --- | --- | -| 2026-09-23 | R-10: round-2 fixes confirmed; R2-02 per the recommendation; R2-03 extended to PDD, online and PD-AF. | -| 2026-09-23 | R-9: code review of PR 36 posted as inline comments; fixes deferred. | -| 2026-09-23 | R-8: D-9 adopted for the C3 witness rule and dense V5; P4 authorized to continue. | -| 2026-09-23 | R-7: ground-truth `topk_softmax` fixed to the four-argument version for the MoE retry. | -| 2026-09-23 | R-6: execute P0–P4 and validate the fix against vLLM on a GPU worker (package P5). | -| 2026-09-23 | R-5: round-1 plan review verified and applied to the records; execution deferred. | -| 2026-09-23 | R-4: D-1..D-5 adopted; push and draft PR authorized. | -| 2026-09-22 | Created from the Step 9 finding W9-01 in `task_2026-09-21_issue26_correctness_pr`; recorded the user's scope decision and the request for a reviewable plan. | - -## Origin - -Found on 2026-09-22 while probing Frontier boundaries for Step 9 of the Issue 26 -correctness PR (package P1(b)). Recorded there as W9-01 -(`task_memory/task_2026-09-21_issue26_correctness_pr/issues.md`). The defect -predates both stacked PRs; the three files involved are byte-identical to -`origin/main` at `1f694f7`. - -## Requests - -`[Original Request]` (2026-09-22, after the W9-01 report with three options and -a recommendation to fix it as a separate correctness item): - -> 采纳你的推荐，继续 - -`[Original Request]` (2026-09-22, interrupting the first source read): - -> 先给出"共享 admission 排序问题"的修复计划,将具体的计划落地到文档，由我审阅 - -`[Original Request]` (2026-09-23, after the first external review of PR 36 at -`a6ec6a6`; the pasted review is recorded finding by finding in `review.md`): - -> 以下是最新review结果，请你核实每个comments，采纳高价值和必要决策，修复完善docs，暂不执行。 - -`[Original Request]` (2026-09-23, after round 1 was applied and pushed): - -> 按照已有plan执行上述修复（该修复需要和在gpu worker上运行的vllm进行合理的对比验证，确保修改的有效性） - -`[Original Request]` (2026-09-23, during P5, after the first GPU run failed on the -MoE `topk_softmax` ABI): - -> 我先提前决策，避免中断任务：topk_softmax 统一修复为4 个参数的版本 - -`[Original Request]` (2026-09-23, answering the two stops of the test report §6): - -> 采纳你的推荐，继续 - -Quality gates the user repeated for every core-module change in this line of -work, carried over verbatim: - -> 对frontier 核心模块的代码的修改和实现上，确保可读性和可维护，任何引入的修改和实现都应该是高价值的（要么对fidelity有收益，要么与模拟功能直接相关，不可替代），禁止hard-coding，禁止临时补丁，禁止过度防御，禁止冗余性设计和实现，禁止使用ai味命名函数和变量。 - -## Decisions - -| Id | Decision | Source | -| --- | --- | --- | -| R-1 | The defect is fixed as a separate correctness item, not inside the Issue 26 feature branch. Step 9's PP>1 packages stay paused until it lands. | user, 2026-09-22 | -| R-2 | Branch `fix/stage-admission-ordering` from `origin/main` `1f694f7`, worktree `/data/ycfeng/Frontier/.worktrees/stage-admission-ordering`. | agent, under R-1 | -| R-3 | No source change before the user reviews `plan.md` and `design.md`. | user, 2026-09-22 | -| R-4 | Plan decisions D-1..D-5 adopted as recommended. Push the branch and open a draft PR so the review happens on the remote; the reviewer resumes from a prepared prompt. | user, 2026-09-23 | -| R-5 | Verify every review finding against the source, adopt the high-value and necessary corrections into the records (dispositions in `review.md`, new decisions D-6 and D-7 in `plan.md`), and do not execute: no P0 run, no source change. The docs commit is pushed to the draft PR under R-4. | user, 2026-09-23 | -| R-6 | Execute P0–P4 as planned. The fix must also be validated against vLLM running on a GPU worker, in a comparison designed to show whether the change is effective (package P5 in `plan.md`). The request authorizes the GPU job within the standing GPU rules below. | user, 2026-09-23 | -| R-7 | The vLLM ground truth uses the four-argument `topk_softmax` (wrapper and call). Applied as the recorded overlay patch `calibration/stage_admission_case_001/inputs/groundtruth_overlay.patch` on the one retry job; the vLLM-BS checkout is unchanged. | user, 2026-09-23 | -| R-8 | Adopt both recommendations (plan D-9): contention witnesses pass on a strictly larger co-execution fraction; V5 gates MoE only and reports dense. Continue to P4. | user, 2026-09-23 | -| R-10 | Fix the recommended round-2 findings (R2-01, R2-04, R2-05, R2-07 to R2-11, R2-14, R2-15), with R2-12 and R2-13 as documentation. R2-02: restate the D-9 rationale with both sources and quantify the start-offset part without a new GPU job. R2-03: add PDD and online cells, plus PD-AF online if needed. R2-06 had no answer: record the pre-merge step only (plan §7). | user, 2026-09-23 | - -## Constraints carried from the parent task - -- `rm` is authorized; `mv`, destructive overwrites, history rewrites, force - pushes and branch or worktree deletion are not. -- Pushing this branch and opening a draft pull request were authorized on - 2026-09-23 (R-4). Merge, marking ready, force-push and history rewrites remain - unauthorized. -- No `Co-Authored-By: Claude` on commits or pull requests. -- Temporary files under `/data/ycfeng/tmp`; the simulator interpreter is - `/data/ycfeng/envs/frontier-py310/bin/python`. -- Never `cd` into the original repository root; use `git -C` and absolute paths. -- GPU work (R-6): StepMind Python `RJobBackend` only, `i-fengyicheng` personal - auth, `charged_group="codesign"` only (`steptron_ci` paused until the user - allows it again), `positive_tags=["H800"]`, submitted from this machine with - local NFS mounts, launcher kept alive, no resubmission while queued. Cloud - volume access is confined to `/mnt/codesign-exp/ycfeng`. Credential values - stay in restricted files and process environments; never print or record - them, and keep shell tracing off. - -`[Original Request]` R-9 (2026-09-23, after P4): - -> review pr36，将review comments提交到该remote repo的pr36上，暂不执行修复。 - -Outcome: round-2 review recorded in `review.md` and posted to PR 36 as one -`COMMENT` review with 15 inline comments. No source or test change. - -`[Original Request]` R-10 (2026-09-23, after the round-2 review). The question -listed the recommended fixes, two options for R2-02 (recommended: restate D-9 -with both sources and measure the post-synchronization start without a new -GPU job), and asked for the scope of R2-03 and R2-06: - -> 确认，执行上上述修复； R2-02 采纳你的推荐；R2-03需要补充 PDD+online（如果你认为pd-af+online有必要，请一并补充） diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/review.md b/task_memory/task_2026-09-22_stage_admission_ordering/review.md deleted file mode 100644 index 636463df..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/review.md +++ /dev/null @@ -1,106 +0,0 @@ -# Stage admission ordering under pipeline parallelism — Review record - -## Modification History - -| Date | Change | -| --- | --- | -| 2026-09-23 | Round 2 dispositions recorded (R-10): 14 findings applied and verified, R2-06 recorded as pre-merge step P6. | -| 2026-09-23 | Round 2: code review of the implementation at `ecff89a` recorded and posted to PR 36; findings verified, fixes deferred by the owner. | -| 2026-09-23 | Created. First external plan review of PR 36 at `a6ec6a6` recorded; each finding re-checked against `1f694f7` source, with a disposition and the place it was applied. | - -## Round 1: plan review of PR 36 at `a6ec6a6` - -| Item | Value | -| --- | --- | -| Component / phase | Plan and design, before P0; no source change on the branch | -| Reviewer | External review agent, started from `review_prompt.md` | -| Inspected by the reviewer | `stage_execution_context.py`, `replica_stage_schduler.py`, `sync_entry.py`, `base_replica_scheduler.py`, `stage_contexts.py`, `stage_wakeup.py`, `batch_stage_end_event.py`, `replica_stage_schedule_event.py`, `base_event.py`, `round_robin_cluster_scheduler.py`, `metrics_store.py`, `batch_stage.py`, the three admission test files, `AGENTS.md`, and the task records | -| Reviewer's recommendation | Conditional GO for option B. P0 may proceed. Before P1: fix C4 and the matrix outcome classes, name the stage ledger for C3, publish the reproduction inputs, and qualify the capacity-1 and option-A claims. | -| Re-check | Every source anchor below re-read on `1f694f7` in this worktree on 2026-09-23. No simulator was run. | -| Owner instruction | "核实每个comments，采纳高价值和必要决策，修复完善docs，暂不执行" (requirements R-5) | - -### Findings and dispositions - -| # | Reviewer verdict | Re-check against source | Disposition | Applied in | -| --- | --- | --- | --- | --- | -| 1 | Agree with the diagnosis; the cited admission loop is the wrong branch | Confirmed. `base_replica_scheduler.py:893` is the unified `DECODE` loop. The co-location reproduction runs the `MONOLITHIC`/`PREFILL` `else` branch at `:1037-1054`, with the loop at `:1039`. Both loops use the same `num_running_batches < num_stages` bound. | Adopted. Anchor corrected. The shape table is labelled author-reported until P0 republishes it from the case inputs. | `design.md` "The defect" and shape table | -| 2 | Agree with B; make `remove(ticket)` explicit; test both sides of the EP boundary | Confirmed. `try_acquire` ends with `popleft()` at `stage_execution_context.py:337`, which would remove the wrong ticket once a non-head ticket can be admitted. `cancel` already uses `self._ready_fifo.remove(ticket)` (`:456`). | Adopted. The sketch removes the admitted ticket with `remove(ticket)`. P2(a) adds a `full0, full1, wave0, full2` contract test. | `design.md` "Recommended rule"; `plan.md` P1, P2(a) | -| 3 | Needs evidence. At capacity 1 the context API does change. The unchanged-behaviour claim belongs to the callers. | Confirmed on four points:
(a) On an idle capacity-1 context with FIFO `[full0, full1]`, the current rule refuses `full1` and B admits it.
(b) `DenseFFNBatchGroup` takes `global_id = _batch_group_creation_counter` (`round_robin_cluster_scheduler.py:1097,1118`). It gets one full-stage ticket (`:1138`) and is queued on the one full-stage scheduler per replica (`:1100`).
(c) EP child batches share one `EP_WAVE` ticket (`:1052-1057`).
(d) New fact: `enqueue_ep_wave` has no other caller. Queued EP waves therefore exist only on `DECODE_FFN` contexts. On `MONOLITHIC`/`PREFILL`/`DECODE`, `EP_WAVE` is only an active-scope transition of owners that were already admitted. | Adopted. The section is retitled and restated as a caller-level condition, with the API-level change stated explicitly. P2(a′) adds the control with two successive dense FFN groups and a neighbouring EP group. No capacity-1 special case is added. | `design.md` "Where behaviour is expected to stay unchanged"; `plan.md` P2(a′) | -| 4 | Needs evidence. Acquisition emits no wake, but the exact stall trace for option A is not established. | Confirmed on four points:
(a) `BatchStageEndEvent` emits the releasing lane's own retry (`batch_stage_end_event.py:139-146`) before its sibling retries (`:148-158`).
(b) `build_stage_wakeup_events` orders siblings by lane key, not by FIFO position (`stage_wakeup.py:30-32`).
(c) The queue key is `_priority_number = (time, id, event_type)` (`base_event.py:63-64`, `simulator.py:1268`). `BaseEvent.__lt__` (`:66-70`) compares type before id.
(d) A refused attempt returns `[]` (`replica_stage_schedule_event.py` "No batch to schedule" branch). | Adopted. A stays rejected on design grounds. The first-draft trace is now labelled an unverified hypothesis and is not pursued, because B does not depend on it. No acquisition wake-up is added. | `design.md` Options table, row A | -| 5 | Disagree with C4 as written | Confirmed. C4 required that all unit tests pass, while G2 expected a baseline failure set of 84 imported from another checkpoint. | Adopted. C4 is replaced with the reviewer's wording, verbatim. G2 now compares node id → outcome per suite against a fresh run of the base source in the same environment. | `plan.md` C4, §4.6 | -| 6 | Needs evidence. The liveness claims are too broad, and a mixed-phase scope boundary is missing. | Confirmed on three points:
(a) `attn_dp=2, PP=2` with 3 requests completes on main (author-run log), so the shape alone does not imply a drain.
(b) The shared forward across mixed prefill and decode source lanes is PR 35 W3 (`65ed8a7`), which is not on main.
(c) New fact: a `MONOLITHIC` request with `decode_tokens=1` completes at the prefill boundary (`request.py:1286-1293,1379-1384`), so it gives a prefill-only witness. | Adopted. The drain condition is restated as a queued-ticket arrangement. Mixed-phase failures become a stop-and-report boundary. C1 witnesses come from a phase-controlled prefill-only group. Composition with PR 35 is checked in the parent task before Step 9 resumes. | `design.md` "The defect", "Scope boundary"; `plan.md` C1, C6, G3a/G3b, §6 | -| 7 | Disagree with the matrix as written | Confirmed on three points:
(a) G3 labelled every `PP>1` cell "drain → complete".
(b) P3 allowed differences only in C3.
(c) DP2/PP3 was replaced by DP4/PP3.
New fact: the node-size rule (`parallel_semantics.py:236-262`) is applied only when materializing `collective_sim` (`cluster.py:203`) and `astra_sim_analytical` (`cluster.py:267`). `analytical` has no such rule, so `attn_dp=2, PP=3` on 6 devices is constructible there. The earlier probe used the default `astra_sim_analytical` (`config.py:2494`), and there it was rejected (parent W9-02). | Adopted. P0 now classifies outcomes into four classes. Acceptance runs on three separate paths: U unchanged, L repaired liveness, T timing. Synthetic cases set `AnalyticalCCBackendConfig` explicitly. A concrete case list is published. | `plan.md` §4.1-§4.4, P3 | -| 8 | Agree; name the ledger and measure overlap duration | Confirmed. `frontier_stage_batch_ledger.jsonl` rows carry `cluster_type`, `replica_id`, `stage_id`, `execution_scope`, `replica_local_id`, `stage_start_ts` and `stage_end_ts` (`metrics_store.py:4401-4422`). `execution_scope` is `ATTN_DP_LANE` for lane rows outside `DECODE_FFN` (`:1510-1521`). Capture defaults to on (`config.py:1202-1205`). The file is written by `plot()`, which runs only with `write_metrics=True` (`metrics_store.py:62-67,2200-2218`). | Adopted, with the reviewer's metric definition. The fixture sets `write_metrics=True`, because the earlier probe's `write_metrics=False` would have produced no ledger. | `plan.md` C3, §4.5 | -| 9 | Needs evidence. P0 must make the evidence reproducible. P2(b) and P2(c) need correcting. | Confirmed. The reproduction scripts exist only in the session scratchpad. P2(b) covered only the first blocked group. P2(c) said "drains on main" for the dense fixture too. | Adopted. P0 now has an artifact list per case, and DRAINED cases get a state report instead of a hash. P2(b) runs through release and restore into the next group, in both lane orders. P2(c) gives per-fixture base expectations. | `plan.md` P0, P2(b), P2(c), §4.3 | -| 10 | Agree; narrow the queue bound | Confirmed. The per-lane bound comes from the admission loops (`:893`, `:1039`). `DECODE_FFN` queues are fed by M2N groups and have no such bound. | Adopted. The bound is restated for the shared-lane contexts only. | `design.md` "Recommended rule" | - -### Items not adopted, and why - -- **A finite event trace for option A (finding 4, optional).** Not produced. It would require implementing A in order to reject it, and the reviewer states that B does not depend on it. The trace stays labelled unverified. -- **None of the required corrections was declined.** - -### New facts found during the re-check (not in the review) - -1. Queued EP waves exist only on `DECODE_FFN` (see finding 3(d)). On the shared-lane contexts where the defect lives, B's "EP wave queued ahead" clause never fires, and B reduces to "admit any queued full-stage ticket within capacity and seal". -2. Sibling wake-ups follow lane-key order (`stage_wakeup.py:30-32`), not FIFO order. At `PP=1` with `attn_dp ≥ 3`, one release can wake two idle siblings whose tickets are queued in the opposite order. The current rule refuses the first sibling woken. So `PP=1` cells with `attn_dp=4` are expected, not guaranteed, to stay unchanged. A difference there stops the work for diagnosis (plan P3). It is not accepted automatically. -3. The ledger needs `write_metrics=True` (see finding 8). - -### Status after round 1 - -Docs corrected. P0 has not started, per the owner's "暂不执行". The next step is the owner's decision to start P0. - -## Round 2: code review of PR 36 at `ecff89a` - -| Item | Value | -| --- | --- | -| Component / phase | Implementation after P4: the rule, P2 tests, case matrix, vLLM comparison tools, and task evidence scripts; diff `1f694f7..ecff89a` | -| Reviewer | `/code-review` skill, run as a forked review agent | -| Inspected by the reviewer | The full PR diff; three touched unit files run on the branch (180 passed); a short script reproduced R2-01 on both trees | -| Re-check | Each finding re-read against the cited source on `ecff89a`, 2026-09-23. R2-01 was reproduced again here. R2-02 numbers were read from `analysis/co_execution_decomposition_sa-pp-20260923b.json`. R2-09 ordering was checked in Python. | -| Owner instruction | "review pr36，将review comments提交到该remote repo的pr36上，暂不执行修复。" (requirements R-9) | -| Posted | https://github.com/NetX-lab/Frontier/pull/36#pullrequestreview-5286523149 (event `COMMENT`, 15 inline comments on `ecff89a`) | - -### Findings - -| Id | Anchor | Verdict | Finding | -| --- | --- | --- | --- | -| R2-01 | `stage_execution_context.py:351` | confirmed, reproduced | `try_acquire` on an already-active full-stage ticket runs off the scan and `remove` raises `ValueError` ("not in deque"); base returned `False`. The only production caller checks `owns()` first. | -| R2-02 | `compare_lanes.py:43` | confirmed | The D-9 rationale ("duration variance, not admission") is not fully supported. Start offsets exceed end offsets in 3 of 6 dense rounds, and `M5_equal_durations` is 0.667–0.911. The union-minus-overlap identity in `decompose_co_execution.py` overcounts disjoint pairs. | -| R2-03 | `stage_admission_matrix.py:255` | plausible | Before/after cases are offline co-location only. There are no PDD or online cells at `attn_dp>1, PP>1`. | -| R2-04 | `compare_lanes.py:99` | confirmed | `vllm_placement` `ok` ignores `unseen`, so missing placement logs still pass. | -| R2-05 | `compare_lanes.py:182` | confirmed | V1 and dense V4 fold the base negative control into the vLLM MATCH status, so a rerun against a fixed base reports MISMATCH. | -| R2-06 | `.gitignore:173` | confirmed | The task-directory exception publishes records into `main` on merge, reversing `26b490a`. D-5 has no pre-merge removal step. | -| R2-07 | `stage_admission_matrix.py:581` | confirmed | No `subprocess.run` timeout in `_run_one` / `_run_recipe_case`. | -| R2-08 | `stage_admission_matrix.py:445` | confirmed | `work/` is shared across sets, so concurrent sets delete each other's outputs. | -| R2-09 | `vllm_burst_driver.py:71` | confirmed | `differing` (sorted as `Path`) is compared with `expected` (sorted as `str`), so some identical sets are rejected. | -| R2-10 | `vllm_burst_driver.py:95` | confirmed | `apply_patch` skips unknown-tag lines without counting them, and `hunks[target] = []` drops a repeated file's earlier hunks. | -| R2-11 | `stage_execution_context.py:342` | confirmed | The scope branch is repeated and the FIFO is scanned twice (simplification). | -| R2-12 | `test_stage_execution_context.py:136` | plausible | Capacity-1 contexts lose the context-level full-stage insertion order. This should be stated as a contract change, not as "unaffected". | -| R2-13 | `stage_execution_context.py:345` | design note | On shared-lane contexts the FIFO no longer orders admission. `queued_tickets` / `admission_seq` still read as an ordered queue there. | -| R2-14 | `evidence/step9_probe/probe_main.py:15` | confirmed | A hard-coded worktree `ROOT` is put first on `sys.path`, so a #35 rerun would import this tree. | -| R2-15 | `calibration/.../analysis/synthetic_check.py:7` | confirmed | A hard-coded scratch `BASE` bypasses `matrix_root()`, and reusable probes live under `task_memory/` rather than `tests/`. | - -The owner first deferred fixes ("暂不执行修复"), then decided (R-10): -"确认，执行上上述修复； R2-02 采纳你的推荐；R2-03需要补充 PDD+online（如果你认为pd-af+online有必要，请一并补充）". - -### Dispositions (R-10) - -Evidence for each row is in the test report §8. - -| Id | Disposition | Where | Verification | -| --- | --- | --- | --- | -| R2-01 | Fixed: an active ticket is refused, as on the base | `1661bf1` | unit test; `after` vs `after-r2` 98/98 byte-identical | -| R2-02 | Rationale restated with both sources; the script's identity is limited to overlapping pairs; a derived barrier-aligned M5 added. No GPU job. | `e35242f`, `plan.md` D-9 (b), test report §5.3 | reran on runs a and b: no disjoint pair; aligned M5 dense 0.66–0.98, MoE 0.988–0.994 | -| R2-03 | Fixed: G8 PDD offline, G9 PDD online, G10 co-location online, G11 PD-AF `PREFILL_PP=2` (50 cases). Online burst cells added because MONOLITHIC/PREFILL place incremental arrivals on lane 0 on `main` (PR 35 W2). | `a8e8d8a` | 12 more base deadlocks repaired; 0 STOP | -| R2-04 | Fixed | `e35242f` | unit test | -| R2-05 | Fixed: rows N1 and N4, `negative_control_holds` | `e35242f` | unit tests; run b rerun PASS, controls hold | -| R2-06 | Recorded as pre-merge step P6; not executed | `plan.md` §7 | waits for the owner | -| R2-07 | Fixed | `a8e8d8a` | a 2 s timeout recorded as `other_failure` | -| R2-08 | Fixed by a set lock; the shared `work/` path stays for byte identity | `a8e8d8a` | a concurrent run fails at once | -| R2-09 | Fixed | `e35242f` | unit test | -| R2-10 | Fixed | `e35242f` | unit tests | -| R2-11 | Fixed with R2-01 | `1661bf1` | as R2-01 | -| R2-12 | Stated as a contract change | `design.md` round-2 notes, PR body | G1 PD-AF and G11 byte-identical | -| R2-13 | Docstring states the FIFO meaning; the EP-only queue variant deferred | `1661bf1`, `design.md` | review | -| R2-14 | Fixed | `e35242f` | C6 probe 6/6 with `PYTHONPATH` only | -| R2-15 | Replaced by `tests/unit/test_stage_admission_pp_tools.py`; `synthetic_check.py` removed | `e35242f` | 9 passed; 7 fail on the `ecff89a` tools | diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md b/task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md deleted file mode 100644 index e2b954b1..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/review_prompt.md +++ /dev/null @@ -1,34 +0,0 @@ -# Resume prompt for the reviewing agent - -## Modification History - -| Date | Change | -| --- | --- | -| 2026-09-23 | Round 2: the prompt now asks the reviewer to verify the round-1 dispositions in `review.md` and the corrected plan. | -| 2026-09-23 | Created for round 1. | - -Copy everything below the line into the review agent's first message. - ---- - -You are continuing the review of draft PR https://github.com/NetX-lab/Frontier/pull/36 on `NetX-lab/Frontier`, branch `fix/stage-admission-ordering`, base `main` at `1f694f7`. This is round 2. In round 1 you reviewed the plan at `a6ec6a6` and gave a conditional GO for option B, with ten findings. The owner had every finding verified against the source and the records corrected. Nothing was executed: no P0 run, no source change. - -Start with the repository's `AGENTS.md`. Then read, under `task_memory/task_2026-09-22_stage_admission_ordering/`, in this order: `review.md` (your findings, the source re-check, the disposition of each, and three new facts found during the re-check), `design.md`, `plan.md`, `requirements.md` (R-5), `progress.md`. - -Background, briefly. One `StageExecutionContext` (`frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`) owns each physical `(replica, stage)` and is shared by that stage's attention-DP lanes. Today `try_acquire` admits only the strict FIFO head. At `PP > 1` a lane holds several queued tickets while it consumes one, so a busy lane's queued ticket at the head can block an idle lane that its own sync room is waiting for. Option B, adopted as D-1: a full-stage ticket is refused only by an EP wave queued ahead of it; EP waves keep the strict head rule; the admitted ticket leaves the FIFO by `remove(ticket)`. - -What to check, in priority order: - -1. For each of the ten findings in `review.md`: is the disposition faithful to what you asked, and is it applied where the table says? Flag anything weakened, misread or missing. -2. The three new facts in `review.md`. Check each against the source: - - Queued EP waves exist only on `DECODE_FFN`, because `enqueue_ep_wave`'s sole caller is `round_robin_cluster_scheduler.py:1052`. - - Sibling wake-ups follow lane-key order (`stage_wakeup.py:30-32`), which makes `PP=1, attn_dp=4` an expected-unchanged class rather than a guaranteed one. - - The stage ledger is written only with `write_metrics=True`. - - Also check one correction made during the re-check: the first draft's dense "lanes serialized" label is withdrawn, because from source the base already overlaps lanes after the first release. P2(c)'s dense assertion was changed to a same-start condition for that reason. -3. `design.md` "Where behaviour is expected to stay unchanged". Is the caller-level condition correct and sufficient for `DECODE_FFN` and `DECODE_ATTN`, and is it stated with the right strength? -4. `plan.md` §4: fixture, case list, outcome classes and signature, acceptance paths U/L/T, the ledger metric, and the test-identity comparison. Can P0 run from this text alone? Are the base hypotheses and paths consistent with C1–C4? Is any stop condition missing? -5. P2(a), (a′), (b) and (c). Does each test fail on the base for the stated reason and pass after P1? Is any of them redundant with an existing test? -6. Fit with the owner's core-module gates: readability, no hard-coding, no temporary patches, no over-defensive branches, no redundant mechanisms, plain domain names. This covers the planned harness `tests/e2e/stage_admission_matrix.py` as well as the one-file rule change. - -Report findings as a numbered list with a source or record anchor (`path:line`) and a verdict per item (agree / disagree / needs evidence). End with a one-paragraph recommendation on whether P0 may start as written. Do not change source or push; the owner decides. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/summary.md b/task_memory/task_2026-09-22_stage_admission_ordering/summary.md deleted file mode 100644 index 2a56ada2..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/summary.md +++ /dev/null @@ -1,77 +0,0 @@ -# Stage admission ordering under pipeline parallelism — Summary - -## Modification History - -| Date | Change | -| --- | --- | -| 2026-09-23 | R-10: round-2 review remediation (14 findings applied, R2-06 pending as P6); PDD, online and PD-AF cells added. | -| 2026-09-23 | Created at P4: fix, tests, P0–P3 and the vLLM comparison complete under D-9. | - -## Overview - -W9-01: with `attn_dp > 1` and `num_pipeline_stages > 1`, a busy lane's queued -ticket at the head of a stage's ready FIFO refused another lane's runnable -batch. MoE runs drained with requests unfinished (admission deadlock). Dense -runs completed but started the lanes one forward apart. - -The fix (plan D-1, option B) changes one predicate in -`StageExecutionContext.try_acquire`. A full-stage ticket is refused only by an -EP wave queued ahead of it. EP waves keep the strict FIFO-head rule. A -ticket that is already active is refused (round 2, R2-01). - -## Deliverables - -| Item | Path / commit | -| --- | --- | -| Rule refactor (round 2) | `1661bf1`: one branch per scope; an active ticket is refused; docstring on FIFO meaning | -| Rule and P2 tests | `dac4e69`: `frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`, `tests/unit/test_stage_execution_context.py`, `tests/unit/test_shared_forward_group_admission.py`, `tests/unit/test_mixed_layer_decode_ffn_scheduling.py`, `tests/integration/test_stage_admission_pipeline_lanes.py` | -| Case matrix | `tests/e2e/stage_admission_matrix.py` (`a054d87`, `5ade853`, `aeeca93`; round 2 `a8e8d8a`: PDD, online and PD-AF groups G8–G11, case timeout, set lock), 148 cases | -| vLLM comparison | `tests/comparison/stage_admission_pp/{vllm_burst_driver.py,run_vllm_worker.sh,compare_lanes.py}` (`799ccb4`, `a1b9819`, `aeeca93`; round 2 `e35242f`: negative-control rows, placement, overlay and patch fixes) and `tests/unit/test_stage_admission_pp_tools.py` | -| Test report | `test_report_2026-09-23_stage_admission_ordering.md` | -| Calibration case | `calibration/stage_admission_case_001/` (manifest, inputs incl. `groundtruth_overlay.patch`, two vLLM runs, `analysis/`) | -| Evidence | `evidence/` (base negative controls, G2 comparisons, path-T explanations for rounds 1 and 2, tool tests on the old tools, Step 9 probe, co-execution decomposition script) | -| Branch / PR | `fix/stage-admission-ordering`, draft PR https://github.com/NetX-lab/Frontier/pull/36 | - -## Validation (observed) - -| Criterion | Result | -| --- | --- | -| C1 | 18 base admission deadlocks (G3a 10, G3b 6, G7 2) complete with requests and tokens conserved | -| C2 | 50/50 unchanged cases byte-identical (30 release recipes, every `PP=1` cell, G5) | -| C3 | 6 T cases identical. The other 8 change start times only: same batches, same component durations, no self-overlap, `peak_lanes ≤ attn_dp`. All 4 witnesses have a strictly larger co-execution fraction (D-9). | -| C4 | `tests/unit` and `tests/integration`: no regression, no new failure, skips and collection errors unchanged | -| C5 | One predicate plus docstrings; no flag, field, fallback, wake-up or special case | -| C6 | Step 9 probe shape MoE `attn_dp=2, moe_ep=2, PP=2` completes 6/6 (base drains) | -| C7 | vLLM DP=2/PP=2 on 4×H800, run `sa-pp-20260923b`: 50 MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5, D-9). MoE: 26/26 rows MATCH. Dense: V1–V4 MATCH in every round. The base fails its negative controls: MoE deadlock, dense pairing and co-start. | - -Round 2 (R-10, test report §8): - -| Check | Result | -| --- | --- | -| Rule refactor | `after` vs `after-r2`: 98/98 cases byte-identical | -| New groups G8–G11 (50) | PDD offline and online, co-location online, PD-AF `PREFILL_PP=2`: U 18 PASS, L 12 PASS (base deadlocks, conserved), T 12 PASS and 4 EXPLAIN (start times only); 0 STOP | -| G2 | no regression, no new failure, skips unchanged | -| C7 rerun (run b vs `after-r2`) | 56 rows: 50 MATCH, 2 INFORMATIONAL, 4 negative controls HOLDS, 0 MISMATCH | -| Harness and tools | timeout, lock, placement, overlay and patch checks verified; tool tests fail 7/9 on the old tools | - -Decisions taken during execution: - -| Id | Decision | -| --- | --- | -| R-7 | The vLLM ground truth uses the four-argument `topk_softmax`, applied as a recorded overlay patch. The checkout is unchanged. | -| R-8 / D-9 | Witnesses are judged by co-execution fraction. V5 gates MoE only. Dense rationale restated at R-10: pre-exchange wait and duration variance, neither admission. | -| R-10 | Round-2 fixes as recommended; PDD and online cells added, plus PD-AF `PP > 1` controls. | - -## Open and deferred work - -- R2-06 / P6: before PR 36 merges, drop the `.gitignore` exception and - untrack this task directory (the parent task keeps copies). It deletes - tracked records, so it waits for the owner's go-ahead. -- On this branch, online Poisson arrivals reach only lane 0 of MONOLITHIC and - PREFILL contexts (PR 35 W2). PR 35's composition check after the - merge-forward reruns G9 and G10 with lane rotation, as well as G3b. - -- PR 35 (`fix/issue26-correctness-pr`) merges this branch forward after it lands and reruns G3b as the composition check with W3. Only then does Step 9 resume (C6). -- `PP=3` with `attn_dp=2` stays rejected by the node-size rule on the default backends (W9-02), outside this fix. -- vLLM-BS: the fork's Python `topk_softmax` still passes five arguments. So does its test `tests/model_executor/test_enabled_custom_ops.py::test_topk_softmax_wrapper_forwards_renormalize`. The four-argument form was applied only as this case's overlay patch. -- Dense per-rank duration variance, which vLLM shows and the dummy predictor lacks, is an execution-time-model topic. It is not part of admission. diff --git a/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md b/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md deleted file mode 100644 index 267c850f..00000000 --- a/task_memory/task_2026-09-22_stage_admission_ordering/test_report_2026-09-23_stage_admission_ordering.md +++ /dev/null @@ -1,386 +0,0 @@ -# Test report — stage admission ordering (P0–P3, P5) - -## Modification History - -| Date | Change | -| --- | --- | -| 2026-09-23 | R-10 round-2 remediation: §8 added (rule refactor, PDD/online/PD-AF cells, tool fixes); §5.2 and §5.3 restated for the N rows and the barrier-aligned M5. | -| 2026-09-23 | D-9 adopted ("采纳你的推荐，继续"): C3 witnesses judged by co-execution fraction, V5 gated on MoE only. Both comparisons rerun (`aeeca93`); all criteria pass. | -| 2026-09-23 | Created. P0–P3 and P5 executed; two plan stop conditions reached (C3 witness metric at `attn_dp=4`, C7 V5 on the dense shape). P4 push held for the user's decision. | - -## 1. Result - -| Criterion | Result | Section | -| --- | --- | --- | -| C1 repaired liveness | PASS: all 10 G3a `admission_deadlock` cases complete with conservation; so do the 6 G3b and 2 G7 MoE deadlocks. | §4.1 | -| C2 unchanged controls | PASS: 50 of 50 U cases byte-identical. | §4.2 | -| C3 timing change | PASS (D-9): 6 T cases identical, 8 differ. All 8 differences are start times only (same batches, same component durations), with no self-overlap and `peak_lanes ≤ attn_dp`. All 4 contention witnesses have a strictly larger co-execution fraction. The first comparison stopped on the original absolute-overlap rule; see §4.3. | §4.3 | -| C4 existing tests | PASS: no base-passed node regresses, no new failure or error, skips and collection errors unchanged. | §4.4 | -| C5 rule shape | PASS by review: one predicate, docstrings state the contract, no flag, field, fallback, wake-up, PP branch, second queue or capacity-1 case. | §3 | -| C6 Step 9 probe | Informational: MoE `attn_dp=2, moe_ep=2, PP=2` completes 6/6 (base drains). `PP=3` stops on the known W9-02 node-size rejection. | §4.5 | -| C7 vLLM comparison | PASS (D-9): 50 rows MATCH, 0 MISMATCH, 2 INFORMATIONAL (dense V5). MoE matches on all 26 rows, V5 included; dense matches V1–V4 in every round. The negative controls fail on the base as planned. The first comparison stopped on dense V5; see §5.3. | §5 | - -Round 2 (R-10, §8): the rule refactor keeps all 98 cases byte-identical; -50 new PDD, online and PD-AF cells pass (12 more base deadlocks repaired, -0 STOP); G2 shows no regression; the vLLM comparison rerun passes with the -negative controls as separate rows that hold. - -Observed facts are separated from inferences. Inferences are marked -"Inference". - -## 2. Environment and commits - -| Item | Value | -| --- | --- | -| Host | `kun-workspace-vgen2` (CPU) | -| Interpreter | `/data/ycfeng/envs/frontier-py310/bin/python`, Python 3.10.6; distribution digest `ecd50ea8…1902620` for both sets | -| Environment | `PYTHONPATH=`, `WANDB_DISABLED=true`, `VIDUR_DISABLE_WANDB=1`, `TMPDIR=/data/ycfeng/tmp/stage_admission_ordering/pytest-tmp` | -| Base set `base` | run at `a054d87` (harness only; `frontier/` identical to `1f694f7`) | -| After set `after` | run at `dac4e69`, tree clean outside `task_memory/`; 98 cases in 90 s with `--jobs 8` | -| Rule commit | `dac4e69` fix(scheduler): order full-stage admission only behind queued EP waves | -| Harness commits | `a054d87`, `5ade853` (matrix), `799ccb4` (vLLM comparison), `a1b9819` (recorded overlay patch), `aeeca93` (D-9 witness and V5 rules) | -| Scratch root | `/data/ycfeng/tmp/stage_admission_ordering/{base,after,base-rerun,base-pytest,after-pytest,step9_probe}` | - -Commands: - -```bash -python -m tests.e2e.stage_admission_matrix run --set after --jobs 8 -python -m tests.e2e.stage_admission_matrix compare --before base --after after \ - --output /data/ycfeng/tmp/stage_admission_ordering/compare_base_after_d9.json # first run: compare_base_after.json -python -m pytest tests/ -q -p no:cacheprovider --continue-on-collection-errors \ - --junitxml=/.xml # suite in {unit, integration}, base and after -python task_memory/.../evidence/explain_t_path.py -python -m tests.comparison.stage_admission_pp.compare_lanes \ - --vllm-run calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b \ - --before base --after after --output calibration/stage_admission_case_001/analysis -``` - -## 3. P1 rule - -`StageExecutionContext.try_acquire` (`frontier/scheduler/replica_stage_scheduler/stage_execution_context.py`): -an EP wave must be the FIFO head; a full-stage ticket is refused only by an EP -wave queued ahead of it; the admitted ticket leaves the FIFO by -`remove(ticket)`. `_validate_ticket` already rejects a ticket that is neither -queued nor active, so the scan always finds the ticket or an earlier wave. -One file, +21/−7 lines. P1 acceptance: `tests/unit/test_stage_execution_context.py` -and `tests/unit/test_shared_forward_group_admission.py` gave 34 passed with no -assertion change. - -## 4. P2 and P3 - -### 4.0 P2 tests and base negative controls - -The new tests were copied into a `git archive 799ccb4` export (rule as on -`1f694f7`) and run there; the log is `evidence/base_negative_controls.log`. - -| Test | Expected on base | Observed on base | After P1 | -| --- | --- | --- | --- | -| (a) `test_full_stage_ticket_passes_queued_full_stage_work_but_not_a_queued_wave` | fails at first assertion | fails at line 111, `try_acquire(full1)` is False | pass | -| (a) `test_queued_ep_wave_orders_full_stage_work_on_both_sides` | pass | pass | pass | -| (a) `test_idle_single_owner_stage_admits_a_later_queued_full_stage_ticket` | fails | fails at line 141 | pass | -| (a′) `test_decode_ffn_dense_groups_keep_counter_order_around_a_queued_ep_wave` | pass | pass | pass | -| (b) `test_idle_lane_is_admitted_behind_a_busy_lane_queued_ticket[0,1]` | fails at the other lane's first admission | both fail at line 71, `pop_batch_if_not_busy()` is None | pass | -| (c) `test_moe_lanes_complete_every_request[G3a-moe-dp2-pp2-n4, G3a-moe-dp4-pp2-n8]` | `admission_deadlock` | both `admission_deadlock` | pass: (4, 64, 4) and (8, 128, 8) | -| (c) `test_dense_lanes_start_in_the_same_first_forward` | fails only the same-start assertion | fails `0.05 == 0.0`; lane 1 runs `[0, 0.05]`, lane 0 starts at `0.05` | pass: both lanes start at 0.0 | - -### 4.1 C1 — path L (18 cases, all PASS) - -Observed `(requests, prefill tokens, decode tokens)` after P1 equals the -generated workload in every case. - -| Cases | Observed | -| --- | --- | -| G3a `dp2-pp{2,3}-n{4,8,12}`, `dp4-pp{2,3}-n{8,12}` (10) | n4: (4, 64, 4); n8: (8, 128, 8); n12: (12, 192, 12) | -| G3b `dp2-pp{2,3}-n{4,8}`, `dp4-pp{2,3}-n8` (6) | n4: (4, 64, 12); n8: (8, 128, 24); no mixed-phase failure | -| G7 MoE `dp2-pp2-n{8,16}` (2) | (8, 2048, 8); (16, 4096, 16) | - -R0 (informational): the three base deadlocks `moe-dp2-pp2-n4`, `moe-dp2-pp2-n6`, -`moe-dp4-pp2-n8` now succeed; `moe-dp2-pp3-n6` remains -`configuration_rejection` (node-size rule, D-6); the other 12 stay `success`. - -### 4.2 C2 — path U (50 cases, all PASS) - -Byte-identical `sha256sums.txt`: G1 30 recipes (10 PD-AF included), `PP=1` -cells of G3a (6), G3b (4) and G4 (4, `attn_dp=4, PP=1` included), G5 6. - -### 4.3 C3 — path T (14 cases) - -Identical hashes (6): G3a/G3b/G4 `dp4-pp{2,3}-n4` (one batch per lane). - -Differing (8). `evidence/explain_t_path.py` checks, per stage and lane, that -the ordered batch list, the forward duration and the `execution_time` -component ledger are equal before and after; output -`evidence/p3_t_path_explanation.json`. All 8: `same_batches_and_component_durations = true`, -no self-overlap, `peak_lanes ≤ attn_dp`. Differing files are the ledger, -`request_metrics.csv` and `system_metrics.json` only. Stage 0: - -| Case | W | multi-lane time before → after | co-execution fraction before → after | peak lanes | first stage-0 starts before → after | Verdict | -| --- | --- | --- | --- | --- | --- | --- | -| G4-dense-dp2-pp2-n4 | | 0.25 → 0.30 | 0.714 → 1.0 | 2 → 2 | {1: 0, 0: 0.05} → all 0 | EXPLAIN | -| G4-dense-dp2-pp2-n8 | W | 0.45 → 0.50 | 0.818 → 1.0 | 2 → 2 | {1: 0, 0: 0.05} → all 0 | EXPLAIN | -| G4-dense-dp2-pp3-n4 | | 0.108 → 0.216 | 0.333 → 1.0 | 2 → 2 | {1: 0, 0: 0.036} → all 0 | EXPLAIN | -| G4-dense-dp2-pp3-n8 | W | 0.216 → 0.396 | 0.375 → 1.0 | 2 → 2 | {1: 0, 0: 0.072} → all 0 | EXPLAIN | -| G4-dense-dp4-pp2-n8 | W | 0.55 → 0.30 | 0.846 → 1.0 | 2 → 4 | {1: 0, 2: 0.05, 3: 0.10, 0: 0.15} → all 0 | EXPLAIN (first run: STOP) | -| G4-dense-dp4-pp3-n8 | W | 0.396 → 0.216 | 0.846 → 1.0 | 2 → 4 | {1: 0, 2: 0.036, 3: 0.072, 0: 0.108} → all 0 | EXPLAIN (first run: STOP) | -| G7-dense-dp2-pp2-n8 | | 0.36 → 0.48 | 0.60 → 1.0 | 2 → 2 | {1: 0, 0: 0.12} → all 0 | EXPLAIN | -| G7-dense-dp2-pp2-n16 | | 0.84 → 0.96 | 0.778 → 1.0 | 2 → 2 | {1: 0, 0: 0.12} → all 0 | EXPLAIN | - -Why the two witnesses fail the stated rule (observed from the ledgers): at -`attn_dp=4` the base admits the lanes two at a time; after P1 all four lanes -start every forward together. The stage busy period shrinks from 0.65 to 0.30 -(PP=2) and from 0.468 to 0.216 (PP=3), so the time with two or more lanes busy -shrinks with it, although it is now the whole busy period. Request E2E for -`dp4-pp2-n8` drops from 500–700 ms to 300–350 ms with identical batches. -Inference: absolute `multi_lane_busy_time` measures overlap only while the -busy period stays the same length; it cannot express "more overlap" when the -fix compresses the timeline, which happens whenever the base serialized more -than two lanes. The first comparison stopped here with nothing adjusted. -Under D-9 the witness condition is the co-execution fraction, which strictly -increases in all four witnesses (0.818, 0.375, 0.846, 0.846 → 1.0); the rerun -(`compare_base_after_d9.json`) gives U 50 PASS, L 18 PASS, T 6 PASS and -8 EXPLAIN, and no STOP. - -### 4.4 C4 — G2 test identities - -| Suite | Base | After | Regressions | New failures | Skip / collection changes | Only after | -| --- | --- | --- | --- | --- | --- | --- | -| `tests/unit` | 84 failed, 3644 passed, 49 skipped, 10 errors | 84 failed, 3650 passed, 49 skipped, 10 errors | 0 | 0 | none; `ERROR` lines identical | the 6 new P2 unit tests, all passed | -| `tests/integration` | 11 passed, 21 skipped, 5 errors | 14 passed, 21 skipped, 5 errors | 0 | 0 | none; `ERROR` lines identical | the 3 new P2(c) tests, all passed | - -Evidence: `evidence/g2_unit_compare.json`, `evidence/g2_integration_compare.json`; -junit XML under the scratch root. - -### 4.5 C6 — Step 9 boundary probe - -The original `probe_main.py` wraps `BaseClusterScheduler.on_replica_batch_end`, -a seam that exists only on the PR 35 branch, so it raises `AttributeError` -on this branch. `evidence/step9_probe/probe_completion.py` reuses its -`build_config` unchanged (a100, 6 requests, 16/3 tokens, Poisson) and reports -completion, one process per shape. - -| Shape | Base | After | -| --- | --- | --- | -| MoE `attn_dp=2, moe_ep=2, PP=1` | — | 6/6 | -| MoE `attn_dp=2, moe_ep=2, PP=2` | drain, "Sequential simulation ended with non-empty scheduler state" | 6/6 | -| MoE `attn_dp=2, moe_ep=2, PP=3` | — | `ValueError`: collective-sim node-size rule (W9-02, unchanged) | -| dense `attn_dp=1, PP=2` | — | 6/6 | - -Composition with PR 35 W3 stays a parent-task check after merge-forward. - -## 5. C7 — vLLM comparison (P5) - -### 5.1 Ground-truth runs - -| Run | RJob | Result | -| --- | --- | --- | -| `sa-pp-20260923a` | `exp-0923-022226-151935`, codesign, 4×H800, creator `i-fengyicheng`, NFS `100.96.128.195:/data/ycfeng/Frontier` | dense complete; MoE failed in `profile_run`: `_moe_C::topk_softmax() expected at most 4 argument(s) but received 5`. Job `Failed`. | -| `sa-pp-20260923b` | `exp-0923-024146-345158`, same shape and mount | MoE and dense complete; job `Succeeded`; worker status 0 | - -Cause of the run-a failure (observed): fork commit `1109c4f16` changed -`vllm/_custom_ops.py::topk_softmax` and the `vllm_topk_softmax` call in -`fused_moe.py` to pass a fifth `renormalize` argument, but the fork's own -`csrc/moe/torch_bindings.cpp` (unchanged from `upstream-v0.10.2`) and the -v0.10.2 image both declare the four-argument op. The user decided on -2026-09-23: "topk_softmax 统一修复为4 个参数的版本". Run b applies -`inputs/groundtruth_overlay.patch` (SHA-256 `8d476789…3a9c81`) to the accepted -overlay: it restores the upstream four-argument wrapper and call. The worker -records `_custom_ops.py` as byte-identical to the image's after the patch. -Numerics are unchanged: `vllm_topk_softmax` renormalizes in Python after the -call in both versions. The checkout `494b9f327` is not modified. - -vLLM run b: 7 rounds per model (1 warmup + 2 bursts × 3), 152 `pp_boundary` records per -model, no preemption, placement records for every request with none -misplaced; `num_gpu_blocks` 600666 (MoE) and 304854 (dense). - -### 5.2 Workflow-gap table (run b) - -`analysis/workflow_gap_table.csv`, `analysis/lane_metrics.json`, -`analysis/workflow_gap_status.json`. - -| Check | MoE n8 | MoE n16 | Dense n8 | Dense n16 | -| --- | --- | --- | --- | --- | -| V1 completion | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | -| N1 / N4 base control (R-10) | N1 HOLDS: base `admission_deadlock` | N1 HOLDS: base `admission_deadlock` | N4 HOLDS: base co-start 1.0 | N4 HOLDS: base co-start 1.0 | -| V2 lane sequences | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH | -| V3 stage-0 pairing | 3/3 MATCH | 3/3 MATCH | 3/3 MATCH; base pairs 0↔3, 2↔5, …, 6↔none | 3/3 MATCH; base shifted by one forward | -| V4 co-start (vLLM / after / base) | 0.009–0.063 / 0.0 / — | 0.005–0.024 / 0.0 / — | 0.008–0.248 / 0.0 / 1.0 | 0.046–0.171 / 0.0 / 1.0 | -| V5 co-execution (vLLM mean / after / base) | 0.976 / 1.0 / — MATCH | 0.948 / 1.0 / — MATCH | 0.706 / 1.0 / 0.600 INFORMATIONAL (first run: MISMATCH) | 0.865 / 1.0 / 0.778 INFORMATIONAL (first run: MISMATCH) | - -Run a (dense only, same scripts): V1–V4 all MATCH; V5 vLLM mean 0.714 (n8) -and 0.685 (n16): MISMATCH under the first rule, INFORMATIONAL under D-9. - -Until R-10 the base controls were folded into V1 and dense V4, so a base -without the defect would have turned those vLLM rows into MISMATCH (R2-05). -The rerun at R-10 (`--after after-r2`) writes them as rows N1 and N4: 56 rows, -50 MATCH, 2 INFORMATIONAL, 4 HOLDS, 0 MISMATCH; status PASS, -`negative_control_holds = true`, placement ok with 0 unseen requests. - -### 5.3 V5 on the dense shape - -`evidence/decompose_co_execution.py` splits the stage-0 non-overlap of each -pair of overlapping forwards into `|Δstart| + |Δend|` -(`analysis/co_execution_decomposition_sa-pp-20260923{a,b}.json`). The identity -holds only for overlapping pairs; at R-10 the script counts disjoint pairs and -checks its pairing against M3. In every round of runs a and b there is no -disjoint pair and no unpaired forward, and the pairing equals M3. The last -column sets both starts of each pair to the later one (R-10, R2-02): vLLM -0.10.2 without CUDA graphs runs the per-forward DP metadata all-reduce inside -`set_forward_context`, after `forward_start_ts`, so neither rank computes -before the later one arrives. The traces carry no timestamp after that -exchange, so this column is derived, not measured. - -| Shape (run b) | vLLM M5 per round | Σ start offsets (ms) | Σ end offsets (ms) | stage-0 duration median (ms), CV | M5, starts aligned to the later one (derived) | -| --- | --- | --- | --- | --- | --- | -| MoE n8 | 0.977, 0.974, 0.977 | 0.32–0.54 | 0.19–0.20 | 5.3–8.5, 0.07–0.09 | 0.991, 0.991, 0.994 | -| MoE n16 | 0.978, 0.937, 0.928 | 0.70–3.83 | 0.26–0.69 | 5.3–7.5, 0.05–0.07 | 0.994, 0.988, 0.989 | -| Dense n8 | 0.657, 0.851, 0.609 | 1.49–2.63 | 0.26–4.40 | 3.0–3.8, 0.13–0.29 | 0.722, 0.977, 0.713 | -| Dense n16 | 0.926, 0.833, 0.837 | 1.02–3.60 | 0.60–2.81 | 2.6–2.7, 0.10–0.19 | 0.970, 0.972, 0.883 | - -Run a, dense, same columns: observed 0.642, 0.739, 0.760 (n8) and 0.537, -0.752, 0.767 (n16); starts aligned 0.739, 0.843, 0.782 and 0.656, 0.945, -0.950. - -Observed: - -- vLLM's dense M5 varies between rounds more than the V5 bound: 0.537–0.926 - across the 12 dense rounds of runs a and b. The n16 means of the two runs - differ by 0.18. -- In every vLLM round the pairing (V3) and the one-to-one lane sequences (V2) - match the after revision, and the first forwards co-start (V4). -- The non-overlap consists of per-pair start offsets and end offsets. Start - offsets exceed end offsets in 3 of the 6 dense rounds of run b. -- With both starts aligned to the later one, M5 rises in every round (dense - 0.66–0.98, MoE 0.988–0.994). What remains in dense is the end offsets. -- MoE ends align within 0.2–0.7 ms in total. - -Inference (restated at R-10, R2-02): in MoE the EP collectives inside each -forward hold the two ranks together, so vLLM's co-execution is close to -Frontier's 1.0. The dense ranks meet once per forward, in the DP all-reduce. -The dense non-overlap has two sources. First, the rank that reaches the -all-reduce first records its wait as busy time, because `forward_start_ts` -precedes the exchange. Second, the host-bound forwards of about 3 ms vary in -duration per rank. Neither is an admission difference: both ranks enter the -same forward, which is what V1–V4 measure, and they match. The dummy -predictor models neither the wait nor the variation, so Frontier's -co-execution is exactly 1.0 whenever the lanes co-start. The first version of -this paragraph named only the duration variation; the start part was there -too. The dense base (0.600, -0.778) is numerically closer to vLLM only because base serialization removes -overlap; its pairing (V3) and co-start (V4) are wrong in every round. - -`compare_lanes.py` labels every `MISMATCH` with the admission owner -`stage_execution_context.py`; on the evidence above, these two rows belong to -the execution-time model instead. The first comparison stopped here with -nothing adjusted. Under D-9 dense V5 is reported, not gated; the rerun gives -`workflow_gap_status.json` status PASS with 0 mismatches. The P5a synthetic -check, now `tests/unit/test_stage_admission_pp_tools.py` (R-10), still flags -a planted late-lane dense round through V3 and V4. - -## 6. Decisions - -Both stops were resolved by D-9 (`plan.md`), adopted by the user on -2026-09-23 ("采纳你的推荐，继续"): - -1. C3: a contention witness passes on a strictly larger co-execution fraction - `multi_lane_busy_time / busy_time`; the self-overlap and `peak_lanes` checks - are unchanged. -2. C7: V5 gates the MoE shape only; the dense value is reported with the - decomposition of §5.3. C7 rests on V1–V4 for both models, V5 for MoE, and the - base negative controls. - -## 7. Verification limits - -- The Frontier side runs the dummy predictor; no latency or duration - parity is claimed (D-8). Dense co-execution against vLLM is therefore not a - gate (D-9). -- vLLM instrumented mode synchronizes after each forward; stage-1 intervals - use a wall/monotonic offset and are informational. -- The vLLM ground truth runs with one recorded overlay patch (§5.1); the fork - checkout still carries the five-argument call and its fork test - `tests/model_executor/test_enabled_custom_ops.py::test_topk_softmax_wrapper_forwards_renormalize`. -- C6 was measured with a completion-only probe because the boundary seam is - on PR 35; the PR 35 composition check is pending in the parent task. -- Round 2 (§8): on this branch, online Poisson arrivals reach only lane 0 of - MONOLITHIC and PREFILL contexts (PR 35's W2 defect on `main`); online - multi-lane coverage here comes from the `-burst` cells. The derived - barrier-aligned M5 of §5.3 is not a measurement. - -## 8. Round-2 remediation (R-10) - -Scope: `plan.md` §7; findings in `review.md` Round 2. Owner instruction: -"确认，执行上上述修复； R2-02 采纳你的推荐；R2-03需要补充 PDD+online（如果你认为pd-af+online有必要，请一并补充）". - -### 8.1 Commits and commands - -| Commit | Content | -| --- | --- | -| `1661bf1` | R2-01, R2-11, R2-13: `try_acquire` refactor, class docstring, unit test | -| `a8e8d8a` | R2-03, R2-07, R2-08: matrix groups G8–G11, cluster-keyed drain report, case timeout, set lock | -| `e35242f` | R2-02, R2-04, R2-05, R2-09, R2-10, R2-14, R2-15: comparison tools, tool unit tests, evidence scripts | - -```bash -# base for the new groups at a8e8d8a, with stage_execution_context.py replaced by -# its 1f694f7 version for the run (run.json: status "M frontier/.../stage_execution_context.py"), then restored -python -m tests.e2e.stage_admission_matrix run --set base --group G8 --group G9 --group G10 --group G11 --jobs 16 -python -m tests.e2e.stage_admission_matrix run --set after-r2 --jobs 16 # a8e8d8a, clean outside task_memory -python -m tests.e2e.stage_admission_matrix compare --before base --after after-r2 \ - --output /data/ycfeng/tmp/stage_admission_ordering/compare_base_after-r2.json -python task_memory/.../evidence/explain_t_path.py after-r2 /compare_base_after-r2.json \ - task_memory/.../evidence/r2_t_path_explanation.json -python -m pytest tests/ -q -p no:cacheprovider --continue-on-collection-errors \ - --junitxml=/after-r2-pytest/.xml -python -m tests.comparison.stage_admission_pp.compare_lanes \ - --vllm-run calibration/stage_admission_case_001/runs/vllm-instrumented/sa-pp-20260923b \ - --before base --after after-r2 --output calibration/stage_admission_case_001/analysis -python task_memory/.../evidence/decompose_co_execution.py dense # and moe dense -``` - -Environment as §2; interpreter digest `ecd50ea8…` for every set. - -### 8.2 Results per finding - -| Finding | Check | Expected | Observed | Result | -| --- | --- | --- | --- | --- | -| R2-01, R2-11 | `try_acquire` on an active full-stage ticket, capacity 2 | `False`, context unchanged | base rule `False`; `dac4e69` raises `ValueError` ("not in deque"); `1661bf1` `False`, ticket still active, FIFO unchanged | PASS | -| R2-01, R2-11 | the three context unit files | pass, no assertion change | 181 passed | PASS | -| R2-01, R2-11 | set `after` vs `after-r2`, 98 cases | byte-identical | 97 success hash files identical; the configuration rejection has the same error (`identity_after_after-r2.json`) | PASS | -| R2-03 | groups G8–G11, 50 cases (§8.3) | §4.4 paths | U 18 PASS, L 12 PASS, T 16: 12 PASS, 4 EXPLAIN; 0 STOP | PASS | -| R2-07 | `run --case-timeout 2` on a recipe case | `other_failure`, set completes | `"case timeout after 2 s"` after 2 s; no child or simulator process left | PASS | -| R2-08 | a second `run` while one is running | fails at once | `RuntimeError: another set is running under …; sets share work/ and run one at a time`, exit 1 | PASS | -| R2-04, R2-05, R2-09, R2-10, R2-15 | `tests/unit/test_stage_admission_pp_tools.py` | pass; the ecff89a tools fail the new checks | 9 passed; on the ecff89a tools 7 failed, 2 passed (the late-lane round and the unexpected-file rejection, which the old tools already handled) (`evidence/r2_tool_tests_on_ecff89a.txt`) | PASS | -| R2-05 | `compare_lanes` rerun on run b vs `after-r2` | PASS with controls holding | 56 rows: 50 MATCH, 2 INFORMATIONAL, 4 HOLDS, 0 MISMATCH; `negative_control_holds = true` (§5.2) | PASS | -| R2-02 | decomposition rerun on runs a and b | identity stated for overlapping pairs; derived aligned M5 | no disjoint or unpaired forward in any round; pairing equals M3; existing fields unchanged; aligned M5 in §5.3 | done | -| R2-14 | `probe_completion.py` with `PYTHONPATH` only | C6 shapes complete | `moe_dp2_pp2` 6/6, `dense_dp1_pp2` 6/6; the probe's resolved config and `R0-moe-dp2-pp2-n6`'s differ only in `metrics_config` | PASS | -| G2 | `tests/unit`, `tests/integration` vs `base-pytest` | no regression | unit 84 failed / 3660 passed / 49 skipped / 10 errors (base 84 / 3644 / 49 / 10); integration 14 passed / 21 skipped / 5 errors (base 11 / 21 / 5). 0 regressions, 0 new failures, 0 skip changes; new node ids only: 16 unit, 3 integration (`evidence/r2_g2_*_compare.json`) | PASS | -| R2-12, R2-13 | documentation | contract stated | `design.md` round-2 note; PR body | done | -| R2-06 | pre-merge step P6 | recorded, not executed | `plan.md` §7 | open | - -### 8.3 New groups (R2-03) - -| Group | Cases | Base | `after-r2` | Paths | -| --- | --- | --- | --- | --- | -| G8 PDD offline | MoE `attn_dp ∈ {2,4}` × `PP ∈ {1,2,3}`, dense `dp1-pp2`, `n=8`, prefill 16 / decode 3 | 4 `admission_deadlock` (MoE `PP > 1`), 3 success | 7 success | U 3 PASS; L 4 PASS | -| G9 PDD online | the G8 shapes at Poisson 20/s; MoE `dp{2,4}-pp{2,3}` burst | 4 `admission_deadlock` (the burst cells), 7 success | 11 success | U 3 PASS; T 4 PASS, identical; L 4 PASS | -| G10 co-location online | MoE (prefill-only) and dense, `dp{2,4}-pp{1,2,3}`, Poisson 20/s and burst; `dp2-pp2` at 5/s and 80/s | 4 `admission_deadlock` (MoE burst `PP > 1`), 24 success | 28 success | U 8 PASS; T 12 PASS, identical, 4 EXPLAIN (dense burst `PP > 1`); L 4 PASS | -| G11 PD-AF | dense and MoE recipes, offline and online, `PREFILL_PP=2` | 4 success | 4 success | U 4 PASS | - -Every L case conserves requests and tokens (8 requests, 128 prefill tokens; -24 decode tokens for PD cells, 8 for prefill-only cells). The 4 EXPLAIN cases -(`evidence/r2_t_path_explanation.json`) run the same batches with the same -component durations as the base; only start times differ. Their co-execution -fraction goes from 0.818, 0.375, 0.846 and 0.846 to 1.0, `peak_lanes ≤ -attn_dp`, and no lane overlaps itself. These are the same values as the -offline G4 cells of the same shapes. - -Observed while running: in the Poisson online cells, every MONOLITHIC and -PREFILL forward runs on lane 0. PDD DECODE lanes all run. `_schedule_batch_mode` -numbers lanes from 0 within each scheduling call, and online arrivals come one -per call. That is PR 35's W2 defect on `main` (fixed on PR 35, not here). The -burst cells deliver all requests at `t=0` in online mode and reach every lane. -They carry the online L and T coverage on this branch. The Poisson cells show -that a multi-lane context with one live lane is unchanged. - -PD-AF: `DECODE_ATTN` requires `attn_dp = 1`, and PD-AF has one `attn_dp` for -every role, so no PD-AF context has more than one lane. The G11 cells are -capacity-1 `PREFILL` contexts at `PP = 2`, where R2-12's contract change -applies. They are byte-identical, offline and online. diff --git a/tests/e2e/stage_admission_matrix.py b/tests/e2e/stage_admission_matrix.py index 628bd2bb..c872b011 100644 --- a/tests/e2e/stage_admission_matrix.py +++ b/tests/e2e/stage_admission_matrix.py @@ -1,10 +1,9 @@ #!/usr/bin/env python3 """Case matrix for stage admission of attention-DP lanes under pipeline parallelism. -Runs the case list of the stage-admission-ordering plan -(``task_memory/task_2026-09-22_stage_admission_ordering/plan.md`` §4) on the -current source tree and writes, for each case, its inputs (``case.json``), its -run provenance (``run.json``) and one outcome artifact: +Runs the case list of ``build_cases`` on the current source tree and writes, +for each case, its inputs (``case.json``), its run provenance (``run.json``) +and one outcome artifact: * ``success``: ``sha256sums.txt`` over the copied metrics tree; * ``admission_deadlock``: ``state_report.json`` read from the live scheduler