[None][fix] Guard Python KV receive ownership and publication - #17720
Conversation
|
/bot run --disable-fail-fast --stage-list "A10-PyTorch-1, A10-PyTorch-2, A10-PyTorch-3" |
|
PR_Github #66363 [ run ] triggered by Bot. Commit: |
|
/bot run --stage-list "CPU-Generic-x86-1" |
|
PR_Github #66377 [ run ] triggered by Bot. Commit: |
|
PR_Github #66363 [ run ] completed with state |
905fb53 to
0304618
Compare
|
/bot run --stage-list "CPU-Generic-x86-1" |
|
PR_Github #66385 [ run ] triggered by Bot. Commit: |
|
PR_Github #66377 [ run ] completed with state |
|
PR_Github #66385 [ run ] completed with state
|
chienchunhung
changed the title
|
/bot run --stage-list "CPU-Generic-x86-1" |
2 similar comments
|
/bot run --stage-list "CPU-Generic-x86-1" |
|
/bot run --stage-list "CPU-Generic-x86-1" |
|
PR_Github #66436 [ run ] triggered by Bot. Commit: |
|
PR_Github #66436 [ run ] completed with state
|
|
/bot run --stage-list "CPU-Generic-x86-1" |
|
PR_Github #66556 [ run ] triggered by Bot. Commit: |
|
PR_Github #66556 [ run ] completed with state
|
0a1aaf7 to
4c37734
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #66849 [ run ] triggered by Bot. Commit: |
4c37734 to
3c37cb4
Compare
|
/bot run --stage-list "CPU-Generic-x86-1" |
|
PR_Github #66895 [ run ] triggered by Bot. Commit: |
|
PR_Github #66849 [ run ] completed with state |
|
PR_Github #66895 [ run ] completed with state
|
|
/bot run --stage-list "CPU-Generic-x86-1" |
ea8bdfb to
f27bea1
Compare
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
c79720a to
6242f98
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #71051 [ run ] triggered by Bot. Commit: |
|
/bot run --disable-fail-fast |
|
PR_Github #71067 [ run ] triggered by Bot. Commit: |
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
|
PR_Github #71051 [ run ] completed with state |
|
/bot run --disable-fail-fast |
|
PR_Github #71072 [ run ] triggered by Bot. Commit: |
|
PR_Github #71067 [ run ] completed with state |
|
PR_Github #71072 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #71141 [ run ] triggered by Bot. Commit: |
|
PR_Github #71141 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #71299 [ run ] triggered by Bot. Commit: |
|
PR_Github #71299 [ run ] completed with state |
5 participants
Summary
DBR-PR1 — receive-side ownership foundation. This PR adds the dormant GEN-side physical-ownership and publication primitive needed for safe, deadline-bounded KV-transfer retirement.
Today a Python KV receive can become logically failed or cancelled while a remote writer or local CUDA operation may still access its destination. That can allow premature cleanup, republish work after cancellation, lose ownership during partial publication, or tear down registered resources during shutdown.
This PR separates logical outcome from physical liveness. It remains default-off:
enforce_physical_ownershipdefaults toFalse, and this PR adds no public configuration or production activation.Safety boundary
When ownership enforcement is enabled, destination reuse is allowed only when:
Failure, cancellation, timeout, or incomplete evidence does not prove quiescence. Writer identities are sealed explicitly for identified cohorts; the qualified immutable/no-retry ADP path may instead seal the selected group by its expected writer count.
flowchart LR U["Destination unpublished"] --> G{"Cancellation / publication gate"} G -- "cancel wins" --> S["SAFE_UNPUBLISHED"] S --> R["Release reservation"] G -- "publication wins" --> C["Seal authorized writers"] C --> P["Publish REQUEST_DATA"] P --> A["Destination may be accessed"] A --> T{"All writers settled<br/>and local work complete?"} T -- "no" --> H["Retain session and allocation"] T -- "yes" --> D["Owner drained<br/>reuse may proceed"]Implementation
resources_drained()predicate succeeds.Validation
The first commit,
fc756e735, contains the three deterministic motivating regressions. Historical RED evidence from a test-only predecessor is PR_Github #66895 / pipeline #54452; it is motivation evidence, not patch-equivalent evidence for the current first commit.At current head
bff832cf1:CPU-Generic-x86-1andtests/unittest/disaggregatedpassed in the exact-head full-CI sequence.The focused module contains 20 ownership regressions covering writer failure, pre-publication cancellation, publication ordering, partial publication, cross-rank drain consensus, shutdown retention, and immutable/no-retry ADP group sealing. Three existing test modules provide compatibility coverage.
Deferred
This PR does not add CTX source/backend-handle ownership, a quiescence deadline, endpoint fail-close/restart, retry or reroute identity, broad topology/backend qualification, in-flight cancellation, or C++ parity. Follow-up #18041 adds the narrow CTX-side bridge and qualified activation; this PR alone changes no production deployment behavior.