Skip to content

Release endpoint admission while waiting for a slot - #5300

Merged
joelteply merged 10 commits into
canaryfrom
10417f6d/endpoint-admission-lock-order
Oct 23, 2026
Merged

joelteply merged 10 commits into
canaryfrom
10417f6d/endpoint-admission-lock-order

Conversation

@joelteply

@joelteply joelteply commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

A slot waiter retained its endpoint read guard while releasing and later reacquiring its adapter semaphore. A later admission could hold that semaphore behind a queued lifecycle writer, completing a lock cycle. Release the entire admission before waiting on the registered slot-release notification, then reacquire in the existing permit-to-reader order and resolve the current generation pool.

Keep two distinct budgets: only endpoint-readiness waits spend cumulative readiness patience; the existing absolute request deadline bounds the whole admission, including permit and slot waits. Both streaming entrypoints pass these distinct budgets instead of pre-minimizing them.

Five regressions cover the lock cycle, transition draining running turns without draining the adapter queue, replacement-generation pool identity after a 300-second slot wait and a 20-second closed endpoint, readiness patience plus cancellation cleanup, and request deadlines during semaphore and all-pinned slot waits. Explicit cumulative debit across multiple successful outer retries is not separately exercised.

Validation: independent source approval at e992346; current 7adeb78 adds only the review-requested deadline comment correction. Fresh exact-head CI is required. No local compilation beside serving. Earlier 56c83d3 CI passed all four then-existing engine regressions (engine 1150 passed, kernel 1230 passed, core 6188 passed); that does not validate the new budget semantics. Merged #5190 (4b82aa8) is inherited at 01a6a1a with independent delta source approval. Current 014f41b only normalizes indentation after that approval (git diff -w empty). Fresh final-head CI is required. Pre-inheritance 7adeb78 CI also passed, validating the new readiness-budget regressions before the telemetry merge.

Baseline limitation: regression-only 7d1ba97 run 39890937094 failed in the old kernel disk-eviction fixture before engine tests ran. This is not an executed red lock-cycle proof. The lock cycle is independently source-traced; Intel live wedge causation remains unproven and needs owner acceptance after reviewed integration.

@joelteply joelteply changed the title Reproduce endpoint drain blocked by adapter admission lock ordering Release endpoint admission while waiting for a slot Oct 23, 2026
@github-actions github-actions Bot added size: L and removed size: M labels Oct 23, 2026
@joelteply
joelteply marked this pull request as ready for review October 23, 2026 03:03
@github-actions github-actions Bot added size: XL and removed size: L labels Oct 23, 2026
@joelteply

Copy link
Copy Markdown
Contributor Author

Exact-head CI attempt1 at014f41b6a hit the job45-minute limit, confirmed by GitHub check annotation. This is not a test failure: core6004, kernel1230, engine1152 and hands192 tests passed; four installed-release CLI cases passed; doc guards logged2 passed as cancellation arrived. Database and binding-drift steps did not run, so the overall gate remains incomplete.

The measured cause was Azure apt mirror download throughput: dependency setup15m21 versus13s in the previous passing run; cmake14MB took a226.6s log gap and libclang30MB a526.3s gap. One failed-job retry at the same reviewed head is running (attempt2). Its dependency setup completed in13s, recovering the lost budget. No source/timeout changes and no local serving-node build.

@joelteply
joelteply merged commit e73ee56 into canary Oct 23, 2026
13 of 15 checks passed
@joelteply
joelteply deleted the 10417f6d/endpoint-admission-lock-order branch October 23, 2026 05:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant