[probe] Disable wait-continuation dedupe by shalabhc · Pull Request #3748 · vercel/workflow

shalabhc · 2026-08-22T02:44:07Z

Not for merge — diagnostic probe

Gives every wait-continuation message a unique idempotency key, so no enqueue is ever collapsed.

A key is still attached rather than omitted, because absence is not neutral: some worlds (world-postgres) serialize key-less workflow messages per run, which would park the continuation behind the handler's own inline step execution. This removes only the collapsing property.

Theory under test

A pending wait is re-encountered on every replay — the runtime is event-sourced, so the workflow body re-executes from the top each time anything wakes the run, and sleep() re-registers the same wait under a replay-stable correlation id. The idempotency key exists so those repeated passes collapse to one delayed message instead of N.

The suspected defect is not that it dedupes, but that the key's lifetime is tied to the queue's retention TTL rather than to the lifetime of the message it suppresses. It should mean "don't enqueue a second copy while one is outstanding"; it actually means "never accept this key again for ~24h". Those agree until the message is consumed — after which nothing is left to protect, but the key stays burnt. A handler that then finds the wait still pending tries to schedule the replacement and is silently refused, with no error, because refusing a duplicate is correct queue behaviour.

That matches the failure signature on main exactly: wait_created with no wait_completed, Status: running, no errors, no handler-failure logs, normal queue metrics.

Expected outcome

Duplicate continuations should be harmless. Replays are idempotent, and the "complete elapsed waits" pass skips waits that already have a wait_completed, so a redundant continuation costs an invocation and does nothing else.

Stalls disappear → the burnt key was losing valid wakeups; the fix is to scope key lifetime to the outstanding message rather than the retention window.
Stalls persist → dedupe is not the mechanism and the loss is further down in delivery.

Costs and risks, since this is not free

More invocations. Each re-observation now enqueues its own continuation, so one wait can fan out into several replays firing near the same deadline. Expected, and the point of the trade.
The delivery-attempt runaway guard is weakened. The module notes each fresh message resets it. Without dedupe there is no ceiling on fresh messages for one wait, so a pathological run could re-enqueue without the guard tripping. Acceptable for a probe, disqualifying for merge as-is.
More concurrent replays of the same run, which raises exposure to any ordering bug that concurrency provokes. Could shift the failure mode rather than remove it — worth reading the results with that in mind.

Tests

wait-continuation.test.ts asserted the dedupe behaviour directly, so its key assertions are inverted here (PROBE: prefixed) and the delay assertions kept. 10/10 pass. Without this the PR would be red for a reason unrelated to the experiment.

wait-completion-replay.test.ts and suspension-handler.test.ts fail with TypeError: globalSingleton is not a function — verified identical on clean main, so pre-existing and unrelated.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview, v0	Aug 22, 2026 2:47am
example-nextjs-workflow-webpack	Ready	Preview, v0	Aug 22, 2026 2:47am
example-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workbench-astro-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workbench-express-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workbench-fastify-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workbench-hono-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workbench-nestjs-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workbench-nitro-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workbench-nuxt-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workbench-python-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workbench-sveltekit-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workbench-tanstack-start-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workbench-vite-workflow	Ready	Preview, v0	Aug 22, 2026 2:47am
workflow-docs	Ready	Preview, v0	Aug 22, 2026 2:47am
workflow-swc-playground	Ready	Preview, v0	Aug 22, 2026 2:47am
workflow-tarballs	Ready	Preview, v0	Aug 22, 2026 2:47am
workflow-web	Ready	Preview, v0	Aug 22, 2026 2:47am

github-actions · 2026-08-22T02:44:24Z

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (47 failed)

astro-node (2 failed):

parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race | wrun_41M0KNWDQ50GZPDA09D96K7JE4 | 🔍 observability
startFromWorkflow - calling start() directly inside a workflow function with hook communication | wrun_41M0KNYNNR0GH7FMF5WRH94CTN | 🔍 observability

example-node (4 failed):

outputStreamWorkflow negative startIndex (reads from end)
hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41M0KNXYZN0GSVTQMMT6H95AYD | 🔍 observability
closureVariableWorkflow - nested step functions with closure variables | wrun_41M0KNYEPV0GTC6D7QP5MY7M53 | 🔍 observability
errorSubclassRoundTripWorkflow - first-class Error subclasses survive every serialization boundary | wrun_41M0KP05N10GS2HWXRVRDCD3GC | 🔍 observability

express-node (4 failed):

outputStreamWorkflow no startIndex (reads all chunks)
error handling retry behavior RetryableError respects custom retryAfter delay
error handling catchability step throw round-trips FatalError with cause chain to workflow catch
hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0KNWD9N0GRV6TMWE9ZF9R2G | 🔍 observability

express-quickjs (4 failed):

outputStreamWorkflow no startIndex (reads all chunks)
outputStreamWorkflow - getTailIndex and getChunks getTailIndex returns correct index after stream completes
outputStreamWorkflow - getTailIndex and getChunks getChunks returns same content as reading the stream
outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_41M0KNT9160GT8YWJP8QZA6P0H | 🔍 observability

hono-node (1 failed):

outputStreamWorkflow positive startIndex (skips first chunk)

nest-node (1 failed):

error handling catchability workflow throw of a non-Error value round-trips verbatim as cause

nest-quickjs (2 failed):

outputStreamWorkflow no startIndex (reads all chunks)
crossContextSerdeWorkflow - classes defined in step code are deserializable in workflow context | wrun_41M0KNZTWD0GKRGDCNMXC6A21P | 🔍 observability

nextjs-turbopack-quickjs (1 failed):

error handling error propagation workflow errors cross-file imports preserve message and stack trace

nextjs-webpack-node (5 failed):

sleepWinsRaceWorkflow | wrun_41M0KNSHN90GP3Y2DK8PQF66Y3 | 🔍 observability
outputStreamWorkflow negative startIndex (reads from end)
outputStreamWorkflow - getTailIndex and getChunks getChunks returns same content as reading the stream
outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_41M0KNT9160GT8YWJP8QZA6P0H | 🔍 observability
error handling serialization failures step-argument serialization failure is catchable in workflow code

nextjs-webpack-quickjs (1 failed):

hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41M0KNXYZN0GSVTQMMT6H95AYD | 🔍 observability

nitro-quickjs (4 failed):

hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0KNWX9H0GQN9EWWHT8S3NB4 | 🔍 observability
hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41M0KNXYZN0GSVTQMMT6H95AYD | 🔍 observability
hookTokenReuseLoopWorkflow - same run recreates a hook with the same token after dispose() | wrun_41M0KNXX5T0GSYTWHZBS0G7MCV | 🔍 observability
closureVariableWorkflow - nested step functions with closure variables | wrun_41M0KNYEPV0GTC6D7QP5MY7M53 | 🔍 observability

nuxt-node (1 failed):

concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0KNW8D40GHTGM1N3XKKK1VW | 🔍 observability

python-node (8 failed):

promiseAllWorkflow | wrun_41M0KNTA1V0GQQ53PKE7Y0KXE2 | 🔍 observability
sleepingWorkflow | wrun_41M0KNS9QG0GM2B8Z7WVNXGRAC | 🔍 observability
parallelSleepWorkflow | wrun_41M0KNSGDZ0GNWN2BBT8S0VY90 | 🔍 observability
nullByteWorkflow | wrun_41M0KNSREV0GMME8GPVQXM6CEY | 🔍 observability
cancelRun - cancelling a running workflow | wrun_41M0KP07FD0GTSRECA2WPK0YCX | 🔍 observability
cancelRun via CLI - cancelling a running workflow | wrun_41M0KP0CSE0GH3QRH486VXXN46 | 🔍 observability
sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0KP0MSE0GNYH89503R4P0JT | 🔍 observability
resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0KP18WY0GSF5SGTWRGYY5A4 | 🔍 observability

sveltekit-node (2 failed):

DurableAgent e2e core basic text response
outputStreamWorkflow no startIndex (reads all chunks)

sveltekit-quickjs (1 failed):

setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

tanstack-start-node (2 failed):

error handling catchability FatalError can be caught and detected with FatalError.is()
AbortController abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook

tanstack-start-quickjs (3 failed):

outputStreamWorkflow - getTailIndex and getChunks getTailIndex returns correct index after stream completes
hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0KNWD9N0GRV6TMWE9ZF9R2G | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0KNW8D40GHTGM1N3XKKK1VW | 🔍 observability

vite-node (1 failed):

thisSerializationWorkflow - step function invoked with .call() and .apply() | wrun_41M0KNZF6V0GNNDTBAPASGSJFV | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0KP3GWG55MXZKQSE6T73VGQ
promiseAllWorkflow | wrun_41M0KNTA1V0GQQ53PKE7Y0KXE2
sleepingWorkflow | wrun_41M0KNS9QG0GM2B8Z7WVNXGRAC
parallelSleepWorkflow | wrun_41M0KNSGDZ0GNWN2BBT8S0VY90
nullByteWorkflow | wrun_41M0KNSREV0GMME8GPVQXM6CEY
cancelRun - cancelling a running workflow | wrun_41M0KP07FD0GTSRECA2WPK0YCX
cancelRun via CLI - cancelling a running workflow | wrun_41M0KP0CSE0GH3QRH486VXXN46
sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0KP0MSE0GNYH89503R4P0JT
resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0KP18WY0GSF5SGTWRGYY5A4

vercel-http-transport (2 failed)

express (1 failed):

outputStreamWorkflow negative startIndex (reads from end)

hono (1 failed):

outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_41M0KNT9160GT8YWJP8QZA6P0H

vercel-ws-transport (8 failed)

express (3 failed):

'hookGetConflictWithParallelStepWorkfl…' - hook.getConflict() does not block step execution | wrun_41M0KNWSCF0GR0ZX2B2D4AEN3Q
hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps | wrun_41M0KNWTPE0GKQV5RTSDCHXKEN
hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token | wrun_41M0KNXPDZ0GSJ6B8EQPJ5QMVE

nextjs-turbopack (1 failed):

error handling serialization failures step-argument serialization failure is catchable in workflow code

vite (4 failed):

writableForwardedFromStepWorkflow | wrun_41M0KNTNBY0GHSQHPR5EQR510W
promiseRaceStressTestWorkflow | wrun_41M0KNTP7E0GW8RRWE1DJ0M82R
hookWithSleepFinalStepWorkflow - step only on final payload | wrun_41M0KP0F7Y0GYWN077AGFQGSMH
AbortController abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

97 flaky tests

abortAnyInWorkflowWorkflow: AbortSignal.any composes signals inside the workflow VM (tanstack-start)
abortExternalSignalWorkflow: signal passed as workflow input (astro)
abortExternalSignalWorkflow: signal passed as workflow input (sveltekit)
abortHookOrderingWorkflow [hook-first-hook-first]: hook.then → addEventListener → resumeHook → abort() (nuxt)
abortHookOrderingWorkflow [listener-first-abort-first]: addEventListener → hook.then → abort() → resumeHook (sveltekit)
abortHookOrderingWorkflow [listener-first-abort-first]: addEventListener → hook.then → abort() → resumeHook (tanstack-start)
abortReasonWorkflow: abort reason preserved across boundaries (nextjs-webpack)
abortSurvivesReplayWorkflow: controller state consistent across replay (fastify)
abortThrowIfAbortedWorkflow: throwIfAborted causes FatalError, no retries (tanstack-start)
abortTimeoutWorkflow: timeout cancels long-running step (fastify)
abortViaHookWorkflow: external hook triggers abort on in-flight step (example)
addTenWorkflow (tanstack-start)
AllInOneService.processNumber - static workflow method using sibling static step methods (astro)
basic text response (nextjs-webpack)
Calculator.calculate - static workflow method using static step methods from another class (tanstack-start)
ChainableService.processWithThis - static step methods using this to reference the class (example)
ChainableService.processWithThis - static step methods using this to reference the class (nextjs-webpack)
completes but prepareCall is not applied (GAP) (nextjs-turbopack)
customSerializationWorkflow - custom class serialization with WORKFLOW_SERIALIZE/WORKFLOW_DESERIALIZE (astro)
distributedAbortController - TTL expiration triggers signal (astro)
errorSubclassRoundTripWorkflow - first-class Error subclasses survive every serialization boundary (nitro)
FatalError fails immediately without retries (tanstack-start)
fetchWorkflow (astro)
fibonacciWorkflow - recursive workflow composition via start() (astro)
fibonacciWorkflow - recursive workflow composition via start() (example)
fibonacciWorkflow - recursive workflow composition via start() (tanstack-start)
getChunks returns same content as reading the stream (astro)
getTailIndex returns correct index after stream completes (example)
getterStepWorkflow - getter functions with "use step" directive (express)
getterStepWorkflow - getter functions with "use step" directive (nest)
health check (queue-based) - workflow endpoint responds to health check messages (astro)
hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (sveltekit)
hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (nextjs-webpack)
hookCleanupTestWorkflow - hook token reuse after workflow completion (express)
hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running (hono)
hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running (nitro)
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-webpack)
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nitro)
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook (tanstack-start)
hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (nest)
hookTokenReuseLoopWorkflow - same run recreates a hook with the same token after dispose() (nitro)
hookWorkflow (nextjs-webpack)
hookWorkflow is not resumable via public webhook endpoint (nextjs-webpack)
importMetaUrlWorkflow - import.meta.url is available in step bundles (example)
nullByteWorkflow (nitro)
outputStreamInsideStepWorkflow - getWritable() called inside step functions (example)
outputStreamInsideStepWorkflow - getWritable() called inside step functions (fastify)
outputStreamInsideStepWorkflow - getWritable() called inside step functions (hono)
outputStreamInsideStepWorkflow - getWritable() called inside step functions (nitro)
outputStreamInsideStepWorkflow - getWritable() called inside step functions (nuxt)
parallelSleepWorkflow (fastify)
parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (tanstack-start)
pathsAliasWorkflow - TypeScript path aliases resolve correctly (astro)
pathsAliasWorkflow - TypeScript path aliases resolve correctly (fastify)
positive startIndex (skips first chunk) (astro)
positive startIndex (skips first chunk) (example)
positive startIndex (skips first chunk) (fastify)
positive startIndex (skips first chunk) (nest)
positive startIndex (skips first chunk) (sveltekit)
promiseAllWorkflow (tanstack-start)
promiseRaceStressTestWorkflow (nitro)
promiseRaceWorkflow (example)
promiseRaceWorkflow (nextjs-webpack)
readableStreamWorkflow (nitro)
regular Error retries until success (express)
resume-or-start route pattern - resumeHook retried after start() reaches the new run (astro)
retainedInterleavingWorkflow (astro)
retainedInterleavingWorkflow (express)
single tool call (nextjs-turbopack)
sleepingWorkflow (astro)
sleepingWorkflow (nest)
sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (astro)
sleepWinsRaceWorkflow (hono)
sleepWinsRaceWorkflow (nest)
sleepWithSequentialStepsWorkflow - sequential steps work with concurrent sleep (control) (nextjs-webpack)
spawnWorkflowFromStepWorkflow - spawning a child workflow using start() inside a step (nextjs-webpack)
start: initial attributes are seeded on run creation (example)
start: initial attributes are seeded on run creation (nitro)
step-argument serialization failure is catchable in workflow code (express)
step-argument serialization failure is catchable in workflow code (sveltekit)
step-return-value serialization failure is catchable in workflow code (example)
step-return-value serialization failure is catchable in workflow code (express)
stepFunctionWithClosureWorkflow - step function with closure variables passed as argument (example)
stepFunctionWithClosureWorkflow - step function with closure variables passed as argument (nest)
StepNotRegisteredError fails the step but workflow can catch it (express)
StepNotRegisteredError fails the step but workflow can catch it (fastify)
StepNotRegisteredError fails the step but workflow can catch it (hono)
stepWinsRaceWorkflow (hono)
validation DX: invalid writes throw catchable FatalErrors naming rule and limit (hono)
webhookWorkflow (nuxt)
workflow throw round-trips FatalError + cause through run_failed event (sveltekit)
WorkflowNotRegisteredError fails the run when workflow does not exist (nextjs-webpack)
WorkflowNotRegisteredError fails the run when workflow does not exist (nuxt)
writableForwardedFromStepWorkflow (hono)
writableForwardedFromStepWorkflow (nuxt)
writableForwardedFromWorkflowWorkflow (example)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

33 infra events

cold-start-warmup · suite warmup (python) · at 02:47:20Z · abandoned wrun_41M0KNQH6P0GNABQVEEMM01V7N · (+7 more)
run-pickup-stall · nullByteWorkflow (python) · at 02:47:36Z · abandoned wrun_41M0KNV5ZZ0GRTCCEYVD2CAT32
run-pickup-stall · promiseAllWorkflow (python) · at 02:47:36Z · abandoned wrun_41M0KNV5ZR0GY426R2B6K39M05
run-pickup-stall · parallelSleepWorkflow (python) · at 02:47:36Z · abandoned wrun_41M0KNV5ZX0GS2D6RGGNY73GB3
run-pickup-stall · sleepingWorkflow (python) · at 02:47:36Z · abandoned wrun_41M0KNV5ZX0GS2D6RGGNY73GB2
run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 02:47:37Z · abandoned wrun_41M0KNV6AW0GPVAZ87WQ36R1VG
run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 02:48:09Z · abandoned wrun_41M0KNW62H0GK43YBWGFE218N2
run-pickup-stall · parallelSleepWorkflow (python) · at 02:48:37Z · abandoned wrun_41M0KNX1B20GWSZT0VH35THFEB
run-pickup-stall · nullByteWorkflow (python) · at 02:48:37Z · abandoned wrun_41M0KNX1AP0GW11E98NZD8GE1R
run-pickup-stall · sleepingWorkflow (python) · at 02:48:37Z · abandoned wrun_41M0KNX1AK0GR0F509WC2PDRRF
run-pickup-stall · promiseAllWorkflow (python) · at 02:48:37Z · abandoned wrun_41M0KNX19R0GS4RF03AHB9MV9K
run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 02:49:27Z · abandoned wrun_41M0KNX5CY0GHX7JJWGKEA3SPF
cold-start-warmup · suite warmup (tanstack-start) · at 02:49:32Z · abandoned wrun_01M0KNYBXZGDNNGK17PY81AD9J
cold-start-warmup · suite warmup (python) · at 02:50:38Z · abandoned wrun_01M0KNXJENS8D4GJ163808TFD0 · (+7 more)
run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 02:50:53Z · abandoned wrun_01M0KP17KB6AZ5P7234PY98WBR
run-pickup-stall · promiseAllWorkflow (python) · at 02:50:53Z · abandoned wrun_01M0KP17KB6AZ5P7234PY98WBS
run-pickup-stall · sleepingWorkflow (python) · at 02:50:53Z · abandoned wrun_01M0KP17KGF86D0EA58BTGNAZT
run-pickup-stall · parallelSleepWorkflow (python) · at 02:50:53Z · abandoned wrun_01M0KP17KKHCN8NH8QRSYT8TQZ
run-pickup-stall · nullByteWorkflow (python) · at 02:50:53Z · abandoned wrun_01M0KP17KNFRXF3A8RGV3A120V
run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 02:51:09Z · abandoned wrun_41M0KNZX9F0GW3T5QMKG3AEE5P
run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 02:51:48Z · abandoned wrun_41M0KNZS690GN8Z3VVZ2FJCY6N
run-pickup-stall · promiseAllWorkflow (python) · at 02:51:53Z · abandoned wrun_01M0KP3273346NS65MD3NF7NSQ
run-pickup-stall · sleepingWorkflow (python) · at 02:51:53Z · abandoned wrun_01M0KP3274A76J0J62X0G5K7TM
run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 02:51:53Z · abandoned wrun_01M0KP3271P9B4WJT9K31VMCE6
run-pickup-stall · parallelSleepWorkflow (python) · at 02:51:53Z · abandoned wrun_01M0KP32797FP3B5MAPS3J6450
run-pickup-stall · nullByteWorkflow (python) · at 02:51:53Z · abandoned wrun_01M0KP327K23DSTGGE3R8X7RCP
run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 02:52:53Z · abandoned wrun_01M0KP4WTR8S1Q1306RYS10EZJ
run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 02:52:53Z · abandoned wrun_01M0KP4WV071DBPRB6RAG7935D
run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 02:52:53Z · abandoned wrun_01M0KP4WV31QA1VYZVSRYB3SY8
run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 02:53:23Z · abandoned wrun_01M0KP5T6H0HSP019G43DPDTR6
run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 02:53:23Z · abandoned wrun_01M0KP5T6KQNDK5F1HDET3NEWE
run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 02:53:53Z · abandoned wrun_01M0KP6QE7WQEG50W9VVNMCC56
run-pickup-stall · pathsAliasWorkflow - TypeScript path aliases resolve correctly (nextjs-webpack) · at 02:55:54Z · abandoned wrun_01M0KPADTAJP9VHT12B721RRXR

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
❌ ▲ Vercel Production	3129	47	1144	4320
✅ 💻 Local Development	3922	0	558	4480
✅ 📦 Local Production	3922	0	558	4480
✅ 🐘 Local Postgres	3922	0	558	4480
✅ 🪟 Windows	320	0	0	320
❌ 🌐 Cross-language Conformance	0	9	132	141
❌ vercel-http-transport	815	2	143	960
✅ vercel-multi-region	27	0	0	27
❌ vercel-ws-transport	545	8	87	640
Total	16602	66	3180	19848

Details by Category

❌ ▲ Vercel Production

App	Passed	Failed	Skipped
❌ astro-node	130	2	28
✅ astro-quickjs	0	0	160
❌ example-node	128	4	28
✅ example-quickjs	132	0	28
❌ express-node	128	4	28
❌ express-quickjs	128	4	28
✅ fastify-node	132	0	28
✅ fastify-quickjs	132	0	28
❌ hono-node	131	1	28
✅ hono-quickjs	132	0	28
❌ nest-node	131	1	28
❌ nest-quickjs	130	2	28
✅ nextjs-turbopack-node	19	0	141
❌ nextjs-turbopack-quickjs	156	1	3
❌ nextjs-webpack-node	152	5	3
❌ nextjs-webpack-quickjs	156	1	3
✅ nitro-node	0	0	160
❌ nitro-quickjs	128	4	28
❌ nuxt-node	131	1	28
✅ nuxt-quickjs	132	0	28
❌ python-node	0	8	152
❌ sveltekit-node	149	2	9
❌ sveltekit-quickjs	150	1	9
❌ tanstack-start-node	130	2	28
❌ tanstack-start-quickjs	129	3	28
❌ vite-node	131	1	28
✅ vite-quickjs	132	0	28

✅ 💻 Local Development

App	Passed	Skipped
✅ astro-stable-node	134	26
✅ astro-stable-quickjs	134	26
✅ express-stable-node	134	26
✅ express-stable-quickjs	134	26
✅ fastify-stable-node	134	26
✅ fastify-stable-quickjs	134	26
✅ hono-stable-node	134	26
✅ hono-stable-quickjs	134	26
✅ nest-stable-node	134	26
✅ nest-stable-quickjs	134	26
✅ nextjs-turbopack-canary-node	141	19
✅ nextjs-turbopack-canary-quickjs	141	19
✅ nextjs-turbopack-stable-node	160	0
✅ nextjs-turbopack-stable-quickjs	160	0
✅ nextjs-webpack-canary-node	141	19
✅ nextjs-webpack-canary-quickjs	141	19
✅ nextjs-webpack-stable-node	160	0
✅ nextjs-webpack-stable-quickjs	160	0
✅ nitro-stable-node	134	26
✅ nitro-stable-quickjs	134	26
✅ nuxt-stable-node	134	26
✅ nuxt-stable-quickjs	134	26
✅ sveltekit-stable-node	153	7
✅ sveltekit-stable-quickjs	153	7
✅ tanstack-start-node	134	26
✅ tanstack-start-quickjs	134	26
✅ vite-stable-node	134	26
✅ vite-stable-quickjs	134	26

✅ 📦 Local Production

App	Passed	Skipped
✅ astro-stable-node	134	26
✅ astro-stable-quickjs	134	26
✅ express-stable-node	134	26
✅ express-stable-quickjs	134	26
✅ fastify-stable-node	134	26
✅ fastify-stable-quickjs	134	26
✅ hono-stable-node	134	26
✅ hono-stable-quickjs	134	26
✅ nest-stable-node	134	26
✅ nest-stable-quickjs	134	26
✅ nextjs-turbopack-canary-node	141	19
✅ nextjs-turbopack-canary-quickjs	141	19
✅ nextjs-turbopack-stable-node	160	0
✅ nextjs-turbopack-stable-quickjs	160	0
✅ nextjs-webpack-canary-node	141	19
✅ nextjs-webpack-canary-quickjs	141	19
✅ nextjs-webpack-stable-node	160	0
✅ nextjs-webpack-stable-quickjs	160	0
✅ nitro-stable-node	134	26
✅ nitro-stable-quickjs	134	26
✅ nuxt-stable-node	134	26
✅ nuxt-stable-quickjs	134	26
✅ sveltekit-stable-node	153	7
✅ sveltekit-stable-quickjs	153	7
✅ tanstack-start-node	134	26
✅ tanstack-start-quickjs	134	26
✅ vite-stable-node	134	26
✅ vite-stable-quickjs	134	26

✅ 🐘 Local Postgres

App	Passed	Skipped
✅ astro-stable-node	134	26
✅ astro-stable-quickjs	134	26
✅ express-stable-node	134	26
✅ express-stable-quickjs	134	26
✅ fastify-stable-node	134	26
✅ fastify-stable-quickjs	134	26
✅ hono-stable-node	134	26
✅ hono-stable-quickjs	134	26
✅ nest-stable-node	134	26
✅ nest-stable-quickjs	134	26
✅ nextjs-turbopack-canary-node	141	19
✅ nextjs-turbopack-canary-quickjs	141	19
✅ nextjs-turbopack-stable-node	160	0
✅ nextjs-turbopack-stable-quickjs	160	0
✅ nextjs-webpack-canary-node	141	19
✅ nextjs-webpack-canary-quickjs	141	19
✅ nextjs-webpack-stable-node	160	0
✅ nextjs-webpack-stable-quickjs	160	0
✅ nitro-stable-node	134	26
✅ nitro-stable-quickjs	134	26
✅ nuxt-stable-node	134	26
✅ nuxt-stable-quickjs	134	26
✅ sveltekit-stable-node	153	7
✅ sveltekit-stable-quickjs	153	7
✅ tanstack-start-node	134	26
✅ tanstack-start-quickjs	134	26
✅ vite-stable-node	134	26
✅ vite-stable-quickjs	134	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	160	0	0
✅ nextjs-turbopack-quickjs	160	0	0

❌ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
❌ python	0	9	132

❌ vercel-http-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
❌ express	131	1	28
❌ hono	131	1	28
✅ nextjs-turbopack	157	0	3
✅ nitro	132	0	28
✅ vite	132	0	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

❌ vercel-ws-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
❌ express	129	3	28
❌ nextjs-turbopack	156	1	3
❌ vite	128	4	28

📋 View full workflow run

github-actions · 2026-08-22T02:44:26Z

📊 Workflow Benchmarks

commit 41a01c5 · Sat, 22 Aug 2026 03:05:06 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	375 (+67%) 🔻	1350 🔴 (+21%) 🔻	1380 🔴 (+21%) 🔻	1438 🔴 (-1.6%)	30
TTFS	stream	424 (+90%) 🔻	1396 🔴 (+28%) 🔻	1425 🔴 (+28%) 🔻	1649 🔴 (+35%) 🔻	30
TTFS	hook + stream	716 (+74%) 🔻	1639 🔴 (+28%) 🔻	1703 🔴 (+30%) 🔻	6016 🔴 (+298%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	534 (-12%)	863 (-49%) 💚	928 (-47%) 💚	1717 (-11%)	10
Fan-out TTLS	Promise.all(100 steps)	1991 (+19%) 🔻	4339 (+33%) 🔻	4938 (+43%) 🔻	7963 (+90%) 🔻	10
STSO	1020 steps (inline)	99 (-4.8%)	122 (-12%)	140 (-10%)	211 (-25%) 💚	1019
WO	1020 steps	122305 (-11%)	122305 (-11%)	122305 (-11%)	122305 (-11%)	1
CRTT	first chunk (pooled)	86 (-9.5%)	146 (+3.5%)	183 (+12%)	195 (-56%) 💚	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	140 (+10%)	119 (-62%)	155 (-92%)	305 (-89%)	80.5 (-48%)	10
size sweep (100/s, 160B-12KB)	123 (±0%)	132 (-29%)	344 (+22%)	617 (+32%)	105 (-45%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	174 (+27%)	110 (-37%)	157 (-73%)	326 (-85%)	165 (-21%)	3
replay eve-gpt-5.6-sol-2000t (1x)	134 (-10%)	116 (-33%)	157 (-90%)	342 (-91%)	299 (-46%)	2
replay eve-gpt-5.6-sol-2000t (2x)	130 (-9%)	149 (-57%)	206 (-92%)	576 (-85%)	356 (-63%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 137971ms → this run 122112ms (Δ -15859ms, -11%)

 50-100 ms  ┃                         main   0  this   1    +1
100-150 ms  ██████████████████████░┃  main 880  this 960   +80
150-200 ms  ┃██                       main 108  this  46   -62
200-250 ms  ┃                         main  18  this   7   -11
250-300 ms  ┃                         main   4  this   3    -1
300-350 ms  ┃                         main   3  this   1    -2
350-400 ms  ┃                         main   3  this   1    -2
400-450 ms  ┃                         main   1  this   0    -1
500-550 ms  ┃                         main   2  this   0    -2

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant  RTT 1ms→5s+             avg         p50         p90         p99     n
control  ······█▇▁····  100.6 (-74%)   96 (-29%)  155 (-92%)  305 (-89%)  3000
sweep    ······█▆▁▁···  115.1 (-24%)   93 (-31%)  344 (+22%)  617 (+32%)  3000
gw 1x    ·····▁█▄▁····   98.1 (-46%)   90 (-30%)  157 (-73%)  326 (-85%)  5295
eve 1x   ·····▁█▆▁▁···  103.5 (-62%)   92 (-23%)  157 (-90%)  342 (-91%)  5186
eve 2x   ·····▁▆█▂▁···  126.9 (-63%)  113 (-38%)  206 (-92%)  576 (-85%)  7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control  █▇▄▂▆▃▃▃▂▁  91–114ms
sweep    ▄▃▃▁█▄▇▆▄▁  92–145ms
gw 1x    ▇▅▄▃▃▅█▁▆▃  82–114ms
eve 1x   ▂▂▂▂▁▁▃█▆▂  91–137ms
eve 2x   ▃▁▂▁▁▂▄█▂▁  107–202ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep  ▄▆█▆▆▂▁  112–118ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control  ▁▇█▁█▆▆▁▆▃  28–32ms
sweep    ▁▅▂▄█▂█▂▁▁  36–55ms
gw 1x    ▄▄▄▁▂▄█▁▄▂  24–33ms
eve 1x   ▃▄▂▁▃▁▃█▃▅  19–29ms
eve 2x   ▇▄▃▁▄▄▆▃▁█  18–24ms

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

github-actions · 2026-08-22T02:47:52Z

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario	outcome	events	virt	replay
✅ `smoke-no-steps`	completed	3	0ms	ok
✅ `smoke-one-step`	completed	6	0ms	ok
✅ `hook-at-step-started`	completed	12	0ms	ok
✅ `hook-at-step-completed`	completed	12	0ms	ok
✅ `hook-at-hook-created`	completed	12	0ms	ok
✅ `deadline-hook-wins`	completed	7	1.0h	ok
✅ `deadline-expires`	completed	7	1.0h	ok
✅ `long-sleep`	completed	11	30.0d	ok
✅ `hook-never-arrives`	stalled	3	0ms	skipped
✅ `step-retries-twice`	completed	10	2.0s	ok
✅ `parallel-steps`	completed	9	0ms	ok
✅ `hook-on-execution-state`	completed	12	0ms	ok
✅ `peek-hook-before-branch`	completed	12	0ms	ok
✅ `peek-hook-after-branch`	completed	12	0ms	ok
✅ `peek-hook-at-registration`	completed	12	0ms	ok
✅ `race-hook-before-probe`	completed	12	0ms	ok
✅ `race-hook-after-probe`	completed	12	0ms	ok
✅ `race-duplicate-delivery`	completed	13	0ms	ok
✅ `attr-hook-before-step`	completed	11	0ms	ok
✅ `attr-hook-after-step`	completed	11	0ms	ok
✅ `attr-from-step-body`	completed	13	0ms	ok
✅ `fork-hook-after-timeout`	completed	14	1.0m	ok
✅ `fork-hook-before-timeout`	completed	14	1.0m	ok
✅ `count-hook-after-timeout`	completed	17	1.0m	ok
✅ `count-hook-before-timeout`	completed	20	1.0m	ok
✅ `stale-read-step-count-fork`	completed	20	1.0m	ok
✅ `stale-read-equal-step-counts`	completed	14	1.0m	ok
✅ `step-vs-step-fork`	completed	12	0ms	ok
✅ `step-vs-step-fork-fenced`	completed	12	0ms	ok
✅ `fence-catches-benign-direction`	completed	12	5ms	ok
✅ `in-flight-before-decision`	completed	17	1.0m	ok
❌ `in-flight-before-decision-counted`	completed	17	1.0m	ok
✅ `in-flight-after-decision`	completed	19	2.0m	ok
✅ `stale-read-step-count-fork-fenced`	completed	20	1.0m	ok
✅ `fork-hook-wins`	completed	13	1.0m	ok
✅ `fork-timeout-wins`	completed	13	1.0m	ok
✅ `unclaimed-payload-under-fork`	completed	17	1.0m	ok
✅ `claimed-payload-under-fork`	completed	17	1.0m	ok
✅ `writers-independent-step-bodies`	completed	12	0ms	ok
✅ `writers-scripted-tempo`	completed	12	0ms	ok
✅ `cancel-mid-step`	cancelled	7	0ms	skipped

Full trace: world-sim.txt

[probe] Disable wait-continuation dedupe

41a01c5

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

vercel Bot deployed to Preview – workbench-python-workflow August 22, 2026 02:44 View deployment

vercel Bot deployed to Preview – workflow-web August 22, 2026 02:44 View deployment

vercel Bot deployed to Preview – workflow-tarballs August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – workbench-nitro-workflow August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – workbench-express-workflow August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – workbench-fastify-workflow August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – workbench-hono-workflow August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – workbench-astro-workflow August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – workbench-vite-workflow August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – workbench-tanstack-start-workflow August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – example-workflow August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – workbench-sveltekit-workflow August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – workbench-nuxt-workflow August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-webpack August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-turbopack August 22, 2026 02:45 View deployment

vercel Bot deployed to Preview – workflow-docs August 22, 2026 02:46 View deployment

vercel Bot deployed to Preview – workflow-swc-playground August 22, 2026 02:47 View deployment

Navigation Menu

Uh oh!

[probe] Disable wait-continuation dedupe - #3748

[probe] Disable wait-continuation dedupe#3748
shalabhc wants to merge 1 commit into
mainfrom
e2e-no-wait-dedupe

shalabhc commented Aug 22, 2026

Uh oh!

changeset-bot Bot commented Aug 22, 2026

Uh oh!

vercel Bot commented Aug 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

shalabhc commented Aug 22, 2026

Not for merge — diagnostic probe

Theory under test

Expected outcome

Costs and risks, since this is not free

Tests

Related

Uh oh!

changeset-bot Bot commented Aug 22, 2026

⚠️ No Changeset found

Uh oh!

vercel Bot commented Aug 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 E2E Test Results

❌ Failed E2E Tests

🌐 Cross-language Conformance (9 failed)

vercel-http-transport (2 failed)

vercel-ws-transport (8 failed)

⚠️ Flaky E2E Tests (passed on retry)

🛠 Infra Events (absorbed by the harness)

E2E Test Summary

Uh oh!

github-actions Bot commented Aug 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Workflow Benchmarks

Uh oh!

github-actions Bot commented Aug 22, 2026

Sim World

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Aug 22, 2026 •

edited

Loading

github-actions Bot commented Aug 22, 2026 •

edited

Loading

github-actions Bot commented Aug 22, 2026 •

edited

Loading