feat(pull): async fan-out join + sync edge adapter — no autonomous trigger is stranded (#2524) - #2532
Conversation
Verified on a live local stack with the real aeroponics fleetNot a fixture run — 25 executions, 4 autonomous triggers, ~$7.79 of real turns. Zero pushed, zero re-deliveries, zero stuck rows. Loops (#2523) — three full cyclesEach cycle was terminal → CAS advance → park on Runs landed on The agent did real work: three sourced aeroponic-tomato papers found, fact-checked via Cron (#2391 + the #2523 scheduler poll)Zero retries, zero bogus terminals. The same table holds a clean A/B on the identical agent and schedule shape: the 14:00 fire on pre-change code is Precision, so this isn't overclaimed: the scheduler logs Fan-out (#2524)Sync, 3 subtasks — returned in 11s, 11ms from last terminal to the caller's aggregate. (Single uvicorn worker here, so the Future always wins; multi-worker falls to the 5s DB poll, as documented.)
The semaphore did not bound execution — it self-released as each dispatch returned QUEUED, and real concurrency became the worker pool. Exactly the documented behaviour, and the reason it needed no branch. (Peak 2 rather than 3 only because
Authorization boundary: Two honest caveats
Not covered
|
Live-tested the last two triggers, and the un-gated paths with the allowlist EMPTYFollow-up to the run above. Same local stack, same worktree mounts, same aeroponics fleet. This session closes the two triggers that comment listed as "not covered", and then answers the question that matters more for merging: what does this PR change on an install with no pilots? 23 executions, $2.37. Zero stuck rows, zero re-deliveries. Every trigger appears on both paths on purpose — the pulled and pushed columns of the same trigger are the A/B. 1.
|
| attempt | result |
|---|---|
POST …/respond |
200, dispatched |
POST …/respond again |
400 Cannot respond to item with status 'responded' |
maybe_dispatch_resume(item, same answer) called directly, in a separate process |
returns None, no dispatch |
schedule_executions count for triggered_by='operator_response' after all three: 1. The claim row is durable, not in-process:
scope | idempotency_key | status
agent:claim-verifier | operator_resume:727dd20…:2a70cbbf746bb874a1e… | completed
The third attempt is the one worth having — the status gate would have hidden a broken key.
2. Inbound a2a (ent#157) — pulled, both methods
a2a_exposed=1 on cornelius, caller authenticating with a user-scoped MCP key as Bearer.
message/send — 10.09s wall, artifact intact:
{"kind":"task","status":{"state":"completed"},
"artifacts":[{"parts":[{"kind":"text","text":"PULL_A2A_SEND_OK"}]}]}row: a2a | success | cornelius#w2 | redeliveries 0 | $0.127
message/stream — working immediately, then a single final frame:
16:39:19.9 data: {"result":{"kind":"status-update","status":{"state":"working"},"final":false}}
16:39:24 data: {"result":{…"status":{"state":"completed"},
"artifacts":[{"parts":[{"text":"PULL_A2A_STREAM_OK"}]}],"final":true}}
row: a2a | success | cornelius#w1 | redeliveries 0
The artifact is identical on both paths — same request, flag off (§3 below), 4.63s instead of 10.09s:
{"kind":"task","status":{"state":"completed"},
"artifacts":[{"parts":[{"kind":"text","text":"PULL_A2A_SEND_OK"}]}]}Tail latency, measured. The documented ~5s is a grid, not a constant — SYNC_WAITER_POLL_INTERVAL = 5.0 and the wake lands on the first poll after the terminal, so the cost is anywhere in [0, 5) depending on where the terminal falls:
| wait started | terminal applied | caller answered | tail | |
|---|---|---|---|---|
message/send |
16:38:56.143 | 16:39:01.833 | 16:39:06.194 | 4.36s |
message/stream |
16:39:19.969 | 16:39:24.331 | 16:39:24.9 | ~0.6s |
Total pull overhead on the send case: 0.07s dispatch + 0.6s claim + 4.36s waiter = 5.0s over the identical push call (10.09 vs 4.63). For a turn measured in minutes that is noise, as the docstring says; for a sub-5s turn it is more than doubling, which is worth knowing before anyone points a chatty A2A client at a pilot agent.
Auth boundary held: unauthenticated POST /a2a/cornelius → 401, non-exposed agent → uniform 404, public card on the exposed agent → 200.
3. Flag-OFF parity — PULL_MODE_PILOT_AGENTS=""
Backend recreated with an empty allowlist and all three agents recreated so TRINITY_PULL_MODE was cleared (verified empty in each container). Every row below has claimed_by_worker IS NULL; not one queued execution or Pull pilot line appeared in the backend log for the whole window.
Loops run all iterations and finalize correctly. Three loops, 8 runs:
loop_wBTC6RmD max_runs=3 delay=15s → completed / max_runs_reached 3/3
loop_W2FS25fI max_runs=3 delay=30s → completed / max_runs_reached 3/3 (restarted mid-loop, below)
loop_35yd6QQV max_runs=5 stop_signal → completed / stop_signal_matched 1/5
Cross-iteration state survives on the push path too — the runs returned PUSH_LOOP_RUN 1, 2, 3, each incrementing the counter it read out of {{previous_response}}.
delay_seconds parks and the 5s sweep dispatches, on push. next_run_at was stamped at exactly +15s each time and cleared by the sweep:
16:45:20 run 1 terminal, park stamped next_run_at = 16:45:35.107 (+15.000s)
16:45:41 park cleared, run 2 dispatched
16:45:45 run 2 terminal, park stamped next_run_at = 16:45:57.710 (+15.000s)
16:46:01 park cleared, run 3 dispatched
16:46:06 completed / max_runs_reached
The sweep is not flag-gated and it does the work on the push path as well. Granularity is the sweep period, as documented — clears landed 4–6s after due.
stop_signal is honoured. max_runs=5, stop_signal="HALT-NOW"; the agent emitted HALT-NOW on run 1 (it took the wrong branch of my prompt — model behaviour, not the platform) and the loop terminated at runs_completed=1 with stop_reason=stop_signal_matched. Gate works; the 1-of-5 is my prompt's fault, not the loop's.
Backend restart mid-loop → resumes, does not go interrupted. Restarted trinity-backend at 16:48:02 while loop_W2FS25fI was parked at runs=1/3, next_run_at=16:48:29:
16:48:02 docker restart trinity-backend
16:48:07 backend healthy → loop still `running`, runs=1/3, park intact
16:48:33 park cleared by the sweep on the fresh backend, run 2 dispatched
16:49:19 completed / max_runs_reached 3/3
Run 2 returned RESTART_LOOP_RUN 2 — it read run 1's response out of the durable last_response across the restart. SELECT status, count(*) FROM agent_loops afterwards: zero interrupted rows. This is the intended behaviour change from dropping mark_orphan_loops_interrupted; flagging it, not fixing it. Worth a release note — an operator who restarts a backend today gets loops marked interrupted, and after this they get loops that quietly carry on.
max_concurrency does bound the push path — but not measurable the way the pull run measured it. See §4; the short version is the A/B:
max_concurrency=1, 5 subtasks |
max_concurrency=5, 5 subtasks |
|
|---|---|---|
| batch wall time | 19.33s | 4.58s |
| completions at | 33.31 / 36.61 / 39.90 / 44.00 / 48.18 | 31.22 / 31.39 / 31.71 |
| spread | 3.3–4.2s apart, one at a time | all 3 within 0.49s |
| outcome | 5/5 success | 3/5 success, 2 failed: Agent at capacity (3/3 parallel tasks running) |
At c=1 execution is strictly serial. At c=5 against the agent's own max_parallel_tasks=3, three ran genuinely concurrently and the excess two were rejected, not queued — which is the dispatch-routing half of flag-OFF parity proved directly: overflow_policy stayed "reject".
Fan-out deadline reports running, not failed — confirmed un-gated. 3 subtasks, max_concurrency=1, timeout_seconds=10:
returned at 10.05s batch_status = deadline_exceeded
d1 completed d2 completed d3 running ← not `failed`
status endpoint immediately after: running [d1 completed, d2 completed, d3 running]
… d3 kept running and landed success on its row …
status endpoint after terminals: completed [d1 completed, d2 completed, d3 completed]
Exactly the documented contract, and it happens with an empty allowlist. Any existing caller that branches on a subtask being failed at the deadline changes behaviour on upgrade — this is the one item in the un-gated list I'd call out loudest in the release notes.
operator_response on push, for parity — same table, one row each:
id status claimed_by_worker response
ZoEUzhEblX_D0o-kInShag success (null) PUSH_OPRESUME_OK
hdXSeg6y5HxPyCEy2zAYqw success claim-verifier#w2 PULL_OPRESUME_OK 2026-09-04
Both audit rows record "status": "success".
4. Instrument correction — the fan-out concurrency query means different things on the two paths
The max_concurrency=1 measurement above cannot be done with the row-interval sweep the earlier comment used, and I want that on the record because it qualifies the "peak concurrency 2" number in it.
db/schedules/executions.py says started_at is written at admission, and the backlog path resets it when the row leaves the queue ("reset started_at so drain records a clean run window"). So:
- push —
started_atis stamped when the row is created, before the fan-out semaphore. All 5 subtasks of thec=1batch carrystarted_atwithin 17ms of each other, and an interval sweep over them returnspeak_concurrency = 5for a batch that was provably serial. The query reads N for any N-task batch regardless ofmax_concurrency. - pull —
started_atis the claim, so the sweep measures real execution. Confirmed in this session's own rows:operator_responsewas created at16:37:23.279and carriesstarted_at = 16:37:28.008, i.e. after the claim.
So peak=2 under pull and any push number are not the same measurement and shouldn't be compared. On push the signal is the completion cadence plus the row span, which is unambiguous here — c=1 spans of 4.38 / 7.67 / 10.96 / 15.05 / 19.23s (each row's span absorbing its own wait behind the semaphore) against c=5 spans of 4.06 / 4.23 / 4.54s.
None of this changes the PR's conclusion about max_concurrency self-releasing under pull. It changes how the next person should measure it.
5. ⚠️ The Alembic chain no longer applies to dev — this branch needs a renumber on rebase
#2526 is already merged (2bebbf58 on dev), and it landed with its revision renamed, because 0050 had been taken by 0050_agent_canvases in the meantime:
dev: 0049_execution_turn_integrity → 0050_agent_canvases → 0051_agent_loops_terminal_driven (head)
#2532: 0049_execution_turn_integrity → 0050_agent_loops_terminal_driven → 0051_execution_fan_out_task_id
0051_execution_fan_out_task_id still carries down_revision = "0050_agent_loops_terminal_driven" — an id that does not exist on dev — and its own number now collides with dev's loops revision. Rebasing without touching it gives a dangling down_revision, and the failure is at backend boot, not at review:
alembic.util.exc.CommandError: Can't locate revision identified by '0051_execution_fan_out_task_id'
(That is a real line from this session — I hit the same class of error when I pointed the base checkout at a DB stamped on this branch's chain.) The rebase needs:
# 0052_execution_fan_out_task_id.py
revision = "0052_execution_fan_out_task_id"
down_revision = "0051_agent_loops_terminal_driven"Worth stating plainly for whoever merges: the loop rewrite is not pending any more, it is live on dev today, so §3's flag-OFF findings — loops surviving restart instead of going interrupted, park granularity becoming the sweep period, the extra indexed read per terminal — already describe dev, not a hypothetical.
Verification
tests/unit/test_2524_fanout_async_join.py
tests/unit/test_2523_loops_terminal_driven.py
tests/unit/test_2048_pull_pilot_reach.py
tests/unit/test_ent329_operator_resume.py
tests/unit/test_157_a2a_inbound_server.py → 142 passed
tests/lint_sys_modules.py → OK (140 violations / 206 baseline, none new)
tests/lint_root_test_placement.py → OK
Caveats
- The two operator-queue items were created through
db.create_operator_queue_itemrather than by an agent writing its own~/.trinity/operator-queue.json. That is the sinkoperator_queue_servicewrites through, and the respond→dispatch path under test starts after ingestion either way — but it is not the agent-authored seam end to end. - One loop failed before I could use it: I recreated the agents and started a loop seconds later, and iteration 1 hit
Agent unreachable — transport circuit breaker open. The breakers were closed again a minute later and the retry was clean. Environmental, not a defect — but noteon_failure="abort"did the right thing and killed the loop on the first failure. - Still nothing here exercises lease expiry or re-delivery —
redelivery_count = 0on all 23 rows.
…igger is stranded (#2524) Completes #1081 Phase 4. After this, every autonomous trigger can run on the durable queue; only the interactive ones stay on push, which is the deliberate Open Question 7 scope cut (#1982/#1989). FAN-OUT. `FanOutService.execute` built a `dict[task_id, FanOutTaskResult]` inside one `asyncio.gather`, so the batch existed only while the request that started it did. A pull-claimed subtask returns nothing to collect (the row is queued and the turn runs later in the agent's worker), and nothing could answer about a batch afterwards — no `async_mode`, no status endpoint, a disconnect lost it. The batch now lives on `schedule_executions`: every subtask row carries `fan_out_id` plus the caller's own `fan_out_task_id` (new column — the id used to be a dict key no async batch could reach), and `build_aggregate` rebuilds the result from the rows. Adds `async_mode` and `GET /api/agents/{name}/fan-out/{fan_out_id}`, which also checks the batch belongs to that agent: `fan_out_id` is opaque but not secret. Two decisions the issue asked for. `max_concurrency` keeps its meaning and needed no branch. The semaphore stays around the `execute_task` call: on push that call spans the whole turn so it paces dispatch as before, and under pull it returns in milliseconds so the worker pool becomes the cap — Phase 5's "capacity becomes physical", by construction. Deleting it, as first planned, would have fired N dispatches at an agent whose `max_parallel_tasks` is 3 and turned the excess into CapacityFull. The outer deadline bounds the WAIT, not the work — a contract change. A still-open subtask now reports `running`, not `failed`; the batch still reports `deadline_exceeded`. A queued or claimed row is not the backend's to cancel, and on push the old cancellation was half-illusory anyway (it abandoned the HTTP call while the agent kept running and billing the turn). After a deadline the status endpoint is the source of truth. A2A + OPERATOR_RESPONSE. These were deferred with "a2a cannot hand back a receipt to poll" — true and beside the point: it does not need a receipt, it needs to BLOCK CORRECTLY while the turn happens elsewhere, which `sync_waiter.wait_for_sync_terminal` already did for `/task`. `dispatch_and_await_terminal` is the adapter: `execute_task`, and on a QUEUED return, wait for that row's terminal and rebuild from it. Nothing signals that waiter on the pull path, so the wake is the 5s DB poll — up to ~5s of extra tail latency on an a2a call against a pilot, deliberately not worth a second signalling path. `PULL_REACHABLE_TRIGGERS` now equals `_AUTONOMOUS_TRIGGERS` and stays an enumerated allow-list on purpose: a structural test forbids deriving it, because that would hand reach to the next autonomous trigger with nobody checking dispatch can deliver it — precisely #2048's defect. `note_unreachable_pull_trigger` is kept and still tested, against a synthetic narrowing. The loop advance (#2523) and fan-out join share one `_terminal_side_effects` shim off `spawn_task_terminal_event`, with separate guards so one raising cannot cost the other its terminal. Migration 0051 + the SQLite twin: `fan_out_task_id`, plus `idx_executions_fan_out_status` — the join COUNTs non-terminal rows for one batch on every fan-out terminal. Refs #1081, #2048, #2391, #2523, ent#157, ent#329. #2392 gets worse from here: `effect_guard` still fails open when the execution id is absent, and a fan-out multiplies that by N. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Tm7UEkd4G9KQZD5oeLRSa
27df798 to
be4d7ac
Compare
Rebased onto
|
…ync-join # Conflicts: # tests/registry.json
|
merge-train 2026-09-09: not on this train. The schema lane is clean and I checked it rather than assuming: SQLite migration registered at Two criticals, the first reproduced by execution. 1. Pre-created
Reproduced against real Failure scenario at defaults ( This is the #2435 class re-entering: every age check is anchored at admission, and the PR introduces a new hidden queue without the Your live runs did not surface it because those batches finished fast enough to slip between 300s cleanup ticks. A 12-task batch of 3-minute turns will not. Fix shape: create the rows 2. The default wait budget ignores With no caller Warnings worth folding into the same pass:
Verified clean, on the record: Invariant #18 holds (the POST still takes Rides the next train once the row lifecycle and the deadline are settled. |
|
merge-train 2026-09-10: not on this train. All 25 checks are green and the schema lane is clean, but there is a reproduced data-integrity regression that CI structurally cannot see. Rides the next train once it's resolved. ❌ Pre-created
|
|
merge-train 2026-09-11: not on this train. The 2026-09-10 ejection (pre-created |
|
Resolve by running |
|
Resolve by merging |
|
merge-train 2026-09-12: not on this train — conflicts with Also: Rides the next train once |
|
merge-train 2026-09-13: not on this train. The previous ejection (see the merge-train comment above) has no push after it — the branch head is unchanged — so the same finding stands. Rides the next train once addressed. |
|
🚧 Alembic head check could not run — this PR conflicts with
Merge Advisory — this check does not block merge. · head_sha: |
|
merge-train: ejected this run — rides the next train once rebased. (1) Alembic fork (#2068 class): |
Resolves conflicts with #2679 (#2670) by adopting its shipped GET /api/agents/{name}/fan-out/{fan_out_id} and dropping this branch's duplicate route, get_status and batch_belongs_to. The GET gains an additive task_id from fan_out_task_id. Review fixes (#2524, merge-train 2026-09-09/10): - Create each subtask row at slot grant inside the max_concurrency semaphore instead of up front. Pre-created RUNNING rows waiting behind the semaphore were bulk-FAILed by the #106 no-session sweep; QUEUED rows would be claimed by claim_next_queued while _dispatch_all also dispatched them. The sync caller now waits for the shielded dispatch, then for queued rows to reach a terminal. - Default wait budget covers ceil(N / min(max_concurrency, max_parallel_tasks)) waves instead of one subtask's bound. - Snapshot subscription_id on fan-out rows (SUB-004). - Carry execute_task's error_code into the sync aggregate. - _fail_subtask gates side effects on the CAS and emits through spawn_task_terminal_event. - A failed fan-out DB poll read no longer escapes the wait. - MCP fan_out: async_mode param, corrected deadline wording. - Docs: architecture/execution.md, api-endpoints.md, requirements/scheduling.md 37.4, fan-out flow. - Real-schema test for the fan-out SQL. Renumber the Alembic revision to 0062_execution_fan_out_task_id off 0061_execution_open_canvas (single head). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vybe
left a comment
vybe
left a comment
There was a problem hiding this comment.
merge-train: validated on #2820 (batch of 6, full suite green; pg-migrations and schema-parity green there, journey-smoke green alongside #2809's J10 fan-out journey — the interaction this batch existed to test). Lane B+schema — /validate-pr + /review. Coverage mutation-proven: 14/14 mutants caught, zero source-text assertions in either new test file. Five invariants verified: the single new terminal writer is CAS-gated and closes its #1804 activity (with the join placed outside the try, so a raising close cannot strand the batch); the Invariant #18 idempotency triple holds; zero-fire on the join is impossible by contradiction; MCP types match the backend contract with optional on both sides; none of the 8 modified existing tests weakened an assertion. Dual-track migration complete (SQLite + Alembic 0062 on 0061, single head before and after). Two post-merge follow-ups noted on the PR: build_aggregate's queued branch and the SELECT column list sit in the seam between the fake-db service tests and the real-schema db tests, where four mutation probes survived.
… nothing — the rebuild's justification was false Two merge-train findings on #2805: 1. dev's head moved to `0062_execution_fan_out_task_id` (#2532) after this branched, and this revision declared the same parent — a live #2068 fork, two heads, `upgrade head` applying zero revisions. Re-chained as `0063_agent_sync_state_git_dir_bytes_bigint` off `0062_execution_fan_out_task_id`; `check_alembic_heads.py` on the merged tree: 64 revisions, 1 head. 2. The SQLite rename-swap rebuild was justified by "the schema-parity suite would go red forever" — disproved by a one-line negative control (registration removed, suite still green): both parity fixtures build from empty, so `init_schema` creates the table in both snapshots and the guard cannot see this column's declared type. A boot-time DROP TABLE of a live table for a CI benefit that does not exist is the wrong trade. The rebuild, its column list and its three tests are gone; a note beside `_migrate_agent_sync_state_git_dir_bytes` says why the SQLite track deliberately carries nothing, a test pins that it stays that way for a reason, and the learnings entry now states the honest lesson: run the negative control before writing a guard's behaviour into a durable file. The seven sibling columns with the same int4-vs-`< 2**63` mismatch are filed as #2827. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VpvcfgWkmQPD7DrDLmATTf
2 participants
Fixes #2524. Parent: #1081 — this completes Phase 4 ("sync edge adapter + async fan-out join"). Siblings: #2513/#2391 (cron), #2526/#2523 (loops, merged).
After this, every autonomous trigger can run on the durable queue.
fan_out, and thena2a+operator_response, were the last three.What was wrong
FanOutService.executebuilt adict[task_id, FanOutTaskResult]fromexecute_task's return values inside oneasyncio.gather. A pull-claimed subtask returns nothing to collect —execute_taskreturns as soon as the row is on the durable queue and the turn runs later, in the agent's worker — so a collector built on the return value reads an empty result for every subtask. That is whyfan_outsat in the stranded half ofPULL_REACHABLE_TRIGGERS(#2048).Unlike loops (#2523), fan-out has a genuinely blocking caller —
POST /fan-outreturns the aggregate — which is why Phase 4 names it separately and why this PR carries an adapter.The change
fan_out_idplus the caller's ownfan_out_task_id(new column), andbuild_aggregate()rebuilds the syncFanOutResultfrom those rows, in input order.max_concurrencysemaphore — never up front. A row created up front and left waiting is a hidden queue every recovery path misreads: asRUNNINGit is bulk-FAILed by the bug: Cleanup service misses 'skipped' executions and slow to detect no-session failures #106 no-session sweep (and matches the watchdog orphan reconcile and the stale sweep); asQUEUEDit is claimable byclaim_next_queued— backlog drain and pull workers — while the service also dispatches it, i.e. a double run.sync_waiter.wait_for_fan_out_batchfor rows that came backqueued— Phase 4's "sync edge adapter", in the shapesync_waiteralready uses for/task.async_modeonFanOutRequest(and on the MCPfan_outtool) →{fan_out_id, status: "accepted", total}immediately. Poll with bug(mcp): fan_out has no gateway-timeout receipt — the third route of the #914 class, and the one that runs longest #2670'sGET /api/agents/{name}/fan-out/{fan_out_id}/get_fan_out_result, which now also returns each subtask'stask_id.fan_outjoinsPULL_REACHABLE_TRIGGERS.Decisions
1.
max_concurrencykeeps its meaning, and needed no branch. The semaphore stays around theexecute_taskcall. On push that call spans the whole turn, so it paces dispatch exactly as before; under pull it returns in milliseconds, so it self-releases and real concurrency becomes the agent's worker pool. Deleting it would turn N concurrent dispatches at an agent withmax_parallel_tasks=3intoCapacityFullfailures.2. The outer deadline bounds the WAIT, not the work — contract change.⚠️ A still-open subtask (including one still waiting for a slot) now reports
status="running", notfailed; the batch still reportsdeadline_exceeded. The deadline never cancels the dispatch. After a deadline the GET is the source of truth.3. With no caller deadline the wait covers the whole batch:
ceil(N / min(max_concurrency, max_parallel_tasks)) × execution_timeout_seconds + 120s. The first version used one subtask's bound, which returneddeadline_exceededon batches the old unboundedgathercompleted.The join
join_fan_out_on_terminalhangs offevent_dispatch_service.spawn_task_terminal_event— the wrapper every CAS-won terminal writer already calls — beside #2523's loop advance, each under its own guard (_terminal_side_effects). Not insideemit_task_terminal_event, which returns early when no subscription matches. One PK read per fleet terminal; a(fan_out_id, status)-indexed COUNT only for fan-out rows. It fires on "no open row", which is safe because the sync waiter is registered only after every subtask has been dispatched._dispatch_allalso calls it directly on a non-QUEUED return (the fast-fail paths write FAILED without a terminal event)._fail_subtask(a raisedexecute_task) is a proper terminal writer: on a won CAS it closes the activity (#1804) and emits throughspawn_task_terminal_event(#1578).a2aandoperator_responsetask_execution_service.dispatch_and_await_terminal:execute_task, and if it comes back QUEUED, wait for that row's terminal and rebuild the result from it. Callers:routers/a2a._run_a2a_task(JSON-RPC contract untouched) andoperator_resume_service(the ent#329 receipt is neverqueued).a2acall against a pilot agent (measured 4.36s / 0.6s in the live run below).PULL_REACHABLE_TRIGGERSnow equals_AUTONOMOUS_TRIGGERSbut stays an enumerated allow-list, with a test that fails if someone derives it.Known limits
execution_idsas evidence, not a manifest), and a backend restart loses the undispatched tail — forasync_modeexactly as for a sync call. Under pull every row exists within milliseconds.error_codeexists only on push results —schedule_executionshas no column for it, so pull subtasks and GET results carry none.sync_waiter's registry is in-process; multi-worker, the 5s DB poll is the wake path.Migration
0062_execution_fan_out_task_id(off0061_execution_open_canvas) + the SQLite twin. One column plusidx_executions_fan_out_status ON schedule_executions(fan_out_id, status).check_alembic_heads.py: 63 revisions, 1 head.Follow-up that gets worse from here
#2392.
effect_guardstill fails open when the execution id is absent, and a fan-out multiplies that by N. Do not run a side-effect-bearing fan-out pilot before it is resolved.What changed after review (2026-09-15)
Merged
origin/dev(conflicts with #2679/#2670 infan_out_service.py,routers/fan_out.py,executions.py,pull_pilot.py,fan-out.md).RUNNINGrows bulk-FAILed by the #106 sweepQUEUED(suggested) was not used:claim_next_queuedhas no trigger filter, so backlog drain / pull workers would claim the rows while_dispatch_allalso dispatches them. Restamp-at-grant would not protect rows while they wait.ceil(N / c)subscription_idmissingexecute_task.error_codealways null_fail_subtaskignores CAS, never emitsspawn_task_terminal_event.started_atorderingbuild_aggregatealways uses input order; the read documents thatstarted_atorder is not a contract.async_modedoesn't survive restart_poll_dbexceptions escapearchitecture/execution.md,architecture/api-endpoints.md,requirements/scheduling.md§37.4; MCPfan_outdescription corrected +async_mode;models.py/test_2048comments.test_2524_fanout_real_schema.py(column, agent-scoped read,count_fan_out_open, the sweep hazard).get_statusandbatch_belongs_towere dropped in favour of #2670's shippedGET(unchanged contract, plus additivetask_id).0062.Tests
tests/unit/test_2524_fanout_async_join.py— join, rows at slot grant, queued path, sync wait on queued rows,max_concurrencypacing,error_code, deadline reports the tailrunning,_fail_subtaskCAS gating, wait budget, sync edge adapter.tests/unit/test_2524_fanout_real_schema.py— the SQL against the real schema.tests/unit/test_inter_agent_timeout_unit.py,test_2670_fan_out_receipt.py,test_2048_pull_pilot_reach.py,test_ent329_operator_resume.py,test_157_a2a_inbound_server.py,test_736_a2a_outbound_call.py— updated / still green.tsc --noEmitclean,npm testgreen.Live-fleet evidence from the first version (pull + push A/B, lease re-delivery and poison-park) is in the comments; the row-lifecycle change above has not been re-run live.
🤖 Generated with Claude Code