fix(execution): equity CLOSE re-checks the live broker position at dispatch (ALP-943)

[jatassi]

Summary

• The 2026-06-09 incident: a PM-directed MRVL CLOSE dispatched 48 minutes after the monitor's re-protection stop had flattened the position; the dispatch trusted the frozen positions projection, tolerated the already-filled protective-leg cancel rejection by design, and sold 4 shares into a flat position — Alpaca opened an unmanaged −4 short (sell_to_open).
• New execution-time drift guard in the shared equity CLOSE dispatch (_close_equity, covering both the PM-directed and engine-envelope paths): re-check the live broker position via the new AccountStateQueries.get_open_position before any leg-cancel RPC — flat or side-flipped → reject with the new position_state_drift code (http_status=0, local guard); live qty below requested → clamp the sell to the live quantity.
• A protective-leg cancel that is permanently rejected now resolves the leg's actual broker state via get_order_by_id: filled / partially-filled aborts the close with the same drift rejection (the protective exit executed); canceled / expired / unknown proceeds as before. If the abort lands after an earlier leg's cancel was confirmed, the ALP-937 NAKED POSITION CRITICAL alert fires before the abort propagates.
• PM path emits a command_abandoned forensic entry for local-guard rejections; engine path returns a rejected SubmissionResult with the trigger not marked seen (the monitor may retry). Runbook § 8.11 documents the new rejection class (operator posture: no action).

Closes https://linear.app/alphamind-jatassi/issue/ALP-943/fixexecution-equity-close-dispatch-executes-against-the-stale

Test plan

• CI (.github/workflows/ci.yml) — expected to fail with the GitHub usage-limit billing sentinel; per docs/agents/ci.md fallback, the full CI chain runs locally (lint chain + full pytest) before merge
• Local lint chain clean (ruff check, ruff format --check, mypy, lint-imports)
• ALP-943 acceptance criteria all met (drift-guard unit + route-through tests in tests/execution/oms/, query-surface tests in tests/execution/broker_adapter/)

🤖 Generated with Claude Code

[linear]

ALP-943 fix(execution): equity CLOSE dispatch executes against the stale positions projection, discarding the broker's filled-leg flat signal — sold 4 MRVL into a flat position → unmanaged −4 short (2026-06-09)

Summary

The 2026-06-09 17:00Z pipeline run (inv-20260609T170000Z-d558684c) decided to close MRVL while it was long 4, and executed that CLOSE at 17:48Z — 48 minutes later, after the monitor's fresh re-protection stop (ALP-938) had already flattened the position at 17:00:06Z. The dispatch sold 4 shares into a flat position; Alpaca classified the sell as sell_to_open and opened an unmanaged, unprotected −4 MRVL short that the projection does not represent. Live state (paper account PA3QJFZPNGCH, verified 2026-06-10 04:02Z): MRVL −4 @ 254.5975, zero open orders, the account's only open position. Cleanup below is still pending.

Corrected from the original report

• This is the PM-directed equity CLOSE path, not the monitor "engine-envelope" path. ENV-SA-2 is a PMStrategistEnvelope id (src/alphamind/commands/pm_envelope.py:352), submitted through the PM submit_envelope MCP wrapper. The monitor-facing engine-envelope path (MON.{session}.{trigger} ids, submit_engine_envelope.py) shares the same downstream _close_equity and inherits the same defect; the fix below covers both.
• The system did not merely fail to consult fresher state — it received broker-confirmed proof the position was flat, twice, and discarded it. The PM's own CANCELs on the re-protection legs (ENV-SA-ORD-1/2) were rejected order is already in "filled" state seconds before the CLOSE, and the CLOSE path's ALP-937 protective-leg cancellation hit the same 422 on the filled stop and proceeded by documented design (src/alphamind/execution/oms/broker_dispatch.py:557-563).
• The orders.status=PENDING "projection gap" is not a second defect. The next run (19:00Z) integrated the stop-out correctly (position CLOSED, honest PnL −314.58) and quarantined the short's two fills by design — fills cannot integrate against a CLOSED position (src/alphamind/execution/write_paths/fill_collection.py:144-149, ALP-761), and a quarantined fill never advances its order's status. The residue is cleanup, not fix scope.
• scripts/reconcile_alpaca_state.py is a read-only diff tool (its module docstring), not a repair tool; the original cleanup step implying it could reconcile is corrected below.

Evidence (prod DB + live Alpaca PA3QJFZPNGCH + archived PM submission log — all UTC)

1. 2026-06-08 15:49Z — bracket entry BUY 8 MRVL @ 301.90 (orders.ORD-MRVL-entry-d167f…, FILLED). Its protective legs died the same day (stop CANCELLED / target EXPIRED — the ALP-937 disease, since fixed); held long 8 unprotected overnight.
2. 2026-06-09 14:12:34Z — PM partial close SELL 4 @ 275.74 (ORD-CLOSE-…-b0a85bf7…, FILLED, cid inv-20260609T133500Z….ENV-SA-1…); remainder long 4, unprotected.
3. 17:00:00Z — inv-20260609T170000Z-d558684c starts; fill collection commits at 17:00:02.45 (invocations.fill_collection_completed_at); positions.POS-MRVL-d167f… reads OPEN / share_count 4.0.
4. 17:00:00–06Z — the monitor's ALP-938 re-protection submits a fresh OCO for the remainder (ORD-RBR-stop-d61fbf…-4, stop 288, qty 4 + ORD-RBR-tp-…-3). Price ~249 is far through the stop: it fills at 17:00:06.19Z, 4 @ 249.415 (broker_event_log.fevt-318e7b03635dd8fd). Broker flat on MRVL at 17:00:06Z. GS was identically re-protected and stopped out the same second (ORD-RBR-stop-3906ef…-5, 1 @ 1011.01).
5. 17:01:22Z — the monitor captures the stop-out fills into broker_event_log + fill_records (fill-27dcd6a036f9f43e, persistence_timestamp 17:01:22, processing_status unprocessed). Per single-writer, positions is untouched — fills integrate only during a fill-collection phase, and this run's had already committed.
6. 17:47–17:48Z — the PM, deliberating since the 17:00:02 snapshot, submits its envelopes (archived decision/portfolio_manager/submission_log.json): ENV-SA-1 closes TSM (legitimate; filled 17:47:10). ENV-SA-ORD-1 and ENV-SA-ORD-2 try to CANCEL the two MRVL re-protection legs; Alpaca rejects both — HTTP 422 order is already in "filled" state. ENV-SA-2 then closes MRVL (quantity="all", market, close_rationale_type risk_management / pm_directed) — accepted by the validator.
7. 17:48:01.13Z — dispatch resolves the CLOSE context from the positions row — share_count 4.0, LONG → SELL 4 (dispatch.py:629-651). The ALP-937 leg-cancellation hits 422 already-filled on the stop leg and proceeds (broker_dispatch.py:580-591, logged at INFO). The pre-commit writeback marks both legs ORDER_CANCELLED with filled_quantity_at_cancellation 0 — false for the stop, which filled 4 (activity_log 17:48:01.128).
8. 17:48:01.5–02.8Z — Alpaca order 582c0a0c-a7fe-4550-92ce-92c8e82e69e3 (cid inv-…d558684c.ENV-SA-2.0.0~…) carries no client-supplied position_intent (order_equity.py:254-261 sends none); Alpaca classifies it sell_to_open (fill payload raw_event_payload.order.position_intent) and fills 3 @ 254.63 + 1 @ 254.50 → −4 short @ 254.5975 avg.
9. 19:00:02Z — the pre_close run (inv-20260609T190000Z-348cc265) integrates the stop-out: position CLOSED, realized_pnl −314.58 — exactly 4×(275.74−301.90) + 4×(249.415−301.90). The short's two fills cannot integrate against the CLOSED position → quarantined (fill_records.processing_status='quarantined', summary fills_quarantined: 2) with two RECONCILIATION_ALERT activity entries; the CLOSE order row stays PENDING.
10. Live Alpaca, 2026-06-10 04:02Z — MRVL −4 @ 254.5975, zero open orders, only open position. Account cash 100,523.70 vs cash_ledger.current_cash_usd 99,505.33 — the 1,018.37 delta is the unbooked short proceeds (quarantined fills book no cash).
11. Code that ran: invocations.git_sha_at_invocation = 420d2daf — includes the ALP-937/938/939/940 fixes; this defect is current-main behavior.

Root cause (confirmed)

1. The positions projection is frozen between fill-collection phases. Fills integrate into positions only during an invocation's fill-collection phase (process_unprocessed_fills, src/alphamind/execution/write_paths/fill_collection.py; the two-phase invariant in execution/CLAUDE.md). Between 17:00:02 and 19:00:02 the MRVL row read OPEN / 4 regardless of broker reality.
2. Intra-invocation flattening is an expected event class, not a freak race. The monitor flattens positions between phases by design (ALP-938 re-protection, any stop). The captured fills sit unprocessed in fill_records (17:01:22) until the next run.
3. CLOSE dispatch trusts the projection at dispatch time. _close_command_context (src/alphamind/decision/portfolio_manager/submit_envelope/dispatch.py:629-651) resolves symbol/qty/side from the positions row ~48 minutes after the snapshot, with no live-broker check; the engine-envelope twin _engine_close_dispatch_kwargs (src/alphamind/execution/oms/submit_engine_envelope.py:489-542) does the same.
4. The broker's flat signal is discarded by design. _cancel_protective_legs (src/alphamind/execution/oms/broker_dispatch.py:544-599) classifies the 422 already-filled cancel rejection as a benign already-terminal leg — its docstring states "the position likely already exited; its shares are already free and the close proceeds" (lines 557-563) — and proceeds with the full-quantity sell.
5. Nothing constrains the order to closing semantics. submit_equity_close (src/alphamind/execution/broker_adapter/order_equity.py:216-263) submits a plain MarketOrderRequest with no position_intent (alpaca-py supports it on OrderRequest); with short_selling_enabled: true, Alpaca executes a sell on a flat position as sell_to_open.

Secondary integrity note (not the fix target): the pre-commit writeback _cancel_equity_protective_legs (src/alphamind/execution/write_paths/command_execution/close.py:162-215) marks legs CANCELLED before broker outcomes exist (ALP-836 pre-commit ordering), so the activity log recorded a "cancellation" of a leg that had actually filled. Fill integration corrects the orders row (it did, at 19:00); the activity-log entry stays false. Repointing that emission is ALP-836 ordering territory, out of scope here.

Why it escaped tests

tests/execution/oms/test_broker_dispatch.py:874 (test_close_equity_tolerates_already_terminal_protective_leg) asserts the tolerate-and-proceed behavior as desired — the ALP-937 design framed already-terminal legs purely as a held-shares non-problem and never re-examined what a FILLED protective leg implies, and the test enshrined that. The broker fake models held_for_orders share reservation but not position existence, so a sell on a flat position succeeds in the fake where real Alpaca opens a short. And no test spans the cross-process drift (monitor stop-out between snapshot and PM dispatch) — it crosses two processes and ~48 minutes of wall clock.

Reading

• src/alphamind/execution/oms/broker_dispatch.py:438-599 — _dispatch_close / _close_equity / _cancel_protective_legs: the fix site. dispatch_command_to_broker already receives queries: AccountStateQueries (line 149) but does not thread it to _dispatch_close.
• src/alphamind/execution/broker_adapter/queries.py — AccountStateQueries (the live surface) and PositionSnapshot (signed qty); get_asset's 404→None precedent at line 586 is the template for the new position getter.
• src/alphamind/execution/broker_adapter/errors.py:17-42 — PermanentRejectionCode Literal + PermanentRejection (code / http_status / alpaca_message): the rejection vocabulary to widen.
• src/alphamind/execution/broker_adapter/order_options.py:90-103 — PermanentRejectionError, the carrier both callers already translate.
• src/alphamind/decision/portfolio_manager/submit_envelope/dispatch.py:261-330, 391-435, 598-651 — _route_one_command (pre-commit → dispatch → teardown), its PermanentRejectionError handling, _close_command_context.
• src/alphamind/execution/oms/submit_engine_envelope.py:402-487 — the monitor-facing engine close sharing _close_equity; its PermanentRejectionError handling at 459-476, and the rejected-submission state rule (trigger not marked seen).
• src/alphamind/execution/broker_adapter/order_equity.py:216-263 — submit_equity_close, whose quantity resolution hoists into _close_equity per Scope (B).
• tests/execution/oms/test_broker_dispatch.py:821-960 — the ALP-937 equity-close tests, including the one this fix rewrites.
• scripts/RUNBOOK_production.md — Living-document rule; the new rejection class lands here.

Scope — execution-time live-position recheck in the shared equity CLOSE dispatch

(A) Live-position query surface. Add get_open_position(symbol: str) -> PositionSnapshot | None to AccountStateQueries (queries.py) and its protocol mirror (protocols.py), wrapping TradingClient.get_open_position with 404 → None exactly like get_asset (queries.py:586-596). Non-404 errors propagate.

(B) The guard in the equity close dispatch. Thread queries through _dispatch_close into _close_equity (dispatch_command_to_broker already receives it). At the top of _close_equity, before any protective-leg cancel RPC, call queries.get_open_position(position_symbol) and resolve the requested close quantity there by hoisting submit_equity_close's resolution (position_qty if command.quantity == "all" else float(command.quantity); submit_equity_close's position_qty parameter becomes the resolved final qty, its only caller being _close_equity). Outcomes: no live position, or the live position's side (sign of PositionSnapshot.qty) contradicts position_side → raise PermanentRejectionError carrying PermanentRejection(code="position_state_drift", http_status=0, alpaca_message=<one line naming symbol, expected side/qty, live qty>); live absolute qty below the requested quantity (beyond a 1e-9 float-dust epsilon) → clamp the sell to the live absolute qty; otherwise proceed unchanged. http_status=0 marks a local, non-HTTP guard rejection. Widen PermanentRejectionCode with "position_state_drift" (errors.py:17-27). Both callers already translate PermanentRejectionError into a rejected SubmissionResult (dispatch.py:391, submit_engine_envelope.py:459) — no caller changes.

(C) Abort on the filled-leg signal. In _cancel_protective_legs, on a permanent cancel rejection, resolve the leg's actual broker state via client.get_order_by_id: status filled or partially_filled → raise the same position_state_drift rejection (the position exited; the close must not proceed); canceled / expired / 404-unknown → proceed as today (benign already-terminal). This closes the residual race between (B)'s check and the cancel RPCs without message-text parsing.

(D) Tests. Rewrite test_close_equity_tolerates_already_terminal_protective_leg into the split pair: already-CANCELED leg with live position present → proceed; already-FILLED leg → position_state_drift rejection, no sell submitted. New tests: live-flat → rejection before any leg-cancel RPC and no order; live side flipped → rejection; live qty below the requested close qty → submitted sell clamped to live qty; through _route_through_broker, the rejection yields a rejected SubmissionResult + abandoned-command entry + pre-commit teardown; through submit_engine_envelope, status rejected with the trigger not marked seen. Extend the broker fake to model position existence and quantity for these cases.

(E) Runbook. Document the position_state_drift rejection class in scripts/RUNBOOK_production.md: a dispatch-time drift guard fired because the position changed between decision and execution; operator posture is no action — the position is already flat or smaller, the PM re-evaluates next invocation, and protection for any remainder is the monitor's ALP-938 job.

Out of scope: equity position_intent on close/protective orders (broker-enforced closing semantics for the residual seconds-wide race — a deliberate rung above this fix; needs paper-API behavior confirmation first); options/strategy close revalidation (options closes already pass buy/sell_to_close intents); ALP-942 (monitor transaction discipline — this fix stands without it); decision-time staleness (the strategist/PM deliberating over the invocation snapshot is the pipeline's design); the pre-commit ORDER_CANCELLED activity emission noted in Root cause.

Acceptance criteria

• AccountStateQueries.get_open_position returns a PositionSnapshot for an open symbol and None on Alpaca's 404, with non-404 errors propagating (unit test, mocked TradingClient).
• An equity CLOSE whose symbol has no live broker position is rejected with code position_state_drift before any leg-cancel or order-submit RPC reaches the broker fake.
• An equity CLOSE whose live position side contradicts the resolved position_side is rejected with code position_state_drift.
• An equity CLOSE whose live absolute qty is below the requested close quantity submits a sell for exactly the live quantity.
• A protective-leg cancel rejected permanently where the leg's broker state is filled aborts the close with position_state_drift and no sell; where the leg is canceled, the close proceeds (the rewritten ALP-937 test pair).
• Through _route_through_broker, a position_state_drift rejection yields a rejected SubmissionResult, an abandoned-command entry, and teardown of the pre-committed order row.
• Through submit_engine_envelope, the same guard yields a rejected SubmissionResult and the trigger is not marked seen (the monitor may retry).
• PermanentRejectionCode includes "position_state_drift"; ruff, mypy, and import-linter pass unchanged.
• scripts/RUNBOOK_production.md documents the position_state_drift rejection class (verified by inspection).

Verification

Scoped runs while implementing: uv run pytest tests/execution/oms/ tests/execution/broker_adapter/ -n auto and uv run pytest tests/decision/portfolio_manager/ -n auto (run the owning layers' full test dirs — close-path fixture fallout spreads beyond named files). Lint chain after every batch. The authoritative full-suite gate is CI on Windows. The runbook criterion is inspection-only.

One-time prod cleanup (manual, Windows prod box; source .env)

As of 2026-06-10 04:02Z the short is still open. Verify current broker/DB state first — every step is conditional on it still holding.

1. Broker — flatten. Buy 4 MRVL to cover (Alpaca UI, or a prod-box script with .env loaded). Do not give the cover order an AlphaMind-namespace client_order_id (no inv-… / MON.… prefix) — it must stay foreign to attribution. There are zero open MRVL orders, so nothing to cancel first.

2. DB — terminal-project the CLOSE order row. UPDATE orders SET status='FILLED', filled_quantity=4, average_fill_price=254.5975, remaining_quantity=0, last_update_timestamp='2026-06-09T17:48:02.750249+00:00' WHERE order_id='ORD-CLOSE-POS-MRVL-d167f23da4805ab0bb5d23859e7e9f88-e80e7da1eae55d95840837e4c96bd87c'. This records what the broker actually did; touch nothing else on the row.

3. DB — leave the quarantine alone. fill-681430f9f313319e and fill-b5b49e1ae159bcf0 stay processing_status='quarantined' and their broker_event_log rows stay — that is the designed record of orphan fills. Do NOT integrate them: positions.POS-MRVL-d167f… must stay CLOSED with realized_pnl_to_date_usd −314.58 (thesis P&L ends at the stop-out); the short round-trip is non-thesis P&L.

4. DB — cash, last. Cash hazard: cash_ledger excludes the short's proceeds (verified: Alpaca 100,523.70 vs ledger 99,505.33 pre-cover) and the quarantined/cover fills will never book them. After the cover, set cash_ledger.current_cash_usd and settled_cash_usd (singleton row id='current') to Alpaca's post-cover cash value — booking the round-trip as a non-thesis cash adjustment.

5. Expect one orphan surface for the cover. The monitor/recovery may capture the cover fill; with no matching order row it surfaces through the unattributed/quarantine channel with one RECONCILIATION_ALERT at the next collection. Acknowledge it; do not integrate.

6. Verify end state. Alpaca: MRVL flat, no open orders. DB: the CLOSE order row FILLED; position CLOSED with PnL −314.58 unchanged; cash_ledger equals Alpaca cash. scripts/reconcile_alpaca_state.py (read-only diff; prod box, .env loaded) reports no MRVL drift, and the next invocation raises no new MRVL alerts beyond step 5's.

Separate observation — do NOT bundle

The strategist/PM deliberated over dead inputs: the GS pending-order assessments (ENV-SA-ORD-3/4) wrote "maintain" rationales for protective orders that had filled or auto-canceled 47 minutes earlier, and the GS/MRVL position assessments evaluated positions already stopped out. This incident cost nothing extra from it — GS escaped a second naked short only because the PM happened to approve HOLD with zero commands. A deterministic pre-PM refresh of position/pending-order state (or marking known-dead inputs in the snapshot) is a decision-layer concern with a different fix surface; file separately if wanted.

Related

ALP-942 — the monitor SQLITE_BUSY_SNAPSHOT crash ran alongside this incident (crashed 17:00:05Z on the re-protection cancel event, restarted 17:01:20Z) and delayed fill capture by ~75 seconds. It did not cause this bug — capture at 17:01:22Z was still 46 minutes before the close — and this fix stands without it.

ALP-937 / ALP-939 — built the protective-leg cancellation this CLOSE path runs; the tolerate-already-terminal design decision corrected here shipped there.

ALP-938 — the auto re-protection whose stop legitimately flattened MRVL (and GS) at 17:00:06Z; correct behavior, and the drift source in this incident.

Review in Linear