fix(data_sources): merge Alpaca indicative NBBO + IEX spot into the options collector (ALP-949)

[jatassi]

Summary

• Polygon's plan entitlement withholds NBBO quotes and underlying price from options chain snapshots, so every options_contract_snapshots row since 2026-04-27 (~30.1M) has NULL bid/ask/last_price/underlying_price — no IVr reference line renders, retrieve_options_chain returns nothing, premium validation rejects every options proposal, and option marks have no price.
• New vendor module data_sources/alpaca/options.py fetches the full chain on Alpaca's free indicative options feed (_OPTIONS_FEED is the single flip point for a future OPRA upgrade) and the underlying spot via one multi-symbol IEX latest-trade request.
• collect_options_chains merges per field at collection time — Polygon wins when present (correct under a future plan upgrade), Alpaca fills the gaps; matched via bare OCC symbol (O: prefix strip). source is polygon+alpaca when the chain fetch succeeded, else polygon.
• Any Alpaca failure (missing creds, vendor/transport error, response-shape change) degrades that scope to Polygon-only rows — never aborts the run. Socket-timeout shim on both clients (mirrors broker_adapter/client_factory.py, ALP-841) so a hung connection can't wedge the collector.
• No schema change, no schedule change, no migration. config/data_sources.yaml already carried the alpaca provider entry (Q1 failover) — no config edit, snapshot unchanged. docs/runbooks/services.md documents the dependency + degraded mode.

Note: two # type: ignore[no-untyped-call] comments on APIError(...) construction in tests, mirroring the existing pattern in tests/execution/broker_adapter/.

Closes https://linear.app/alphamind-jatassi/issue/ALP-949/options-chain-snapshots-carry-no-nbbo-underlying-price-polygon-plan

Test plan

• CI (.github/workflows/ci.yml) green on the PR — lint on Linux + full pytest on Windows
• Local lint chain clean (ruff check, ruff format --check, mypy, lint-imports)
• ALP-949 acceptance criteria all met (merge fill, Polygon precedence, per-underlying failure isolation, spot-failure degradation, bare-OCC keying, config suite green, runbook updated)

🤖 Generated with Claude Code

[linear]

ALP-949 Options chain snapshots carry no NBBO / underlying price (Polygon plan entitlement) — merge Alpaca indicative quotes + IEX spot into the options collector

Symptom

The first scheduled run under ALP-948 (inv-20260611T133500Z-22430078, 2026-06-11 market_open) rendered zero extended TICKER: price | IVr NN | exp: … reference-price lines for all 79 active-universe tickers — pipeline.log: analyst input bundle assembled (mode=normal, chars=3390, options_context_tickers=0) (the log line at src/alphamind/decision/analyst/runner.py:232). The analyst correctly followed the doctrine ("no extended line → propose equity only") and never called retrieve_options_chain. Without this fix the ALP-948 motivating symptom (every archived proposal is equity) persists indefinitely.

Evidence (prod DB + live Polygon + live Alpaca, 2026-06-11)

• Prod DB (immutable SMB snapshot): latest MU batch 2026-06-11T13:08:00Z — 10,392 rows; bid/ask/last_price/underlying_price non-null counts all 0; implied_volatility/delta 9,044; open_interest 10,392. Matches the original report's whole-table audit (0 usable rows of ~30.1M since the table's first batch 2026-04-27).
• Live Polygon probe (current plan key): GET /v3/snapshot/options/MU result objects carry exactly day, details, greeks, open_interest, underlying_asset — no last_quote, no last_trade, and underlying_asset contains only ticker (no price). The entitlement split is confirmed at the API itself: greeks/OI included, NBBO quotes/trades/underlying price withheld on this tier.
• Collector parsing is correct: src/alphamind/data_sources/polygon/options.py:153-178 maps last_quote.bid/ask, last_trade.price, underlying_asset.price with None fallbacks — the fields are simply never present.
• Live Alpaca probe (paper keys from .env, after the 2026-06-11 close): OptionHistoricalDataClient.get_option_chain(OptionChainRequest(underlying_symbol="MU", feed="indicative")) returned 10,392 contracts — exactly Polygon's chain size — with a two-sided latest_quote on 10,392/10,392 (bid/ask + sizes, quote timestamps at the close), latest_trade on 8,880, IV+greeks on 6,004. Alpaca option snapshots carry no open interest, no volume, no underlying price (model fields: symbol, latest_trade, latest_quote, implied_volatility, greeks).
• feed="opra" fails with APIError: "OPRA agreement is not signed". Despite the wording, there is no standalone agreement to sign: real-time OPRA on Alpaca is bundled into the paid Algo Trader Plus market-data subscription (~$99/month, dashboard "Plans & Features" → "Upgrade to AlgoTrader Plus"; the OPRA agreement is signed inside that subscription flow). On the Basic (free) data plan, indicative is the only available options feed — a free real-time derivative of OPRA (quotes indicative real-time; trades 15-min-delayed derivatives).
• Underlying spot: one multi-symbol StockLatestTradeRequest(feed=IEX) returned live trades for MU/NVDA/AAPL on these keys (free feed; same surface the codebase already uses at src/alphamind/execution/continuous_monitor/safety_core/price_feed.py:115).

Root cause (confirmed as reported)

The ALP-948 feature code is correct and fully wired. The Polygon options snapshot response on the current API plan never includes last_quote / last_trade / underlying_asset.price, so every options_contract_snapshots row has bid, ask, last_price, underlying_price NULL since the table's first batch. Blast radius, all downstream of those NULL columns:

(A) IVr context. SqlOptionsChainReader._chain_slice (src/alphamind/state/repository/options_chain_read.py:358-412) returns None for every underlying — spot is None because no row carries underlying_price — so build_options_context is always empty and no extended reference line can render.

(B) Chain tool. retrieve_options_chain returns the no-chain-data message for every underlying (same _chain_slice path; even past spot, _quote_from_row at options_chain_read.py:482-515 rejects every contract for the missing two-sided quote).

(C) Premium validation. latest_quote (options_chain_read.py:416-446) SQL-filters to usable rows (bid/ask/IV/delta non-null) → always None → any options proposal is rejected (option_contract_unknown), so even a hand-written options command cannot pass validate_analyst_output.

(D) Option position marks. src/alphamind/state/repository/sql_option_price_provider.py prices option positions from bid/ask midpoint with last_price fallback — both NULL, so the continuous monitor's greeks-refresh/pricing path for any future options position is equally inert.

Why it escaped tests

The collector tests fabricate Polygon snapshots via make_option_snapshot (tests/data_sources/_fakes/polygon.py:243-265), which populates last_quote, last_trade, and underlying_asset.price — the sanctioned vendor-API fake encodes fields the production plan does not deliver, and plan entitlement is observable only against the live API. ALP-948's decision-layer tests seed options_contract_snapshots rows directly with populated bid/ask, so the read path was proven against data that production never writes.

Reading

1. src/alphamind/data_sources/polygon/options.py — the collector to extend: collect_options_chains run shape + per-underlying error isolation (:56-121), _build_rows merge point (:124-179), upsert helpers.
2. src/alphamind/data_sources/polygon/client.py:36 — the data-layer credential convention (env var read directly) the new Alpaca module mirrors.
3. src/alphamind/state/repository/options_chain_read.py — the consumer contract the written columns must satisfy: _quote_from_row usability predicate (:482-515), latest_quote SQL filter (:429-438), spot selection (:391-399).
4. src/alphamind/state/repository/sql_option_price_provider.py — second consumer (mid primary, last_price fallback): explains why a missing Alpaca latest_trade is acceptable when the quote is present.
5. src/alphamind/persistence/models.py:253-286 — OptionsContractSnapshots: every needed column already exists; no Alembic migration.
6. src/alphamind/execution/continuous_monitor/safety_core/price_feed.py — existing alpaca-py stock-data precedent (DataFeed.IEX, StockLatestQuoteRequest shape).
7. config/data_sources.yaml + src/alphamind/config/models/data_sources.py — provider registry and its validators (api_key_env must exist in .env.example; ALPACA_PAPER_KEY/ALPACA_PAPER_SECRET already do, lines 9-10).
8. config/collector_schedule.yaml:10 — polygon.options cadence (8,38 9-16 * * mon-fri), unchanged.
9. tests/data_sources/_fakes/polygon.py:243-265 and tests/data_sources/polygon/test_polygon.py:410-520 — fake + test conventions to mirror for the Alpaca fake.
10. docs/runbooks/services.md — collector service runbook the change must update.

Scope — merge Alpaca indicative NBBO + IEX underlying spot into the existing polygon.options collection task

Alpaca cannot replace Polygon (no open interest / volume in its option snapshots — the chain filter's OI floor needs Polygon) and Polygon cannot supply quotes on the current plan — the two complement exactly, so the fix is a merge at collection time. No schema change, no schedule change, no new collector task.

(A) New vendor module. src/alphamind/data_sources/alpaca/options.py (plus empty __init__.py), following the per-vendor data_sources/ convention. Contents: a frozen dataclass OptionQuote with fields bid: float | None, ask: float | None, last_price: float | None; fetch_chain_quotes(underlying: str, *, _client: Any = None) -> dict[str, OptionQuote] fetching the full unfiltered chain via OptionHistoricalDataClient.get_option_chain(OptionChainRequest(underlying_symbol=..., feed=_OPTIONS_FEED)) — keys are Alpaca's bare OCC symbols (e.g. MU260612P01062500), bid/ask from latest_quote.bid_price/ask_price, last_price from latest_trade.price, each None when the sub-object is absent; fetch_underlying_trades(tickers: list[str], *, _client: Any = None) -> dict[str, float] via one multi-symbol StockLatestTradeRequest(symbol_or_symbols=tickers, feed=DataFeed.IEX) on StockHistoricalDataClient. _OPTIONS_FEED = OptionsFeed.INDICATIVE is a module-level constant — the single flip point if the operator later subscribes to Alpaca's Algo Trader Plus market-data plan (paid; unlocks the OPRA feed). Credentials: read ALPACA_PAPER_KEY / ALPACA_PAPER_SECRET env vars directly (the market-data plane accepts paper keys — verified live; mirrors the PolygonClient convention).

(B) Merge in collect_options_chains (src/alphamind/data_sources/polygon/options.py). Fetch the spot map once per run before the underlying loop (a failure logs a warning and leaves underlying_price None for the whole batch without aborting). Per underlying, fetch Alpaca chain quotes wrapped in with_retries(RetryShape.important) and a try/except mirroring the existing Polygon continue isolation — an Alpaca failure writes that underlying's Polygon-only rows exactly as today. _build_rows gains keyword params alpaca_quotes: Mapping[str, OptionQuote] | None and underlying_price: float | None; for bid/ask/last_price the Polygon-parsed value wins when present, the Alpaca value fills when Polygon's is None (today Polygon is always None, so Alpaca always fills; precedence stays correct under a future Polygon plan upgrade). The Alpaca lookup key is details.ticker with the O: prefix stripped. underlying_price is stamped on every row of the underlying. source is polygon+alpaca for every row of an underlying whose Alpaca chain fetch succeeded, else polygon — audit-only; nothing reads source programmatically (verified). IV and greeks stay Polygon-sourced (better coverage on the probe: 9,044 vs 6,004 of 10,392 for MU; keeps the q3 ATM-IV baseline single-sourced).

(C) Provider registry entry. Add to config/data_sources.yaml providers: alpaca with api_key_env: ALPACA_PAPER_KEY, rate_limit_per_minute: 200, retry_shape: important. The registry is validated config, not runtime wiring. A config value change can shift the resolved-config snapshot — run tests/config/test_snapshot.py and re-pin expected if it does.

(D) Tests. New fake tests/data_sources/_fakes/alpaca.py (fake option-chain + stock-trade clients injected via _client, the sanctioned broker-API boundary) and merge tests alongside the existing options tests in tests/data_sources/polygon/test_polygon.py. Existing make_option_snapshot defaults stay as-is (they now exercise Polygon precedence); merge tests pass quote-less overrides (bid=None, ask=None, last_price=None, underlying_price=None) to model the production response shape.

(E) Runbook. docs/runbooks/services.md collector documentation gains the new Alpaca data dependency of the polygon.options task (indicative options feed + IEX spot) and its degraded mode: on Alpaca failure the task still writes quote-less rows with source polygon, visible in collector.log warnings.

Out of scope: no Alembic migration (all columns exist); no backfill of the ~30.1M historical quote-less rows (readers take only the latest snapshot per contract, and latest_quote already SQL-filters to usable rows); no OPRA feed switch (gated on the paid Algo Trader Plus subscription — if purchased later, the flip is the one-line _OPTIONS_FEED constant); no change to config/options_chain.yaml read policy; the reference-price line's own price source is untouched.

Acceptance criteria

• With a Polygon fake delivering quote-less snapshots (the production shape) and an Alpaca fake delivering chain quotes and a spot price, collect_options_chains writes snapshot rows with bid, ask, last_price, underlying_price populated and source set to polygon+alpaca.
• A Polygon-delivered non-None bid/ask/last_price survives the merge unchanged (Polygon precedence per field).
• An Alpaca chain-fetch failure for one underlying still writes that underlying's quote-less Polygon rows with source polygon and does not affect other underlyings in the same run.
• A failed underlying-spot fetch leaves underlying_price None for the batch without aborting collection.
• fetch_chain_quotes keys its result by bare OCC symbol, and the merge matches Polygon contract_ticker rows via the O: prefix strip (test with a contract present on both sides and one missing from the Alpaca chain).
• config/data_sources.yaml carries the alpaca provider entry and the config test suite passes (snapshot re-pinned if shifted).
• docs/runbooks/services.md documents the collector's Alpaca dependency and the quote-less degraded mode.

Verification

uv run pytest tests/data_sources/ -n auto and uv run pytest tests/config/ -n auto; the lint chain (ruff check, ruff format, mypy, lint-imports); CI full suite on Windows is the authoritative gate. The runbook criterion is verified by inspection.

Post-deploy production confirmation (operator, next market-hours collection): SELECT COUNT(*), SUM(bid IS NOT NULL), SUM(underlying_price IS NOT NULL) FROM options_contract_snapshots WHERE snapshot_ts > '<deploy-ts>' shows non-zero usable counts; the next scheduled invocation logs options_context_tickers > 0 and its archived decision/analyst/user_message.md shows at least one extended | IVr line (IVr n/a is a pass — the rank renders n/a until the ATM-IV baseline calibrates, input_bundle.py:186); retrieve_options_chain returns contracts for a liquid name (MU/NVDA).

One-time prod data cleanup

No bulk cleanup needed. The historical quote-less rows are inert: chain reads use only the latest snapshot per contract, latest_quote SQL-filters to usable rows, and the IV/greeks columns (always populated) continue to feed the ATM-IV baseline unchanged.

Separate observation — do NOT bundle

Collector data-quality monitoring gap: run tracking counts rows_written only, so a payload column NULL across 100% of ~30.1M rows stayed invisible for six weeks. A NULL-rate check on collected columns is a distinct follow-up with its own root cause — file separately if wanted.

Related

ALP-948 — this issue unblocks its production behavior; the feature code itself needs no change.

Review in Linear