Say why a stream stopped, and give each wait its own setting

[elias-ba]

Description

Two problems with one cause: nothing downstream could tell what had gone wrong, because the reason was thrown away before anyone could read it.

Tesla's Finch adapter halts a streamed body identically for a mid-stream error, a mid-stream timeout and a clean end. It returns nil in all three cases and binds the error to an unused variable. So a hung Apollo, a severed connection and a genuinely short answer were the same event to us, and all three reported "Stream ended without complete response". They now say three different things.

We use our own copy of that adapter which keeps the reason. A sentinel inside the stream is not possible, because the SSE middleware concatenates elements as binaries and would fail on anything else; but the unfold runs in the calling process, so the reason is left there and read back once the stream has been consumed. The copy also passes request_timeout through to Finch, which the 1.18.3 we are pinned to drops.

The copy is not upstream's file with two edits. It also loses upstream's build/4 clauses, so a multipart or streamed request body would raise rather than work, and it keeps only the newer of upstream's two version-gated error clauses. That is fine for Apollo, which sends JSON, and the moduledoc says what the module is for. Upstream has since fixed the reason and released it, so this whole file goes away when we bump Tesla, tracked in #5080. It is excluded from coverage on the way out, since it is upstream's code and only the lines we changed are ours to test.

The reason arrives as either Finch's transport struct or Mint's, and both carry it under the same key, so it is taken out before being matched on. Matching one struct silently missed the other, which put a dropped connection back on the generic sentence: the exact conflation this exists to end.

APOLLO_TIMEOUT is replaced by three settings that each answer one question: how long to wait to reach Apollo, how long a silence mid-answer is acceptable, and how long a whole request may take. One number could not answer all three, so it had to be sized for the longest, which meant a completely hung Apollo held the user on a spinner for five minutes. The staging and prod config is renamed in the same change, and all three have defaults so nothing has to be set.

Oban's drain window is derived from those three rather than fixed at the six minutes #5069 set, so raising one of them cannot leave a deploy killing AI jobs before they finish. OBAN_SHUTDOWN_GRACE_PERIOD_MS overrides it, and the boot warning now names it.

Apollo also gets its own Finch pool, which is how the connect timeout takes effect at all. An AI stream holds a connection for the length of an answer, and sharing the default pool meant a handful of them tied up connections other outbound calls were waiting on. It is pinned to http1 deliberately: on http2 Finch reinterprets the idle setting as a deadline for the whole request.

The adapter tests run over a raw socket rather than the Tesla mock, and are the first thing in the suite to exercise the lazy stream path at all, which is why this went unnoticed. Checked by reverting the two lines that keep the reason: the tests then report nil where they expect a timeout.

Closes #4882

Validation steps

1. Point Apollo at something that accepts the connection and then goes quiet
   (nc -l <port> is enough), set APOLLO_IDLE_TIMEOUT_MS=5000, and send a
   message. You should get an error in about five seconds rather than waiting
   out the whole request budget.
2. Read what was recorded, since the panel does not render it yet:

   import Ecto.Query
   Lightning.Repo.one(
     from m in Lightning.AiAssistant.ChatMessage,
       order_by: [desc: m.inserted_at], limit: 1
   ).failure_message

   For the stall above it should say the assistant stopped responding partway
   through. Kill the listener mid-answer instead and it should say the
   connection was lost. Those two used to read the same.
3. Check the settings are actually in force:

   Lightning.Config.apollo(:connect_timeout)
   Lightning.Config.apollo(:idle_timeout)
   Lightning.Config.apollo(:request_timeout)

4. Set APOLLO_REQUEST_TIMEOUT_MS above the drain window and restart, to see
   the boot warning name OBAN_SHUTDOWN_GRACE_PERIOD_MS.

Additional notes for the reviewer

1. lib/lightning/tesla/adapter/finch.ex is a copy of upstream's adapter and is most of the diff. Upstream has since fixed and released the bug behind it, so the file goes when we bump Tesla (#5080); it is excluded from coverage for that reason.
2. On AI assistant: streaming errors are hidden behind a generic message #4882: this gives the three failures three different sentences, which was the main complaint. It deliberately does not pass Apollo's own words through, since some of them carry internal hostnames and paths. #5072 is where that call is made.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

• I have used Claude Code
• I have used another model
• I have not used AI

You can read more details in our
Responsible AI Policy

Pre-submission checklist

• I have performed an AI review of my code (we recommend using /review
  with Claude Code)
• I have implemented and tested all related authorization policies.
  (e.g., :owner, :admin, :editor, :viewer) — none touched here
• I have updated the changelog. (for the whole stack, in #5074)
• I have ticked a box in "AI usage" in this PR

[github-actions]

I have enough context. The changes are:

• Config: split Apollo timeout into connect/idle/request timeouts
• ai_assistant.ex: preserve partial stream content when stream dies mid-response
• chat_message.ex: add failure_category and failure_message fields
• message_processor.ex: pass failure reasons; handle nil message case
• stuck_message_reaper.ex: new Oban cron worker that clears abandoned :processing messages
• apollo_client.ex: use custom Finch adapter with new timeout settings
• application.ex: dedicated Apollo Finch pool; separate telemetry handler ids
• bootstrap.ex: add env vars for new timeouts; add reaper to cron; raise shutdown_grace_period
• Custom Lightning.Tesla.Adapter.Finch: preserves stream error via process dictionary
• ai_assistant_channel.ex: include failure_category/failure_message in broadcasts
• Two migrations: add failure columns; partial index for stuck messages

Now scoping each check:

S0 (project scoping): The reaper is an internal Oban cron job (like Lightning.Janitor) — not a user-facing entry point, so no membership check applies. Its broadcast_message_error/1 posts to ai_session:<session_id>, whose channel already authorizes subscription via authorize_session_access (lib/lightning_web/channels/ai_assistant_channel.ex:39). The channel changes add fields to existing broadcasts on topics already gated by that auth. save_partial_response writes to the same session that made the request. No new query touches project-scoped resources without going through already-authorized paths.

S1 (authorization): No new user-triggered CRUD actions. The reaper isn't user-invoked; the channel change adds payload fields to existing broadcasts (no new handle_in).

S2 (audit trail): Chat messages are operational/session data, not in the audited config-resource set (credentials, project settings, workflows, webhook auth methods, oauth clients, version control). No matching audit module exists or is expected here.

Security Review ✅

• S0 (project scoping): New stream-error handling writes to the same authorized session, and the reaper/channel broadcasts go to ai_session:<id> topics that the channel's existing authorize_session_access already gates.
• S1 (authorization): N/A, no new user-triggered CRUD actions — the reaper is an internal Oban cron job and the channel change only adds fields to existing broadcasts.
• S2 (audit trail): N/A, chat-message status changes are operational data, not in the audited config-resource set (credentials, project settings, workflows, OAuth clients, etc.).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.5%. Comparing base (6874963) to head (5434625).

Additional details and impacted files

@@                       Coverage Diff                       @@
##           timeout-keep-partial-response   #5073     +/-   ##
===============================================================
- Coverage                           90.6%   90.5%   -0.1%     
===============================================================
  Files                                422     422             
  Lines                              20066   20075      +9     
===============================================================
- Hits                               18178   18175      -3     
- Misses                              1888    1900     +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.