Skip to content

Stop two AI tests depending on timing they do not control - #5078

Merged
elias-ba merged 0 commit into
timeout-keep-status-timelinefrom
timeout-stable-message-order
Aug 17, 2026
Merged

Stop two AI tests depending on timing they do not control#5078
elias-ba merged 0 commit into
timeout-keep-status-timelinefrom
timeout-stable-message-order

Conversation

@elias-ba

Copy link
Copy Markdown
Contributor

Description

Two tests that passed alone and failed in a full run. That is the worst kind of red, because it lands on whoever pushes next rather than on whoever caused it. Both surfaced while working on the stack below this one, and neither is caused by it.

The AI chat tests picked a reply out of a session with List.last/1. Messages come back ordered by inserted_at alone and the schema keeps that to the second, so a question and the answer to it written inside the same second tie and come back either way round. Against a real model the answer takes seconds and the order holds; in a test both writes land in the same millisecond, so it was a coin toss that usually landed the right way up. They now pick by role, which is what each of them meant.

The ordering itself is deliberately left alone. It is only reachable when two messages share a second, which needs a reply to arrive instantly, and fixing it properly means a migration to microsecond timestamps. Worth doing, not worth doing here.

The credentials schema test asserted that nothing at all was logged. with_log/1 captures everything logged in the window, including from the async tests running beside it, so a warning anywhere else in the suite failed it. It now asserts the absence of the one message that would mean the types were not recognised, which is what the test is actually about.

Closes #__

Validation steps

  1. mix test test/lightning/ai_assistant/ai_assistant_test.exs test/lightning/credentials/schema_test.exs
  2. Revert assistant_message/1 back to List.last/1 and run the full suite a few times; the global chat meta assertion at ai_assistant_test.exs:3338 fails intermittently.

AI Usage

  • I have used Claude Code

Pre-submission checklist

  • I have performed an AI review of my code
  • I have ticked a box in "AI usage" in this PR

@github-project-automation github-project-automation Bot moved this to New Issues in Core Aug 17, 2026
@github-actions

Copy link
Copy Markdown

Security Review ✅

  • S0 (project scoping): No new/modified queries or web-layer entry points accept project_id from params; all changes operate on already-authorized sessions loaded in join/3 (lib/lightning_web/channels/ai_assistant_channel.ex:39), and the reaper broadcasts to session-scoped PubSub topics that only authorized subscribers receive (lib/lightning/ai_assistant/message_processor.ex:334).
  • S1 (authorization): No new handle_in events or controller actions are introduced; the modified message_status_changed handler only reformats broadcasts on the socket topic authorized at join time.
  • S2 (audit trail): N/A — the changes touch ai_chat_messages (user-generated chat content, not project/instance configuration), which is not subject to audit coverage in the existing codebase.

@codecov

codecov Bot commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.6%. Comparing base (88b5c6c) to head (dbe62b1).

Additional details and impacted files
@@                      Coverage Diff                       @@
##           timeout-keep-status-timeline   #5078     +/-   ##
==============================================================
+ Coverage                          90.5%   90.6%   +0.1%     
==============================================================
  Files                               422     421      -1     
  Lines                             20090   20012     -78     
==============================================================
- Hits                              18188   18138     -50     
+ Misses                             1902    1874     -28     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@elias-ba
elias-ba force-pushed the timeout-keep-status-timeline branch from be7f4bd to 25cad92 Compare August 17, 2026 05:34
@elias-ba
elias-ba force-pushed the timeout-stable-message-order branch 2 times, most recently from 1894d0c to dbe62b1 Compare August 17, 2026 09:24
@elias-ba
elias-ba merged commit dbe62b1 into timeout-keep-status-timeline Aug 17, 2026
1 check was pending
@elias-ba
elias-ba force-pushed the timeout-keep-status-timeline branch from 25cad92 to 88b5c6c Compare August 17, 2026 09:25
@elias-ba
elias-ba deleted the timeout-stable-message-order branch August 17, 2026 09:25
@github-project-automation github-project-automation Bot moved this from New Issues to Done in Core Aug 17, 2026
@elias-ba
elias-ba restored the timeout-stable-message-order branch August 17, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant