Stop two AI tests depending on timing they do not control - #5079
Open
elias-ba wants to merge 1 commit into
Open
Conversation
Both passed alone and failed in a full run, which is the worst kind of red: it lands on whoever pushed next rather than on whoever caused it. The chat tests picked a reply with List.last/1. Messages come back ordered by inserted_at alone and the schema keeps that to the second, so a question and its answer written inside the same second tie and come back either way round. Against a real model the answer takes seconds and the order holds; here both writes land in the same millisecond, so it was a coin toss that usually landed right. They now pick by role, which is what each of them meant. The ordering itself is left alone. It is only reachable when two messages share a second, which needs a reply to arrive instantly, and changing it means a migration to microsecond timestamps. Worth doing, not worth doing here. The schema test asserted that nothing at all was logged. with_log/1 captures everything logged in the window, including from the async tests running beside it, so any warning anywhere failed it. It now asserts the absence of the one message that would mean the types were not recognised, which is what the test is about.
Security Review ✅
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two tests that passed alone and failed in a full run, which is the worst kind of red: it lands on whoever pushes next rather than on whoever caused it.
The chat tests picked a reply with
List.last/1, but messages are ordered byinserted_atand that is stored to the second, so a question and its answer written in the same millisecond tie and come back either way round. They now pick by role, which is what each of them meant. The ordering itself is left alone; that needs a migration and is filed as #5081.The credentials schema test asserted that nothing at all was logged, which any warning from a neighbouring async test would break. It now asserts the absence of the one message it actually cares about.
Validation steps
mix test test/lightning/ai_assistant/ai_assistant_test.exs test/lightning/credentials/schema_test.exsList.last/1back in place ofassistant_message/1and run the full suite a few times. The global chat assertions fail on some runs and not others.Additional notes for the reviewer
AI Usage
Please disclose whether you've used AI anywhere in this PR (it's cool, we just want to know!):
You can read more details in our Responsible AI Policy
Pre-submission checklist
/reviewwith Claude Code):owner,:admin,:editor,:viewer)