e2e round 2: re-verify, fix locator drift, add evaluator-review coverage - #7
Merged
Conversation
Two TestEvaluatorReviewFlow assertions were tautological (asserted 'x or True', always passing regardless of actual UI state). Also added TestEvaluatorOverrideFields: the override risk/reason controls (EvaluatorReviewPage.override_result's targets) had zero test coverage before this — only presence of the review section/button was checked. Stops short of submit_review() since there's no un-submit mutation to undo an irreversible PENDING_REVIEW -> COMPLETED transition on shared sandbox data.
Root-caused via live network capture: the frontend's actual GetAudits response returns an identical, non-recency-ordered set of ~10 rows across four separate fresh-draft creations, regardless of page reload or wait time -- a newly created draft is unreachable from the list's landing page. xfail test_new_draft_appears_in_draft_tab_with_wizard_link referencing the new row (reproduced 4/4 in isolation, no concurrent load).
REVIEW_SECTION and SUBMIT_REVIEW_BUTTON were written against the old
markup (an 'Evaluator Review' heading, a 'Submit Review' button). The
Jul 2026 single-page redesign replaced both with plain prose ('...
pending review by the evaluator.' / 'Ready to submit? ...') and a bare
'Submit' button -- confirmed live against a real PENDING_REVIEW audit
(id 1908). This was previously masked by tautological 'assert x or True'
test assertions (fixed in the prior commit); strengthening those
assertions surfaced the drift, not a product defect. Verified fix
against the live backend: TestEvaluatorReviewFlow now 3/3 passing.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/e2e/suite (543 tests, all 12 shard groups) to verify nothing regressed since the Phase 13 triage — all clean, zero unexplained failures.docs/app_bugs.md): the evaluations list's default query doesn't sort by recency — a newly created draft never shows up on the list page. Root-caused via live GraphQL capture, reproduced 4/4 in isolation.locators/evaluator_review_locators.py— stale against the July single-page redesign (old markup assumed a distinct "Evaluator Review" section/"Submit Review" button that no longer exist).assert x or Truechecks in the evaluator-review flow (tests/e2e/test_bulk_evaluation_flow.py) that always passed regardless of real state, and addedTestEvaluatorOverrideFields— first real coverage of the override risk-dropdown/reason-textarea controls.Test plan
tests/e2e/run clean locally (--splits 12 --group N -n 1 --reruns 2)locators/evaluator_review_locators.pyfix verified against a live PENDING_REVIEW audit (3/3 passing)main(post PR Path-filter CI: only run the test suites that changed #6 merge)