Skip to content

Navigator: a step that changed nothing and sent no request does not resolve - #302

Merged
DavertMik merged 5 commits into
mainfrom
fix-navigator-false-success
Nov 12, 2026
Merged

DavertMik merged 5 commits into
mainfrom
fix-navigator-false-success

Conversation

@DavertMik

@DavertMik DavertMik commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Navigator.resolveState without an expectedUrl (the path interact() takes) set resolved = attempt.ok. The first code block that ran without throwing ended the loop as a success, even if it did nothing.

Seen in Langfuse session ThoroughIndustrialSalmon765: the Tester asked interact to open project settings. Four blocks failed, then I.click({role:'link', text:'Project'}) ran fine but most likely hit another visible "Project" link on the same page. URL, ARIA and HTML were all unchanged, yet interact returned success: true ("Successfully executed: Navigate to project settings…"). The Tester then improvised a direct amOnPage.

Fix

When no URL is expected, a block that ran without an error is checked before it counts as done. It counts as done when any of these holds:

  1. It sent a request to the app. Action now counts XHR/fetch requests as they start during the attempt (sentRequests), so a slow request that has not answered yet still counts. "The app" means the base URL's host and its subdomains (isSameHostFamily), so analytics and other third-party calls are ignored.
  2. The page changed: a fresh capture differs from the page before the block (Diff.hasChanges(): URL, ARIA, HTML).
  3. Something happens within 1s after that capture: the app sends a request (waitForAppRequest, new), or the DOM changes (waitForDomChange, existing) and a second capture shows a difference. This covers effects that land after the readiness wait, which can end after ~350ms of DOM quiet.

If none holds, the attempt counts as failed: it is logged, added to the batch failures and fed back to the AI with Ran without error but nothing on the page changed, mirroring the existing Reached … but the page state did not change handling. No prompt text changed.

The "XHR/fetch to the app" check was written out in Action and XhrCapture; it is now one isAppRequest() in url-matcher.ts, used by both and by waitForAppRequest().

Trade-off

The 1s settle only runs when a block looks like it did nothing, so the normal path costs nothing. Steps that are done yet change nothing and send no request (hover, scroll, copy to clipboard) still count as failed and the Navigator moves to the next solution. An app that polls its own backend can make a no-op click look like it did something; that only makes the Navigator accept a click, as it always did before this PR.

Not in this PR

The same session showed the Navigator using the ARIA snapshot's {img} placeholder as a link name ({role:'link', text:'{img}'}), which can never match. Fixing that needs prompt wording, so it is left for a separate change once the wording is agreed.

Tests

  • A block that runs fine but changes nothing and sends nothing does not resolve; its failure is fed back and no flow is saved.
  • A block that sent an app request resolves even though the page is unchanged.
  • A page that changes only after the first capture resolves.
  • A request to the app during the settle resolves; a third-party request (analytics.example.net) does not.
  • bun test tests/integration/ passes (175). The 8 Provider/Langfuse-env failures in tests/unit/ fail the same way on clean main. Lint and tsc clean on the changed files.
  • Touches src/ai/, so regression is queued behind the approval gate. Not started.

🤖 Generated with Claude Code

Without an expected URL, resolveState accepted the first code block that ran
without an error. A click on the wrong element (same page, nothing changed)
came back to the Tester as a successful interact(). Compare the page after the
block with the page before it and treat an unchanged page as a failed attempt,
fed back to the AI like the existing "page state did not change" case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DenysKuchma

Copy link
Copy Markdown
Collaborator

One question: capture() runs right after the attempt — a delayed effect (animation, background request) could be misread as "nothing changed". Is there a settle before capture, or is the retry feedback enough?

Before calling a step "nothing changed", count it as done when it sent an
XHR/fetch to the app's own domain (subdomains included, third parties such as
analytics excluded), or when the page changes or the app sends a request
within 1s after the first capture. Action now counts app requests as they
start, so a slow request that has not answered yet is still seen.

The "XHR/fetch to the app" check moves into isAppRequest() and is shared by
Action, XhrCapture and the new waitForAppRequest().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DavertMik DavertMik changed the title Navigator: a step that changed nothing does not resolve Navigator: a step that changed nothing and sent no request does not resolve Nov 11, 2026
DavertMik and others added 2 commits November 12, 2026 02:07
…method

attemptChangedPage() runs the request, capture and settle checks as early
returns in order, so the loop body only records progress or the failure.
Same behaviour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uccess

# Conflicts:
#	CHANGELOG.md
#	src/ai/navigator.ts
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Explorbot Self-Regression

Commit 881f1a3 · run

Scenario Result Time Previous median Change
basic (native) PASS 4m 2s 5m 7s -21% faster
experience: control OK — failed as expected 54s 30s +80% slower
experience: seeded PASS 53s 44s +20% slower
slow (native) PASS 2m 14s 2m 14s 0%
gated (native) PASS 2m 15s 2m 2s +11%
staged-login (native) PASS 31s 32s -3%
gated-no-auth (native) PASS 17s 21s -19%

Change compares it with the median passing time of the last 5 runs; ±20% or more is flagged.

Run details

  • basic (native) — PASS (4m 2s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=4, wellFormed=true, keywords=7/3); scenarios: PASS (tests=8/5, features=4/3); tests passed: PASS (5 passed, 0 failed (reporter: 5 passed, 0 failed))
  • experience: control — PASS (54s): control: OK — failed as expected (0 passed, 0 failed)
  • experience: seeded — PASS (53s): seeded: PASS (1 passed, 0 failed)
  • slow (native) — PASS (2m 14s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=2, wellFormed=true, keywords=7/3); scenarios: PASS (tests=8/5, features=4/3); tests passed: PASS (1 passed, 0 failed (reporter: 1 passed, 0 failed))
  • gated (native) — PASS (2m 15s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=4, wellFormed=true, keywords=7/3); scenarios: PASS (tests=7/5, features=3/3); tests passed: PASS (1 passed, 0 failed (reporter: 1 passed, 0 failed))
  • staged-login (native) — PASS (31s): reached target: PASS; navigator alone: PASS (captain handovers: 0)
  • gated-no-auth (native) — PASS (17s): stopped: PASS (exited on its own before the 180s timeout); captain handovers: PASS (1/1)

Session analysis — basic (native):

Session Analysis

Issue listing works: title search, status filtering, opening a detail page, and creating an issue from the top navigation all behave as expected. The toolbar creation path could not be confirmed because its test never asserts the new row in the action log.

Coverage

What works

  • Create issue from top navigation — ET-2 Create a new issue from the top navigation and verify it appears in the issues list
  • Open issue detail — ET-3 Open an existing issue from its title link and verify the issue detail page shows that issue
  • Filter issues by status — ET-4 Filter the issues list by a status and verify only issues with that status are shown
  • Search issues by title — ET-5 Search issues by title text and verify the list shows only matching issues

Execution Issues

  • ET-1 Create a new issue from the in-page toolbar and verify it appears in the issues list — No post-submit table assertion in log; passed on notes.

Session analysis — slow (native):

Session Analysis

Issue creation works end to end: a new issue submitted from the drawer form appears in the issues list while existing issues remain visible. No defects or friction were observed on this path.

Coverage

  • Pages: /issues, /issues?new=1
  • Features: Issue creation via the New Issue drawer form, issues list display

What works

  • Issue creation — ET-1 Create a new issue with a title and verify it appears in the issues list

Session analysis — gated (native):

Session Analysis

The toolbar New Issue flow creates an issue that appears in the issues table with the entered title, priority and bug label. The create form has no Status field, so the status expectation was not tested as written.

Coverage

  • Pages: /workspace, /issues, /issues?new=1 (New Issue drawer)
  • Features: New Issue creation from toolbar, issues table listing

UX issues

  • New Issue form - No Status field, but the scenario expects the new row to reflect a chosen status (ET-1 Create a new issue from the toolbar and verify it appears in the issues list)

Execution Issues

  • ET-1 Create a new issue from the toolbar and verify it appears in the issues list - status check matched generic "open" text, not a chosen value
  • ET-1 Create a new issue from the toolbar and verify it appears in the issues list - issue count check has no matching log step

The merge left both PRs' pushes in formatSessionLog: #303's numbered push
into the current page group (with #300's messages appended after it), and
#300's leftover groups.get(), which crashed now that groups is an array.
Every Pilot session-log test failed on main.

Append the messages before the one numbered push, and start the landed
page's group after a navigating action, so the action stays under the
page it ran on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DavertMik
DavertMik merged commit e489011 into main Nov 12, 2026
3 checks passed

This branch was successfully deployed

1 active deployment
regression — 5798f06d Deployed Nov 12, 2026 by DavertMik via regression #215
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants