Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
## 2026-11-12

### Changes

- Updated CodeceptJS to 4.2.4, which ships chained step options. When a locator matches several elements, the agent now picks one as `I.click('Delete').at(2)` instead of `I.click('Delete', step.opts({ elementIndex: 2 }))` — same behavior, modern syntax.
- Generated tests now record the element picked for an ambiguous click as `I.click('Delete').at(2)`, so they stay short and replay without failing on multiple matches.
- [Prima] `do` and `check` now see the knowledge for the current page: knowledge files, `--knowledge` and `--auth`. Before, a sign-in step blocked with "no auth details" even when `--auth` was given.
## 2026-11-11

Expand All @@ -19,6 +22,7 @@
- **`EXPLORBOT_URL`** with a path or query (`https://app.example.com/projects/demo/?preview=pr-42`) now sets the base path and pinned query, the same as `web.url` in a config file or `--base-url`. Before, relative navigation was appended to the whole URL and opened pages like `/projects/demo/?preview=pr-42/projects/demo/settings`.
- [Pilot] The action history Pilot reviews before a pass/fail verdict is now in the order things happened. Returning to a page shows it as a new visit instead of merging it into the first one, and checks made right after a navigation appear under the page they ran on. Actions are numbered across the whole log. A check from an earlier visit is no longer mistaken for the final state.
- **Pilot sees notifications when it decides pass/fail** — A toast, alert or other message that appeared after an action is now part of the action log Pilot reads for its final verdict and for reset decisions. A test whose confirmation closed before it could be checked is no longer failed for showing "no confirmation".

## 2026-11-10

### Configuration
Expand Down
118 changes: 53 additions & 65 deletions bun.lock

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@
"bash-tool": "^1.3.15",
"chalk": "^5.6.2",
"cli-highlight": "^2.1.11",
"codeceptjs": "^4.2.0",
"codeceptjs": "^4.2.4",
"commander": "^14.0.1",
"debug": "^4.4.3",
"dedent": "^1.6.0",
Expand Down
8 changes: 6 additions & 2 deletions src/action.ts
Original file line number Diff line number Diff line change
Expand Up @@ -593,8 +593,12 @@ export const attachStepLogger = (target: ExecutedStep[], assertionsTarget?: Arra

const executed: ExecutedStep = { command: step.toCode(), success: !error };
if (Object.keys(step.opts || {}).length) {
const args = [step.humanizeArgs(), `step.opts(${JSON.stringify(step.opts)})`].filter(Boolean).join(', ');
executed.command = `${step.prefix}${step.actor}.${step.title}(${args})${step.suffix}`;
const { elementIndex, ...rest } = step.opts;
const args = [step.humanizeArgs()];
if (Object.keys(rest).length) args.push(`step.opts(${JSON.stringify(rest)})`);
let command = `${step.prefix}${step.actor}.${step.title}(${args.filter(Boolean).join(', ')})`;
if (elementIndex !== undefined) command += `.at(${JSON.stringify(elementIndex)})`;
executed.command = `${command}${step.suffix}`;
}
if (error) {
executed.error = errorToString(error);
Expand Down
2 changes: 1 addition & 1 deletion src/ai/pilot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1257,7 +1257,7 @@ export class Pilot implements Agent {
- "overlay: none" but Tester targets an overlay → overlay closed; re-trigger.
- "region:" in <state> → a large area appeared in place without navigation (subview, wizard step, panel). Direct Tester to act inside it; the rest of the page is still usable.
- Action SUCCESS but ariaDiff empty → may have worked without visible DOM change; check result message.
- MultipleElementsFound → nothing was clicked. Tell Tester to reuse the same locator with step.opts({ elementIndex: N }) from the numbered elements list.
- MultipleElementsFound → nothing was clicked. Tell Tester to reuse the same locator with .at(N) chained to it, N from the numbered elements list.
- Wrong page (settings vs feature) → getVisitedStates() then back() or reset(). Don't try breadcrumbs (SPA back-nav is unreliable).
- Click SUCCESS but executed locator ≠ explanation intent, or "skipped" attempts present → wrong element clicked.
- form(I.type()) SUCCESS but "element" shows a button/link → keys went to wrong element; click the input first.
Expand Down
6 changes: 3 additions & 3 deletions src/ai/rules.ts
Original file line number Diff line number Diff line change
Expand Up @@ -348,11 +348,11 @@ export const actionRule = dedent`
If nothing works, use visualClick() — it locates the target in a screenshot before clicking it.

When a click result reports several matches, pick one from its numbered list by position rather than guessing a new locator.
Reuse the same locator with step.opts({ elementIndex: N }) as the LAST argument. N is the "Element N" number.
Reuse the same locator and chain .at(N) to the command. N is the "Element N" number. Negative N counts from the end.

<example>
I.click('Remove', step.opts({ elementIndex: 2 }));
I.click({ role: 'link', text: 'Details' }, '.panel', step.opts({ elementIndex: 1 }));
I.click('Remove').at(2);
I.click({ role: 'link', text: 'Details' }, '.panel').at(1);
</example>

For checkboxes, prefer I.checkOption/I.uncheckOption over I.click.
Expand Down
14 changes: 7 additions & 7 deletions src/ai/tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,12 @@ export function createCodeceptJSTools({ explorer, stateManager, judge }: ToolDep
Container narrows search area. Use when page has multiple matching elements.
Example: Page has 3 "Delete" buttons in different rows:
I.click("Delete", ".row-1") - clicks Delete inside element with class row-1
I.click(locator, step.opts({ elementIndex: N })) - click the Nth element the locator matches
I.click(locator).at(N) - click the Nth element the locator matches
Use after a result reported several matches. N is the "Element N" number from that list.
step.opts() always goes LAST, after the container when there is one.
.at() chains to the command, after the container when there is one.
Example: the result listed 3 "Delete" buttons and you want the second:
I.click("Delete", step.opts({ elementIndex: 2 }))
I.click("Delete", ".table", step.opts({ elementIndex: 2 })) - same, narrowed to a container
I.click("Delete").at(2)
I.click("Delete", ".table").at(2) - same, narrowed to a container

IMPORTANT: This tool ONLY accepts click commands. For typing text, use form() tool.
CRITICAL: All commands MUST target the SAME element using different locators.
Expand All @@ -69,7 +69,7 @@ export function createCodeceptJSTools({ explorer, stateManager, judge }: ToolDep
2. I.click(text, container) - when the container is verified - e.g. I.click("Save", ".modal")
3. I.click(CSS, container) - e.g. I.click("#btn", ".modal")
4. I.click(CSS) or I.click(XPath) - when locator already includes context (ID, XPath)
After a result reporting multiple matches, reuse that locator with step.opts({ elementIndex: N }) as the last argument.
After a result reporting multiple matches, reuse that locator with .at(N) chained to the command.
`),
explanation: z.string().describe('Why you are clicking this element'),
}),
Expand Down Expand Up @@ -1332,7 +1332,7 @@ export async function failedToolResult(action: string, message: string, data?: R
{ intent }
);
if (!index) return result;
result.suggestion = `Element ${index} is the one meant. Repeat the action with step.opts({ elementIndex: ${index} }) as the last argument.`;
result.suggestion = `Element ${index} is the one meant. Repeat the action with the same locator and .at(${index}) chained to it.`;
return result;
}

Expand All @@ -1349,7 +1349,7 @@ function getMultipleElementsSuggestion(): string {
return dedent`
Multiple elements matched your locator, so that command did nothing — it selected no element and acted on none.
Read the numbered elements list and act on the one you meant by its number:
reuse the same locator with step.opts({ elementIndex: N }) as the last argument.
reuse the same locator with .at(N) chained to the command.
A match reported as not visible can never be acted on — pick one that is.
If none of them is the element you want, narrow the locator with a container or its full unique text.
If the matches cannot be told apart, click the one you mean by appearance with visualClick().
Expand Down
8 changes: 4 additions & 4 deletions tests/unit/click-ambiguity.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ describe('click on an ambiguous locator', () => {
expect(result.multipleElementsDetected).toBe(true);
expect(result.elements).toContain('Element 1:');
expect(result.elements).toContain('Element 2:');
expect(result.suggestion).toContain('elementIndex');
expect(result.suggestion).toContain('.at(N)');
expect(result.elements).not.toContain('Identical to element');
});

Expand Down Expand Up @@ -113,9 +113,9 @@ describe('judging an ambiguous match', () => {
};
const judgePicking = (index: number) => judgeAnswering((options) => options[index]).judge;

it('names the element the judge picks, by elementIndex', async () => {
it('names the element the judge picks, by .at(N)', async () => {
const result = await failedToolResult('form', 'Multiple elements (2) found', {}, multipleElementsError(), judgePicking(1), 'Toggle the control');
expect(result.suggestion).toContain('elementIndex: 2');
expect(result.suggestion).toContain('.at(2)');
});

it('keeps the numbered list when the judge rejects', async () => {
Expand Down Expand Up @@ -149,7 +149,7 @@ describe('judging an ambiguous match', () => {
const { deps, action } = fakeDeps(() => multipleElementsError());
let state = { url: '/settings', html: '<html><body></body></html>', ariaSnapshot: '- switch', id: 'before' };
deps.stateManager.getCurrentState = () => state;
const executed = `I.click({"role":"switch"}, step.opts({"elementIndex":2}))`;
const executed = `I.click({"role":"switch"}).at(2)`;
action.attemptExactElementIndex = async () => {
action.lastError = null;
action.executedSteps = [{ command: executed, success: true }];
Expand Down
2 changes: 1 addition & 1 deletion tests/unit/executed-steps.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ describe('executed steps', () => {

detachSteps();

expect(steps[0]?.command).toBe('I.click("Manual", step.opts({"elementIndex":2}))');
expect(steps[0]?.command).toBe('I.click("Manual").at(2)');
});

it('reports a successful command even when a later one fails', () => {
Expand Down
Loading