Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,12 @@
- Coordinate-first resolved element activation: iOS/macOS runner interaction pattern where a selector or text query resolves the semantic `XCUIElement`, then activation uses the element's resolved center coordinate when a frame is available. This keeps target selection semantic while avoiding `XCUIElement.tap()` post-action element re-resolution after normal navigation. tvOS remains focus/remote-driven.
- Interaction dispatch path: one concrete route an interaction command takes to the device (runtime selector/ref resolution, direct iOS selector, native ref via web clickRef, coordinate, maestro non-hittable fallback). Every path classifies every guarantee in the ADR 0011 registry.
- Gesture plan: typed, platform-neutral normalization of one- or two-contact gesture intent into bounded pointer trajectories. Contact topology is separate from motion; two-contact intent remains pan/pinch/rotate/transform even when native injection shares one executor. See ADR 0013.
- Android planned-touch executor: Android-local adapter seam that accepts `AndroidTouchPlan`—the
platform-neutral `GesturePlan` plus Android's stationary long-press plan—and selects the paired
provider-native touch/viewport adapter or bundled instrumentation-helper adapter. Scroll and
long-press retain their command semantics and only share physical touch execution through this
seam. Helper long-press executes its absolute stationary path without a viewport probe; provider
long-press receives its paired provider-owned viewport. See ADR 0013.
- Multi-touch geometry: the internal initial span and angle plus centroid translation, scale, and rotation used to build both contact trajectories. Geometry is viewport-aware and fails early when the requested motion cannot fit; it is not a public tuning surface.
- Guarantee cell: one (dispatch path, guarantee) entry in `src/contracts/interaction-guarantees.ts`, classified as runtime/runner/delegated/inapplicable/waived. Completeness is a compile error; honesty is gate-tested.
- Owned waiver: a `gap:`-prefixed waived cell carrying a `trackingIssue` URL. Waivers are diffable debt with an owner, never folklore.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@
public final class MultiTouchInstrumentation extends Instrumentation {
private static final String PROTOCOL = "android-multitouch-helper-v1";
private static final String HELPER_API_VERSION = "1";
private static final int MIN_DURATION_MS = 16;
private static final int MAX_DURATION_MS = 10_000;
private static final int MIN_DURATION_MS = 0;
private static final int MAX_DURATION_MS = 120_000;
private Bundle arguments;

@Override
Expand Down Expand Up @@ -204,7 +204,8 @@ private static PointerPath[] readPointers(JSONArray pointers, String kind, int d
throw new IllegalArgumentException("Planned sample coordinates must be finite");
}
long offsetMs = (long) rawOffsetMs;
if (offsetMs <= previousOffsetMs) {
if (offsetMs < previousOffsetMs
|| (offsetMs == previousOffsetMs && durationMs != 0)) {
throw new IllegalArgumentException("Planned sample offsets must be strictly increasing");
}
parsed[sampleIndex] = new PointerSample(offsetMs, (float) x, (float) y);
Expand Down Expand Up @@ -307,7 +308,7 @@ private static MotionEvent motionEvent(

private static int requireDuration(int value) {
if (value < MIN_DURATION_MS || value > MAX_DURATION_MS) {
throw new IllegalArgumentException("durationMs must be between 16 and 10000");
throw new IllegalArgumentException("durationMs must be between 0 and 120000");
}
return value;
}
Expand Down
26 changes: 18 additions & 8 deletions docs/adr/0013-unified-gesture-plans.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@ Two-contact geometry also differed by platform. Android used a fixed radius whil
radius from the app frame. Neither path validated every planned point against the active
interaction viewport before injection.

`scroll` remains separate because it owns viewport/edge traversal, content-state verification, and
runner fallback policy rather than a single physical gesture.
`scroll` remains a separate command because it owns viewport/edge traversal and content-state
verification rather than a single physical gesture. Its Android physical movement still lowers to
the Android planned-touch executor.

## Decision

Expand Down Expand Up @@ -54,11 +55,16 @@ no established automation use case justifies a public tuning surface.

Platform adapters consume the canonical plan:

- Android sends the plan to provider-native touch injection when available, otherwise to the
bundled instrumentation helper. The helper injects the exact planned pointer samples. A
two-contact plan never falls back to `adb input swipe`; issue #690 separately owns removal of the
existing one-contact fallback. The snapshot helper is stopped before local gesture
instrumentation because Android permits only one instrumentation owner of `UiAutomation`.
- Android's `executeAndroidTouchPlan` adapter seam sends planned touch, including gesture plans plus
the physical movement for scroll and long-press, to provider-native touch injection when
available, otherwise to the bundled instrumentation helper. The helper injects the exact planned
pointer samples. A stationary long-press needs no viewport on the helper path; the executor adds
the paired provider-owned viewport only for provider-native touch. Android touch execution never
falls back to `adb input swipe`. Public scroll durations below one 16 ms planner frame normalize
to that physical minimum and report the executed duration. Scroll evidence reports absolute
injected coordinates against zero-origin extents that include the viewport offset. The snapshot helper is stopped
before local gesture instrumentation because Android permits only one instrumentation owner of
`UiAutomation`.
- iOS converts every planned point to native orientation and feeds the exact arrays to the existing
private XCTest event bridge. macOS lowers a one-contact plan to its drag executor and tvOS lowers
it to remote direction. Core admission and the Apple adapter both consume the same shared
Expand Down Expand Up @@ -93,6 +99,9 @@ selectors or refs and therefore cannot claim element-targeting guarantees.
- One-finger pan remains the default and explicit two-finger pan retains pan intent.
- The active viewport is resolved for each gesture, so rotation, keyboard, and window changes do
not use stale geometry.
- On bare ADB, Android scroll and long-press require the bundled touch helper and `UiAutomation`;
helper installation or runtime failure is surfaced directly rather than degrading to an
approximate `adb input swipe`.
- Pointer plans are larger than scalar requests but bounded by duration and the 16 ms sample
cadence; deleting duplicate scalar executors offsets the package cost.

Expand All @@ -106,4 +115,5 @@ selectors or refs and therefore cannot claim element-targeting guarantees.
fling/swipe as a quick directional throw and pan as deliberate timed translation.
- Add `two-finger-pan`: rejected because pointer count is topology, not a new motion intent.
- Expose span/angle controls: rejected until a concrete automation use case needs them.
- Consolidate scroll: rejected because edge/content verification and fallback policy are distinct.
- Consolidate scroll command semantics: rejected because edge/content verification is distinct;
only its Android physical touch execution is shared.
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ test('request envelopes deviating from the default are bounded, reviewed sets',
install: 180_000,
reinstall: 180_000,
install_source: 180_000,
longpress: 210_000,
test: 'unbounded',
};
for (const descriptor of commandDescriptors) {
Expand Down
8 changes: 7 additions & 1 deletion src/core/command-descriptor/registry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -703,7 +703,13 @@ export const RAW_COMMAND_DESCRIPTORS = [
daemon: { route: 'interaction', replayScopedAction: true, androidBlockingDialogGuard: true },
dispatch: {},
capability: { apple: APPLE_SIM_AND_DEVICE, android: ANDROID_ALL, linux: LINUX_DEVICE },
timeoutPolicy: interactionTimeoutPolicy('longpress'),
timeoutPolicy: {
...SETTLE_FLAG_PRESERVE_DAEMON_TIMEOUT_POLICY,
// Android's cold path may inspect/install the helper, hand off a running
// snapshot helper, hold for 120 seconds, then use 15 seconds of helper
// completion overhead. Keep that complete route inside the envelope.
envelopeMs: 210_000,
},
postActionObservation: postActionObservation('longpress'),
batchable: true,
},
Expand Down
6 changes: 3 additions & 3 deletions src/core/interactors/android.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ import {
typeAndroid,
} from '../../platforms/android/input-actions.ts';
import {
performGestureAndroid,
executeAndroidTouchPlan,
readAndroidGestureViewport,
} from '../../platforms/android/multitouch-helper.ts';
} from '../../platforms/android/touch-executor.ts';
import {
readAndroidClipboardText,
writeAndroidClipboardText,
Expand Down Expand Up @@ -53,7 +53,7 @@ export function createAndroidInteractor(device: DeviceInfo): Interactor {
type: (text, delayMs) => typeAndroid(device, text, delayMs),
fill: (x, y, text, delayMs) => fillAndroid(device, x, y, text, delayMs),
scroll: (direction, options) => scrollAndroid(device, direction, options),
performGesture: (plan) => performGestureAndroid(device, plan),
performGesture: (plan) => executeAndroidTouchPlan(device, plan),
gestureViewport: () => readAndroidGestureViewport(device),
screenshot: (outPath, options) => screenshotAndroid(device, outPath, options),
snapshot: async (options) => {
Expand Down
32 changes: 21 additions & 11 deletions src/daemon/__tests__/recording-gestures.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -108,26 +108,36 @@ test('scroll augmentation preserves explicit reference frame from platform resul
assert.equal(augmented.y2, 240);
});

test('scroll augmentation preserves explicit pixel travel coordinates', () => {
test('scroll visualization preserves absolute travel in its zero-origin reference frame', () => {
const session = makeSession();
session.snapshot = undefined;

const augmented = augmentScrollVisualizationResult(session, 'scroll', ['down'], {
direction: 'down',
pixels: 240,
x1: 201,
y1: 557,
x2: 201,
y2: 317,
referenceWidth: 402,
referenceHeight: 874,
x1: 211,
y1: 577,
x2: 211,
y2: 337,
referenceWidth: 412,
referenceHeight: 894,
}) as Record<string, unknown>;

assert.equal(augmented.x1, 201);
assert.equal(augmented.y1, 557);
assert.equal(augmented.x2, 201);
assert.equal(augmented.y2, 317);
recordTouchVisualizationEvent(session, 'scroll', ['down'], augmented, {}, 1_500);

assert.equal(augmented.x1, 211);
assert.equal(augmented.y1, 577);
assert.equal(augmented.x2, 211);
assert.equal(augmented.y2, 337);
assert.equal(augmented.pixels, 240);
const event = session.recording?.gestureEvents[0];
assert.equal(event?.kind, 'scroll');
assert.equal(event?.referenceWidth, 412);
assert.equal(event?.referenceHeight, 894);
assert.equal(event?.x, 211);
assert.equal(event?.y, 577);
assert.equal(event?.x2, 211);
assert.equal(event?.y2, 337);
});

test('gesture recording prefers native runner timing when available', () => {
Expand Down
180 changes: 100 additions & 80 deletions src/platforms/android/__tests__/input-actions.test.ts
Original file line number Diff line number Diff line change
@@ -1,50 +1,120 @@
import { test } from 'vitest';
import assert from 'node:assert/strict';
import { promises as fs } from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import {
fillAndroid,
longPressAndroid,
rotateAndroid,
scrollAndroid,
swipeAndroid,
typeAndroid,
} from '../input-actions.ts';
import type { DeviceInfo } from '../../../kernel/device.ts';
import { AppError } from '../../../kernel/errors.ts';
import { withScriptedAdb } from '../../../__tests__/test-utils/mocked-binaries.ts';
import { ANDROID_EMULATOR } from '../../../__tests__/test-utils/index.ts';
import { withAndroidAdbProvider, type AndroidTouchInjector } from '../adb-executor.ts';

test('scrollAndroid supports explicit pixel travel distance', async () => {
await withScriptedAdb(
'agent-device-android-scroll-pixels-',
[
'#!/bin/sh',
'printf "%s\\n" "$@" >> "$AGENT_DEVICE_TEST_ARGS_FILE"',
'if [ "$1" = "-s" ]; then',
' shift',
' shift',
'fi',
'if [ "$1" = "shell" ] && [ "$2" = "wm" ] && [ "$3" = "size" ]; then',
' echo "Physical size: 1080x1920"',
' exit 0',
'fi',
'exit 0',
'',
].join('\n'),
async ({ argsLogPath, device }) => {
const result = await scrollAndroid(device, 'down', { pixels: 240, durationMs: 120 });
const args = await fs.readFile(argsLogPath, 'utf8');
test('scrollAndroid plans explicit pixel travel through semantic touch injection', async () => {
const touchCalls: Parameters<AndroidTouchInjector>[0][] = [];
const result = await withAndroidAdbProvider(
{
exec: async () => {
throw new Error('adb must not run');
},
gestureViewport: async () => ({ x: 10, y: 20, width: 1080, height: 1920 }),
touch: async (request) => {
touchCalls.push(request);
return { injected: true };
},
},
{ serial: ANDROID_EMULATOR.id },
async () => await scrollAndroid(ANDROID_EMULATOR, 'down', { pixels: 240, durationMs: 120 }),
);

assert.equal(touchCalls.length, 1);
const touch = touchCalls[0]!;
assert.equal(touch.intent, 'pan');
assert.deepEqual(touch.pointers[0]?.samples[0]?.point, { x: 550, y: 1100 });
assert.deepEqual(touch.pointers[0]?.samples.at(-1)?.point, { x: 550, y: 860 });
assert.equal(result.pixels, 240);
assert.equal(result.durationMs, 120);
assert.equal(result.referenceWidth, 1090);
assert.equal(result.referenceHeight, 1940);
assert.equal(result.x1, 550);
assert.equal(result.y1, 1100);
assert.equal(result.x2, 550);
assert.equal(result.y2, 860);
assert.equal(result.backend, 'provider-native-touch');
assert.equal(result.injected, true);
});

assert.match(args, /shell\ninput\nswipe\n540\n1080\n540\n840\n120\n/);
assert.doesNotMatch(args, /uiautomator|dump/);
assert.equal(result.pixels, 240);
assert.equal(result.durationMs, 120);
assert.equal(result.referenceWidth, 1080);
assert.equal(result.referenceHeight, 1920);
test('scrollAndroid accepts sub-frame public durations at the Android planner minimum', async () => {
const touchCalls: Parameters<AndroidTouchInjector>[0][] = [];
const results = await withAndroidAdbProvider(
{
exec: async () => {
throw new Error('adb must not run');
},
gestureViewport: async () => ({ x: 0, y: 0, width: 1080, height: 1920 }),
touch: async (request) => {
touchCalls.push(request);
},
},
{ serial: ANDROID_EMULATOR.id },
async () => {
const outputs: Record<string, unknown>[] = [];
for (const durationMs of [0, 15]) {
outputs.push(await scrollAndroid(ANDROID_EMULATOR, 'down', { durationMs }));
}
return outputs;
},
);

assert.deepEqual(
touchCalls.map((call) => call.durationMs),
[16, 16],
);
assert.deepEqual(
results.map((result) => result.durationMs),
[16, 16],
);
});

test('longPressAndroid sends a stationary semantic touch plan', async () => {
const touchCalls: Parameters<AndroidTouchInjector>[0][] = [];
const result = await withAndroidAdbProvider(
{
exec: async () => {
throw new Error('adb must not run');
},
gestureViewport: async () => ({ x: 10, y: 20, width: 300, height: 500 }),
touch: async (request) => {
touchCalls.push(request);
},
},
{ serial: ANDROID_EMULATOR.id },
async () => await longPressAndroid(ANDROID_EMULATOR, 30, 40, 750),
);

assert.deepEqual(touchCalls, [
{
topology: 'single',
intent: 'longPress',
durationMs: 750,
viewport: { x: 10, y: 20, width: 300, height: 500 },
pointers: [
{
pointerId: 0,
samples: [
{ offsetMs: 0, point: { x: 30, y: 40 } },
{ offsetMs: 750, point: { x: 30, y: 40 } },
],
},
],
},
]);
assert.equal(result.backend, 'provider-native-touch');
});

test('rotateAndroid locks auto-rotate and sets user rotation', async () => {
await withScriptedAdb(
'agent-device-android-rotate-landscape-left-',
Expand All @@ -59,56 +129,6 @@ test('rotateAndroid locks auto-rotate and sets user rotation', async () => {
);
});

test('swipeAndroid invokes adb input swipe with duration', async () => {
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'agent-device-swipe-test-'));
const adbPath = path.join(tmpDir, 'adb');
const argsLogPath = path.join(tmpDir, 'args.log');
await fs.writeFile(
adbPath,
'#!/bin/sh\nprintf "%s\\n" "$@" > "$AGENT_DEVICE_TEST_ARGS_FILE"\nexit 0\n',
'utf8',
);
await fs.chmod(adbPath, 0o755);

const previousPath = process.env.PATH;
const previousArgsFile = process.env.AGENT_DEVICE_TEST_ARGS_FILE;
process.env.PATH = `${tmpDir}${path.delimiter}${previousPath ?? ''}`;
process.env.AGENT_DEVICE_TEST_ARGS_FILE = argsLogPath;

const device: DeviceInfo = {
platform: 'android',
id: 'emulator-5554',
name: 'Pixel',
kind: 'emulator',
booted: true,
};

try {
await swipeAndroid(device, 10, 20, 30, 40, 250);
const args = (await fs.readFile(argsLogPath, 'utf8')).trim().split('\n').filter(Boolean);
assert.deepEqual(args, [
'-s',
'emulator-5554',
'shell',
'input',
'swipe',
'10',
'20',
'30',
'40',
'250',
]);
} finally {
process.env.PATH = previousPath;
if (previousArgsFile === undefined) {
delete process.env.AGENT_DEVICE_TEST_ARGS_FILE;
} else {
process.env.AGENT_DEVICE_TEST_ARGS_FILE = previousArgsFile;
}
await fs.rm(tmpDir, { recursive: true, force: true });
}
});

test('typeAndroid chunks ASCII input text for shell fallback', async () => {
await withScriptedAdb(
'agent-device-android-type-ascii-chunked-',
Expand Down
Loading
Loading