diff --git a/docs/architecture/computer-use-control.md b/docs/architecture/computer-use-control.md index 78a89fbaad..6708131c42 100644 --- a/docs/architecture/computer-use-control.md +++ b/docs/architecture/computer-use-control.md @@ -430,8 +430,12 @@ For one unmodified left ImageXy click, native application-root AX hit testing ma select an explicitly pressable button, checkbox, radio button or link. The hit must belong to the exact bound window and contain the point. Do not rank nearby rectangles, climb ancestors or replace double/right/modifier clicks with AXPress. -Only explicit ActionUnsupported permits a different delivery path; unknown AX -outcomes stop without a duplicate click. +For an attempted AX mutation, only explicit ActionUnsupported permits a different +delivery path; unknown mutation outcomes stop without a duplicate click. The +optional read-only point lookup has a local 300 ms messaging timeout. An absent, +unsupported or temporarily unresponsive AX hit test retains the independently +validated window-coordinate route; permission, identity and unknown errors stay +visible. This is a route choice before input, never a retry of an uncertain action. Text/key `focus` uses retained AX identity or native hit testing and semantic focus for text controls. Other eligible targets use the exact-window directed @@ -463,7 +467,10 @@ The directed mouse source is `CombinedSessionState`, with modifier flags set explicitly to the requested modifiers. A private mouse source itself caused AppKit to raise inactive windows in the strict fixture; changing only that source is therefore part of the routing contract, not a reason to add Command. Keyboard -events retain a private source. Event field 58 is set to 1 for window routing; it +events retain a private source. A chord uses one source, unique modifiers and +cumulative press/release flags, ending with no modifiers held. Native modifier +events are `FlagsChanged`, so cancellation tracks and releases them as well as +ordinary keys. Event field 58 is set to 1 for window routing; it is not a click-group identifier and must not be repurposed as one. These event fields are native adapter details, not application-specific rules. @@ -477,10 +484,15 @@ window-addressed down/up focus pair is placed outside the content frame at local observed target coordinates and original modifiers. There is no implicit Command, global foreground switch, window raise/restore or human-cursor warp. -The preparation cache includes generation, PID, window ID, human foreground -identity and activation epoch, so preparation is not repeated for every key or -batch step while stale activation state cannot be reused. There is no app-name -rule or guessed content-control coordinate. Preparation does not depend on an AX +The preparation cache includes generation, PID, process-serial identity, window +ID, human foreground identity, global activation epoch and target-local loss +epoch. A PID-scoped, listen-only event tap invalidates preparation when native +deactivation or key-focus loss occurs, even when the human foreground PID does +not change. It returns every event unchanged and never suppresses input. A +missing or disabled observer disables cache reuse while keeping directed input +available; it does not trigger foreground takeover. Stop, target replacement and +capture failure remove the matching observer. There is no app-name rule or +guessed content-control coordinate. Preparation does not depend on an AX key-state acknowledgement, which normal AppKit windows can omit; fixtures inspect their own `NSWindow.keyWindow` directly. @@ -597,6 +609,132 @@ Local fixtures and unsupported-host unit tests are not remote end-to-end evidenc ## Verification and acceptance +### Background input evidence, 2026-10-17 + +REA inspected the installed Codex 26.1013.31256 artifact and its `@oai/sky` +0.7.8 macOS client. The native `SkyComputerUseService` SHA-256 is +`f21c31c0ee5df419dbd165bf76ebd65c64d864c6fec5b6c6c3a23c42ff3d2e13`. +JavaScript inventory/analysis covered 41 files without parse failures; 1,513 +dynamic/external semantic relationships remained unknown. Native evidence is +Mach-O metadata plus symbol-correlated ARM64 disassembly, not recovered Swift +source: a configured native decompiler was unavailable. + +The inspected paths establish these implementation facts: + +- The macOS client keeps a transport per API version, serializes requests and + uses length-prefixed JSON messages over a Unix socket, with request IDs and + deadlines. Input receipts and subsequent application-state reads are separate. +- `SynthesizedEvent.send(to:)` at `0x10075ad0c` calls the PID-directed event + wrapper. Mouse construction supplies a window number and window-local point. + PID targeting alone is not the entire background-input mechanism. +- `SyntheticAppFocusEnforcer` distinguishes actual foreground ownership from + whether the target believes itself active/focused. `enforceActiveState` at + `0x1007479fc` constructs target-local activation events; `startMonitoring` at + `0x100747348` observes target-process events and updates that local state. +- `safelyEnforceActiveState` at `0x1007482a8` also contains temporary event taps + and conditional foreground restoration: `0x100748c64` calls + `activateWithOptions:` on the saved foreground application. Its tap callbacks + include event suppression/redirection. Static inspection does not establish + that every Codex route avoids foreground changes. OpenBitFun retains its + stricter background contract and does not adopt those branches. + +The REA JavaScript parent Evidence ID is +`ev_a10e7629df3bce8ec8fe8be5d0d9a870f775b7d798e8eb363f4b718c31a9de02`; +the Mach-O Evidence ID is +`ev_299cc4d31a7f4b5776f8c39358856b799661ae9229faee6e45abcb2bc5c13f20`. +These are static shipped-artifact findings, separate from the real-application +trials below. They do not recover hidden model prompts or establish a general +success rate. Third-party artifacts are not build or runtime dependencies. + +The focused Desktop Computer Use unit suite passed 181 tests with nine opt-in +native tests ignored. These cover the new optional AX read policy, actual native +chord event flags and cancellation bookkeeping, target-loss cache invalidation, +and foreground-admission guidance. Objective-C strict syntax checking also +passed. This does not establish native delivery or application compatibility. + +Both local native acceptance commands passed: `--orchestration` and `--busy-ax`. +The full Tool fixture induces real target-local deactivation without a global +foreground switch. Subsequent ordinary input prepares focus again and reaches +the original canvas, without repeating its content click or changing another +field. The `--busy-ax` fixture pauses only its disposable target to trigger real +AX messaging failure; after resumption, exactly one pointer down/up pair reaches +the button and activates it once. The mouse witness records dequeued events, +including those consumed inside standard controls' tracking loops, without +changing their input behavior or overriding `acceptsFirstMouse`. + +Both runs retained independent foreground/key, stacking, observer-input and +physical-cursor assertions. The full fixture also passed semantic Unicode input, +modifier shortcuts, scroll, preview revocation and Stop cleanup. Earlier runs +blocked by the hardware-idle gate were inconclusive and are not counted as +passes. This is native evidence for the dedicated fixtures on this macOS host, +not a broad application success-rate benchmark or cross-Space guarantee. +Windows/Linux native delivery and all four remote scenarios were not exercised +by this change. + +The subsequent systematic native comparison also covered capture selection, +helper-process target resolution, first-click policy, scrolling, dragging, +menus/sheets and cancellation. The reference uses desktop-independent window +capture on an inspected ScreenCaptureKit path; it also has other capture routes. +Its input target resolver can follow out-of-process AX targets, whereas +OpenBitFun keeps input within the authorized captured PID/window. Copying that +resolver would require an explicit authority design. Reference menu/window +observers and drag APIs do not establish equivalent coverage of system popups, +all helper processes or cancellation races. + +For text, `+[SAIVirtualKeyPress keyPressesForString:]` at `0x10024c2e4` searches +the current keyboard layout with `UCKeyTranslate`. At `0x10024c428`, an unmatched +UTF-16 unit is logged and skipped; this method has no Unicode-only fallback. +The later event converter's optional Unicode field does not remove that +limitation. OpenBitFun retains direct Unicode scalar delivery and a private +keyboard source. Fixed US-keyboard shortcut mapping and the local 8 ms text +pacing policy are not claims of reference parity or non-US layout coverage. + +Real-application trials used disposable local documents and isolated browser / +editor profiles. OpenBitFun actions went through the production +`ComputerUseTool` and real main-thread run loop using the devtools JSONL probe. +Reference actions used the installed Codex public native app API; the two +backends never sent input concurrently to the same application. Results were +checked in the returned AX tree and, for editors, the saved file, rather than +inferred from successful action receipts. + +| Application / UI family | OpenBitFun observed result | Codex observed result | +|---|---|---| +| TextEdit 1.20 / AppKit | Image-coordinate focus, select-all, full `OpenBitFun background 文本🙂`, and save | Click, select-all and save worked; `typeText` retained only `Codex background ` in this trial | +| Microsoft Edge 155.0.4283.60 / Chromium | One additional button activation, full Unicode input, and anchored scroll | One button activation and anchored scroll; `typeText` retained only ASCII | +| VS Code 1.117.0 / Electron | Image-coordinate editor focus, full Unicode, Command+Shift+Left replacement and save produced exactly `OpenBitFun shortcut 文本🙂` | Editor click, select-all and save worked; the saved text contained only ASCII | +| Calculator 11.0 / native controls | Background clear plus directed key sequence `17 + 25` produced `42` | Background clear plus the equivalent key sequence produced `42` | + +These are individual compatibility observations on this host/version, not a +success-rate benchmark or a claim that every Codex text path drops Unicode. +The reference's static layout-lookup branch is consistent with the observed +loss, but those actions were not instruction-traced. A planned LibreOffice +trial could not launch and is not counted as acceptance. + +An independent passive witness sampled frontmost PID, physical cursor, AX +focused-window identity and relative stacking at a requested 10 ms interval, +with activation notifications and sampling-gap records. It requested no OS +permissions, created no window and posted no input. No trial recorded the +target application becoming the human foreground. During the OpenBitFun Edge +and Calculator intervals, foreground PID and cursor were unchanged and the +target stayed below the foreground window; available AX focus samples retained +the same window ID. Some AX reads timed out or became stale and remain unknown. +TextEdit / VS Code intervals also contained real user cursor movement or +switches among other foreground applications, so they are not an unchanged- +desktop proof. The Codex Edge click/text interval contained one target-above- +foreground stacking sample between below samples with the same foreground PID +and AX window ID. Its cause was not isolated; it is not evidence of invariant +window order. Polling can miss transients; the controlled fixture's independent +observer-input, key-window and exact-delivery checks remain necessary. + +Cancellation cleanup now retains and revalidates process identity, confirms the +original window still belongs to that process, and uses current window geometry +for mouse release routing. A missing target or failed window-local route cannot +fall back to an unbound event. These checks reduce stale-target delivery; they +do not make the identity-check-to-post interval atomic. The focused unit tests +cover identity changes, routing failure, window changes and serialized cleanup +before capture teardown, while native Stop +acceptance exercises ordinary cleanup. + ### Latency design evidence, 2026-10-13 The installed `openai-codex-electron` 26.1007.30912 artifact (`@oai/cua` 0.2.6, diff --git a/scripts/fixtures/computer-use-background-witness.m b/scripts/fixtures/computer-use-background-witness.m new file mode 100644 index 0000000000..d415d5bacc --- /dev/null +++ b/scripts/fixtures/computer-use-background-witness.m @@ -0,0 +1,372 @@ +// Independent, read-only foreground witness for any macOS control backend. +// Build: clang -fobjc-arc -fblocks -Wall -Wextra -Werror -framework AppKit \ +// -framework ApplicationServices computer-use-background-witness.m -o witness +// Run: witness --target-pid PID [--target-window ID] --output /tmp/witness.json +// SIGUSR1 requests a new baseline; wait for baseline.sequence to advance before +// an action. SIGTERM/SIGINT write the final observation and exit. No NSApplication, +// windows, activation, event posting, screenshots, or permission prompts are used. +// Window metadata is reduced to PIDs, IDs and stacking; titles are never read. + +#import +#import +#include +#include +#include +#include +#include +#include +#include +#include +#include + +typedef AXError (*OBFAXWindowIDFunction)(AXUIElementRef, CGWindowID *); + +static double monotonicSeconds(void) { + struct timespec value; + clock_gettime(CLOCK_MONOTONIC, &value); + return value.tv_sec + value.tv_nsec / 1e9; +} + +static BOOL writePrivateJSON(NSDictionary *value, NSString *path) { + NSError *error = nil; + NSData *data = [NSJSONSerialization dataWithJSONObject:value + options:NSJSONWritingPrettyPrinted | NSJSONWritingSortedKeys error:&error]; + if (!data) { + fprintf(stderr, "JSON serialization failed: %s\n", error.localizedDescription.UTF8String); + return NO; + } + char *temporary = strdup([[path stringByAppendingString:@".XXXXXX"] fileSystemRepresentation]); + int fd = mkstemp(temporary); // 0600 from creation; rename keeps the file private. + BOOL ok = fd >= 0; + size_t offset = 0; + while (ok && offset < data.length) { + ssize_t written = write(fd, (const char *)data.bytes + offset, data.length - offset); + if (written < 0 && errno == EINTR) continue; + if (written <= 0) { ok = NO; break; } + offset += (size_t)written; + } + if (fd >= 0 && close(fd) != 0) ok = NO; + if (ok && rename(temporary, path.fileSystemRepresentation) != 0) ok = NO; + if (!ok) { + fprintf(stderr, "Cannot write witness output: %s\n", strerror(errno)); + unlink(temporary); + } + free(temporary); + return ok; +} + +static NSDictionary *focusedWindow(pid_t pid, OBFAXWindowIDFunction windowIDFunction) { + if (!AXIsProcessTrusted()) return @{@"status": @"permission_denied", @"pid": @(pid), @"window_id": NSNull.null}; + if (pid <= 0) return @{@"status": @"no_frontmost_application", @"pid": @(pid), @"window_id": NSNull.null}; + if (!windowIDFunction) return @{@"status": @"window_id_api_unavailable", @"pid": @(pid), @"window_id": NSNull.null}; + AXUIElementRef app = AXUIElementCreateApplication(pid); + // AX runs off the sampling queue. A slow foreground app must not suppress + // cursor/activation evidence. This timeout changes only this local AX client. + AXUIElementSetMessagingTimeout(app, 0.02f); + CFTypeRef window = NULL; + AXError result = AXUIElementCopyAttributeValue(app, kAXFocusedWindowAttribute, &window); + CGWindowID windowID = kCGNullWindowID; + if (result == kAXErrorSuccess && window && CFGetTypeID(window) == AXUIElementGetTypeID()) { + AXUIElementSetMessagingTimeout((AXUIElementRef)window, 0.02f); + result = windowIDFunction((AXUIElementRef)window, &windowID); + } else if (result == kAXErrorSuccess) { + result = kAXErrorNoValue; + } + if (window) CFRelease(window); + CFRelease(app); + return @{@"status": result == kAXErrorSuccess && windowID != kCGNullWindowID ? @"available" : @"unknown", + @"pid": @(pid), @"window_id": windowID ? @(windowID) : NSNull.null, + @"ax_error": @(result)}; +} + +@interface OBFBackgroundWitness : NSObject +@property pid_t targetPID; +@property CGWindowID targetWindow; +@property NSString *outputPath; +@property double interval; +@property double duration; +@property double started; +@property double lastSample; +@property double lastFlush; +@property double maxSampleGap; +@property double pendingBaseline; +@property NSUInteger sampleCount; +@property NSUInteger lateSampleCount; +@property NSUInteger axSampleCount; +@property NSUInteger baselineSequence; +@property BOOL axInFlight; +@property BOOL stopping; +@property BOOL done; +@property BOOL writeFailed; +@property NSDictionary *initial; +@property NSDictionary *current; +@property NSDictionary *baseline; +@property NSDictionary *latestAX; +@property NSDictionary *previousState; +@property NSMutableArray *changes; +@property NSMutableArray *activations; +@property NSMutableArray *baselines; +@property NSMutableArray *samplingGaps; +@property id activationObserver; +@property dispatch_queue_t axQueue; +@property dispatch_source_t terminateSource; +@property dispatch_source_t interruptSource; +@property dispatch_source_t resetSource; +@property OBFAXWindowIDFunction windowIDFunction; +- (void)sample; +- (void)flush:(BOOL)finished; +- (void)requestStop; +@end + +@implementation OBFBackgroundWitness +- (double)elapsedMilliseconds { return (monotonicSeconds() - self.started) * 1000.0; } + +- (NSDictionary *)stackingForPID:(pid_t)frontPID focused:(NSDictionary *)focused { + CFArrayRef raw = CGWindowListCopyWindowInfo(kCGWindowListOptionOnScreenOnly | kCGWindowListExcludeDesktopElements, + kCGNullWindowID); + if (!raw) return @{@"status": @"window_list_unavailable"}; + NSArray *windows = CFBridgingRelease(raw); + NSNumber *focusedID = [focused[@"status"] isEqual:@"available"] ? focused[@"window_id"] : nil; + NSInteger referenceIndex = -1, targetIndex = -1; + CGWindowID referenceID = focusedID.unsignedIntValue; + CGWindowID selectedTargetID = self.targetWindow; + NSInteger referenceLayer = 0, targetLayer = 0; + for (NSUInteger index = 0; index < windows.count; index++) { + NSDictionary *entry = windows[index]; + pid_t pid = [entry[(id)kCGWindowOwnerPID] intValue]; + CGWindowID windowID = [entry[(id)kCGWindowNumber] unsignedIntValue]; + NSInteger layer = [entry[(id)kCGWindowLayer] integerValue]; + if (pid == frontPID && ((focusedID && windowID == focusedID.unsignedIntValue) || + (!focusedID && referenceIndex < 0 && layer == 0))) { + referenceIndex = (NSInteger)index; + referenceID = windowID; + referenceLayer = layer; + } + if (pid == self.targetPID && ((self.targetWindow && windowID == self.targetWindow) || + (!self.targetWindow && targetIndex < 0 && layer == 0))) { + targetIndex = (NSInteger)index; + selectedTargetID = windowID; + targetLayer = layer; + } + } + NSString *relation = @"unknown"; + if (referenceIndex >= 0 && targetIndex >= 0) { + relation = targetIndex < referenceIndex ? @"above" : (targetIndex > referenceIndex ? @"below" : @"same"); + } + return @{@"status": @"available", @"reference_source": focusedID ? @"ax_focused_window" : @"frontmost_pid_first_layer_zero_window", + @"reference_window_id": referenceID ? @(referenceID) : NSNull.null, + @"reference_index": referenceIndex >= 0 ? @(referenceIndex) : NSNull.null, + @"reference_layer": referenceIndex >= 0 ? @(referenceLayer) : NSNull.null, + @"target_window_id": selectedTargetID ? @(selectedTargetID) : NSNull.null, + @"target_index": targetIndex >= 0 ? @(targetIndex) : NSNull.null, + @"target_layer": targetIndex >= 0 ? @(targetLayer) : NSNull.null, + @"target_relative_to_reference": relation}; +} + +- (void)requestAXForPID:(pid_t)pid { + if (self.axInFlight || self.stopping) return; + self.axInFlight = YES; + dispatch_async(self.axQueue, ^{ + @autoreleasepool { + double began = [self elapsedMilliseconds]; + NSDictionary *focused = focusedWindow(pid, self.windowIDFunction); + double completed = [self elapsedMilliseconds]; + dispatch_async(dispatch_get_main_queue(), ^{ + self.latestAX = @{@"started_ms": @(began), @"completed_ms": @(completed), @"focused_window": focused}; + self.axSampleCount++; + self.axInFlight = NO; + if (self.stopping) { + [self sample]; + [self flush:YES]; + self.done = YES; + } + }); + } + }); +} + +- (void)sample { + double now = monotonicSeconds(); + double elapsed = (now - self.started) * 1000.0; + if (self.lastSample > 0) { + double gap = now - self.lastSample; + self.maxSampleGap = MAX(self.maxSampleGap, gap); + if (gap > self.interval * 2.0) { + self.lateSampleCount++; + [self.samplingGaps addObject:@{@"from_ms": @((self.lastSample - self.started) * 1000.0), + @"to_ms": @(elapsed), @"duration_ms": @(gap * 1000.0)}]; + } + } + self.lastSample = now; + self.sampleCount++; + pid_t frontPID = NSWorkspace.sharedWorkspace.frontmostApplication.processIdentifier; + CGEventRef event = CGEventCreate(NULL); + CGPoint cursor = event ? CGEventGetLocation(event) : CGPointMake(NAN, NAN); + if (event) CFRelease(event); + BOOL cursorAvailable = isfinite(cursor.x) && isfinite(cursor.y); + NSDictionary *focused = self.latestAX[@"focused_window"]; + if (!focused || [focused[@"pid"] intValue] != frontPID) { + focused = @{@"status": @"pending", @"pid": @(frontPID), @"window_id": NSNull.null}; + } else if (elapsed - [self.latestAX[@"completed_ms"] doubleValue] > 100.0) { + focused = @{@"status": @"stale", @"pid": @(frontPID), @"window_id": NSNull.null}; + } + NSDictionary *state = @{@"frontmost_pid": @(frontPID), + @"cursor": @{@"status": cursorAvailable ? @"available" : @"unknown", + @"x": cursorAvailable ? @(cursor.x) : NSNull.null, + @"y": cursorAvailable ? @(cursor.y) : NSNull.null}, + @"focused_window": focused, @"stacking": [self stackingForPID:frontPID focused:focused]}; + self.current = @{@"monotonic_ms": @(elapsed), @"sample": @(self.sampleCount), @"state": state, + @"ax_read_started_ms": self.latestAX[@"started_ms"] ?: NSNull.null, + @"ax_read_completed_ms": self.latestAX[@"completed_ms"] ?: NSNull.null}; + if (!self.initial) self.initial = self.current; + if (self.previousState && ![self.previousState isEqual:state]) { + NSMutableArray *fields = [NSMutableArray array]; + for (NSString *key in @[@"frontmost_pid", @"cursor", @"focused_window", @"stacking"]) { + if (![self.previousState[key] isEqual:state[key]]) [fields addObject:key]; + } + [self.changes addObject:@{@"observation": self.current, @"changed_fields": fields}]; + } + self.previousState = state; + // A reset is acknowledged only after a new AX read started after the request. + // The AX result may legitimately be unknown or permission_denied. + if (self.pendingBaseline >= 0 && self.latestAX && + [self.latestAX[@"started_ms"] doubleValue] >= self.pendingBaseline && + [self.latestAX[@"focused_window"][@"pid"] intValue] == frontPID) { + self.baselineSequence++; + self.baseline = @{@"sequence": @(self.baselineSequence), @"requested_ms": @(self.pendingBaseline), + @"observation": self.current, @"change_index": @(self.changes.count), + @"activation_index": @(self.activations.count), @"sampling_gap_index": @(self.samplingGaps.count)}; + [self.baselines addObject:self.baseline]; + self.pendingBaseline = -1; + [self flush:NO]; + printf("BASELINE %lu %.3f\n", (unsigned long)self.baselineSequence, elapsed); + fflush(stdout); + } + [self requestAXForPID:frontPID]; + if (now - self.lastFlush >= 0.25) [self flush:NO]; +} + +- (void)flush:(BOOL)finished { + NSDictionary *document = @{@"schema_version": @1, @"witness_pid": @(getpid()), + @"target_pid": @(self.targetPID), @"target_window_id": self.targetWindow ? @(self.targetWindow) : NSNull.null, + @"finished": @(finished), @"elapsed_ms": @([self elapsedMilliseconds]), + @"permissions": @{@"accessibility_trusted": @(AXIsProcessTrusted()), + @"screen_recording_preflight": @(CGPreflightScreenCaptureAccess()), + @"permission_requested": @NO}, + @"availability": @{@"ax_window_id_function": @(self.windowIDFunction != NULL), + @"key_window_id": @"not_exposed_by_public_ax; focused_window_is_reported", + @"activation_notifications": @YES}, + @"sampling": @{@"requested_interval_ms": @(self.interval * 1000.0), @"samples": @(self.sampleCount), + @"ax_samples": @(self.axSampleCount), @"max_gap_ms": @(self.maxSampleGap * 1000.0), + @"gaps_over_twice_interval": @(self.lateSampleCount), + @"ax_stale_threshold_ms": @100}, + @"limits": @[@"Observations do not identify whether a change came from the user, a backend, or another process.", + @"Polling can miss changes between samples; AX reads and window-list reads are not atomic.", + @"AX focused window is asynchronous; read timestamps and unavailable states must be checked.", + @"Stacking describes listed on-screen windows, not visual occlusion or other Spaces.", + @"Without a supplied target window, the first on-screen layer-zero target window is used."], + @"start": self.initial ?: (id)NSNull.null, @"current": self.current ?: (id)NSNull.null, + @"baseline": self.baseline ?: (id)NSNull.null, @"baselines": self.baselines, + @"changes": self.changes, @"activation_notifications": self.activations, @"sampling_gaps": self.samplingGaps}; + if (!writePrivateJSON(document, self.outputPath)) self.writeFailed = YES; + self.lastFlush = monotonicSeconds(); +} + +- (void)requestStop { + if (self.stopping) return; + self.stopping = YES; + if (!self.axInFlight) { + [self sample]; + [self flush:YES]; + self.done = YES; + } +} +@end + +static dispatch_source_t signalSource(int value, dispatch_block_t handler) { + signal(value, SIG_IGN); + dispatch_source_t source = dispatch_source_create(DISPATCH_SOURCE_TYPE_SIGNAL, (uintptr_t)value, 0, + dispatch_get_main_queue()); + dispatch_source_set_event_handler(source, handler); + dispatch_resume(source); + return source; +} + +static BOOL parseNumber(const char *value, unsigned long long maximum, unsigned long long *result) { + if (!value || !*value || *value == '-') return NO; + char *end = NULL; + errno = 0; + unsigned long long parsed = strtoull(value, &end, 10); + if (errno || !end || *end || !parsed || parsed > maximum) return NO; + *result = parsed; + return YES; +} + +int main(int argc, const char *argv[]) { + @autoreleasepool { + OBFBackgroundWitness *witness = [OBFBackgroundWitness new]; + witness.interval = 0.01; + witness.pendingBaseline = 0; + for (int index = 1; index < argc; index++) { + NSString *argument = @(argv[index]); + if ([argument isEqual:@"--help"]) { + puts("Usage: witness --target-pid PID [--target-window ID] --output PATH [--duration-ms MS] [--interval-ms MS]"); + puts("SIGUSR1: new baseline; SIGTERM/SIGINT: final output and clean exit. No UI or input is generated."); + return 0; + } + if (index + 1 >= argc) { fprintf(stderr, "Missing argument value\n"); return 2; } + const char *value = argv[++index]; + unsigned long long number = 0; + if ([argument isEqual:@"--output"]) witness.outputPath = [@(value) stringByStandardizingPath]; + else if ([argument isEqual:@"--target-pid"] && parseNumber(value, INT_MAX, &number)) witness.targetPID = (pid_t)number; + else if ([argument isEqual:@"--target-window"] && parseNumber(value, UINT32_MAX, &number)) witness.targetWindow = (CGWindowID)number; + else if ([argument isEqual:@"--duration-ms"] && parseNumber(value, UINT_MAX, &number)) witness.duration = number / 1000.0; + else if ([argument isEqual:@"--interval-ms"] && parseNumber(value, 1000, &number) && number >= 5) witness.interval = number / 1000.0; + else { fprintf(stderr, "Invalid argument: %s\n", argument.UTF8String); return 2; } + } + if (!witness.targetPID || !witness.outputPath.length) { + fprintf(stderr, "--target-pid and --output are required; see --help\n"); + return 2; + } + witness.started = monotonicSeconds(); + witness.changes = [NSMutableArray array]; + witness.activations = [NSMutableArray array]; + witness.baselines = [NSMutableArray array]; + witness.samplingGaps = [NSMutableArray array]; + witness.axQueue = dispatch_queue_create("org.openbitfun.background-witness.ax", DISPATCH_QUEUE_SERIAL); + witness.windowIDFunction = (OBFAXWindowIDFunction)dlsym(RTLD_DEFAULT, "_AXUIElementGetWindow"); + witness.activationObserver = [NSWorkspace.sharedWorkspace.notificationCenter + addObserverForName:NSWorkspaceDidActivateApplicationNotification object:nil queue:NSOperationQueue.mainQueue + usingBlock:^(NSNotification *notification) { + NSRunningApplication *application = notification.userInfo[NSWorkspaceApplicationKey]; + [witness.activations addObject:@{@"monotonic_ms": @([witness elapsedMilliseconds]), + @"activated_pid": @(application.processIdentifier), + @"sampled_frontmost_pid": @(NSWorkspace.sharedWorkspace.frontmostApplication.processIdentifier)}]; + }]; + witness.terminateSource = signalSource(SIGTERM, ^{ [witness requestStop]; }); + witness.interruptSource = signalSource(SIGINT, ^{ [witness requestStop]; }); + witness.resetSource = signalSource(SIGUSR1, ^{ + if (!witness.stopping) witness.pendingBaseline = [witness elapsedMilliseconds]; + }); + [witness sample]; + [witness flush:NO]; + if (witness.writeFailed) return 1; + NSTimer *timer = [NSTimer scheduledTimerWithTimeInterval:witness.interval repeats:YES block:^(NSTimer *unused) { + (void)unused; + if (!witness.stopping) [witness sample]; + if (witness.duration > 0 && monotonicSeconds() - witness.started >= witness.duration) [witness requestStop]; + }]; + timer.tolerance = 0; + while (!witness.done) { + @autoreleasepool { + [[NSRunLoop mainRunLoop] runMode:NSDefaultRunLoopMode beforeDate:[NSDate dateWithTimeIntervalSinceNow:0.05]]; + } + } + [timer invalidate]; + [NSWorkspace.sharedWorkspace.notificationCenter removeObserver:witness.activationObserver]; + dispatch_source_cancel(witness.terminateSource); + dispatch_source_cancel(witness.interruptSource); + dispatch_source_cancel(witness.resetSource); + return witness.writeFailed ? 1 : 0; + } +} diff --git a/scripts/fixtures/computer-use-input-controls.m b/scripts/fixtures/computer-use-input-controls.m index 9c30cb91af..f8f2472cfa 100644 --- a/scripts/fixtures/computer-use-input-controls.m +++ b/scripts/fixtures/computer-use-input-controls.m @@ -29,8 +29,12 @@ static void publish(void) { @interface OBFControlsApp : NSApplication @end @implementation OBFControlsApp -- (void)sendEvent:(NSEvent *)event { - if (event.type == NSEventTypeLeftMouseDown || event.type == NSEventTypeLeftMouseUp) { +- (NSEvent *)nextEventMatchingMask:(NSEventMask)mask untilDate:(NSDate *)expiration inMode:(NSRunLoopMode)mode dequeue:(BOOL)dequeue { + NSEvent *event = [super nextEventMatchingMask:mask untilDate:expiration inMode:mode dequeue:dequeue]; + // Observe actual dequeued events, including a standard control's tracking + // loop. NSButton consumes mouse-up inside mouseDown:, so sendEvent: alone + // cannot witness both halves of an otherwise correctly delivered click. + if (dequeue && (event.type == NSEventTypeLeftMouseDown || event.type == NSEventTypeLeftMouseUp)) { NSPoint local = event.locationInWindow; NSPoint content = [targetWindow.contentView convertPoint:local fromView:nil]; NSView *hit = event.windowNumber == targetWindow.windowNumber ? [targetWindow.contentView hitTest:content] : nil; @@ -38,6 +42,9 @@ - (void)sendEvent:(NSEvent *)event { BOOL outsideFrame = !NSPointInRect(screen, targetWindow.frame); [events addObject:@{@"type":@(event.type), @"window_id":@(event.windowNumber), @"target_key_before":@(targetWindow.keyWindow), @"target_active_before":@(NSApp.active), @"outside_frame":@(outsideFrame), @"time":@(NSProcessInfo.processInfo.systemUptime), @"flags":@(event.modifierFlags), @"x":@(local.x), @"y":@(local.y), @"hit":NSStringFromClass(hit.class) ?: @"none"}]; } + return event; +} +- (void)sendEvent:(NSEvent *)event { [super sendEvent:event]; publish(); } diff --git a/scripts/test-macos-input-controls.mjs b/scripts/test-macos-input-controls.mjs index 888b0123e4..de49e4c460 100644 --- a/scripts/test-macos-input-controls.mjs +++ b/scripts/test-macos-input-controls.mjs @@ -6,9 +6,10 @@ import { spawn } from 'node:child_process'; import { fileURLToPath } from 'node:url'; assert.equal(process.platform, 'darwin', 'directed input fixture requires macOS'); const latency = process.argv.includes('--latency'); -const orchestration = process.argv.includes('--orchestration') || process.argv.includes('--semantic') || latency; +const busyAx = process.argv.includes('--busy-ax'); +const orchestration = process.argv.includes('--orchestration') || process.argv.includes('--semantic') || latency || busyAx; const semantic = process.argv.includes('--semantic') || orchestration; -assert.ok(process.argv.length === 2 || (process.argv.length === 3 && semantic), 'Usage: node scripts/test-macos-input-controls.mjs [--semantic | --orchestration | --latency]'); +assert.ok(process.argv.length === 2 || (process.argv.length === 3 && semantic), 'Usage: node scripts/test-macos-input-controls.mjs [--semantic | --orchestration | --latency | --busy-ax]'); const root = fileURLToPath(new URL('../', import.meta.url)); const run = (command, args, env = process.env, onSpawn) => new Promise((resolve, reject) => { const child = spawn(command, args, { cwd: root, stdio: 'inherit', windowsHide: true, env }); @@ -118,6 +119,7 @@ try { OPENBITFUN_INPUT_FIXTURE_APP: join(directory, 'OpenBitFun Input Target.app'), OPENBITFUN_INPUT_FIXTURE_RESULT: result, OPENBITFUN_INPUT_OBSERVER_RESULT: `${result}.observer`, OPENBITFUN_INPUT_HOST_READY: hostReady, + ...(busyAx ? { OPENBITFUN_INPUT_BUSY_AX: '1' } : {}), }; // Initialize AppKit before creating the foreground observer. Otherwise the // test host's own startup can deactivate it before the first tool action. @@ -173,7 +175,7 @@ try { assert.equal(event.flags & (1 << 20), 0, 'focus preparation must not add Command'); } } - console.log(semantic ? 'PASS: native AX hit testing and semantic input operate on covered controls without raising the target' : 'PASS: standard controls and non-AX canvas preserve input semantics and foreground state'); + console.log(busyAx ? 'PASS: real AX timeout retains one directed click with foreground and cursor unchanged' : semantic ? 'PASS: native AX hit testing and semantic input operate on covered controls without raising the target' : 'PASS: standard controls and non-AX canvas preserve input semantics and foreground state'); } finally { if (harness && harness.exitCode === null) await new Promise(resolve => { harness.once('exit', resolve); harness.kill('SIGTERM'); }); if (observer && observer.exitCode === null) await new Promise(resolve => { observer.once('exit', resolve); observer.kill('SIGTERM'); }); diff --git a/src/apps/desktop/src/computer_use/AGENTS.md b/src/apps/desktop/src/computer_use/AGENTS.md index 898f51da5c..286345b825 100644 --- a/src/apps/desktop/src/computer_use/AGENTS.md +++ b/src/apps/desktop/src/computer_use/AGENTS.md @@ -80,12 +80,15 @@ The directed-input route can change a target application's internal focus without changing the human foreground application. Its window-addressed focus pair at local `(-8,-8)` must remain outside content; the requested gesture must reach content exactly once with unchanged modifiers. Do not validate it -using a fixture that overrides `acceptsFirstMouse`. Record observer active/key, +using a fixture that overrides `acceptsFirstMouse`. Record dequeued mouse events +so standard controls' tracking loops cannot hide a delivered mouse-up. Record observer active/key, frontmost PID, stacking order, real text destination and content event counts. Do not use missing AX key-state attributes as a substitute for native focus validation: fixtures inspect their own `NSWindow.keyWindow`. Preparation caches must include generation, PID, window ID, human foreground identity and activation -epoch, and must not add app-name rules or content-coordinate guesses. An off-frame routing +epoch plus a passive target-local loss epoch, and must not add app-name rules or +content-coordinate guesses. Target-local event monitoring must be listen-only; +unavailable monitoring disables cache reuse, not background input. An off-frame routing position is distinct from the actual user-requested content target. No after-action raise/restore is allowed. Stop may deactivate a target only when it is not the user's actual foreground application. Prototype success is not @@ -129,7 +132,12 @@ node scripts/test-macos-input-controls.mjs --semantic # Full Tool image-coordinate button activation/text focus and wrong-field guard. # Target/observer have distinct app bundles; AppKit initializes before the # foreground baseline. Activation history catches transient focus changes. +# Also induces target-local loss while foreground stays fixed, then verifies +# the next ordinary input prepares focus again without another content click. node scripts/test-macos-input-controls.mjs --orchestration +# Real AX messaging timeout in a stopped disposable process, then one directed +# click after resumption; retains exact delivery and foreground/cursor checks. +node scripts/test-macos-input-controls.mjs --busy-ax # Identical five-input sequence: five tool calls vs one batch, with default # images and AX-only observations. Fresh isolated app/host per case. node scripts/test-macos-input-controls.mjs --latency @@ -141,6 +149,30 @@ node scripts/test-macos-input-controls.mjs node scripts/test-macos-control-roundtrip.mjs ``` +For explicitly selected disposable real applications, the devtools JSONL probe +uses the same production Tool path and main-thread run loop. It accepts only one +startup PID (optionally one window), background observations and app actions; +it rejects activation, global input, clipboard and scripts. Do not use personal +application windows or infer delivery from `ok: true`. Observe after errors +before choosing another action; the probe never retries input automatically. + +```bash +cargo test -p openbitfun-desktop --features devtools --test computer_use_native_roundtrip --no-run --message-format=json +# Use the executable path emitted above; EOF or the total deadline stops control. +OPENBITFUN_CUA_PROBE_PID= OPENBITFUN_CUA_PROBE_OUTPUT_DIR=/tmp/cua-probe +# stdin JSONL: {"id":"observe","input":{"action":"get_app_state","include_image":true}} +cargo test -p openbitfun-desktop --features devtools --lib native_control_roundtrip::probe +# Independent read-only witness; no app launch, permission prompts or input. +clang -fobjc-arc -fblocks -Wall -Wextra -Werror -framework AppKit -framework ApplicationServices scripts/fixtures/computer-use-background-witness.m -o /tmp/cua-witness +/tmp/cua-witness --target-pid --target-window --output /tmp/cua-witness.json +``` + +Wait for the witness's baseline before input. `SIGUSR1` requests a new baseline; +wait for its sequence to advance. `SIGTERM` flushes the final record. Check +sampling gaps and unknown AX reads, and separate concurrent human activity from +backend-caused changes. The witness records IDs/coordinates, not titles, text or +pixels; it cannot independently prove key-window state or causal attribution. + The browser fixture runs production snapshot/resolver JavaScript in Chromium, then passes the actual DOM payload to compiled Rust presentation tests. The optional OCR step compiles the Desktop test target and invokes macOS Vision on diff --git a/src/apps/desktop/src/computer_use/control_session.rs b/src/apps/desktop/src/computer_use/control_session.rs index bc2c91a2a0..84ccbeaaa6 100644 --- a/src/apps/desktop/src/computer_use/control_session.rs +++ b/src/apps/desktop/src/computer_use/control_session.rs @@ -324,9 +324,7 @@ pub(crate) fn input_allowed() -> Result<(), String> { pub(crate) fn foreground_allowed() -> Result<(), String> { input_allowed()?; if locked()?.admission.snapshot.mode != ControlMode::Foreground { - return Err( - "[FOREGROUND_REQUIRED] Seat input requires start_control with mode=foreground".into(), - ); + return Err(admission::FOREGROUND_REQUIRED.into()); } Ok(()) } diff --git a/src/apps/desktop/src/computer_use/control_session/admission.rs b/src/apps/desktop/src/computer_use/control_session/admission.rs index 647152c135..be595eba6b 100644 --- a/src/apps/desktop/src/computer_use/control_session/admission.rs +++ b/src/apps/desktop/src/computer_use/control_session/admission.rs @@ -5,6 +5,8 @@ use openbitfun_agent_tools::computer_use_control::{ ControlActionPolicy, ControlMode, ControlSharing, ControlSnapshot, SharingPhase, }; +pub(super) const FOREGROUND_REQUIRED: &str = "[FOREGROUND_REQUIRED] This global action uses the human's foreground application or cursor. Keep background control and use app_click, app_type_text, app_key_chord, app_scroll or app_drag with observed application/window targets. Do not request foreground takeover to recover from this error. Use start_control with mode=foreground only when the user explicitly requests taking over the visible desktop."; + #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) struct ControlToken { pub(super) generation: u64, @@ -428,7 +430,7 @@ impl Admission { return Err("[CONTROL_OBSERVE_ONLY] Start a control session to send input".into()); } if policy.foreground && mode != ControlMode::Foreground { - return Err("[FOREGROUND_REQUIRED] This action uses desktop focus or seat input; explicitly start_control with mode=foreground".into()); + return Err(FOREGROUND_REQUIRED.into()); } let sequence = self .snapshot @@ -483,6 +485,15 @@ mod tests { assert!(state.snapshot.sharing.is_none()); state.bind("pid:1/window:10".into()).unwrap(); state.release(observe, true); + for action in ["click", "type_text", "key_chord", "scroll", "drag"] { + let error = state.admit("task", action, 1).unwrap_err(); + assert!(error.starts_with("[FOREGROUND_REQUIRED]"), "{error}"); + assert!(error.contains("app_click, app_type_text, app_key_chord")); + assert!(error.contains("Do not request foreground takeover")); + assert!(error.contains("only when the user explicitly requests")); + assert_eq!(state.snapshot.mode, ControlMode::Background); + assert!(!state.busy); + } let input = state.admit("task", "app_type_text", 1).unwrap(); assert!(state.check_input(input).is_ok()); state.release(input, true); diff --git a/src/apps/desktop/src/computer_use/desktop_host/native_control_probe.rs b/src/apps/desktop/src/computer_use/desktop_host/native_control_probe.rs new file mode 100644 index 0000000000..544da3c63d --- /dev/null +++ b/src/apps/desktop/src/computer_use/desktop_host/native_control_probe.rs @@ -0,0 +1,362 @@ +//! Opt-in JSONL driver for the real ComputerUseTool, compiled only by devtools. +//! It narrows test scope; it does not grant or replace production authority. +use super::{ + ComputerUseHost, ComputerUseTool, DesktopComputerUseHost, StopOnExit, Tool, ToolResult, + ToolUseContext, OWNER, +}; +use base64::Engine; +use futures::FutureExt; +use openbitfun_core::agentic::tools::computer_use_host::AppInputAction; +use serde_json::{json, Value}; +use std::io::{BufRead, Write}; +use std::path::PathBuf; +use std::sync::Arc; +use std::time::Duration; + +#[derive(Clone, Copy)] +struct Scope { + pid: i32, + window: Option, +} + +fn normalize(mut input: Value, scope: Scope) -> Result { + let object = input + .as_object_mut() + .ok_or("Probe input must be an object")?; + let action = object + .get("action") + .and_then(Value::as_str) + .ok_or("action is required")? + .to_owned(); + if !matches!( + action.as_str(), + "start_control" + | "stop_control" + | "control_status" + | "list_windows" + | "get_app_state" + | "get_app_shortcuts" + | "describe_screen" + | "screenshot" + | "app_click" + | "app_type_text" + | "app_key_chord" + | "app_scroll" + | "app_drag" + | "app_batch" + | "app_wait_for" + ) { + return Err("PROBE_SCOPE_DENIED: Only target-scoped observation and app input are allowed; global input, activation, launch, clipboard and scripts are unavailable".into()); + } + if object + .get("mode") + .is_some_and(|mode| mode.as_str() != Some("background")) + { + return Err("PROBE_SCOPE_DENIED: Only background mode is allowed".into()); + } + if action == "start_control" { + object.insert("mode".into(), json!("background")); + } + let mut selected_window = scope.window; + if let Some(selector) = object.get("app") { + let selector = selector + .as_object() + .ok_or("PROBE_SCOPE_DENIED: app must be a PID/window selector")?; + if selector + .keys() + .any(|key| key != "pid" && key != "window_id") + || selector + .get("pid") + .is_some_and(|pid| pid.as_i64() != Some(scope.pid as i64)) + { + return Err("PROBE_SCOPE_DENIED: The process selected at startup cannot change".into()); + } + if let Some(window) = selector.get("window_id") { + let window = window + .as_u64() + .filter(|id| *id > 0 && *id <= u32::MAX as u64) + .ok_or("PROBE_SCOPE_DENIED: window_id must be a positive macOS window ID")?; + if scope.window.is_some_and(|allowed| allowed != window) { + return Err( + "PROBE_SCOPE_DENIED: The window selected at startup cannot change".into(), + ); + } + selected_window = Some(window); + } + } + let mut selector = json!({"pid":scope.pid}); + if let Some(window) = selected_window { + selector["window_id"] = json!(window); + } + object.insert("app".into(), selector); + // Even without a fixed startup window, observation must remain on the + // selected/bound window rather than dump every document in that process. + object.insert("focus_window_only".into(), json!(true)); + if action == "app_batch" { + for step in object + .get("steps") + .and_then(Value::as_array) + .ok_or("steps must be an array")? + { + // The production enum denies unknown fields and has no selector, + // mode, activation or global-seat variant. + let action: AppInputAction = serde_json::from_value(step.clone()) + .map_err(|error| format!("PROBE_SCOPE_DENIED: {error}"))?; + action + .validate() + .map_err(|error| format!("PROBE_SCOPE_DENIED: {error}"))?; + } + } + Ok(input) +} + +fn write_json(value: &Value) -> Result<(), String> { + let mut output = std::io::stdout().lock(); + serde_json::to_writer(&mut output, value).map_err(|error| error.to_string())?; + writeln!(output) + .and_then(|_| output.flush()) + .map_err(|error| error.to_string()) +} + +fn observation(results: Vec, directory: Option<&PathBuf>, sequence: u64) -> Value { + let mut contents = Vec::new(); + let mut images = Vec::new(); + for result in results { + contents.push(result.content()); + if let ToolResult::Result { + image_attachments: Some(attachments), + .. + } = result + { + for attachment in attachments { + let mut image = json!({"mime_type":attachment.mime_type}); + let saved = directory.map(|directory| -> Result { + let extension = match attachment.mime_type.as_str() { + "image/jpeg" => "jpg", + "image/png" => "png", + "image/webp" => "webp", + _ => return Err("Unsupported image MIME type".into()), + }; + let path = directory.join(format!( + "probe-{}-{sequence}-{}.{}", + std::process::id(), + images.len(), + extension + )); + let bytes = base64::engine::general_purpose::STANDARD + .decode(&attachment.data_base64) + .map_err(|error| error.to_string())?; + let mut file = std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(&path) + .map_err(|error| error.to_string())?; + file.write_all(&bytes).map_err(|error| error.to_string())?; + Ok(path) + }); + match saved { + Some(Ok(path)) => image["path"] = json!(path), + other => { + image["data_base64"] = json!(attachment.data_base64); + if let Some(Err(error)) = other { + // Input may already have happened. A file-output + // problem must not masquerade as failed delivery. + image["save_error"] = json!(error); + } + } + } + images.push(image); + } + } + } + let failed = contents.iter().find(|body| { + body.get("ok") == Some(&Value::Bool(false)) + || body.get("success") == Some(&Value::Bool(false)) + }); + let mut response = json!({"ok":failed.is_none(),"images":images}); + if let Some(body) = failed { + response["error"] = body.get("error").cloned().unwrap_or_else(|| body.clone()); + response["input_may_have_been_submitted"] = json!(true); + } + let first = contents.first().cloned().unwrap_or(Value::Null); + response["data"] = if first.get("ok") == Some(&Value::Bool(true)) && first.get("data").is_some() + { + first["data"].clone() + } else { + first + }; + if contents.len() > 1 { + response["results"] = json!(contents); + } + response +} + +pub(super) async fn run() { + if let Err(error) = run_inner().await { + let _ = write_json(&json!({"event":"error","ok":false,"error":error})); + } +} + +async fn run_inner() -> Result<(), String> { + let pid = std::env::var("OPENBITFUN_CUA_PROBE_PID") + .ok() + .and_then(|value| value.parse::().ok()) + .filter(|pid| *pid > 0) + .ok_or("OPENBITFUN_CUA_PROBE_PID must identify one explicit target process")?; + let window = std::env::var("OPENBITFUN_CUA_PROBE_WINDOW_ID") + .ok() + .map(|value| { + value + .parse::() + .ok() + .filter(|id| *id > 0 && *id <= u32::MAX as u64) + .ok_or("Invalid OPENBITFUN_CUA_PROBE_WINDOW_ID") + }) + .transpose()?; + let seconds = std::env::var("OPENBITFUN_CUA_PROBE_TIMEOUT_SECS") + .unwrap_or_else(|_| "600".into()) + .parse::() + .ok() + .filter(|seconds| (1..=1800).contains(seconds)) + .ok_or("OPENBITFUN_CUA_PROBE_TIMEOUT_SECS must be between 1 and 1800")?; + let directory = std::env::var_os("OPENBITFUN_CUA_PROBE_OUTPUT_DIR").map(PathBuf::from); + if let Some(directory) = &directory { + if !directory.is_absolute() { + return Err("OPENBITFUN_CUA_PROBE_OUTPUT_DIR must be absolute".into()); + } + std::fs::create_dir_all(directory).map_err(|error| error.to_string())?; + } + let scope = Scope { pid, window }; + let identity = crate::computer_use::process_identity::ProcessIdentity::read(pid)?; + let host = Arc::new(DesktopComputerUseHost::new()); + let mut context = ToolUseContext::for_tool_listing(None, None); + context.session_id = Some(OWNER.into()); + context.computer_use_host = Some(host.clone()); + context.primary_model_facts.supports_image_inputs = true; + context.primary_model_facts.api_format = "anthropic".into(); + let tool = ComputerUseTool::new(); + let _cleanup = StopOnExit; + let deadline = tokio::time::Instant::now() + Duration::from_secs(seconds); + let (sender, mut receiver) = tokio::sync::mpsc::channel(1); + std::thread::spawn(move || { + for line in std::io::stdin().lock().lines() { + if sender + .blocking_send(line.map_err(|error| error.to_string())) + .is_err() + { + break; + } + } + }); + write_json( + &json!({"event":"ready","pid":pid,"window_id":window,"mode":"background","timeout_seconds":seconds}), + )?; + let mut sequence = 0u64; + let reason = loop { + let line = match tokio::time::timeout_at(deadline, receiver.recv()).await { + Ok(Some(line)) => line, + Ok(None) => break "eof", + Err(_) => break "deadline", + }; + sequence += 1; + let started = std::time::Instant::now(); + let parsed = line.and_then(|line| { + serde_json::from_str::(&line).map_err(|error| error.to_string()) + }); + let id = parsed + .as_ref() + .ok() + .and_then(|request| request.get("id")) + .cloned() + .unwrap_or_else(|| json!(sequence)); + let input = parsed + .and_then(|request| normalize(request.get("input").cloned().unwrap_or(request), scope)); + let action = input + .as_ref() + .ok() + .and_then(|input| input.get("action")) + .cloned() + .unwrap_or(Value::Null); + let mut response = match input.and_then(|input| identity.validate().map(|_| input)) { + Err(error) => json!({"ok":false,"error":error,"input_may_have_been_submitted":false}), + Ok(input) => { + let call = + std::panic::AssertUnwindSafe(tool.call_impl(&input, &context)).catch_unwind(); + match tokio::time::timeout_at(deadline, call).await { + Ok(Ok(Ok(results))) => observation(results, directory.as_ref(), sequence), + Ok(Ok(Err(error))) => { + json!({"ok":false,"error":error.to_string(),"input_may_have_been_submitted":true}) + } + Ok(Err(_)) => { + let _ = host.stop_control(OWNER).await; + json!({"ok":false,"error":"PROBE_TOOL_PANIC: Control revoked; observe before deciding whether any input is needed","input_may_have_been_submitted":true}) + } + Err(_) => { + write_json( + &json!({"id":id,"ok":false,"action":action,"elapsed_ms":started.elapsed().as_millis(),"input_may_have_been_submitted":true,"error":"PROBE_DEADLINE: Control cancelled; any already submitted input must not be repeated without observation"}), + )?; + break "deadline"; + } + } + } + }; + response["id"] = id; + response["action"] = action; + response["elapsed_ms"] = json!(started.elapsed().as_millis()); + write_json(&response)?; + }; + let stopped = host + .stop_control(OWNER) + .await + .map_err(|error| error.to_string())?; + write_json(&json!({"event":"stopped","reason":reason,"state":stopped.state})) +} + +#[cfg(test)] +mod tests { + use super::*; + #[test] + fn probe_enforces_startup_process_window_and_background_scope() { + let scope = Scope { + pid: 42, + window: Some(17), + }; + for input in [ + json!({"action":"click"}), + json!({"action":"activate_app"}), + json!({"action":"start_control","mode":"foreground"}), + json!({"action":"get_app_state","app":{"pid":43}}), + json!({"action":"app_click","app":{"pid":42,"window_id":18}}), + json!({"action":"get_app_state","app":{"bundle_id":"untrusted"}}), + json!({"action":"app_batch","steps":[{"action":"app_key_chord","keys":["return"],"app":{"pid":43}}]}), + ] { + assert!(normalize(input, scope).is_err()); + } + let read = normalize( + json!({"action":"get_app_state","focus_window_only":false}), + scope, + ) + .unwrap(); + assert_eq!(read["app"], json!({"pid":42,"window_id":17})); + assert_eq!(read["focus_window_only"], true); + assert_eq!( + normalize(json!({"action":"start_control"}), scope).unwrap()["mode"], + "background" + ); + } + #[test] + fn probe_without_fixed_window_allows_same_process_window_reselection() { + let scope = Scope { + pid: 42, + window: None, + }; + let read = normalize(json!({"action":"list_windows"}), scope).unwrap(); + assert_eq!(read["app"], json!({"pid":42})); + let read = normalize( + json!({"action":"get_app_state","app":{"window_id":19}}), + scope, + ) + .unwrap(); + assert_eq!(read["app"], json!({"pid":42,"window_id":19})); + } +} diff --git a/src/apps/desktop/src/computer_use/desktop_host/native_control_roundtrip_tests.rs b/src/apps/desktop/src/computer_use/desktop_host/native_control_roundtrip_tests.rs index 57ad0de5a0..f21e0783f5 100644 --- a/src/apps/desktop/src/computer_use/desktop_host/native_control_roundtrip_tests.rs +++ b/src/apps/desktop/src/computer_use/desktop_host/native_control_roundtrip_tests.rs @@ -6,6 +6,9 @@ use openbitfun_core::agentic::tools::implementations::ComputerUseTool; use serde_json::{json, Value}; use std::sync::Arc; +#[path = "native_control_probe.rs"] +mod probe; + const OWNER: &str = "native-control-roundtrip-fixture"; async fn select_and_grant_fixture(host: &DesktopComputerUseHost, pid: i32, window: Option) { let pending = crate::computer_use::control_session::request_share( @@ -215,6 +218,10 @@ async fn wait_for_observer(path: &str) -> Value { } pub(crate) async fn run() { + if std::env::var_os("OPENBITFUN_CUA_PROBE_PID").is_some() { + probe::run().await; + return; + } if std::env::var_os("OPENBITFUN_INPUT_FIXTURE_PID").is_some() { if let Ok(case) = std::env::var("OPENBITFUN_INPUT_LATENCY_CASE") { run_latency_case(&case).await; @@ -613,6 +620,79 @@ async fn run_covered_semantic() { / global["height"].as_f64().unwrap(); json!({"kind":"image_xy","x":x.round() as i32,"y":y.round() as i32,"screenshot_id":meta["screenshot_id"]}) }; + if std::env::var_os("OPENBITFUN_INPUT_BUSY_AX").is_some() { + // Pause only the disposable fixture, after observing its exact window. + // The real AX IPC must time out; no mock alters the production query. + // Resume independently even if the tool panics, so cleanup cannot leave + // a stopped process. A semantic-only result fails the raw-event checks. + assert_eq!(unsafe { libc::kill(pid, libc::SIGSTOP) }, 0); + struct ResumeFixture(i32); + impl Drop for ResumeFixture { + fn drop(&mut self) { + unsafe { + libc::kill(self.0, libc::SIGCONT); + } + } + } + let _resume_on_error = ResumeFixture(pid); + let resumed = std::thread::spawn(move || { + std::thread::sleep(std::time::Duration::from_millis(1200)); + unsafe { + libc::kill(pid, libc::SIGCONT); + } + }); + let result = call( + &tool, + &context, + json!({ + "action":"app_click", "app":{"pid":pid}, + "target":image_target(&observation,"button") + }), + ) + .await; + resumed.join().unwrap(); + assert_eq!(result["completed_steps"], 1, "{result}"); + for _ in 0..25 { + if read(&path)["button_actions"] == 2 { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + } + let target = read(&path); + assert_eq!( + target["button_actions"], 2, + "busy-app input must arrive exactly once" + ); + let events = target["events"].as_array().unwrap(); + let content_events: Vec<_> = events + .iter() + .filter(|event| event["outside_frame"] == false) + .collect(); + assert_eq!( + content_events.len(), + 2, + "the timed-out read must select one pointer down/up, never replay AXPress: {target}" + ); + assert_eq!(content_events[0]["type"], 1); + assert_eq!(content_events[1]["type"], 2); + assert_eq!(target["foreign_actions"], 0); + assert_eq!( + crate::computer_use::macos_bg_input::frontmost_pid_macos(), + foreground + ); + let observer = read(&observer_path); + assert_eq!(observer["active"], true); + assert_eq!(observer["key_window"], true); + assert_eq!(observer["target_ahead"], false); + assert_eq!(observer["cursor"], initial_observer["cursor"]); + assert_eq!( + observer["activation_history"].as_array().unwrap().len(), + baseline_activations + ); + call(&tool, &context, json!({"action":"stop_control"})).await; + eprintln!("PASS real AX messaging timeout retains one window-directed click without foreground takeover or replay"); + return; + } eprintln!( "Before image button: observer={} frontmost={:?}", read(&observer_path), @@ -743,6 +823,56 @@ async fn run_covered_semantic() { eprintln!( "PASS non-AX canvas receives one unmodified click and text without foreground change" ); + let focus_event_count = |value: &Value| { + value["events"] + .as_array() + .unwrap() + .iter() + .filter(|event| event["outside_frame"] == true) + .count() + }; + assert_eq!( + focus_event_count(&state), + 2, + "one initial focus pair must serve the entire batch" + ); + // Induce a real target-local deactivation without changing the global + // foreground. The next ordinary Tool input must notice the stale local + // focus and prepare it again, preserving the original first responder. + let identity = crate::computer_use::macos_skylight::process_identity(pid).unwrap(); + assert!(crate::computer_use::macos_skylight::set_local_activation( + identity, + initial["window_id"].as_u64().unwrap() as u32, + false + )); + for _ in 0..25 { + if read(&path)["active"] == false { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + } + assert_eq!( + read(&path)["active"], + false, + "fixture must really lose local focus" + ); + assert_eq!( + crate::computer_use::macos_bg_input::frontmost_pid_macos(), + foreground + ); + let recovered = call( + &tool, + &context, + json!({"action":"app_type_text","app":{"pid":pid},"text":"recovered"}), + ) + .await; + assert_eq!(recovered["completed_steps"], 1, "{recovered}"); + let recovered_state = read(&path); + assert_eq!(recovered_state["canvas_text"], "canvas🙂recovered"); + assert_eq!(recovered_state["field_text"], "native-control🙂中文"); + assert_eq!(recovered_state["canvas_downs"], 1); + assert_eq!(focus_event_count(&recovered_state), 4); + eprintln!("PASS target-local focus loss invalidates cached preparation without global activation or a repeated content click"); // Human preview input uses a separate host instance but the same native // sharing resource, lease and background guard as the Agent. // Exercise that optional API separately, after the entire Agent task has diff --git a/src/apps/desktop/src/computer_use/macos_ax_dump.rs b/src/apps/desktop/src/computer_use/macos_ax_dump.rs index 10c141fff4..5ac6286876 100644 --- a/src/apps/desktop/src/computer_use/macos_ax_dump.rs +++ b/src/apps/desktop/src/computer_use/macos_ax_dump.rs @@ -43,6 +43,7 @@ type AXValueRef = *const c_void; unsafe extern "C" { fn AXUIElementCreateApplication(pid: i32) -> AXUIElementRef; fn AXUIElementGetPid(element: AXUIElementRef, pid: *mut i32) -> i32; + fn AXUIElementSetMessagingTimeout(element: AXUIElementRef, timeout: f32) -> i32; fn AXUIElementCopyElementAtPosition( element: AXUIElementRef, x: f32, @@ -263,6 +264,27 @@ pub(crate) fn is_application_menu_item(target: AxRef) -> bool { } } +/// Hit testing is a read-only optimization for an independently addressed +/// window coordinate. A busy app or missing AX implementation does not make +/// directed pointer input unavailable. Never apply this policy to AX writes or +/// observed node indices: their outcome/identity must remain authoritative. +fn optional_point_hit_available(status: i32) -> OpenBitFunResult { + match status { + 0 => Ok(true), + // Failure / CannotComplete / AttributeUnsupported / NoValue / + // NotImplemented. None of these read results submitted an action. + -25200 | -25204 | -25205 | -25208 | -25212 => { + log::debug!(target: "computer_use::ax", "Optional AX point lookup unavailable (status={status}); retaining window-directed pointer route"); + Ok(false) + } + // Keep permission denial, invalid references/arguments and unknown + // errors visible instead of treating them as missing accessibility. + _ => Err(OpenBitFunError::tool(format!( + "AX_HIT_TEST_FAILED: Application hit testing failed (status={status})" + ))), + } +} + /// Native application-scoped hit testing ignores other applications' covering /// windows. The result must belong to the captured window; another same-app /// window is never substituted. Rectangle ranking is not a valid fallback. @@ -282,21 +304,17 @@ pub(crate) fn retained_target_at_point( if app.is_null() { return Ok(None); } + // This optional query must not impose the AX default multi-second + // messaging timeout on every visual click into a busy application. + // The timeout is local to this temporary application reference. + AXUIElementSetMessagingTimeout(app, 0.3); let mut hit = std::ptr::null(); let status = AXUIElementCopyElementAtPosition(app, x as f32, y as f32, &mut hit); ax_release(app as CFTypeRef); if status != 0 { ax_release(hit as CFTypeRef); - // Only a documented absence of hit-testing/content permits a - // visual fallback. Messaging/permission failures are not evidence - // that a pointer click will focus the intended text field. - return if matches!(status, -25208 | -25212) { - Ok(None) - } else { - Err(OpenBitFunError::tool(format!( - "AX_HIT_TEST_FAILED: Application hit testing failed (status={status})" - ))) - }; + optional_point_hit_available(status)?; + return Ok(None); } if hit.is_null() { return Ok(None); @@ -1116,6 +1134,23 @@ fn quote_clip(s: &str, max: usize) -> String { mod tests { use super::*; + #[test] + fn optional_point_hit_read_failure_preserves_directed_input_route() { + assert!(optional_point_hit_available(0).unwrap()); + for status in [-25200, -25204, -25205, -25208, -25212] { + assert!(!optional_point_hit_available(status).unwrap(), "{status}"); + } + } + + #[test] + fn optional_point_hit_keeps_denial_and_invalid_target_errors() { + for status in [-25201, -25202, -25203, -25211, -99999] { + let error = optional_point_hit_available(status).unwrap_err(); + assert!(error.to_string().contains("AX_HIT_TEST_FAILED")); + assert!(error.to_string().contains(&status.to_string())); + } + } + #[test] #[ignore = "run scripts/test-native-ax-context.mjs with macOS Accessibility permission"] fn native_ax_fixture_round_trips_tree_and_cached_targets() { diff --git a/src/apps/desktop/src/computer_use/macos_bg_input.rs b/src/apps/desktop/src/computer_use/macos_bg_input.rs index ae2f74e450..c5fdb07a6e 100644 --- a/src/apps/desktop/src/computer_use/macos_bg_input.rs +++ b/src/apps/desktop/src/computer_use/macos_bg_input.rs @@ -148,12 +148,160 @@ fn flags_from(mods: &[BgModifier]) -> CGEventFlags { .fold(CGEventFlags::CGEventFlagNull, |acc, m| acc | m.flag()) } -#[derive(Clone, Copy, PartialEq)] +/// Build one complete chord before dispatch. Modifier flags describe the keys +/// still held at each event, so releasing several modifiers cannot reintroduce +/// a previously released key. Aliases must never produce duplicate presses. +fn key_chord_events( + modifiers: &[BgModifier], + key: u16, + label: &str, +) -> OpenBitFunResult> { + let mut unique = Vec::with_capacity(modifiers.len()); + for modifier in modifiers { + if !unique.contains(modifier) { + unique.push(*modifier); + } + } + let source = private_source(label)?; + let mut events = Vec::with_capacity(unique.len() * 2 + 2); + let mut append = |keycode, down, flags| -> OpenBitFunResult<()> { + let event = CGEvent::new_keyboard_event(source.clone(), keycode, down).map_err(|_| { + OpenBitFunError::tool(format!("CGEvent chord creation failed ({label})")) + })?; + event.set_flags(flags); + events.push(event); + Ok(()) + }; + for (index, modifier) in unique.iter().enumerate() { + append(modifier.keycode(), true, flags_from(&unique[..=index]))?; + } + let flags = flags_from(&unique); + append(key, true, flags)?; + append(key, false, flags)?; + for (index, modifier) in unique.iter().enumerate().rev() { + append(modifier.keycode(), false, flags_from(&unique[..index]))?; + } + Ok(events) +} + +#[derive(Clone, Copy, Debug, PartialEq)] enum HeldInput { Key(u16, bool), - Mouse(u8, f64, f64, u32, f64, f64), + Mouse { + button: u8, + point: [f64; 2], + window: u32, + }, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +struct InputOwner { + pid: i32, + identity: super::macos_skylight::ProcessIdentity, +} + +static HELD_INPUTS: std::sync::Mutex> = + std::sync::Mutex::new(Vec::new()); +static INPUT_RELEASE: std::sync::Mutex<()> = std::sync::Mutex::new(()); + +fn verify_input_identity( + owner: InputOwner, + current: Option, +) -> OpenBitFunResult<()> { + if current != Some(owner.identity) { + return Err(OpenBitFunError::tool( + "[CONTROL_TARGET_CHANGED] The input target process identity changed or is unavailable", + )); + } + Ok(()) +} + +fn record_keyboard_input( + held: &mut Vec<(InputOwner, HeldInput)>, + target: InputOwner, + event: &CGEvent, + authenticated: bool, +) { + let code = event + .get_integer_value_field(core_graphics::event::EventField::KEYBOARD_EVENT_KEYCODE) + as u16; + let down = match event.get_type() { + CGEventType::KeyDown => true, + CGEventType::KeyUp => false, + CGEventType::FlagsChanged => { + // CoreGraphics creates FlagsChanged for modifier key-down AND + // key-up. The flag for that key identifies the transition. + let modifier = match code { + 54 | 55 => BgModifier::Command, + 56 | 60 => BgModifier::Shift, + 58 | 61 => BgModifier::Option, + 59 | 62 => BgModifier::Control, + 63 => BgModifier::Fn, + _ => return, + }; + event.get_flags().contains(modifier.flag()) + } + _ => return, + }; + if down { + if !held.iter().any(|(owner, input)| { + *owner == target && matches!(input, HeldInput::Key(c, _) if *c == code) + }) { + held.push((target, HeldInput::Key(code, authenticated))); + } + } else { + held.retain(|(owner, input)| { + !(*owner == target && matches!(input, HeldInput::Key(c, _) if *c == code)) + }); + } +} + +fn record_mouse_input( + held: &mut Vec<(InputOwner, HeldInput)>, + target: InputOwner, + event: &CGEvent, + window: u32, +) { + let kind = event.get_type(); + let button = match kind { + CGEventType::LeftMouseDown | CGEventType::LeftMouseDragged | CGEventType::LeftMouseUp => 0, + CGEventType::RightMouseDown + | CGEventType::RightMouseDragged + | CGEventType::RightMouseUp => 1, + CGEventType::OtherMouseDown + | CGEventType::OtherMouseDragged + | CGEventType::OtherMouseUp => 2, + _ => return, + }; + let point = event.location(); + let latest = HeldInput::Mouse { + button, + point: [point.x, point.y], + window, + }; + match kind { + CGEventType::LeftMouseDown | CGEventType::RightMouseDown | CGEventType::OtherMouseDown => { + held.push((target, latest)); + } + CGEventType::LeftMouseDragged + | CGEventType::RightMouseDragged + | CGEventType::OtherMouseDragged => { + for (owner, input) in held.iter_mut() { + if *owner == target + && matches!(input, HeldInput::Mouse { button: held_button, .. } if *held_button == button) + { + // Cancellation keeps the last global point actually sent; + // its window-local route is rebuilt from current geometry. + *input = latest; + } + } + } + _ => held.retain(|(owner, input)| { + !(*owner == target + && matches!(input, HeldInput::Mouse { button: held_button, .. } if *held_button == button)) + }), + } } -static HELD_INPUTS: std::sync::Mutex> = std::sync::Mutex::new(Vec::new()); /// Every gesture releases its own unfinished presses on error or cancellation. /// Actions hold an exclusive control lease, so no other action can own entries. @@ -167,42 +315,98 @@ impl Drop for InputReleaseGuard { /// Cleanup is allowed after revocation: it only releases events actually posted /// by this host, using the same directed delivery backend. pub(super) fn release_held_inputs() { - let Ok(mut held) = HELD_INPUTS.lock() else { - return; + drain_and_release_inputs(&INPUT_RELEASE, &HELD_INPUTS, |owner, input| { + if let Err(error) = release_held_input( + owner, + input, + super::macos_skylight::process_identity, + super::macos_capture::window_bounds, + route_mouse_to_window, + |pid, event, authenticated| { + if !authenticated + || !super::macos_skylight::post_to_pid(pid, event.as_ptr().cast(), true) + { + event.post_to_pid(pid); + } + }, + ) { + warn!( + target: "computer_use::bg_input", + "Skipped directed input release for pid={}: {}", + owner.pid, + error + ); + } + }); +} + +fn drain_and_release_inputs( + release: &std::sync::Mutex<()>, + held: &std::sync::Mutex>, + mut release_input: impl FnMut(InputOwner, HeldInput), +) { + // Stop calls cleanup before destroying capture. Even an empty second + // cleanup must wait for a prior drain to finish its window lookup and post. + // This mutex protects sequencing only, so poisoning must not bypass it. + let _release = release.lock().unwrap_or_else(|error| error.into_inner()); + // Lock order is release -> held (drain only) -> capture. No capture caller + // holds its lock while entering cleanup, and posting never takes release. + let held = match held.lock() { + Ok(mut held) => std::mem::take(&mut *held), + Err(_) => return, }; - if held.is_empty() { - return; + for (owner, input) in held { + release_input(owner, input); } - let Ok(source) = private_source("cancel_release") else { - return; - }; - for (pid, input) in held.drain(..) { - let event = match input { - HeldInput::Key(code, _) => CGEvent::new_keyboard_event(source.clone(), code, false), - HeldInput::Mouse(button, x, y, _, _, _) => { - let (kind, button) = match button { - 0 => (CGEventType::LeftMouseUp, CGMouseButton::Left), - 1 => (CGEventType::RightMouseUp, CGMouseButton::Right), - _ => (CGEventType::OtherMouseUp, CGMouseButton::Center), - }; - mouse_source("cancel_mouse_release") - .map_err(|_| ()) - .and_then(|source| { - CGEvent::new_mouse_event(source, kind, CGPoint::new(x, y), button) - }) - } - }; - if let Ok(event) = event { - if let HeldInput::Mouse(_, _, _, window, wx, wy) = input { - let _ = route_mouse_to_window(&event, window, wx, wy); - } - event.set_flags(CGEventFlags::CGEventFlagNull); - let auth = matches!(input, HeldInput::Key(_, true)); - if !auth || !super::macos_skylight::post_to_pid(pid, event.as_ptr().cast(), true) { - event.post_to_pid(pid); - } +} + +fn release_held_input( + owner: InputOwner, + input: HeldInput, + mut identity: impl FnMut(i32) -> Option, + window_bounds: impl FnOnce(i32, u32) -> Result<[f64; 4], String>, + route: impl FnOnce(&CGEvent, u32, f64, f64) -> OpenBitFunResult<()>, + post: impl FnOnce(i32, &CGEvent, bool), +) -> OpenBitFunResult<()> { + verify_input_identity(owner, identity(owner.pid))?; + let event = match input { + HeldInput::Key(code, _) => { + CGEvent::new_keyboard_event(private_source("cancel_release")?, code, false).map_err( + |_| OpenBitFunError::tool("Cancellation keyboard release creation failed"), + )? } - } + HeldInput::Mouse { + button, + point: [x, y], + window, + } => { + let current = window_bounds(owner.pid, window).map_err(OpenBitFunError::tool)?; + let (kind, button) = match button { + 0 => (CGEventType::LeftMouseUp, CGMouseButton::Left), + 1 => (CGEventType::RightMouseUp, CGMouseButton::Right), + _ => (CGEventType::OtherMouseUp, CGMouseButton::Center), + }; + let event = CGEvent::new_mouse_event( + mouse_source("cancel_mouse_release")?, + kind, + CGPoint::new(x, y), + button, + ) + .map_err(|_| OpenBitFunError::tool("Cancellation mouse release creation failed"))?; + // Moving or resizing the same window must not strand a posted + // button-down. Keep the last global point instead of translating + // it with the window: its new local point can be outside content, + // so cancellation does not follow a moved control to click it. + route(&event, window, current[0], current[1])?; + event + } + }; + event.set_flags(CGEventFlags::CGEventFlagNull); + // Event construction and routing can cross a process exit. Check again + // immediately before PID-directed delivery; never release into a reused PID. + verify_input_identity(owner, identity(owner.pid))?; + post(owner.pid, &event, matches!(input, HeldInput::Key(_, true))); + Ok(()) } fn route_mouse_to_window(event: &CGEvent, window: u32, wx: f64, wy: f64) -> OpenBitFunResult<()> { @@ -231,6 +435,13 @@ fn post_directed(pid: i32, event: &CGEvent, authenticated: bool) -> OpenBitFunRe .lock() .map_err(|_| OpenBitFunError::tool("Input state lock poisoned"))?; super::control_session::input_allowed().map_err(OpenBitFunError::tool)?; + let identity = super::macos_skylight::process_identity(pid).ok_or_else(|| { + OpenBitFunError::tool("[CONTROL_TARGET_CHANGED] The input target process is unavailable") + })?; + let owner = InputOwner { pid, identity }; + for (held_owner, _) in held.iter().filter(|(held_owner, _)| held_owner.pid == pid) { + verify_input_identity(*held_owner, Some(identity))?; + } let window_id = super::macos_capture::bound_window_id(pid).map_err(|_| { OpenBitFunError::tool( "[CONTROL_TARGET_CHANGED] The captured application window is unavailable", @@ -238,8 +449,9 @@ fn post_directed(pid: i32, event: &CGEvent, authenticated: bool) -> OpenBitFunRe })?; super::control_session::target_allowed(&format!("pid:{pid}/window:{window_id}")) .map_err(OpenBitFunError::tool)?; - let [wx, wy, width, height] = + let bounds = super::macos_capture::window_bounds(pid, window_id).map_err(OpenBitFunError::tool)?; + let [wx, wy, width, height] = bounds; if matches!(event.get_type(), CGEventType::ScrollWheel) { let point = super::control_session::snapshot() .pointer @@ -259,70 +471,16 @@ fn post_directed(pid: i32, event: &CGEvent, authenticated: bool) -> OpenBitFunRe ) { route_mouse_to_window(event, window_id, wx, wy)?; } + verify_input_identity(owner, super::macos_skylight::process_identity(pid))?; if !authenticated || !super::macos_skylight::post_to_pid(pid, event.as_ptr().cast(), true) { event.post_to_pid(pid); } super::macos_capture::note_input(pid); - let point = event.location(); match event.get_type() { - CGEventType::KeyDown => { - let code = event - .get_integer_value_field(core_graphics::event::EventField::KEYBOARD_EVENT_KEYCODE) - as u16; - if !held - .iter() - .any(|(p, i)| *p == pid && matches!(i, HeldInput::Key(c, _) if *c == code)) - { - held.push((pid, HeldInput::Key(code, authenticated))); - } - } - CGEventType::KeyUp => { - let code = event - .get_integer_value_field(core_graphics::event::EventField::KEYBOARD_EVENT_KEYCODE) - as u16; - held.retain(|(p, i)| !(*p == pid && matches!(i, HeldInput::Key(c, _) if *c == code))); + CGEventType::KeyDown | CGEventType::KeyUp | CGEventType::FlagsChanged => { + record_keyboard_input(&mut held, owner, event, authenticated); } - CGEventType::LeftMouseDown | CGEventType::RightMouseDown | CGEventType::OtherMouseDown => { - let button = match event.get_type() { - CGEventType::LeftMouseDown => 0, - CGEventType::RightMouseDown => 1, - _ => 2, - }; - held.push(( - pid, - HeldInput::Mouse(button, point.x, point.y, window_id, wx, wy), - )); - } - CGEventType::LeftMouseDragged - | CGEventType::RightMouseDragged - | CGEventType::OtherMouseDragged => { - let button = match event.get_type() { - CGEventType::LeftMouseDragged => 0, - CGEventType::RightMouseDragged => 1, - _ => 2, - }; - for (owner, input) in held.iter_mut() { - if *owner == pid { - if let HeldInput::Mouse(held_button, x, y, _, _, _) = input { - if *held_button == button { - *x = point.x; - *y = point.y; - } - } - } - } - } - CGEventType::LeftMouseUp | CGEventType::RightMouseUp | CGEventType::OtherMouseUp => { - let button = match event.get_type() { - CGEventType::LeftMouseUp => 0, - CGEventType::RightMouseUp => 1, - _ => 2, - }; - held.retain(|(p, i)| { - !(*p == pid && matches!(i, HeldInput::Mouse(b, _, _, _, _, _) if *b == button)) - }); - } - _ => {} + _ => record_mouse_input(&mut held, owner, event, window_id), } Ok(()) } @@ -739,7 +897,7 @@ pub(super) fn bg_type_text(pid: i32, text: &str) -> OpenBitFunResult<()> { text.chars().count(), text.len() ); - // Single source for the whole string (Codex parity): keeps the kernel + // Single private source for the whole string: keeps the synthetic // keyboard state coherent and avoids the per-char allocation cost. let src = private_source("type_text")?; // We send one event per Unicode scalar to keep individual events small @@ -759,10 +917,8 @@ pub(super) fn bg_type_text(pid: i32, text: &str) -> OpenBitFunResult<()> { })?; ev2.set_string_from_utf16_unchecked(&buf); post_keyboard(pid, &ev2)?; - // 8ms inter-key gap matches Codex / native typing rates and avoids - // dropped chars in Chromium webviews and SwiftUI multi-line fields - // that throttle their keystroke handler. 1ms (the previous value) - // was reliably losing ~5–10% of CJK glyphs in informal smoke tests. + // Pace direct Unicode events so controls can process each scalar. + // This is the local delivery policy, not a reference-backend timing guarantee. thread::sleep(Duration::from_millis(8)); } Ok(()) @@ -779,43 +935,8 @@ pub(super) fn bg_key_chord(pid: i32, modifiers: &[BgModifier], key: u16) -> Open "bg_key_chord.enter pid={} keycode={} modifiers={:?}", pid, key, modifiers ); - let flags = flags_from(modifiers); - // Single source across the whole chord — required for the modifier - // latch state to survive between mod_down → key_down → key_up → mod_up. - let src = private_source("key_chord")?; - - // Press modifiers. - for m in modifiers { - let ev = CGEvent::new_keyboard_event(src.clone(), m.keycode(), true) - .map_err(|_| OpenBitFunError::tool("CGEvent ModDown failed".to_string()))?; - ev.set_flags(flags); - post_keyboard(pid, &ev)?; - } - // Press main key. - { - let ev = CGEvent::new_keyboard_event(src.clone(), key, true) - .map_err(|_| OpenBitFunError::tool("CGEvent KeyDown failed".to_string()))?; - ev.set_flags(flags); - post_keyboard(pid, &ev)?; - } - { - let ev = CGEvent::new_keyboard_event(src.clone(), key, false) - .map_err(|_| OpenBitFunError::tool("CGEvent KeyUp failed".to_string()))?; - ev.set_flags(flags); - post_keyboard(pid, &ev)?; - } - // Release modifiers in reverse press order. - for m in modifiers.iter().rev() { - let ev = CGEvent::new_keyboard_event(src.clone(), m.keycode(), false) - .map_err(|_| OpenBitFunError::tool("CGEvent ModUp failed".to_string()))?; - // Drop this modifier from the flag set as we release it. - let remaining = modifiers - .iter() - .copied() - .filter(|x| x != m) - .collect::>(); - ev.set_flags(flags_from(&remaining)); - post_keyboard(pid, &ev)?; + for event in key_chord_events(modifiers, key, "key_chord")? { + post_keyboard(pid, &event)?; } Ok(()) } @@ -1055,37 +1176,8 @@ pub(super) fn bg_key_chord_no_auth( "bg_key_chord_no_auth.enter pid={} keycode={} modifiers={:?}", pid, key, modifiers ); - let flags = flags_from(modifiers); - let src = private_source("key_chord_no_auth")?; - - for m in modifiers { - let ev = CGEvent::new_keyboard_event(src.clone(), m.keycode(), true) - .map_err(|_| OpenBitFunError::tool("CGEvent ModDown (no_auth) failed".to_string()))?; - ev.set_flags(flags); - post_keyboard_no_auth(pid, &ev)?; - } - { - let ev = CGEvent::new_keyboard_event(src.clone(), key, true) - .map_err(|_| OpenBitFunError::tool("CGEvent KeyDown (no_auth) failed".to_string()))?; - ev.set_flags(flags); - post_keyboard_no_auth(pid, &ev)?; - } - { - let ev = CGEvent::new_keyboard_event(src.clone(), key, false) - .map_err(|_| OpenBitFunError::tool("CGEvent KeyUp (no_auth) failed".to_string()))?; - ev.set_flags(flags); - post_keyboard_no_auth(pid, &ev)?; - } - for m in modifiers.iter().rev() { - let ev = CGEvent::new_keyboard_event(src.clone(), m.keycode(), false) - .map_err(|_| OpenBitFunError::tool("CGEvent ModUp (no_auth) failed".to_string()))?; - let remaining = modifiers - .iter() - .copied() - .filter(|x| x != m) - .collect::>(); - ev.set_flags(flags_from(&remaining)); - post_keyboard_no_auth(pid, &ev)?; + for event in key_chord_events(modifiers, key, "key_chord_no_auth")? { + post_keyboard_no_auth(pid, &event)?; } Ok(()) } @@ -1183,7 +1275,9 @@ pub(super) fn parse_key_spec(spec: &str) -> OpenBitFunResult<(Vec, u for p in head { let m = BgModifier::from_str(p) .ok_or_else(|| OpenBitFunError::tool(format!("unknown modifier in key spec: {}", p)))?; - mods.push(m); + if !mods.contains(&m) { + mods.push(m); + } } let kc = keycode_for_named(last) .or_else(|| { @@ -1215,7 +1309,9 @@ pub(super) fn parse_key_sequence(keys: &[String]) -> OpenBitFunResult<(Vec OpenBitFunResult<(Vec Option { Some(match name.to_ascii_lowercase().as_str() { "return" | "enter" => 36, @@ -1262,9 +1358,9 @@ pub(super) fn keycode_for_named(name: &str) -> Option { }) } -/// Map a single ASCII character to the **US-keyboard** keycode. This is the -/// same table Codex / enigo use; the user's actual keymap is irrelevant for -/// our chord injection because we set explicit modifier flags ourselves. +/// Map a single ASCII character to a US-keyboard virtual keycode for chords. +/// Text entry uses the separate Unicode path; these physical key positions +/// do not establish layout-independent shortcut behavior. pub(super) fn keycode_for_char(c: char) -> Option { let upper = c.to_ascii_uppercase(); Some(match upper { @@ -1573,9 +1669,8 @@ const CHROMIUM_BUNDLE_KEYWORDS: &[&str] = &[ "com.operasoftware.operaprofiles", ]; -/// Returns `true` when the bundle_id indicates a Chromium-based or -/// Electron-based application. These apps need the `bg_click_chromium` -/// 5-event recipe for reliable background clicks. +/// Legacy classification used by the foreground-compatible click route. +/// Background window-directed delivery must not depend on this application list. pub(super) fn is_chromium_electron(bundle_id: Option<&str>) -> bool { if let Some(bid) = bundle_id { let lc = bid.to_ascii_lowercase(); @@ -1614,6 +1709,454 @@ pub(super) fn bundle_id_for_pid(pid: i32) -> Option { mod tests { use super::*; + fn input_owner() -> InputOwner { + InputOwner { + pid: 42, + identity: super::super::macos_skylight::ProcessIdentity([0, 11]), + } + } + + fn held_mouse() -> HeldInput { + HeldInput::Mouse { + button: 0, + point: [120.0, 140.0], + window: 17, + } + } + + #[test] + fn cancellation_serializes_an_empty_stop_cleanup_after_an_inflight_release() { + use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + use std::sync::{mpsc, Arc, Mutex, TryLockError}; + + let release = Arc::new(Mutex::new(())); + let held = Arc::new(Mutex::new(vec![(input_owner(), held_mouse())])); + let completed = Arc::new(AtomicBool::new(false)); + let releases = Arc::new(AtomicUsize::new(0)); + let (drained_tx, drained_rx) = mpsc::channel(); + let (resume_tx, resume_rx) = mpsc::channel(); + let (contending_tx, contending_rx) = mpsc::channel(); + + let first = { + let release = Arc::clone(&release); + let held = Arc::clone(&held); + let completed = Arc::clone(&completed); + let releases = Arc::clone(&releases); + thread::spawn(move || { + drain_and_release_inputs(&release, &held, |_, _| { + // Pause at the exact original race: input was drained but + // its geometry lookup and native release have not finished. + drained_tx.send(()).unwrap(); + resume_rx.recv().unwrap(); + releases.fetch_add(1, Ordering::SeqCst); + completed.store(true, Ordering::SeqCst); + }); + }) + }; + drained_rx.recv().unwrap(); + // The held-state lock is free while cleanup consults capture. + let drained_and_unlocked = held.try_lock().is_ok_and(|held| held.is_empty()); + let second = { + let release = Arc::clone(&release); + let held = Arc::clone(&held); + let completed = Arc::clone(&completed); + thread::spawn(move || { + // This happens while the first callback is blocked on resume, + // proving contention without a sleep or scheduler deadline. + let blocked = matches!(release.try_lock(), Err(TryLockError::WouldBlock)); + contending_tx.send(blocked).unwrap(); + drain_and_release_inputs(&release, &held, |_, _| { + panic!("The first cleanup already owns the only release") + }); + // stop_capture may tear down the native capture only here. + completed.load(Ordering::SeqCst) + }) + }; + let blocked = contending_rx.recv().unwrap(); + resume_tx.send(()).unwrap(); + first.join().unwrap(); + let teardown_follows_release = second.join().unwrap(); + assert!(drained_and_unlocked); + assert!(blocked, "The cleanup lock must span drain through release"); + assert!(teardown_follows_release); + assert_eq!(releases.load(Ordering::SeqCst), 1); + } + + #[test] + fn cancellation_never_posts_to_a_missing_or_reused_process() { + let owner = input_owner(); + let replacement = super::super::macos_skylight::ProcessIdentity([0, 12]); + for current in [None, Some(replacement)] { + for input in [ + HeldInput::Key(55, false), + HeldInput::Key(55, true), + held_mouse(), + ] { + let result = release_held_input( + owner, + input, + |pid| { + assert_eq!(pid, owner.pid); + current + }, + |_, _| panic!("Process identity must be checked before window lookup"), + |_, _, _, _| panic!("A replacement process must not be routed"), + |_, _, _| panic!("A replacement process must not receive a release"), + ); + assert!(result + .unwrap_err() + .to_string() + .contains("CONTROL_TARGET_CHANGED")); + } + } + } + + #[test] + fn cancellation_skips_an_unavailable_original_window() { + let owner = input_owner(); + let result = release_held_input( + owner, + held_mouse(), + |_| Some(owner.identity), + |pid, window| { + assert_eq!((pid, window), (owner.pid, 17)); + Err("Original window is unavailable".to_string()) + }, + |_, _, _, _| panic!("An unavailable window must not be routed"), + |_, _, _| panic!("An unavailable window must not receive a release"), + ); + assert!(result.is_err()); + } + + #[test] + fn cancellation_releases_once_after_window_move_or_resize() { + let owner = input_owner(); + for current in [[200.0, 100.0, 800.0, 600.0], [100.0, 100.0, 10.0, 10.0]] { + let mut posts = 0; + release_held_input( + owner, + held_mouse(), + |_| Some(owner.identity), + |pid, window| { + assert_eq!((pid, window), (owner.pid, 17)); + Ok(current) + }, + |event, window, wx, wy| { + assert_eq!((window, wx, wy), (17, current[0], current[1])); + let point = event.location(); + let local = (point.x - wx, point.y - wy); + assert!(local.0 < 0.0 || local.0 >= current[2] || local.1 >= current[3]); + route_mouse_to_window(event, window, wx, wy) + }, + |pid, event, authenticated| { + posts += 1; + assert_eq!(pid, owner.pid); + assert!(!authenticated); + assert_eq!(event.get_type() as u32, CGEventType::LeftMouseUp as u32); + let point = event.location(); + assert_eq!((point.x, point.y), (120.0, 140.0)); + }, + ) + .unwrap(); + assert_eq!(posts, 1); + } + } + + #[test] + fn cancellation_does_not_fall_back_after_window_routing_failure() { + let owner = input_owner(); + let result = release_held_input( + owner, + held_mouse(), + |_| Some(owner.identity), + |_, _| Ok([100.0, 100.0, 800.0, 600.0]), + |event, window, wx, wy| { + assert_eq!(event.get_type() as u32, CGEventType::LeftMouseUp as u32); + assert_eq!((window, wx, wy), (17, 100.0, 100.0)); + Err(OpenBitFunError::tool("Window-local routing unavailable")) + }, + |_, _, _| panic!("An unrouted release must never be posted"), + ); + assert!(result + .unwrap_err() + .to_string() + .contains("routing unavailable")); + } + + #[test] + fn cancellation_rechecks_process_identity_after_native_event_preparation() { + let owner = input_owner(); + let replacement = super::super::macos_skylight::ProcessIdentity([0, 12]); + for current in [None, Some(replacement)] { + for input in [HeldInput::Key(55, true), held_mouse()] { + let mut reads = 0; + let result = release_held_input( + owner, + input, + |_| { + reads += 1; + if reads == 1 { + Some(owner.identity) + } else { + current + } + }, + |_, _| Ok([100.0, 100.0, 800.0, 600.0]), + route_mouse_to_window, + |_, _, _| panic!("Process exit during preparation must prevent posting"), + ); + assert_eq!(reads, 2); + assert!(result + .unwrap_err() + .to_string() + .contains("CONTROL_TARGET_CHANGED")); + } + } + } + + #[test] + fn cancellation_keeps_last_drag_point_with_current_window_routing() { + use core_graphics::event::EventField; + let owner = input_owner(); + let mut held = vec![]; + let down = CGEvent::new_mouse_event( + mouse_source("held_down_test").unwrap(), + CGEventType::LeftMouseDown, + CGPoint::new(120.0, 140.0), + CGMouseButton::Left, + ) + .unwrap(); + record_mouse_input(&mut held, owner, &down, 17); + let drag = CGEvent::new_mouse_event( + mouse_source("held_drag_test").unwrap(), + CGEventType::LeftMouseDragged, + CGPoint::new(450.0, 260.0), + CGMouseButton::Left, + ) + .unwrap(); + record_mouse_input(&mut held, owner, &drag, 17); + let current_bounds = [600.0, 500.0, 900.0, 700.0]; + assert_eq!(held.len(), 1); + let mut posts = 0; + release_held_input( + held[0].0, + held[0].1, + |_| Some(owner.identity), + |_, _| Ok(current_bounds), + |event, window, wx, wy| { + assert_eq!((window, wx, wy), (17, 600.0, 500.0)); + route_mouse_to_window(event, window, wx, wy) + }, + |pid, event, authenticated| { + posts += 1; + assert_eq!(pid, owner.pid); + assert!(!authenticated); + assert_eq!(event.get_type() as u32, CGEventType::LeftMouseUp as u32); + let point = event.location(); + assert_eq!((point.x, point.y), (450.0, 260.0)); + assert_eq!(event.get_flags(), CGEventFlags::CGEventFlagNull); + assert_eq!( + event.get_integer_value_field(EventField::EVENT_SOURCE_STATE_ID), + 0 + ); + for field in [51, 91, 92] { + assert_eq!(event.get_integer_value_field(field), 17); + } + assert_eq!(event.get_integer_value_field(58), 1); + }, + ) + .unwrap(); + assert_eq!(posts, 1); + } + + #[test] + fn cancellation_preserves_keyboard_backend_without_consulting_windows() { + use core_graphics::event::EventField; + let owner = input_owner(); + for authenticated in [false, true] { + let mut posts = 0; + release_held_input( + owner, + HeldInput::Key(35, authenticated), + |_| Some(owner.identity), + |_, _| panic!("Keyboard cleanup does not require the original window"), + |_, _, _, _| panic!("Keyboard cleanup must not use mouse routing"), + |pid, event, backend| { + posts += 1; + assert_eq!((pid, backend), (owner.pid, authenticated)); + assert_eq!(event.get_type() as u32, CGEventType::KeyUp as u32); + assert_eq!( + event.get_integer_value_field(EventField::KEYBOARD_EVENT_KEYCODE), + 35 + ); + assert_eq!(event.get_flags(), CGEventFlags::CGEventFlagNull); + let source = event.get_integer_value_field(EventField::EVENT_SOURCE_STATE_ID); + assert_ne!(source, CGEventSourceStateID::CombinedSessionState as i64); + assert_ne!(source, CGEventSourceStateID::HIDSystemState as i64); + }, + ) + .unwrap(); + assert_eq!(posts, 1); + } + } + + fn chord_event_states(events: &[CGEvent]) -> Vec<(u32, i64, CGEventFlags)> { + use core_graphics::event::EventField; + // Private sources receive their own native ID; -1 is the constructor + // request, not the stored event ID. Neither shared source may be used. + let source_id = events[0].get_integer_value_field(EventField::EVENT_SOURCE_STATE_ID); + assert_ne!(source_id, CGEventSourceStateID::CombinedSessionState as i64); + assert_ne!(source_id, CGEventSourceStateID::HIDSystemState as i64); + events + .iter() + .map(|event| { + // Inspect the actual native events without posting to any app + // or the global input seat. + assert_eq!( + event.get_integer_value_field(EventField::EVENT_SOURCE_STATE_ID), + source_id + ); + ( + event.get_type() as u32, + event.get_integer_value_field(EventField::KEYBOARD_EVENT_KEYCODE), + event.get_flags(), + ) + }) + .collect() + } + + #[test] + fn key_chord_events_release_only_still_held_modifiers() { + let command = CGEventFlags::CGEventFlagCommand; + let shift = CGEventFlags::CGEventFlagShift; + let option = CGEventFlags::CGEventFlagAlternate; + let none = CGEventFlags::CGEventFlagNull; + for label in ["key_chord", "key_chord_no_auth"] { + let events = key_chord_events( + &[BgModifier::Command, BgModifier::Shift, BgModifier::Option], + 35, + label, + ) + .unwrap(); + assert_eq!( + chord_event_states(&events), + vec![ + (CGEventType::FlagsChanged as u32, 55, command), + (CGEventType::FlagsChanged as u32, 56, command | shift), + ( + CGEventType::FlagsChanged as u32, + 58, + command | shift | option + ), + (CGEventType::KeyDown as u32, 35, command | shift | option), + (CGEventType::KeyUp as u32, 35, command | shift | option), + (CGEventType::FlagsChanged as u32, 58, command | shift), + (CGEventType::FlagsChanged as u32, 56, command), + (CGEventType::FlagsChanged as u32, 55, none), + ], + "{label} must finish with every modifier released" + ); + } + } + + #[test] + fn key_chord_events_normalize_duplicate_modifier_aliases() { + let (spec, key) = parse_key_spec("cmd+command+shift+SHIFT+p").unwrap(); + let (sequence, sequence_key) = parse_key_sequence(&[ + "cmd".into(), + "command".into(), + "shift".into(), + "SHIFT".into(), + "p".into(), + ]) + .unwrap(); + assert_eq!(spec, vec![BgModifier::Command, BgModifier::Shift]); + assert_eq!(sequence, spec); + assert_eq!(sequence_key, key); + let expected = key_chord_events(&spec, key, "unique").unwrap(); + // Direct native callers also receive normalization, independently of + // how a model-facing key specification was parsed. + let duplicate = key_chord_events( + &[ + BgModifier::Command, + BgModifier::Command, + BgModifier::Shift, + BgModifier::Shift, + ], + key, + "duplicate", + ) + .unwrap(); + assert_eq!( + chord_event_states(&duplicate), + chord_event_states(&expected) + ); + assert_eq!(duplicate.len(), 6); + } + + #[test] + fn key_chord_events_leave_following_unmodified_key_clear() { + let modified = + key_chord_events(&[BgModifier::Command, BgModifier::Shift], 35, "modified").unwrap(); + assert_eq!( + modified.last().unwrap().get_flags(), + CGEventFlags::CGEventFlagNull + ); + let following = key_chord_events(&[], 0, "following").unwrap(); + assert_eq!( + chord_event_states(&following), + vec![ + ( + CGEventType::KeyDown as u32, + 0, + CGEventFlags::CGEventFlagNull + ), + (CGEventType::KeyUp as u32, 0, CGEventFlags::CGEventFlagNull), + ] + ); + } + + #[test] + fn key_chord_events_track_modifier_releases_for_cancellation() { + let owner = input_owner(); + let events = key_chord_events( + &[BgModifier::Command, BgModifier::Shift, BgModifier::Option], + 35, + "cancellation", + ) + .unwrap(); + for authenticated in [false, true] { + let mut held = vec![]; + for event in &events[..3] { + record_keyboard_input(&mut held, owner, event, authenticated); + } + // If Stop arrives before the main key, cleanup must already own + // all modifier releases, using the exact delivery backend. + assert_eq!( + held, + vec![ + (owner, HeldInput::Key(55, authenticated)), + (owner, HeldInput::Key(56, authenticated)), + (owner, HeldInput::Key(58, authenticated)), + ] + ); + for event in &events[3..6] { + record_keyboard_input(&mut held, owner, event, authenticated); + } + assert_eq!( + held, + vec![ + (owner, HeldInput::Key(55, authenticated)), + (owner, HeldInput::Key(56, authenticated)), + ] + ); + for event in &events[6..] { + record_keyboard_input(&mut held, owner, event, authenticated); + } + assert!(held.is_empty()); + } + } + /// Runs against scripts/fixtures/computer-use-directed-input.m only. #[test] fn addressed_mouse_source_preserves_button_and_requested_modifiers() { diff --git a/src/apps/desktop/src/computer_use/macos_capture.m b/src/apps/desktop/src/computer_use/macos_capture.m index b9cccbfa12..2d68c20a3c 100644 --- a/src/apps/desktop/src/computer_use/macos_capture.m +++ b/src/apps/desktop/src/computer_use/macos_capture.m @@ -6,6 +6,7 @@ #import #import #include +#include #include #include extern void obf_control_native_stopped(uint64_t generation, uint64_t revision, int32_t pid, uint32_t windowID, const char *reason); @@ -35,6 +36,148 @@ uint64_t obf_capture_activation_epoch(void) { return atomic_load_explicit(&obfActivationEpoch, memory_order_acquire); } +// WindowServer can remove a target's internal focus without changing the +// human's foreground application. Observe that target's events passively; this +// tap never suppresses, edits, reroutes or posts an event. Native event subtype +// field 83 distinguishes AppKit activation and process focus notifications. +bool obf_focus_event_is_loss(uint32_t type, int64_t subtype) { + return (type == 13 && subtype == 2) || type == 19 || + (type == 21 && (subtype == 0x1000 || subtype == 0x4000)); +} + +@interface OBFFocusLossObserver : NSObject { +@public + _Atomic(uint64_t) epoch; + _Atomic(bool) ready; + CFMachPortRef tap; + CFRunLoopSourceRef source; +} +@property(nonatomic) int32_t pid; +@property(nonatomic) uint32_t windowID; +@property(nonatomic) uint64_t generation; +@property(nonatomic) uint32_t identityHigh; +@property(nonatomic) uint32_t identityLow; +- (void)install; +- (void)invalidate; +@end + +static CGEventRef obf_focus_loss_event(CGEventTapProxy proxy, CGEventType type, + CGEventRef event, void *context) { + (void)proxy; + OBFFocusLossObserver *observer = (__bridge OBFFocusLossObserver *)context; + if (type == kCGEventTapDisabledByTimeout || type == kCGEventTapDisabledByUserInput) { + // Missing events make the cached state unknown. Keep input available, + // but never reuse preparation against this observer again. + atomic_store_explicit(&observer->ready, false, memory_order_release); + } else if (event && obf_focus_event_is_loss(type, + CGEventGetIntegerValueField(event, (CGEventField)83))) { + atomic_fetch_add_explicit(&observer->epoch, 1, memory_order_release); + } + return event; +} + +@implementation OBFFocusLossObserver +- (instancetype)init { + if ((self = [super init])) { + atomic_init(&epoch, 1); + atomic_init(&ready, false); + } + return self; +} +- (void)install { + CGEventMask mask = CGEventMaskBit(13) | CGEventMaskBit(19) | CGEventMaskBit(21); + tap = CGEventTapCreateForPid(self.pid, kCGTailAppendEventTap, + kCGEventTapOptionListenOnly, mask, obf_focus_loss_event, (__bridge void *)self); + if (!tap) return; + source = CFMachPortCreateRunLoopSource(kCFAllocatorDefault, tap, 0); + if (!source) { + CFMachPortInvalidate(tap); + CFRelease(tap); + tap = NULL; + return; + } + // CFRunLoop source registration is thread-safe. It does not synchronously + // dispatch to AppKit, so callers may hold Rust FOCUS or CAPTURE locks. + atomic_store_explicit(&ready, CGEventTapIsEnabled(tap), memory_order_release); + CFRunLoopAddSource(CFRunLoopGetMain(), source, kCFRunLoopCommonModes); + CFRunLoopWakeUp(CFRunLoopGetMain()); +} +- (void)invalidate { + // The callback and final removal run on the main runloop. The queued block + // retains this observer until no callback can use its unretained context. + if (source) { + CFRunLoopRemoveSource(CFRunLoopGetMain(), source, kCFRunLoopCommonModes); + CFRelease(source); + source = NULL; + } + if (tap) { + CFMachPortInvalidate(tap); + CFRelease(tap); + tap = NULL; + } +} +@end + +static NSLock *obf_focus_loss_lock(void) { + static NSLock *lock; + static dispatch_once_t once; + dispatch_once(&once, ^{ lock = [[NSLock alloc] init]; }); + return lock; +} +static OBFFocusLossObserver *obfFocusLossObserver; + +static void obf_focus_loss_dispose(OBFFocusLossObserver *observer) { + if (!observer) return; + atomic_store_explicit(&observer->ready, false, memory_order_release); + dispatch_async(dispatch_get_main_queue(), ^{ [observer invalidate]; }); +} + +uint64_t obf_focus_loss_epoch(int32_t pid, uint32_t windowID, uint64_t generation, + uint32_t identityHigh, uint32_t identityLow, bool start) { + NSLock *lock = obf_focus_loss_lock(); + [lock lock]; + OBFFocusLossObserver *observer = obfFocusLossObserver; + BOOL matches = observer && observer.pid == pid && observer.windowID == windowID && + observer.generation == generation && observer.identityHigh == identityHigh && + observer.identityLow == identityLow; + OBFFocusLossObserver *old = nil; + if (!matches && start) { + old = observer; + observer = [[OBFFocusLossObserver alloc] init]; + observer.pid = pid; + observer.windowID = windowID; + observer.generation = generation; + observer.identityHigh = identityHigh; + observer.identityLow = identityLow; + obfFocusLossObserver = observer; + [observer install]; + matches = YES; + } + uint64_t value = matches && atomic_load_explicit(&observer->ready, memory_order_acquire) + ? atomic_load_explicit(&observer->epoch, memory_order_acquire) : 0; + [lock unlock]; + obf_focus_loss_dispose(old); + return value; +} + +static void obf_focus_loss_stop_matching(int32_t pid, uint32_t windowID, uint64_t generation, + bool checkIdentity, uint32_t identityHigh, uint32_t identityLow) { + NSLock *lock = obf_focus_loss_lock(); + [lock lock]; + OBFFocusLossObserver *observer = obfFocusLossObserver; + BOOL matches = observer && observer.pid == pid && observer.windowID == windowID && + observer.generation == generation && (!checkIdentity || + (observer.identityHigh == identityHigh && observer.identityLow == identityLow)); + if (matches) obfFocusLossObserver = nil; + [lock unlock]; + if (matches) obf_focus_loss_dispose(observer); +} + +void obf_focus_loss_stop(int32_t pid, uint32_t windowID, uint64_t generation, + uint32_t identityHigh, uint32_t identityLow) { + obf_focus_loss_stop_matching(pid, windowID, generation, true, identityHigh, identityLow); +} + @interface OBFWindowCapture : NSObject @property(nonatomic, strong) SCStream *stream; @property(nonatomic, strong) SCStreamConfiguration *configuration; @@ -75,6 +218,7 @@ - (void)fail:(NSString *)reason { self.pixels = nil; [self.condition broadcast]; [self.condition unlock]; + obf_focus_loss_stop_matching(self.pid, self.windowID, self.generation, false, 0, 0); } - (void)stream:(SCStream *)stream didStopWithError:(NSError *)error { [self fail:[NSString stringWithFormat:@"CAPTURE_STOPPED: %@", error.localizedDescription]]; diff --git a/src/apps/desktop/src/computer_use/macos_input_focus.rs b/src/apps/desktop/src/computer_use/macos_input_focus.rs index 4e04a7fe0c..91e35530b3 100644 --- a/src/apps/desktop/src/computer_use/macos_input_focus.rs +++ b/src/apps/desktop/src/computer_use/macos_input_focus.rs @@ -12,6 +12,26 @@ use openbitfun_core::util::errors::{OpenBitFunError, OpenBitFunResult}; use std::sync::Mutex; use std::time::Duration; +unsafe extern "C" { + fn obf_focus_loss_epoch( + pid: i32, + window: u32, + generation: u64, + identity_high: u32, + identity_low: u32, + start: bool, + ) -> u64; + fn obf_focus_loss_stop( + pid: i32, + window: u32, + generation: u64, + identity_high: u32, + identity_low: u32, + ); + #[cfg(test)] + fn obf_focus_event_is_loss(event_type: u32, subtype: i64) -> bool; +} + #[derive(Clone, Copy)] struct Focus { pid: i32, @@ -20,16 +40,45 @@ struct Focus { generation: u64, foreground: i32, activation_epoch: u64, + loss_epoch: u64, } impl Focus { - fn reusable(self, next: Self) -> bool { + fn same_target(self, next: Self) -> bool { self.pid == next.pid && self.identity == next.identity && self.window == next.window && self.generation == next.generation + } + fn reusable(self, next: Self) -> bool { + self.same_target(next) && self.foreground == next.foreground && next.activation_epoch != 0 && self.activation_epoch == next.activation_epoch + && next.loss_epoch != 0 + && self.loss_epoch == next.loss_epoch + } + fn observed_loss_epoch(self, start: bool) -> u64 { + unsafe { + obf_focus_loss_epoch( + self.pid, + self.window, + self.generation, + self.identity.0[0], + self.identity.0[1], + start, + ) + } + } + fn stop_observer(self) { + unsafe { + obf_focus_loss_stop( + self.pid, + self.window, + self.generation, + self.identity.0[0], + self.identity.0[1], + ) + } } } static FOCUS: Mutex> = Mutex::new(None); @@ -47,6 +96,7 @@ fn cleanup(focus: Focus) { pub(super) fn stop() { if let Ok(mut state) = FOCUS.lock() { if let Some(focus) = state.take() { + focus.stop_observer(); cleanup(focus); } } @@ -82,20 +132,19 @@ pub(super) fn prepare(pid: i32) -> OpenBitFunResult<()> { let mut state = FOCUS .lock() .map_err(|_| OpenBitFunError::tool("Background input focus lock poisoned"))?; - let focus = Focus { + let mut focus = Focus { pid, identity, window, generation, foreground, activation_epoch, + loss_epoch: 0, }; + focus.loss_epoch = focus.observed_loss_epoch(false); if state.is_some_and(|previous| previous.reusable(focus)) { return Ok(()); } - if let Some(old) = state.take() { - cleanup(old); - } if ![x, y, width, height].iter().all(|v| v.is_finite()) || width <= 0.0 || height <= 0.0 { return Err(OpenBitFunError::tool( "[CONTROL_TARGET_CHANGED] Invalid window geometry for background focus", @@ -122,14 +171,26 @@ pub(super) fn prepare(pid: i32) -> OpenBitFunResult<()> { pair.push(event); } check(pid, window)?; + if let Some(old) = state.take() { + if !old.same_target(focus) { + old.stop_observer(); + cleanup(old); + } + } + // Observe only this process, without suppressing or changing its events. + // An unavailable/disabled tap returns zero, which disables cache reuse but + // does not prevent normal directed input. No main-thread wait is involved. + focus.loss_epoch = focus.observed_loss_epoch(true); // Register cleanup before attempting any native state mutation. *state = Some(focus); if !super::macos_skylight::set_local_activation(identity, window, true) { + focus.stop_observer(); state.take(); return Err(OpenBitFunError::tool("[BACKGROUND_INPUT_UNAVAILABLE] Target-local input activation failed; no content input was sent")); } std::thread::sleep(Duration::from_millis(20)); if let Err(error) = check(pid, window) { + focus.stop_observer(); cleanup(focus); state.take(); return Err(error); @@ -144,6 +205,7 @@ pub(super) fn prepare(pid: i32) -> OpenBitFunResult<()> { // missing AX attribute into a blanket rejection of visual-only controls. std::thread::sleep(Duration::from_millis(10)); if let Err(error) = check(pid, window) { + focus.stop_observer(); cleanup(focus); state.take(); return Err(error); @@ -165,6 +227,7 @@ mod tests { generation: 1, foreground: 33, activation_epoch: 4, + loss_epoch: 1, }; assert!(original.reusable(original)); assert!(!original.reusable(Focus { @@ -193,7 +256,43 @@ mod tests { generation: 1, foreground: 33, activation_epoch: 0, + loss_epoch: 1, }; assert!(!pending.reusable(pending)); } + + #[test] + fn target_loss_invalidates_focus_without_a_foreground_switch() { + let prepared = Focus { + pid: 11, + identity: super::super::macos_skylight::ProcessIdentity([0, 11]), + window: 22, + generation: 1, + foreground: 33, + activation_epoch: 4, + loss_epoch: 1, + }; + assert!(prepared.reusable(prepared)); + assert!(!prepared.reusable(Focus { + loss_epoch: 2, + ..prepared + })); + assert!(!prepared.reusable(Focus { + loss_epoch: 0, + ..prepared + })); + } + + #[test] + fn only_native_focus_loss_events_invalidate_preparation() { + // AppKit app deactivation, process deactivation and key-focus loss. + for (kind, subtype) in [(13, 2), (19, 0), (21, 0x1000), (21, 0x4000)] { + assert!(unsafe { super::obf_focus_event_is_loss(kind, subtype) }); + } + // Our own activation, focus-return notifications and content input are + // not losses and therefore cannot force preparation on every action. + for (kind, subtype) in [(13, 1), (20, 0), (21, 0x8000), (1, 0), (10, 0)] { + assert!(!unsafe { super::obf_focus_event_is_loss(kind, subtype) }); + } + } } diff --git a/src/apps/desktop/tests/computer_use_native_roundtrip.rs b/src/apps/desktop/tests/computer_use_native_roundtrip.rs index 7395706a04..a187072831 100644 --- a/src/apps/desktop/tests/computer_use_native_roundtrip.rs +++ b/src/apps/desktop/tests/computer_use_native_roundtrip.rs @@ -6,9 +6,10 @@ fn main() { use std::time::{Duration, Instant}; if std::env::var_os("OPENBITFUN_ROUNDTRIP_FIXTURE_PID").is_none() && std::env::var_os("OPENBITFUN_INPUT_FIXTURE_PID").is_none() + && std::env::var_os("OPENBITFUN_CUA_PROBE_PID").is_none() { eprintln!( - "Run node scripts/test-macos-control-roundtrip.mjs to launch the isolated fixture" + "Run node scripts/test-macos-control-roundtrip.mjs, or set OPENBITFUN_CUA_PROBE_PID for the scoped JSONL probe" ); std::process::exit(2); } @@ -38,7 +39,17 @@ fn main() { }); let _ = sender.send(result.is_ok()); }); - let deadline = Instant::now() + Duration::from_secs(45); + let limit = if std::env::var_os("OPENBITFUN_CUA_PROBE_PID").is_some() { + std::env::var("OPENBITFUN_CUA_PROBE_TIMEOUT_SECS") + .ok() + .and_then(|value| value.parse::().ok()) + .filter(|seconds| (1..=1800).contains(seconds)) + .unwrap_or(600) + + 15 // Leave the worker time to revoke control and clean up. + } else { + 45 + }; + let deadline = Instant::now() + Duration::from_secs(limit); loop { match receiver.try_recv() { Ok(true) => return, @@ -46,7 +57,7 @@ fn main() { Err(mpsc::TryRecvError::Empty) => {} } if Instant::now() >= deadline { - eprintln!("FAIL native tool roundtrip exceeded 45 seconds while the real main run loop was active"); + eprintln!("FAIL native tool driver exceeded {limit} seconds while the real main run loop was active"); std::process::exit(1); } unsafe {