Summary
Add clipboard and cross-app workflow tools.
Motivation
The existing shell tool provides general command execution, but there is no dedicated capability for cross-app workflows. Office productivity often requires: reading/writing the system clipboard, automating desktop applications via AppleScript (macOS) or PowerShell (Windows), and browser automation for web-based tools. Currently the agent must rely on raw shell commands, which are platform-specific and fragile.
Proposed Solution
Create cross-app workflow tools:
- Clipboard: Read and write to the system clipboard (pbcopy/pbpaste on macOS, clip on Windows, xclip/xsel on Linux)
- Desktop Automation: Execute AppleScript (macOS), PowerShell (Windows), or Automator scripts for application control
- Browser Automation: Launch browsers, navigate pages, fill forms, extract data, take screenshots via Puppeteer or Playwright
- Application Control: Open/close applications, switch windows, interact with native UI elements
Each tool should follow the existing tool pattern in src/tools/ — a zod schema, an impl function, and registration in index.js with appropriate permissions (filesystem:exec, process:spawn).
Alternatives Considered
- Shell-based commands directly: platform-specific, no structured error handling, no browser automation capability.
- Rely on the shell tool: works but lacks the abstraction and safety guarantees of a dedicated tool.
OpenSpec Note
This project uses OpenSpec for feature development. If this request is approved, I will:
- Run /opsx:propose to generate a full proposal with specs and tasks
- Iterate on the design before any code is written
- Follow the task-driven implementation workflow
Additional Context
This should integrate with the URL management gap (#785) — e.g., browser automation for link validation or form filling. The existing shell tool provides the foundation; these tools would add structured, platform-aware abstractions.
Dependencies
- Clipboard: Native via shell commands (pbcopy/pbpaste, clip, xclip/xsel) — no npm dependency required. Cross-platform detection via os.platform().
- Browser automation: playwright (v1.40+) — supports Chromium, Firefox, WebKit. Includes bundled browser binaries. System dependencies: libatk-bridge2.0-0, libdrm2, libxkbcommon0, etc. (handled by playwright install).
- Desktop automation: Platform-specific shell commands (osascript for AppleScript, powershell.exe for Windows). No npm dependency.
- Application control: Platform-specific via shell or Playwright's process management APIs.
Testing Strategy
- Unit tests: Mock platform detection, clipboard command execution, and Playwright browser launch. Verify Zod schema validation for all inputs.
- Browser automation: Use Playwright's --headless mode in CI. Test navigation, form filling, screenshot capture against a local test server (or a public test page like example.com).
- Clipboard: Test read/write round-trip on the current platform. Skip on CI if no display server available (use process.env.DISPLAY check).
- Edge cases: Empty clipboard, binary data in clipboard, browser navigation timeouts, form fields not found, application not installed.
Security Considerations
- Browser automation: Playwright runs in an isolated browser context. Disable --enable-automation banner. Do not allow arbitrary JS execution in the target page context without explicit user consent.
- Desktop automation: AppleScript and PowerShell can execute arbitrary commands. Validate script content against a safe subset (no do shell script, no file deletion, no network commands) OR require explicit user confirmation before execution.
- Process spawning: All shell commands for clipboard/desktop automation must use child_process.spawn with shell: true and input sanitization. Never pass unsanitized user input to shell commands.
- Screenshot output: Save to a designated directory with filesystem:write permission. Validate file paths against allowlist per AGENTS.md 1.2.
- Headless mode: Default to headless browser operation. Headful mode requires a display server and should be opt-in.
Implementation Notes
- Split into two tools: clipboard (simple, cross-platform, no system dependencies) and browser (Playwright-based, platform-specific, heavier). Desktop automation and application control can be added as actions within the browser tool or as a separate desktop tool later.
- Clipboard tool: Simple wrapper around platform-specific commands. Zod schema: { action: "read" | "write", content?: string }.
- Browser tool: Zod schema: { action: "navigate" | "screenshot" | "fill" | "extract" | "click", url?: string, selector?: string, value?: string, outputPath?: string }.
- Timeouts: All browser operations should have configurable timeouts (default: 30s for navigation, 10s for interactions).
- State management: Browser sessions should be tracked (open pages, cookies) and reusable across tool calls within a single agent turn.
- Application control: Defer to a follow-up PR. Start with clipboard and browser automation as the MVP.
- CI considerations: Playwright requires system dependencies. Add npx playwright install --with-deps to CI setup. Skip clipboard tests on headless CI.
Summary
Add clipboard and cross-app workflow tools.
Motivation
The existing shell tool provides general command execution, but there is no dedicated capability for cross-app workflows. Office productivity often requires: reading/writing the system clipboard, automating desktop applications via AppleScript (macOS) or PowerShell (Windows), and browser automation for web-based tools. Currently the agent must rely on raw shell commands, which are platform-specific and fragile.
Proposed Solution
Create cross-app workflow tools:
Each tool should follow the existing tool pattern in src/tools/ — a zod schema, an impl function, and registration in index.js with appropriate permissions (filesystem:exec, process:spawn).
Alternatives Considered
OpenSpec Note
This project uses OpenSpec for feature development. If this request is approved, I will:
Additional Context
This should integrate with the URL management gap (#785) — e.g., browser automation for link validation or form filling. The existing shell tool provides the foundation; these tools would add structured, platform-aware abstractions.
Dependencies
Testing Strategy
Security Considerations
Implementation Notes