Skip to content

feat: add clipboard and cross-app workflow tools #787

Description

@avoidwork

Summary

Add clipboard and cross-app workflow tools.

Motivation

The existing shell tool provides general command execution, but there is no dedicated capability for cross-app workflows. Office productivity often requires: reading/writing the system clipboard, automating desktop applications via AppleScript (macOS) or PowerShell (Windows), and browser automation for web-based tools. Currently the agent must rely on raw shell commands, which are platform-specific and fragile.

Proposed Solution

Create cross-app workflow tools:

  • Clipboard: Read and write to the system clipboard (pbcopy/pbpaste on macOS, clip on Windows, xclip/xsel on Linux)
  • Desktop Automation: Execute AppleScript (macOS), PowerShell (Windows), or Automator scripts for application control
  • Browser Automation: Launch browsers, navigate pages, fill forms, extract data, take screenshots via Puppeteer or Playwright
  • Application Control: Open/close applications, switch windows, interact with native UI elements

Each tool should follow the existing tool pattern in src/tools/ — a zod schema, an impl function, and registration in index.js with appropriate permissions (filesystem:exec, process:spawn).

Alternatives Considered

  • Shell-based commands directly: platform-specific, no structured error handling, no browser automation capability.
  • Rely on the shell tool: works but lacks the abstraction and safety guarantees of a dedicated tool.

OpenSpec Note

This project uses OpenSpec for feature development. If this request is approved, I will:

  1. Run /opsx:propose to generate a full proposal with specs and tasks
  2. Iterate on the design before any code is written
  3. Follow the task-driven implementation workflow

Additional Context

This should integrate with the URL management gap (#785) — e.g., browser automation for link validation or form filling. The existing shell tool provides the foundation; these tools would add structured, platform-aware abstractions.

Dependencies

  • Clipboard: Native via shell commands (pbcopy/pbpaste, clip, xclip/xsel) — no npm dependency required. Cross-platform detection via os.platform().
  • Browser automation: playwright (v1.40+) — supports Chromium, Firefox, WebKit. Includes bundled browser binaries. System dependencies: libatk-bridge2.0-0, libdrm2, libxkbcommon0, etc. (handled by playwright install).
  • Desktop automation: Platform-specific shell commands (osascript for AppleScript, powershell.exe for Windows). No npm dependency.
  • Application control: Platform-specific via shell or Playwright's process management APIs.

Testing Strategy

  • Unit tests: Mock platform detection, clipboard command execution, and Playwright browser launch. Verify Zod schema validation for all inputs.
  • Browser automation: Use Playwright's --headless mode in CI. Test navigation, form filling, screenshot capture against a local test server (or a public test page like example.com).
  • Clipboard: Test read/write round-trip on the current platform. Skip on CI if no display server available (use process.env.DISPLAY check).
  • Edge cases: Empty clipboard, binary data in clipboard, browser navigation timeouts, form fields not found, application not installed.

Security Considerations

  • Browser automation: Playwright runs in an isolated browser context. Disable --enable-automation banner. Do not allow arbitrary JS execution in the target page context without explicit user consent.
  • Desktop automation: AppleScript and PowerShell can execute arbitrary commands. Validate script content against a safe subset (no do shell script, no file deletion, no network commands) OR require explicit user confirmation before execution.
  • Process spawning: All shell commands for clipboard/desktop automation must use child_process.spawn with shell: true and input sanitization. Never pass unsanitized user input to shell commands.
  • Screenshot output: Save to a designated directory with filesystem:write permission. Validate file paths against allowlist per AGENTS.md 1.2.
  • Headless mode: Default to headless browser operation. Headful mode requires a display server and should be opt-in.

Implementation Notes

  • Split into two tools: clipboard (simple, cross-platform, no system dependencies) and browser (Playwright-based, platform-specific, heavier). Desktop automation and application control can be added as actions within the browser tool or as a separate desktop tool later.
  • Clipboard tool: Simple wrapper around platform-specific commands. Zod schema: { action: "read" | "write", content?: string }.
  • Browser tool: Zod schema: { action: "navigate" | "screenshot" | "fill" | "extract" | "click", url?: string, selector?: string, value?: string, outputPath?: string }.
  • Timeouts: All browser operations should have configurable timeouts (default: 30s for navigation, 10s for interactions).
  • State management: Browser sessions should be tracked (open pages, cookies) and reusable across tool calls within a single agent turn.
  • Application control: Defer to a follow-up PR. Start with clipboard and browser automation as the MVP.
  • CI considerations: Playwright requires system dependencies. Add npx playwright install --with-deps to CI setup. Skip clipboard tests on headless CI.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions