diff --git a/.gitignore b/.gitignore index af1c91e..994c727 100644 --- a/.gitignore +++ b/.gitignore @@ -35,3 +35,8 @@ __pycache__/ /npm/bin/computer-use-linux-darwin-* /npm/bin/computer-use-linux-win32-* /npm/bin/computer-use-linux-cosmic +/.agent-shell/ + + +# Agent docs +.agent-docs diff --git a/README.md b/README.md index 139a98a..d0a3ff9 100644 --- a/README.md +++ b/README.md @@ -36,11 +36,13 @@ The crate was extracted from [`codex-desktop-linux`](https://github.com/avifenes MCP tools exposed by the server: **Diagnostics** + - `doctor` — single-shot JSON readiness report (platform, portals, accessibility, windowing, input, readiness summary, and a capability map of available backends) - `setup_accessibility` — enables GNOME's `org.gnome.desktop.interface toolkit-accessibility` setting so toolkit apps expose AT-SPI trees - `setup_window_targeting` — installs and enables the bundled GNOME Shell extension when `org.gnome.Shell.Introspect` is locked down **Discovery** + - `list_apps` — running desktop apps visible to the AT-SPI registry - `list_windows` — compositor windows with title, app id, wm_class, focus state, client type (Wayland/X11), and bounds - `focused_window` — the window currently holding keyboard focus @@ -50,6 +52,7 @@ MCP tools exposed by the server: Screenshot payloads are size-bounded by default before they are returned to the MCP host: max 1920 px width/height and 2 MiB image bytes, with hard caps even when callers request more. Agents that need more detail can pass `max_width`, `max_height`, `max_bytes`, `scale`, `format: "jpeg"`, or `quality`, preferably with a window target or crop. PNG remains the default; JPEG lets callers trade lossless pixels for a smaller payload before the byte cap forces further resizing. Returned screenshot metadata includes `coordinate_width`, `coordinate_height`, `scale`, `format`, and `quality` so callers can convert from a downscaled preview to desktop coordinate pixels. **Input** + - `click` — by element index, semantic selector, or desktop coordinate pixels - `drag` — desktop coordinate drag (start / end) - `scroll` — page-based scroll on an element or at a pixel location @@ -59,10 +62,12 @@ Screenshot payloads are size-bounded by default before they are returned to the Targeted `press_key`/`type_text` results append focused-element feedback from AT-SPI (role, name, editable) and warn when no editable element holds focus. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. `get_app_state` returns a compact readiness block by default; pass `verbose: true` for the full diagnostics report. **Semantic actions** + - `perform_action` — invoke any AT-SPI action exposed by an element (`Press`, `Activate`, `Toggle`, …); defaults to the primary action - `set_value` — write to a settable accessibility element (text fields, sliders, spinners) **Navigation** + - `activate_window` — focus a window by `window_id`, `pid`, `app_id`, `wm_class`, `title`, or terminal selectors - `move_window` / `resize_window` — reposition or resize a window in desktop coordinates (GNOME Shell extension backend); useful to recover windows that are partially off-screen @@ -223,6 +228,24 @@ Edit `~/.config/Claude/claude_desktop_config.json`: Restart Claude Desktop. The tools should appear in the tools list. +### Pi Coding Agent + +```bash +pi install npm:pi-mcp-adapter +pi install npm:@agent-sh/computer-use-linux +``` + +Restart pi or run `/reload`. The MCP proxy tool `mcp()` will have the desktop tools available: + +``` +mcp({ server: "computer-use-linux" }) # list all tools +mcp({ search: "windows" }) # search for window tools +mcp({ tool: "computer_use_linux_doctor" }) # run readiness check +mcp({ tool: "computer_use_linux_list_windows" }) # list desktop windows +``` + +The extension auto-registers the computer-use-linux MCP server into pi-mcp-adapter's config. If the binary is not found, check the [Pi setup guide](skills/computer-use-linux/references/pi-setup.md). + ### Hermes Agent Install the companion Hermes skill so Hermes has the desktop-specific runbook: diff --git a/package.json b/package.json index 700d3f1..de3a48b 100644 --- a/package.json +++ b/package.json @@ -19,7 +19,8 @@ "wayland", "accessibility", "hermes-agent", - "codex" + "codex", + "pi-package" ], "os": [ "linux" @@ -35,10 +36,16 @@ "LICENSE", "README.md", "skills/computer-use-linux/SKILL.md", + "skills/computer-use-linux/references/", "npm/README.md", "npm/bin/computer-use-linux.js", - "npm/install.js" + "npm/install.js", + "pi/" ], + "pi": { + "extensions": ["./pi/extension/index.ts"], + "skills": ["./skills/computer-use-linux/SKILL.md"] + }, "scripts": { "postinstall": "node npm/install.js", "pack:check": "npm pack --dry-run", diff --git a/pi/extension/index.ts b/pi/extension/index.ts new file mode 100644 index 0000000..1597157 --- /dev/null +++ b/pi/extension/index.ts @@ -0,0 +1,262 @@ +/** + * Pi coding agent extension for computer-use-linux. + * + * Discovers the computer-use-linux binary and registers it as an MCP server + * for pi-mcp-adapter by writing to ~/.pi/agent/mcp.json. + * + * Prerequisite: pi-mcp-adapter (npm:pi-mcp-adapter) must be installed. + */ + +import type { ExtensionAPI } from "@earendil-works/pi-coding-agent"; +import { + accessSync, + constants, + existsSync, + mkdirSync, + readFileSync, + writeFileSync, +} from "node:fs"; +import { homedir } from "node:os"; +import { dirname, join } from "node:path"; +import { env } from "node:process"; + +// --------------------------------------------------------------------------- +// Constants +// --------------------------------------------------------------------------- + +const PACKAGE_NAME = "@agent-sh/computer-use-linux"; +const MCP_SERVER_NAME = "computer-use-linux"; + +/** + * Path to pi's agent-level MCP config. pi-mcp-adapter reads from this file + * when it starts up, alongside ~/.config/mcp/mcp.json and .mcp.json. + */ +function getPiAgentMcpConfigPath(): string { + const configured = env.PI_CODING_AGENT_DIR?.trim(); + if (configured) { + return join(configured, "mcp.json"); + } + return join(homedir(), ".pi", "agent", "mcp.json"); +} + +// --------------------------------------------------------------------------- +// Binary discovery +// --------------------------------------------------------------------------- + +/** + * Find the computer-use-linux binary. Cached after first successful lookup. + */ +let _binaryPath: string | null | undefined; + +function findBinary(): string | null { + if (_binaryPath !== undefined) return _binaryPath; + + // 1) Environment variable override + const fromEnv = env.COMPUTER_USE_LINUX_BIN; + if (fromEnv && existsSync(fromEnv)) { + _binaryPath = fromEnv; + return fromEnv; + } + + // 2) Bundled by the npm wrapper's postinstall + // The npm wrapper downloads binaries to npm/bin/ next to the JS wrapper. + // We derive the path relative to our own installed location. + const candidates = [ + // Bundled binary for this platform + join( + __dirname, + "..", + "..", + "npm", + "bin", + `computer-use-linux-linux-${process.arch}`, + ), + // Direct npm global install + join(__dirname, "..", "..", "npm", "bin", "computer-use-linux"), + ]; + for (const candidate of candidates) { + const resolved = join(candidate); + if (existsSync(resolved)) { + _binaryPath = resolved; + return resolved; + } + } + + // 3) PATH + const pathDirs = (env.PATH || "").split(":"); + for (const dir of pathDirs) { + const candidate = join(dir, "computer-use-linux"); + try { + if (existsSync(candidate)) { + accessSync(candidate, constants.X_OK); + _binaryPath = candidate; + return candidate; + } + } catch { + // Skip inaccessible or non-executable entries + } + } + + _binaryPath = null; + return null; +} + +// --------------------------------------------------------------------------- +// MCP config management +// --------------------------------------------------------------------------- + +interface McpServerEntry { + command: string; + args: string[]; + lifecycle?: string; +} + +interface McpConfig { + mcpServers?: Record; +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function readMcpConfig(path: string): McpConfig | null { + // Only return empty if the file genuinely doesn't exist. + // If the file exists but can't be read or parsed, return null + // so the caller knows not to overwrite it. + if (!existsSync(path)) return {}; + + try { + const raw = readFileSync(path, "utf-8"); + const parsed = JSON.parse(raw); + if (!isRecord(parsed)) { + console.error( + `${PACKAGE_NAME}: MCP config at ${path} contains a ${typeof parsed}` + + (Array.isArray(parsed) ? " (array)" : "") + + ", expected a JSON object. Skipping.", + ); + return null; + } + return parsed as McpConfig; + } catch (error) { + console.error( + `${PACKAGE_NAME}: failed to read existing MCP config at ${path}:`, + error instanceof Error ? error.message : String(error), + ); + return null; + } +} + +function writeMcpConfig(path: string, config: McpConfig): void { + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, JSON.stringify(config, null, 2) + "\n", "utf-8"); +} + +/** + * Result of ensuring the computer-use-linux entry in the MCP config. + * - "updated": entry was written or updated successfully + * - "unchanged": entry already matches, no write needed + * - "failed": config could not be read or written + */ +type EnsureResult = "updated" | "unchanged" | "failed"; + +function ensureServerEntry( + configPath: string, + binaryPath: string, +): EnsureResult { + const config = readMcpConfig(configPath); + + // If config exists but can't be read, don't overwrite it + if (config === null) return "failed"; + + // Validate that mcpServers is a plain object before mutating + const servers = + config.mcpServers != null && isRecord(config.mcpServers) + ? (config.mcpServers as Record) + : {}; + + const existing = servers[MCP_SERVER_NAME]; + + // Check if entry already exists and matches + if ( + existing && + existing.command === binaryPath && + existing.args?.length === 1 && + existing.args[0] === "mcp" + ) { + return "unchanged"; // No change needed + } + + // Add or update the entry + servers[MCP_SERVER_NAME] = { + command: binaryPath, + args: ["mcp"], + }; + + try { + writeMcpConfig(configPath, { ...config, mcpServers: servers }); + return "updated"; // Config was updated + } catch (error) { + console.error( + `${PACKAGE_NAME}: failed to write MCP config to ${configPath}:`, + error instanceof Error ? error.message : String(error), + ); + return "failed"; // Write failed + } +} + +// --------------------------------------------------------------------------- +// Extension entry point +// --------------------------------------------------------------------------- + +export default function (pi: ExtensionAPI) { + // Resolve binary path eagerly (before session_start) so it's cached. + const binaryPath = findBinary(); + + pi.on("session_start", async (_event, ctx) => { + try { + if (!binaryPath) { + ctx.ui.notify?.( + `${PACKAGE_NAME}: computer-use-linux binary not found. ` + + "Install it with 'npm install -g @agent-sh/computer-use-linux' " + + "or set COMPUTER_USE_LINUX_BIN.", + "warning", + ); + return; + } + + // Write/update the MCP config that pi-mcp-adapter reads + const configPath = getPiAgentMcpConfigPath(); + const result = ensureServerEntry(configPath, binaryPath); + + if (result === "updated" && ctx.hasUI) { + ctx.ui.notify?.( + `${PACKAGE_NAME}: MCP server configured at ${configPath}. ` + + "Run /reload if pi-mcp-adapter is already installed.", + "info", + ); + } + + if (result === "failed" && ctx.hasUI) { + ctx.ui.notify?.( + `${PACKAGE_NAME}: failed to configure MCP server at ${configPath}. ` + + "Check the console logs and ensure the file is writable.", + "error", + ); + } + + // Check that pi-mcp-adapter is available (its tools are registered) + if (!pi.getAllTools().some((t) => t.name === "mcp")) { + ctx.ui.notify?.( + `${PACKAGE_NAME}: pi-mcp-adapter not detected. ` + + "Install it with 'pi install npm:pi-mcp-adapter' then /reload.", + "warning", + ); + } + } catch (error) { + console.error( + `${PACKAGE_NAME}: unexpected error in session_start handler:`, + error instanceof Error ? error.message : String(error), + ); + } + }); +} diff --git a/skills/computer-use-linux/SKILL.md b/skills/computer-use-linux/SKILL.md index 573ecb2..5ab4a30 100644 --- a/skills/computer-use-linux/SKILL.md +++ b/skills/computer-use-linux/SKILL.md @@ -1,6 +1,6 @@ --- name: computer-use-linux -description: "Use when Hermes needs Linux desktop observation or control through the computer-use-linux MCP server." +description: "Linux desktop observation and control via the computer-use-linux MCP server: accessibility trees, screenshots, window targeting, and input synthesis (click, type, scroll). Works with any MCP host." author: agent-sh license: MIT platforms: [linux] @@ -8,28 +8,29 @@ platforms: [linux] # computer-use-linux -Use `computer-use-linux` when Hermes needs to observe or operate a local Linux desktop through MCP: inspect the accessibility tree, list/focus windows, take screenshots, click, scroll, type, press keys, or invoke AT-SPI actions. +Use `computer-use-linux` when an agent needs to observe or operate a local Linux desktop through MCP: inspect the accessibility tree, list/focus windows, take screenshots, click, scroll, type, press keys, or invoke AT-SPI actions. ## When to Use Use this skill when: -- The user wants Hermes to control a Linux GUI app. + +- The user wants the agent to control a Linux GUI app. - You need desktop state from AT-SPI, screenshots, or compositor window metadata. -- You are configuring the `computer-use-linux` MCP server for Hermes. +- You are configuring the `computer-use-linux` MCP server for your agent. - A desktop action needs target-aware input instead of blind shell commands. Do not use this for remote browsers, websites, or headless automation when a browser-specific tool is available. Do not assume desktop actions are safe just because the MCP connection works. ## Install -Preferred install for Hermes users: +Preferred install: ```bash npm install -g @agent-sh/computer-use-linux computer-use-linux doctor | jq .readiness ``` -Rust users can install the same server from crates.io: +Rust users can install from crates.io: ```bash cargo install computer-use-linux @@ -47,42 +48,23 @@ computer-use-linux doctor | jq .readiness On GNOME Wayland, log out and back in after `setup-window-targeting` if the GNOME Shell extension was newly installed. -## Configure Hermes +## Configure Your Agent -Add the server with the Hermes MCP CLI: +The `computer-use-linux` binary is an MCP server. Configure it as a stdio MCP server in your agent of choice: -```bash -hermes mcp add computer-use-linux --command computer-use-linux --args mcp -hermes mcp test computer-use-linux -hermes mcp configure computer-use-linux +```json +{ + "command": "computer-use-linux", + "args": ["mcp"] +} ``` -`configure` opens Hermes' tool-selection UI for this MCP server. - -The generated config should look like this: - -```yaml -mcp_servers: - computer-use-linux: - command: computer-use-linux - args: ["mcp"] - timeout: 120 - connect_timeout: 30 -``` - -If the binary is not on `PATH`, pass the absolute path to `--command`. - -Hermes registers tools using the `mcp__` pattern. With this config, tool names are prefixed as `mcp_computer_use_linux_`, for example: +If the binary is not on `PATH`, use the absolute path (typically `~/.local/bin/computer-use-linux` or the npm global bin directory). -| MCP tool | Hermes tool name | -| --- | --- | -| `doctor` | `mcp_computer_use_linux_doctor` | -| `get_app_state` | `mcp_computer_use_linux_get_app_state` | -| `list_windows` | `mcp_computer_use_linux_list_windows` | -| `click` | `mcp_computer_use_linux_click` | -| `type_text` | `mcp_computer_use_linux_type_text` | +### Host-specific guides -Restart Hermes after changing MCP config. +- [Hermes setup](references/hermes-setup.md) +- [Pi coding agent setup](references/pi-setup.md) ## Procedure @@ -110,7 +92,6 @@ Run: ```bash computer-use-linux doctor | jq .readiness -hermes chat --toolsets mcp-computer-use-linux -q "List the current desktop windows." ``` Ready output should have: @@ -121,4 +102,4 @@ Ready output should have: - `can_send_development_input: true` - `blockers: []` -If Hermes does not expose the tools, check startup logs for MCP discovery errors and confirm the server name in `config.yaml` is exactly `computer-use-linux`. +Then test with your agent by calling the `doctor` tool or asking the agent to list desktop windows. diff --git a/skills/computer-use-linux/references/hermes-setup.md b/skills/computer-use-linux/references/hermes-setup.md new file mode 100644 index 0000000..83ad540 --- /dev/null +++ b/skills/computer-use-linux/references/hermes-setup.md @@ -0,0 +1,41 @@ +--- +name: hermes-setup +description: "Hermes agent setup for the computer-use-linux MCP server." +--- + +# Hermes Setup + +Add the server with the Hermes MCP CLI: + +```bash +hermes mcp add computer-use-linux --command computer-use-linux --args mcp +hermes mcp test computer-use-linux +hermes mcp configure computer-use-linux +``` + +`configure` opens Hermes' tool-selection UI for this MCP server. + +The generated config should look like this: + +```yaml +mcp_servers: + computer-use-linux: + command: computer-use-linux + args: ["mcp"] + timeout: 120 + connect_timeout: 30 +``` + +If the binary is not on `PATH`, pass the absolute path to `--command`. + +Hermes registers tools using the `mcp__` pattern. With this config, tool names are prefixed as `mcp_computer_use_linux_`, for example: + +| MCP tool | Hermes tool name | +| --- | --- | +| `doctor` | `mcp_computer_use_linux_doctor` | +| `get_app_state` | `mcp_computer_use_linux_get_app_state` | +| `list_windows` | `mcp_computer_use_linux_list_windows` | +| `click` | `mcp_computer_use_linux_click` | +| `type_text` | `mcp_computer_use_linux_type_text` | + +Restart Hermes after changing MCP config. diff --git a/skills/computer-use-linux/references/pi-setup.md b/skills/computer-use-linux/references/pi-setup.md new file mode 100644 index 0000000..e438112 --- /dev/null +++ b/skills/computer-use-linux/references/pi-setup.md @@ -0,0 +1,57 @@ +--- +name: pi-setup +description: "Pi coding agent setup for the computer-use-linux MCP server." +--- + +# Pi Setup + +## Prerequisites + +You need both of these installed: + +```bash +pi install npm:pi-mcp-adapter +pi install npm:@agent-sh/computer-use-linux +``` + +> **Note:** `pi-mcp-adapter` is a separate extension that provides MCP protocol support in pi. The `@agent-sh/computer-use-linux` package auto-registers the desktop MCP server into pi-mcp-adapter's config. + +After installing both, restart pi or run `/reload`. + +## Verify + +The `mcp()` proxy tool should now be available. Call it from any prompt: + +```bash +mcp({ search: "doctor" }) +``` + +Or start with the readiness check (note the server-prefixed tool name): + +```bash +mcp({ tool: "computer_use_linux_doctor" }) +``` + +Search for available tools: + +```bash +mcp({ server: "computer-use-linux" }) +``` + +## If the binary is not found + +The extension looks for `computer-use-linux` in this order: + +1. `COMPUTER_USE_LINUX_BIN` environment variable +2. The npm-bundled binary (from `npm install -g @agent-sh/computer-use-linux`) +3. `$PATH` + +If none are found, install it: + +```bash +npm install -g @agent-sh/computer-use-linux +# or +cargo install computer-use-linux +``` + +Then restart pi or run `/reload`.