Skip to content

feat: add AgentKit CUA computer-use toolset - #1219

Open
helenyang88 wants to merge 1 commit into
volcengine:mainfrom
helenyang88:feat/agentkit-cua-stdio
Open

helenyang88 wants to merge 1 commit into
volcengine:mainfrom
helenyang88:feat/agentkit-cua-stdio

Conversation

@helenyang88

Copy link
Copy Markdown

Summary

  • Add ComputerUseToolset / create_computer_use_toolset in veadk/tools/builtin_tools/computer_use.py, launching the independently distributed agentkit-cua-mcp server over stdio and exposing its desktop tools to ADK agents.
  • Bind each agent invocation to a cloud desktop through ADK state, reject parallel calls, propagate cancellation, refresh the tool list after ensure_sandbox, and return screenshots as ADK image parts.
  • Distribute the MCP server as a versioned public wheel (SHA256 pinned) started via uvx by default; CUA_MCP_WHEEL_URL or explicit connection_params can override it.
  • Add veadk/tools/computer_use/ helpers used by the toolset.

Compatibility

  • New public Python surface only (ComputerUseToolset, create_computer_use_toolset); no existing API, configuration, or generated project is changed.
  • Tool discovery failures fail closed and always keep ensure_sandbox available.

Verification

  • uv run --extra dev pytest -q tests/tools/cua_mcp -> 56 passed
  • uv run --extra dev pre-commit run --files <changed files> -> passed (ruff check/format, secret scan)
  • Branch is based on origin/main (38d3c6e)
  • tests/tools/cua_mcp skips automatically when the optional agentkit-cua-mcp wheel is not installed

Risks

  • Depends on the external agentkit-cua-mcp wheel at runtime; wheel URL changes require a ?revision= bump to avoid stale uv caches.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant