Skip to content

Add UIA action-click tools (list_elements, perform_action, get_element_state) - #2

Merged
bitbound merged 1 commit into
mainfrom
bitbound-uia-action-click-tools
Oct 30, 2026
Merged

bitbound merged 1 commit into
mainfrom
bitbound-uia-action-click-tools

Conversation

@bitbound

Copy link
Copy Markdown
Owner

Summary

Adds three Windows-only MCP tools so agents can activate UI Automation controls through their patterns without moving the real pointer. Pattern activation is exact and works while the window is covered, complementing the pixel tools; click stays the fallback for custom-drawn surfaces. Closes #1.

Approach

  • list_elements(window) resolves the window through the existing WindowsWindowService, walks the UIA control view with max_depth / max_elements bounds and optional control_type / name filters, and caches matched elements under session-scoped ids (e1, e2, ...). Each line reports control type, name, automation id, bounding rect in physical px, and supported patterns.
  • perform_action drives invoke / toggle / expand / collapse / set_value / select / scroll_into_view against a cached id. get_element_state reads Name/Value/Toggle/RangeValue/ExpandCollapse/selection without a screenshot.
  • Hand-written COM interop (Uia/UiaInterop.cs) in the repo's existing raw P/Invoke style. No new NuGet dependency. FlaUI 5.x is net10.0-windows-only while this project targets plain net10.0, so it was ruled out.
  • Registered in the existing OperatingSystem.IsWindows() block in Program.cs. README and the agent skill document the new tools, including a "try UIA before pixels" tip.

Reviewer notes

  • COM interface member order is UIAutomationClient.h vtable order. Uncalled slots are padded placeholders and trailing slots are omitted (safe prefix rule); BB0001 is suppressed in that file only.
  • Ids never reuse numbers. Re-listing a window drops its previous entries, and stale elements map COM failures to a loud "re-run list_elements" message.
  • All UIA calls run on thread-pool threads (already MTA, as UIA requires) behind a 10 s timeout that abandons rather than aborts a hung provider. Flagged in code comments as a first-iteration trade-off.

Verification

  • dotnet build clean, 0 warnings; all 117 tests pass. 21 new unit tests cover the pure seams: cache id numbering and window replacement, action parsing/validation, and output formatting.
  • Live smoke through the packaged build (dnx -y over stdio JSON-RPC) against real Notepad windows: filtered and unfiltered tree walk with truncation reporting, set_value verified by reading the value back, scroll_into_view, and every error path (unknown id, unknown action, missing value, unknown window).

Fixes: #1

…t_state)

Windows-only tools that activate UI Automation control patterns directly, so agents can click, toggle, and expand controls without moving the pointer. Hand-written COM interop, session-scoped element ids with explicit staleness, MTA execution behind a provider timeout. Pixel click stays the documented fallback for custom-drawn surfaces.
@bitbound
bitbound merged commit 96316fb into main Oct 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add UI Automation action-click tools (click without moving the pointer)

1 participant