Repository navigation
Add UIA action-click tools (list_elements, perform_action, get_element_state) - #2
Merged
Merged
Conversation
…t_state) Windows-only tools that activate UI Automation control patterns directly, so agents can click, toggle, and expand controls without moving the pointer. Hand-written COM interop, session-scoped element ids with explicit staleness, MTA execution behind a provider timeout. Pixel click stays the documented fallback for custom-drawn surfaces.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds three Windows-only MCP tools so agents can activate UI Automation controls through their patterns without moving the real pointer. Pattern activation is exact and works while the window is covered, complementing the pixel tools;
clickstays the fallback for custom-drawn surfaces. Closes #1.Approach
list_elements(window)resolves the window through the existingWindowsWindowService, walks the UIA control view withmax_depth/max_elementsbounds and optionalcontrol_type/namefilters, and caches matched elements under session-scoped ids (e1,e2, ...). Each line reports control type, name, automation id, bounding rect in physical px, and supported patterns.perform_actiondrives invoke / toggle / expand / collapse / set_value / select / scroll_into_view against a cached id.get_element_statereads Name/Value/Toggle/RangeValue/ExpandCollapse/selection without a screenshot.Uia/UiaInterop.cs) in the repo's existing raw P/Invoke style. No new NuGet dependency. FlaUI 5.x isnet10.0-windows-only while this project targets plainnet10.0, so it was ruled out.OperatingSystem.IsWindows()block inProgram.cs. README and the agent skill document the new tools, including a "try UIA before pixels" tip.Reviewer notes
UIAutomationClient.hvtable order. Uncalled slots are padded placeholders and trailing slots are omitted (safe prefix rule);BB0001is suppressed in that file only.Verification
dotnet buildclean, 0 warnings; all 117 tests pass. 21 new unit tests cover the pure seams: cache id numbering and window replacement, action parsing/validation, and output formatting.dnx -yover stdio JSON-RPC) against real Notepad windows: filtered and unfiltered tree walk with truncation reporting,set_valueverified by reading the value back,scroll_into_view, and every error path (unknown id, unknown action, missing value, unknown window).Fixes: #1