Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
ce8b517
feat: add experimental hybrid readonly UI orchestration
linnnn89 Oct 6, 2026
a47b6b3
feat(client): bind readonly UI workflows to session lifecycle
linnnn89 Oct 6, 2026
7929cb9
test(hybrid): measure real model readonly workflow costs
linnnn89 Oct 6, 2026
e798f99
feat(client): expose trusted readonly recipes in skill sessions
linnnn89 Oct 6, 2026
5003ad6
fix(hybrid): reject incomplete model turns and invalid evidence
linnnn89 Oct 6, 2026
eb6c63f
docs(hybrid): record observed defects and validation limits
linnnn89 Oct 6, 2026
a27c7b3
feat(hybrid): report failed costs and enforce conditional observations
linnnn89 Oct 6, 2026
ccac194
feat(experiments): bound final JSON correction and record recovery costs
linnnn89 Oct 6, 2026
e7350fa
feat(hybrid): validate model recipe parameters and bound corrections
linnnn89 Oct 6, 2026
ef98cb6
fix(hybrid): distinguish tool invocation from final findings
linnnn89 Oct 6, 2026
91e074b
docs(hybrid): record stale local delivery receipt
linnnn89 Oct 6, 2026
281f837
docs(hybrid): prepare actionable client validation handoff
linnnn89 Oct 6, 2026
67c4dce
docs(hybrid): record client and real-app validation with navigation gap
linnnn89 Oct 6, 2026
751d3fe
feat(hybrid): verify explicit expansion before reading child state
linnnn89 Oct 6, 2026
0163e7c
feat(hybrid): discover a unique collapsed group before navigation
linnnn89 Oct 6, 2026
70a378f
feat(hybrid): return selectable navigation candidates with live valid…
linnnn89 Oct 6, 2026
6149ee8
docs(hybrid): reconcile current navigation gaps and request boundaries
linnnn89 Oct 6, 2026
77f4ab4
docs: reassess hybrid UI constraints and iteration priorities
linnnn89 Oct 6, 2026
dc3eec3
fix: allow auxiliary UI property gaps in expansion workflows
linnnn89 Oct 6, 2026
1be61ef
fix: resume expanded UI targets with local navigation evidence
linnnn89 Oct 6, 2026
5989c6b
feat: scope UI inspection and expansion to parent paths
linnnn89 Oct 6, 2026
3d3d019
docs: define hybrid main merge acceptance line
linnnn89 Oct 7, 2026
b8c5ce2
feat: finish scoped client UI workflows for 0.17.0
linnnn89 Oct 7, 2026
2f3afab
docs: link 0.17.0 merge evidence to PR 52
linnnn89 Oct 7, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# Changelog

## 0.17.0 (unreleased)

- Add opt-in client workflows through the persistent JSON Skill session: bounded readonly-ui steps, checkbox-audit over an explicit set of controls, and expand-ui for one explicit or selected Group. Preserve live observations, conditional reads, target identity, cancellation and owned-process cleanup; unknown or incomplete evidence cannot produce successful findings. No model API or background workflow runs unless explicitly invoked.
- Add wincode_ui_set_expanded and optional scopePath for inspect/review/setExpanded. Paths resolve each observed ancestor uniquely, and failures never fall back to whole-window operations. Scoped requests require inspectionVersion 5; older Hosts are rejected, and scoped actions check capability before dispatch.
- Carry scopePath through expand-ui discovery, selection, expansion, readback and nextRequest. Explicit parent requests inspect the child inside that parent from the first read; final child readback resolves the parent again. Duplicate names or IDs in another region cannot supply the result. Each request expands at most once; an already expanded parent needs no extra action. Keep missing, ambiguous, incomplete and unknown outcomes visible without automatic replay or recursive navigation.
- Keep click/type scopes unchanged. Navigation events and focus requests may bring the application forward even though the helper does not explicitly activate or restore its window. Historical model experiments and performance limits remain separate from functional validation; no universal accuracy, latency or background guarantee is claimed.

## 0.16.0 (unreleased)

- Add semantic Windows UI actions as `wincode_ui_click` and `wincode_ui_type` (keyboard `type` or `setValue`). A control is acted on only when one bounded search proves the exact selector unique inside the target window; disabled controls, unconfirmed keyboard focus, read-only values, unsupported patterns and ambiguous or incomplete searches are refused with explicit codes. Only UI Automation patterns and keyboard input are used — no coordinate mouse simulation, no activation, restore or z-order change of the target window. The helper inspection structure is now version 3, so an older helper can no longer report a requested action as a successful inspection. Results report the pattern used, the target identity and the accepted input length without echoing the input text; actions are audited as `click`/`type`/`setValue` rather than `inspect`.
Expand Down
16 changes: 10 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@

[Setup / 配置指南](WinCode-Skill制作与MCP配置指南.md) · [Code / 代码分析](skills/wincode/references/code.md) · [UI inspection / UI 检查](skills/wincode/references/ui.md) · [Changelog / 版本记录](CHANGELOG.md)

The persistent JSON Skill session also supports explicit readonly workflows, checkbox summaries and one-parent expansion with observed ancestor paths. See [client workflows and validation boundaries / 客户端工作流与验证边界](docs/hybrid-readonly-orchestration.md). 这些流程通过显式调用启用;支持按实际父路径定位、一次展开与子控件状态读回,不自动递归导航或调用模型 API。

## English

WinCode is a local server implementing the Model Context Protocol (MCP) for AI coding agents. It combines Windows UI Automation (UIA), code navigation and .NET project analysis, so an agent can inspect a running application and investigate its source code through the same connection.
Expand All @@ -26,7 +28,7 @@ WinCode is a local server implementing the Model Context Protocol (MCP) for AI c
- **UI inspection with source navigation:** Inspect a control, find candidate XAML declarations and related C# code, then read the relevant source lines. File paths, line numbers and content hashes make the findings traceable.
- **Faster, more accurate inspection in everyday use:** In our day-to-day Windows/.NET development, WinCode makes UI inspection and source navigation faster and more accurate than screenshot-based Computer Use workflows. Direct access to structured control properties and source locations reduces reliance on image interpretation and repeated interaction. Targeted queries and compact responses also reduce the amount of data the model needs to process.
- **Project and code analysis:** Explore declared solution and project references, search text and symbols, read selected code, and assess change impact. Built-in text analysis works by default; optional Roslyn integration provides compiler-backed C# symbol and reference analysis.
- **Semantic UI actions:** `wincode_ui_click` and `wincode_ui_type` act on exactly one control that a single bounded search proves unique. They use UI Automation patterns and keyboard input only—never coordinate mouse simulation—and never activate, restore or raise the target window. Disabled controls, ambiguous or incomplete matches, read-only values and unconfirmed keyboard focus are refused rather than guessed.
- **Semantic UI actions:** `wincode_ui_click` and `wincode_ui_type` act on exactly one control that a single bounded search proves unique. They use UI Automation patterns and keyboard input only—never coordinate mouse simulation—and do not explicitly activate, restore or raise the target window. Control events and navigation may still bring the application forward. Disabled controls, ambiguous or incomplete matches, read-only values and unconfirmed keyboard focus are refused rather than guessed.

A recorded test with a 222-node window reduced response text from approximately **62 KB to 1.6 KB** by querying a specific control instead of returning the full tree. See the [test record](docs/codex_worklog.md).

Expand Down Expand Up @@ -142,6 +144,7 @@ Use actual paths and line numbers from the search result. `lineRanges` selects k
| `wincode_ui_list_windows` | List visible top-level windows with process and title filters. |
| `wincode_ui_inspect` | Read controls and optional states or screenshots. |
| `wincode_ui_click` | Click one uniquely matched control through UI Automation patterns. |
| `wincode_ui_set_expanded` | Explicitly expand/collapse one uniquely matched enabled control; optional scopePath confines it to observed ancestors. Requires inspectionVersion 4, or 5 with a path. Verify its state afterwards. |
| `wincode_ui_type` | Type text into, or set the value of, one uniquely matched control. |
| `wincode_ui_review` | Inspect UI and return candidate XAML/C# source locations. |
| `wincode_hello_world` | Read instance identity, workspace binding, capabilities and known status. |
Expand All @@ -158,7 +161,7 @@ Use actual paths and line numbers from the search result. `lineRanges` selects k

### Scope and limitations

- **Desktop access:** UI inspection is read-only: it does not click controls, type text or read input-field values. The separate `wincode_ui_click` and `wincode_ui_type` tools are destructive: they act on one uniquely matched control through UI Automation patterns or keyboard input, never mouse simulation, and never activate or restore the target window. They refuse disabled controls, ambiguous or incomplete matches, read-only values and unconfirmed keyboard focus: `type` requests keyboard focus on the control (which may bring its window to the front), so use it only when the user has approved foreground interaction, and prefer `mode:"setValue"` for background work. Background mode requires both PID and HWND, supports non-minimized windows, and does not activate or restore the target. UI inspection requires an interactive Windows desktop session and does not support headless operation.
- **Desktop access:** UI inspection is read-only: it does not click controls, type text or read input-field values. The separate `wincode_ui_click` and `wincode_ui_type` tools are destructive: they act on one uniquely matched control through UI Automation patterns or keyboard input, never mouse simulation, and do not explicitly activate or restore the target window. Application events may still bring it forward. They refuse disabled controls, ambiguous or incomplete matches, read-only values and unconfirmed keyboard focus: `type` requests keyboard focus on the control (which may bring its window to the front), so use it only when the user has approved foreground interaction, and prefer `mode:"setValue"` for background work. Background mode requires both PID and HWND, supports non-minimized windows, and does not activate or restore the target. UI inspection requires an interactive Windows desktop session and does not support headless operation.
- **UIA and visual rendering:** Available properties depend on the application's UIA provider. WPF is covered by the project's desktop tests; other frameworks and custom-rendered controls may expose less information. Colors, icons and rendering quality require visual review. Background screenshots use `PrintWindow` without screen-capture fallback; check the returned capture-quality indicators.
- **Source mapping:** XAML and C# matches identify candidate source locations, with `runtimeSourceVerified: false`. The tool does not verify that the running build matches the source, resolve dynamic bindings or runtime templates, or determine the active `DataContext`.
- **Analysis coverage:** Project structure analysis reads `.sln` and `.csproj` declarations without MSBuild evaluation. Text-based references are heuristic. Review completeness, omissions and diagnostics before drawing conclusions; no matches in a limited scan do not establish absence across the project.
Expand All @@ -167,7 +170,7 @@ Use actual paths and line numbers from the search result. `lineRanges` selects k

### Development and documentation

Current source version: **0.16.0**. See [CHANGELOG](CHANGELOG.md) for version history and migration notes. Windows 11 x64 is the reference platform; ports to other operating systems require adaptation and separate validation.
Current source version: **0.17.0**. See [CHANGELOG](CHANGELOG.md) for version history and migration notes. Windows 11 x64 is the reference platform; ports to other operating systems require adaptation and separate validation.

```powershell
npm run check # Builds, core regression, stdio integration and delivery verification
Expand Down Expand Up @@ -195,7 +198,7 @@ WinCode 是面向 AI 编程智能体的本地模型上下文协议(Model Conte
- **后台 UI 检查:**无需激活目标窗口或切换键盘焦点,即可读取运行中应用的控件信息。智能体检查目标窗口时,用户可以继续使用其他应用。
- **支持纯文本大语言模型:**以结构化 JSON 返回控件名称、层级、属性和状态。通过支持 MCP 的智能体客户端,DeepSeek 等以纯文本方式使用的模型也能检查桌面界面,无需输入图像;截图为可选功能。
- **结合源码分析 UI:**检查运行时控件,查找可能对应的 XAML 声明和相关 C# 代码,再读取具体源码。结果包含文件路径、行号和内容哈希,便于核查。
- **语义化 UI 操作:**`wincode_ui_click` 与 `wincode_ui_type` 只操作同一次有界搜索证明唯一的控件;仅使用 UI Automation 模式与键盘输入,不做坐标鼠标模拟,也不激活、还原或置顶目标窗口。禁用控件、歧义或不完整的匹配、只读值以及无法确认的键盘焦点都会被拒绝,而不是靠猜测继续。
- **语义化 UI 操作:**`wincode_ui_click` 与 `wincode_ui_type` 只操作同一次有界搜索证明唯一的控件;仅使用 UI Automation 模式与键盘输入,不做坐标鼠标模拟,不主动调用激活、还原或置顶;控件事件与导航仍可能让应用进入前台。禁用控件、歧义或不完整的匹配、只读值以及无法确认的键盘焦点都会被拒绝,而不是靠猜测继续。
- **实际使用中更快、更准确:**在日常 Windows/.NET 开发中,使用 WinCode 检查 UI 和定位源码,比基于截图的 Computer Use 工作流更快、更准确。通过直接获取结构化的控件属性和源码位置,可以减少对图像识别的依赖和反复交互;配合定向查询与精简响应,还能减少模型需要处理的数据量。
- **项目与代码分析:**查看解决方案和项目中声明的引用关系,搜索文本与符号,按需读取代码,并评估变更影响。默认提供内置文本分析,可选的 Roslyn 集成支持基于编译器语义的 C# 符号与引用分析。

Expand Down Expand Up @@ -313,6 +316,7 @@ npm run delivery:verify
| `wincode_ui_list_windows` | 列出可见顶层窗口,支持按进程和标题筛选。 |
| `wincode_ui_inspect` | 读取控件信息,以及可选的状态或截图。 |
| `wincode_ui_click` | 通过 UI Automation 模式点击唯一命中的控件。 |
| `wincode_ui_set_expanded` | 明确展开/折叠唯一控件,可用 scopePath 限定实际父范围;需要 Host 取证版本 4,带路径需版本 5。操作后独立读回状态。 |
| `wincode_ui_type` | 向唯一命中的控件输入文本,或直接写入其值。 |
| `wincode_ui_review` | 检查 UI 并返回 XAML/C# 源码候选位置。 |
| `wincode_hello_world` | 读取实例身份、工作区绑定、能力及已知状态。 |
Expand All @@ -329,7 +333,7 @@ npm run delivery:verify

### 适用范围与限制

- **桌面访问:**UI 检查为只读操作,不点击控件、不输入文本,也不读取输入框的值。单独的 `wincode_ui_click` 与 `wincode_ui_type` 是破坏性操作:只通过 UI Automation 模式或键盘输入操作唯一命中的控件,不使用鼠标坐标模拟,也不激活或还原目标窗口。禁用控件、歧义或不完整的匹配、只读值以及无法确认的键盘焦点都会被拒绝;`type` 会向目标控件索取键盘焦点(可能把该窗口带到前台),因此只在用户授权前台交互时使用,后台写入优先 `mode:"setValue"`。后台模式需同时指定 PID 和 HWND,仅支持未最小化的窗口,检查过程中不激活或还原目标窗口。该功能需要交互式 Windows 桌面会话,不支持在无头环境(Headless)中运行。
- **桌面访问:**UI 检查为只读操作,不点击控件、不输入文本,也不读取输入框的值。单独的 `wincode_ui_click` 与 `wincode_ui_type` 是破坏性操作:只通过 UI Automation 模式或键盘输入操作唯一命中的控件,不使用鼠标坐标模拟,不主动调用激活或还原;应用事件仍可能让窗口进入前台。禁用控件、歧义或不完整的匹配、只读值以及无法确认的键盘焦点都会被拒绝;`type` 会向目标控件索取键盘焦点(可能把该窗口带到前台),因此只在用户授权前台交互时使用,后台写入优先 `mode:"setValue"`。后台模式需同时指定 PID 和 HWND,仅支持未最小化的窗口,检查过程中不激活或还原目标窗口。该功能需要交互式 Windows 桌面会话,不支持在无头环境(Headless)中运行。
- **UIA 与视觉效果:**可读取的属性取决于目标应用的 UIA 提供程序。项目的桌面测试覆盖 WPF,其他框架和自绘控件可能提供较少的信息。颜色、图标和渲染质量需要结合图像检查。后台截图使用 `PrintWindow`,不回退到屏幕截图;应检查返回的截图质量提示。
- **源码映射:**XAML 和 C# 的匹配结果提供了可能相关的源码位置,`runtimeSourceVerified` 为 `false`。工具不验证运行版本与源码是否一致,不解析动态绑定或运行时模板,也不确定当前的 `DataContext`。
- **分析范围:**项目结构分析仅静态读取 `.sln` 和 `.csproj` 中的声明,不进行 MSBuild 项目评估。文本引用搜索采用启发式方法。应结合完整性、省略项和诊断信息判断结果;在有限范围内未找到匹配,并不代表整个项目中不存在匹配内容。
Expand All @@ -338,7 +342,7 @@ npm run delivery:verify

### 开发与文档

当前源码版本为 **0.16.0**。版本历史和迁移说明见 [CHANGELOG](CHANGELOG.md)。项目以 Windows 11 x64 为基准平台,移植至其他操作系统需要适配并单独验证。
当前源码版本为 **0.17.0**。版本历史和迁移说明见 [CHANGELOG](CHANGELOG.md)。项目以 Windows 11 x64 为基准平台,移植至其他操作系统需要适配并单独验证。

```powershell
npm run check # 构建、核心回归、stdio 集成和交付校验
Expand Down
4 changes: 2 additions & 2 deletions WinCode-下一轮工程化迭代计划书.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WinCode 后续测试计划

更新:2026-09-12(北京时间)。当前版本为 **0.16.0**;按需会话改动基于 `af9c1b6`,语义 UI 操作改动基于 `3d283ed`。版本和 CI 状态见 [README](README.md),实现方式见[架构说明](WinCode-架构与数据流说明.md),历史结果见[工作日志](docs/codex_worklog.md)。
更新:2026-10-07(北京时间)。当前源码版本为 **0.17.0**,hybrid 客户端父范围接入与合并收口见 [专项计划](docs/hybrid-next-iteration-plan.md#main-合并验收线)。本文继续跟踪原有覆盖限制和条件性测试;历史数量不作为当前项目重新测量结果。版本与实际 CI/交付状态见 [README](README.md) 和 [工作日志](docs/codex_worklog.md)。

固定工作区、保持已加载的 Host、请求数量限制、独立的设计时输出目录、共享缓存回归,以及 PR #40/#41 的代码导航、UI 精简输出和测试修正已合并,Node 22/24 和 CodeQL 检查通过。后续本地验收已完成 TavernDesk 的 8 个上下文场景、6 个 UI 产品任务、两个独立 Gateway 的审计争用及恢复,以及实际 Codex 连接中的单项目 Roslyn 流程。对应 PR 的交付状态见工作日志;这些测试完成项从待办移除,不扩大为所有项目或所有 UI 环境均已验证。

Expand All @@ -24,6 +24,6 @@

继续使用独立 Gateway、每连接固定项目,默认使用 local-text,按需配置 Roslyn。Roslyn 不自动释放,用户可手动释放;本地开发和测试以 Windows 11 x64 为准。

自动空闲释放、pause/resume、自启动、全局停止、审计日志自动轮转、跨实例清理缓存、共享 Host 或数据库、通用租约状态机、跨平台移植和 UI 写操作均暂不开发。
自动空闲释放、pause/resume、自启动、全局停止、审计日志自动轮转、跨实例清理缓存、共享 Host 或数据库、通用租约状态机、跨平台移植均暂不开发。语义 click/type 已实现,0.17.0 增加显式展开和有限客户端导航;父路径暂不扩展到 click/type,自动递归导航按实际需求另定任务。

测试完成后,删除对应待办并将结果记入工作日志。GitHub Release 尚未发布,源码合并和 CI 通过不能替代发布或实际客户端测试。
Loading
Loading