Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
242 changes: 195 additions & 47 deletions docs/architecture/computer-use-control.md

Large diffs are not rendered by default.

16 changes: 8 additions & 8 deletions docs/interactive-capabilities/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
<!-- generated by scripts/generate-interactive-capabilities.mjs; do not edit -->
# OpenBitFun 功能与设置目录 / OpenBitFun Features & Settings

OpenBitFun Playbook 当前包含 **22 个功能**和 **21 个设置页**,共 **43 个**用户可理解的条目、**324 项**有源码证据的子能力。每个条目有独立 Markdown,并直接服务于说明书网站、OpenBitFun 全局搜索和 `OpenBitFunControl` 智能体工具。
OpenBitFun Playbook 当前包含 **22 个功能**和 **21 个设置页**,共 **43 个**用户可理解的条目、**327 项**有源码证据的子能力。每个条目有独立 Markdown,并直接服务于说明书网站、OpenBitFun 全局搜索和 `OpenBitFunControl` 智能体工具。

OpenBitFun Playbook currently contains **22 features**, **21 settings pages**, and **324** source-backed sub-capabilities across **43** user-facing entries. Every entry has its own Markdown page and directly powers the website, in-app global search, and the `OpenBitFunControl` agent tool.
OpenBitFun Playbook currently contains **22 features**, **21 settings pages**, and **327** source-backed sub-capabilities across **43** user-facing entries. Every entry has its own Markdown page and directly powers the website, in-app global search, and the `OpenBitFunControl` agent tool.

## 唯一事实源 / Single source of truth

Expand All @@ -27,20 +27,20 @@ OpenBitFun Playbook currently contains **22 features**, **21 settings pages**, a
- Generated per-item interaction audit: `docs/interactive-capabilities/technical/product-control-open-audit.json`
- Generated low-level audit map: `docs/interactive-capabilities/technical/tauri-command-map.json`

说明书、网站、搜索和智能体只看“功能 + 设置 + 子能力”。每项子能力都必须引用已注册 Tauri Command 或可解析的源码标记;这些证据不会进入公开目录。当前 **664** 个 Tauri 命令只用于实现覆盖审计。产品 UI 交互源码会在生成和检查时扫描并校验,但不会保存成随普通 UI 改动频繁变化的版本化快照。
说明书、网站、搜索和智能体只看“功能 + 设置 + 子能力”。每项子能力都必须引用已注册 Tauri Command 或可解析的源码标记;这些证据不会进入公开目录。当前 **673** 个 Tauri 命令只用于实现覆盖审计。产品 UI 交互源码会在生成和检查时扫描并校验,但不会保存成随普通 UI 改动频繁变化的版本化快照。

Docs, website, search, and agents see only features, settings, and documented sub-capabilities. Every sub-capability must reference a registered Tauri command or a resolvable source marker; evidence is stripped from public projections. The **664** Tauri commands remain implementation-audit evidence only. Product UI interaction sources are scanned and validated during generation and checks, but are not stored as a versioned snapshot that churns with ordinary UI changes.
Docs, website, search, and agents see only features, settings, and documented sub-capabilities. Every sub-capability must reference a registered Tauri command or a resolvable source marker; evidence is stripped from public projections. The **673** Tauri commands remain implementation-audit evidence only. Product UI interaction sources are scanned and validated during generation and checks, but are not stored as a versioned snapshot that churns with ordinary UI changes.

## 控制边界 / Control boundary

- 每个子能力都明确标记为直接控制、委托给专用智能体工具、需交互打开或不支持;“打开页面”不会再被统计成“智能体已控制”。当前覆盖:直接 **49**、委托 **62**、需交互 **213**、不支持 **0**。
- 每个子能力都明确标记为直接控制、委托给专用智能体工具、需交互打开或不支持;“打开页面”不会再被统计成“智能体已控制”。当前覆盖:直接 **49**、委托 **62**、需交互 **216**、不支持 **0**。
- 稳定行为声明为带 JSON 输入契约的 `operations` 或 `options`,并绑定原生产品控制 Provider;智能体不接触原始 Tauri Command。
- `OpenBitFunControl list` 和 `search` 都返回带 `nextCursor` 的精简分页结果;目录可持续增长,不靠固定总量上限。完整目录和 324 项子能力都不会写入 system prompt。
- `OpenBitFunControl list` 和 `search` 都返回带 `nextCursor` 的精简分页结果;目录可持续增长,不靠固定总量上限。完整目录和 327 项子能力都不会写入 system prompt。
- 目录发现与契约读取不依赖 React 或可见窗口。普通配置型 option 统一由 Product Assembly 的共享 ConfigService 执行器读、写并回读,因此 Desktop、CLI 与 Headless 表面走同一份实现;只有宿主原生 operation/provider option 和界面导航按表面注册适配器,缺失时必须明确返回不可用,禁止静默回退本机。只读智能体只能发现和读取目录。

- Every documented item is classified as direct control, delegated Agent control, interactive opening, or unsupported; opening a page is never counted as direct control. Current coverage is **49 direct**, **62 delegated**, **213 interactive**, and **0 unsupported**.
- Every documented item is classified as direct control, delegated Agent control, interactive opening, or unsupported; opening a page is never counted as direct control. Current coverage is **49 direct**, **62 delegated**, **216 interactive**, and **0 unsupported**.
- Stable behavior becomes a typed `operation` or `option` with a JSON input contract and a native product-control provider. Agents never receive raw Tauri commands.
- `OpenBitFunControl list` and `search` return compact pages with a `nextCursor`; the catalog can grow without a fixed total-size ceiling. Neither the full catalog nor its 324 documented items enters the system prompt.
- `OpenBitFunControl list` and `search` return compact pages with a `nextCursor`; the catalog can grow without a fixed total-size ceiling. Neither the full catalog nor its 327 documented items enters the system prompt.
- Discovery and contract lookup do not depend on React or a visible window. Ordinary config-backed options are read, written, and read back by one Product Assembly ConfigService executor shared by Desktop, CLI, and headless surfaces. Only host-native operations/provider options and presentation routes install surface adapters; missing adapters return explicit unavailability without local fallback. Read-only agents may only discover and inspect entries.

## 防腐化门禁 / Anti-drift gates
Expand Down
196 changes: 190 additions & 6 deletions docs/interactive-capabilities/capabilities.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"title": "OpenBitFun Playbook",
"origin": "https://playbook.openbitfun.com",
"source": "src/shared/interactive-capabilities/catalog.json",
"digest": "09837a3f9f09ecf1d1b2e3a60059824d6e8afa979c0b022cef05daa63221c04b",
"digest": "09e162e571595bbe38f6c2e10e061e0f7243baae9e6db56a2e5b8948304a0e32",
"ownerDigest": "c989ace53614c2a526fefe64816fdd6895ec71edcc28f44b6f80d1edb75bb33e",
"searchAcceptance": [
{
Expand Down Expand Up @@ -138,11 +138,11 @@
"features": 22,
"settings": 21,
"userFacing": 43,
"documentedItems": 324,
"documentedItems": 327,
"controlCoverage": {
"direct": 49,
"delegated": 62,
"interactive": 213,
"interactive": 216,
"unsupported": 0
}
},
Expand Down Expand Up @@ -4869,7 +4869,10 @@
"default-cdp",
"agent-actions",
"control-session-preview",
"stop-control-session"
"stop-control-session",
"app-sharing",
"preview-input",
"remembered-control"
],
"kind": "query",
"risk": "read",
Expand Down Expand Up @@ -5322,6 +5325,144 @@
"pageId": "tools.execution"
}
},
{
"id": "feature.computer-use:open:app-sharing",
"capabilityId": "feature.computer-use",
"itemIds": [
"app-sharing"
],
"kind": "open",
"risk": "ui",
"executionHost": "presentationSurface",
"availability": {
"desktop": {
"available": true
},
"cli": {
"available": false,
"reason": "This delivery profile has no live presentation surface"
},
"peer": {
"available": true,
"requiredCapabilities": [
"product_control_v1",
"product_control_presentation_v1"
]
},
"remoteControl": {
"available": true
},
"detachedDispatch": {
"available": false,
"reason": "This delivery profile has no live presentation surface"
}
},
"inputSchema": {
"type": "object",
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"additionalProperties": true
},
"openReason": "visualSelection",
"presentationTarget": {
"kind": "settings",
"pageId": "tools.execution"
}
},
{
"id": "feature.computer-use:open:preview-input",
"capabilityId": "feature.computer-use",
"itemIds": [
"preview-input"
],
"kind": "open",
"risk": "ui",
"executionHost": "presentationSurface",
"availability": {
"desktop": {
"available": true
},
"cli": {
"available": false,
"reason": "This delivery profile has no live presentation surface"
},
"peer": {
"available": true,
"requiredCapabilities": [
"product_control_v1",
"product_control_presentation_v1"
]
},
"remoteControl": {
"available": true
},
"detachedDispatch": {
"available": false,
"reason": "This delivery profile has no live presentation surface"
}
},
"inputSchema": {
"type": "object",
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"additionalProperties": true
},
"openReason": "visualSelection",
"presentationTarget": {
"kind": "settings",
"pageId": "tools.execution"
}
},
{
"id": "feature.computer-use:open:remembered-control",
"capabilityId": "feature.computer-use",
"itemIds": [
"remembered-control"
],
"kind": "open",
"risk": "ui",
"executionHost": "presentationSurface",
"availability": {
"desktop": {
"available": true
},
"cli": {
"available": false,
"reason": "This delivery profile has no live presentation surface"
},
"peer": {
"available": true,
"requiredCapabilities": [
"product_control_v1",
"product_control_presentation_v1"
]
},
"remoteControl": {
"available": true
},
"detachedDispatch": {
"available": false,
"reason": "This delivery profile has no live presentation surface"
}
},
"inputSchema": {
"type": "object",
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"additionalProperties": true
},
"openReason": "visualSelection",
"presentationTarget": {
"kind": "settings",
"pageId": "tools.execution"
}
},
{
"id": "feature.skills:query",
"capabilityId": "feature.skills",
Expand Down Expand Up @@ -21879,13 +22020,15 @@
"启用或停用 Computer Use",
"连接或断开受支持的浏览器",
"检查并打开系统权限设置",
"查看当前控制目标与预览,并随时停止控制"
"查看当前控制目标与预览,并随时停止控制",
"先选择共享 App,再授权输入;可记住授权或随时撤销"
],
"highlightsEn": [
"Enable or disable Computer Use",
"Connect or disconnect a supported browser",
"Inspect and open OS permission settings",
"Inspect the current control target and preview, and stop control at any time"
"Inspect the current control target and preview, and stop control at any time",
"Select an App share before granting input; remember or revoke the grant"
],
"items": [
{
Expand Down Expand Up @@ -22002,6 +22145,39 @@
"Use ComputerUse stop_control to release the tool-owned session. Users can also stop from the executing Desktop control card; its button addresses the displayed generation rather than a subsequently created session."
]
}
},
{
"id": "app-sharing",
"titleZh": "选择 App 窗口并单独授权或撤销控制",
"titleEn": "Select an App window and separately grant or revoke control",
"control": {
"kind": "open",
"reasonCode": "visualSelection",
"reasonZh": "共享窗口及控制模式由用户明确选择;智能体只能请求共享,不能代替用户授权。",
"reasonEn": "Window selection and input mode require an explicit user choice; agents can request sharing but cannot grant themselves control."
}
},
{
"id": "preview-input",
"titleZh": "在共享预览中点击、拖动、滚动及输入",
"titleEn": "Click, drag, scroll and type through the shared preview",
"control": {
"kind": "open",
"reasonCode": "visualSelection",
"reasonZh": "预览输入需要用户授权,并绑定共享窗口、图像坐标及授权版本。",
"reasonEn": "Preview input requires user authorization and is bound to the shared window, image coordinates and authorization revision."
}
},
{
"id": "remembered-control",
"titleZh": "按应用执行文件和控制模式记住或忘记授权",
"titleEn": "Remember or forget control by application executable and input mode",
"control": {
"kind": "open",
"reasonCode": "visualSelection",
"reasonZh": "记住授权只在用户选定并共享同一应用后生效,应用文件或输入模式改变时需要重新授权。",
"reasonEn": "Remembered control applies only after the user shares the same application; changed executable contents or input modes require a new grant."
}
}
],
"stepsZh": [
Expand Down Expand Up @@ -22049,10 +22225,12 @@
"连接或断开受支持的浏览器",
"检查并打开系统权限设置",
"查看当前控制目标与预览,并随时停止控制",
"先选择共享 App,再授权输入;可记住授权或随时撤销",
"Enable or disable Computer Use",
"Connect or disconnect a supported browser",
"Inspect and open OS permission settings",
"Inspect the current control target and preview, and stop control at any time",
"Select an App share before granting input; remember or revoke the grant",
"检查 Computer Use 可用性、辅助功能和屏幕录制授权状态",
"Check Computer Use availability plus accessibility and screen-recording permission status",
"请求桌面控制所需的系统权限",
Expand All @@ -22071,6 +22249,12 @@
"Inspect the current target, mode, and authorized capture preview in the control card",
"停止当前桌面控制会话并释放捕获与输入资源",
"Stop the current desktop-control session and release capture and input resources",
"选择 App 窗口并单独授权或撤销控制",
"Select an App window and separately grant or revoke control",
"在共享预览中点击、拖动、滚动及输入",
"Click, drag, scroll and type through the shared preview",
"按应用执行文件和控制模式记住或忘记授权",
"Remember or forget control by application executable and input mode",
"打开桌面控制设置",
"带我配置浏览器控制",
"Open desktop-control settings",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ Allow agents to operate a browser or local UI while explicitly managing accessib
- Inspect the current target, mode, and authorized capture preview in the control card
- **由专用智能体工具控制 / Delegated Agent tool** · `ComputerUse` · 停止当前桌面控制会话并释放捕获与输入资源
- Stop the current desktop-control session and release capture and input resources
- **智能体可定位入口,需交互完成 / Agent opens; interaction required** · 选择 App 窗口并单独授权或撤销控制
- Select an App window and separately grant or revoke control
- **智能体可定位入口,需交互完成 / Agent opens; interaction required** · 在共享预览中点击、拖动、滚动及输入
- Click, drag, scroll and type through the shared preview
- **智能体可定位入口,需交互完成 / Agent opens; interaction required** · 按应用执行文件和控制模式记住或忘记授权
- Remember or forget control by application executable and input mode

## 怎么用 / How to use it

Expand Down
Loading
Loading