-
Notifications
You must be signed in to change notification settings - Fork 39
docs(research): record zero-resident diagnostic #307
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
seekskyworld
wants to merge
1
commit into
openpi-dev:main
Choose a base branch
from
seekskyworld:docs/issue-20-diagnostic-retrospective
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
50 changes: 50 additions & 0 deletions
50
docs/research/OPENPI_ZERO_RESIDENT_SURFACE_DIAGNOSTIC_2026-08-30.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| # OpenPI 零常驻工具面三臂诊断复盘 | ||
|
|
||
| > 状态:validated(历史诊断复盘;不是正式 Benchmark 或产品 Decision) | ||
| > | ||
| > 创建日期:2026-08-30 | ||
| > | ||
| > 最后核验:2026-08-30 | ||
| > | ||
| > 关联 Issue:[#20](https://github.com/openpi-dev/openpi/issues/20)、[#19](https://github.com/openpi-dev/openpi/issues/19) | ||
|
|
||
| ## 结论边界 | ||
|
|
||
| OpenPI 从多个常驻入口收敛到普通 turn 零常驻 OpenPI 工具,消除了稳定的 schema 和 Skill catalogue 静态税。现有 DeepSeek 0731 ARM64 source-build-derived 三臂诊断显示 OpenPI、Bare Pi、Maka 在小样本中通过数相同,但轨迹 token、turn 和 wall time 仍有较大方差。 | ||
|
|
||
| 这组材料证明了工具面收敛方向和隔离/账本的重要性,不能证明 OpenPI 在总体质量、成本或速度上优于其他 Harness,也不能把结果归因给某一个 runtime 机制。 | ||
|
|
||
| ## 已核验观察 | ||
|
|
||
| - 早期约 31 个模型工具会让普通任务支付完整高级能力 surface;该批任务很少实际采用 Subagent、Workflow、Background、Goal 或 Task。 | ||
| - 7 个 compact entry 能降低工具数量,但仍可能把能力入口和 Skill catalogue 带入每个首请求。 | ||
| - capability gateway 与后续 zero-resident surface 将高级能力改为按明确意图加载,能力组在 Session 内保持稳定。 | ||
| - 三臂诊断的 candidate、HOME、TMP、Session 和 verifier 隔离经历多轮修复;早期越界访问和 verifier 可见性问题会使正式排名失效。 | ||
| - 固定首请求 payload 后,独立模型采样仍可能产生完全不同的错误恢复轨迹;静态税与动态轨迹必须分账。 | ||
|
|
||
| ## 证据账本 | ||
|
|
||
| 每个诊断 cell 应记录:OpenPI/Pi/Maka 完整源码身份、provider/model/thinking、任务与 verifier 身份、candidate 隔离根、首请求 payload hash、工具 schema、请求与工具调用数、input/output/cache/reasoning usage、wall time、protocol/verifier 结果、基础设施失败和完整 trace 位置。 | ||
|
|
||
| `cacheRead`、provider cost 和未冻结的模型账单字段不能在缺少明确口径时自行推导。任何 candidate root 外访问、verifier 泄漏、残留进程或 artifact 不完整都应分类为 harness invalid,而不是模型失败。 | ||
|
|
||
| ## 当前产品边界 | ||
|
|
||
| - 普通父 Session 只保留 Pi 原生核心工具;OpenPI 能力按需加载。 | ||
| - capability discovery 是渐进机制,不是固定 workflow 或模型名称路由器。 | ||
| - 搜索、委派、Workflow、Background、Session 等组的生命周期仍由各自 owner 管理。 | ||
| - 结果投影、重试和 artifact 保留必须与模型上下文、用户 UI 分离。 | ||
| - 在新的配对证据达到预注册门槛前,不因单次诊断扩大默认 prompt 或工具面。 | ||
|
|
||
| ## 后续实验门槛 | ||
|
|
||
| 先完成干净 workspace、网络和 verifier 隔离,再使用 AB/BA 顺序反转与多次重复比较 Bare Pi、OpenPI Explicit、指定 capability profile 和固定版本 OMP。至少同时报告质量、首请求静态税、总 usage、轨迹长度、机制采用率和失败分类。 | ||
|
|
||
| 只有在普通 coding 赛道质量非劣且成本边界通过,并且编排价值赛道确实观察到能力采用带来的净收益,才可提出新的默认行为变更。 | ||
|
|
||
| ## 来源与关联 | ||
|
|
||
| - [Issue #20](https://github.com/openpi-dev/openpi/issues/20):零常驻表面与 DeepSeek 三臂诊断原始讨论。 | ||
| - [Issue #19](https://github.com/openpi-dev/openpi/issues/19):从 compact entry 到 capability gateway 的设计分析。 | ||
| - [Issue #45](https://github.com/openpi-dev/openpi/issues/45):模型能力 × Harness 强度配对研究协议。 | ||
| - [OpenPI documentation contract](../README.md):研究记录状态与证据边界。 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,5 +8,6 @@ The following records predate [`Decision 0001`](../decisions/0001-documentation- | |
|
|
||
| - [`CLAUDE_CODE_WORKFLOW_FANOUT_POLICY_2026-08-23.md`](CLAUDE_CODE_WORKFLOW_FANOUT_POLICY_2026-08-23.md) — official-source research on dynamic fan-out and bounded execution. | ||
| - [`CLAUDE_CODE_WORKFLOW_RUNTIME_CONTRACT_2026-08-23.md`](CLAUDE_CODE_WORKFLOW_RUNTIME_CONTRACT_2026-08-23.md) — version-scoped Workflow contract interview and evidence boundary. | ||
| - [`OPENPI_ZERO_RESIDENT_SURFACE_DIAGNOSTIC_2026-08-30.md`](OPENPI_ZERO_RESIDENT_SURFACE_DIAGNOSTIC_2026-08-30.md) — Issue #19/#20 的零常驻工具面与三臂诊断复盘。 | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 本列表声明文档早于 Decision 0001、尚未迁移到新 metadata 契约。但新增记录创建日期为 2026-08-30,Decision 0001 已于 2026-08-29 生效。这个位置会错误表达其历史豁免身份。 请移到当前研究小节,按现有 docs/README.md 约定补适用来源边界、相关 PR 和替代关系;无需新增格式框架或批量迁移旧文档。 |
||
|
|
||
| When research changes a project constraint, preserve the adopted choice in a Decision. Amend or supersede a historical record rather than silently rewriting its original conclusion. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
此记录定位为 validated 历史复盘,但这里换成了 Bare Pi / Explicit / capability profile / OMP。来源 #20 §13 的后续是优先做 SQLite/Cancel 固定身份匹配重复,满足稳定阈值后再做 runtime 单变量;关联 #45 §7 的四 Harness 则是 Bare Pi / OpenPI Explicit / OMP / Maka,不包含替代 Maka 的 profile 臂。
新组合可以作为建议,但不能不加说明地代替来源的后续路径。请保留或链接原方案,把新增 profile 方案明确标成新建议并说明理由即可;不要求在本 PR 新跑实验或另造框架。