test(chat): align operation fixture and recovery smoke - #5418
huangruiteng merged 5 commits into
Conversation
Signed-off-by: Lihua <1017343802@qq.com>
Signed-off-by: Lihua <1017343802@qq.com>
Signed-off-by: Lihua <1017343802@qq.com>
Signed-off-by: Lihua <1017343802@qq.com>
Signed-off-by: Lihua <1017343802@qq.com>
|
Exact-head CI attribution for
The clean order is to review and merge #5418 first, then sync #5375 so its coverage fix runs with the six-stage assertion. #5409 and #5389 can sync after those baseline fixes land. No merge action was taken. |
|
Updated CI triage for
|
huangruiteng
left a comment
There was a problem hiding this comment.
No blocking finding. No product behavior, authority or protocol change. The remaining uncertainty is the reported load-sensitive browser race: the immutable base also passed the focused browser scenario in this run, so I do not claim to have reproduced that timing failure or qualified the entire browser suite.
动机
本次按 pull_request_review_execution_contract_v2(policy 13)独立评审完整 base-to-head:850268bffc6f7123e88f578d3743f8f6da5c5c73 → d701822beb6892cdf7d9062b24bcc5ccc6779730。目标是修复已有验收的陈旧期望和故障注入竞争,不是新增恢复机制。实际独立 fixture 已输出六阶段;基线的四阶段断言失败,因此修正期望有明确仓库价值。相比新增 runner 或重写生产恢复 owner,延续两个已有测试是更小、完整且可逆的修复。
改动思路
失败开关在 reload 前启用,避免恢复完成后才注入;卡片等待上限从 5 秒调整到 15 秒,以容纳已有历史重试,而非修改产品退避或承诺新的性能指标。新增断言要求开关已被消费,防止场景“绿色但没有测试失败”。Python 测试继续用 -S 执行独立 fixture,不引入测试框架依赖。
具体改动
test_browser_operation_fixture_needs_no_test_framework(tests/test_chat_operation_actions.py:81)校验prepared → delivered → confirmed → waiting → unknown → reconciled,保留 Goal、executor 和未知结果/对账断言。浏览器confirmed-operations.mjs也实际消费前两阶段,并非为适配偶然输出添加字段。chatTodoProposalScenario.run(examples/personal-workspace-browser/chat-todo-proposal.mjs:124–150)仅改变故障注入位置、等待预算及消费检查;仍检查唯一幂等键、原 Goal、用户确认及后续 reload 不重复 apply。- 已沿生产
projectRecoveredTurnProposals和 completed-history effect 检查恢复路径:读取已有 action,失败返回未完成并由既有有限退避重试;preview 不授予 apply 权限。模拟 API 只提供输入/503,恢复与展示执行真实前端代码。
独立本地验证:
- 精确 head 的
uv run --extra test python -m pytest -q tests/test_chat_operation_actions.py:41 passed;同命令不可变 base:40 passed / 1 failed,仅上述陈旧四阶段断言失败。 - 原生
chat-todo-proposal浏览器验收:base、head 开发版均通过;head 打包版通过。headnpm run build:chat和uv run --extra test python scripts/chat_bundle.py verify --source通过。 - 独立负向探针只在内存中绕过 fixture 的故障分支、保留未消费开关,产品代码未改;真实 Chromium 场景准确在
Recovery did not exercise the transient preview failure断言失败。这证明新增检查能拒绝“故障未发生”的伪通过,不证明历史负载问题已复现。 - 改动 Python 的 Ruff、JavaScript 语法检查和 whole-diff whitespace 检查通过。
对主干的风险
贡献 diff 两文件 +8/-3,都是现有测试;无生产状态分类、默认开关、通用控制面措辞、guidance/obligation 或 actor authority 变更。未新增并行决策 owner,不需要语言迁移。长期重复恢复、离开 Goal 后返回及下一次 reload 的既有幂等/确认断言保留,用户操作路径和权限保持不变。
浏览器证据边界是“真实开发/打包前端 + 隔离合成 API/stream”,不是真实模型、部署后端或已安装版本验收。未跑完整 pytest、完整浏览器套件或 PostgreSQL;没有读取、等待或用远端 CI 代替本地评审。作者描述的全套浏览器失败和覆盖率问题不宣称已由本 PR 修好。
耐久性/重复扫描:检查了两个 immutable revision 的 fixture、相邻恢复场景、现有 runner、package smoke 命令和 Python workflow 调用,并查看作者近期 30 个 PR(包括 #5440 的独立 coverage 边界)。本改动复用既有生产验收,没有新建同形 smoke 或临时工具。future-facing pass 考虑了是否需要按请求绑定故障计数器或 TS 抽象;当前局部场景已有明确时序和消费断言,没有足以支持额外共用机制的缺口,暂不扩展。
我的整体评价
APPROVE。明确的陈旧断言修复,加上能经负向探针验证的既有恢复场景加固,构成完整测试维护结果;不把更大的等待值等同于产品性能修复。不替整个恢复路线、全套 CI 或上层路线图宣布完成。批准后应单独读回 effective blocking reviews,只有逐项独立确认旧 finding 已解决且拥有撤销权限时才撤销;本结论不授权 merge。
English verdict
English verdict: APPROVE
APPROVE — independently reviewed exact head d701822beb6892cdf7d9062b24bcc5ccc6779730. The stale lifecycle expectation is reproduced at the immutable base and corrected at head. Native development and packaged browser acceptance pass; an independent guard mutation demonstrates that the new assertion rejects an unexercised transient failure. The base browser also passed, so the historical timing failure is not claimed reproduced. This is existing-test maintenance, not a production latency, authority, protocol or default-policy change. Synthetic API/stream boundaries and unrun whole-suite/backend validation are explicitly retained. No merge authorization.
Goal and gap
The standalone operation fixture emits
preparedanddeliveredbeforeconfirmed, while its Python assertion expected only four later stages. In exact-head CI, the Personal Workspace browser smoke also timed out waiting for a recovered Todo card. That scenario armed its transient preview failure after reopening Goal Chat, allowing recovery to race the injection; its 5-second card deadline was tight for the history retry's 3-second backoff under CI load.Change
d701822beintegrates current main850268bff; the contribution diff is limited to the two test files.Validation
tests/test_chat_operation_actions.py: 41 passed on the integrated source.chat-todo-proposalreal browser scenario passed in development with coverage and in the packaged Chat build.team-evidencetimed out waiting for its active-mode label; that scenario passed when run alone. This separate load-sensitive failure remains unclassified, so the complete suite is not claimed green.node --checkandgit diff --checkpassed. Exact-head remote CI passed the four Python shards and dashboard acceptance, but aggregatepytestfailed while reporting coverage for a disposable semantic-probe fixture (No source for code). #5440 owns that narrow repair. CI is red; maintainer review and merge remain pending.中文
独立操作夹具会先输出
prepared、delivered,原 Python 断言却只接受后四个阶段。上一精确版本 CI 还在恢复后的 Todo 卡片可见性检查处超时:浏览器场景直到重新打开 Goal Chat 才启用一次性预览失败,可能与恢复完成竞争;在 CI 负载下,5 秒等待也不足以覆盖历史重试的 3 秒退避。本 PR 校正六阶段断言,并在重载前启用预览失败,确认故障确实被触发,给恢复重试后的卡片最多 15 秒显示。产品界面、持久化和权限行为未修改。签署合并
d701822be纳入最新 main850268bff,相对主干仅改动两个测试文件。整合源码上的 Python 文件测试 41 项通过;Chat 打包构建与校验通过;
chat-todo-proposal在带覆盖率的开发浏览器和打包浏览器均通过。完整开发浏览器套件曾运行,前 16 个场景通过,随后team-evidence等待运行状态标签超时;该场景单独运行通过,因此不能宣称完整套件已通过。语法和差异检查通过。精确版本远端 CI 的四个 Python 分片与 dashboard acceptance 已通过,但覆盖率汇总读取已清理的语义探针临时源码时失败(No source for code);独立的 #5440 负责该修复。CI 仍为红色,维护者评审与合并待完成。