feat(examples): add evaluation + optimization closed-loop pipeline - #139
feat(examples): add evaluation + optimization closed-loop pipeline#139coder-mtj wants to merge 43 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #139 +/- ##
==========================================
Coverage ? 88.43489%
==========================================
Files ? 491
Lines ? 46035
Branches ? 0
==========================================
Hits ? 40711
Misses ? 5324
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
补充了设计文档和开发过程记录：
测试覆盖：189 tests，14 测试文件，6 维度（单元/集成/大规模/边界/回归/性能） 所有 CI checks 通过。如有遗漏或需要调整的地方请告知。 |
Implements trpc-group#91 — reproducible Evaluation + Optimization pipeline: - Config loading (optimizer.json + evalsets validated) - Baseline evaluation (fake mode with trace evalsets + SDK path) - Failure attribution (10 categories: tool errors, rubric, format, etc.) - Multi-dimensional gate (improvement threshold, critical cases, cost budget) - Validation set comparison (new passes/failures, overfitting detection) - JSON + Markdown report with full audit trail - 6 train+val evalset cases (3 optimizable, 1 degrading, 1 format, 1 edge) - 35 tests covering config, baseline, attribution, gate, validation, report, integration Signed-off-by: coder-mtj <coder-mtj@users.noreply.github.com>
…overage - Add pipeline/optimize.py: GEPA optimization wrapper (fake + live modes) - Add pipeline/tracing.py: audit trail with seed/timing/cost/reproduce - Add agent/ package: calculator agent for optimization testing - Update run_pipeline.py: integrate new modules, AuditTracer, enhanced CLI - Split monolithic test file into 14 focused test files - Expand from 35 to 189 tests (5.4x increase) - Add 6-dimensional test coverage: unit, integration, mock data, edge/boundary, regression, performance - Enhance evalsets: 34 train + 16 val + 12 holdout cases (multi-domain: math, reasoning, tool calls, Chinese, CJK, format) - Add DESIGN.md and README.md with architecture documentation - All 189 tests passing, pipeline verified end-to-end in fake mode
…de no-op - 新增 pipeline/comparator.py：分层评测规则（纯数字/contains/带单位/格式/工具） - 修复 run_baseline_fake 空转：比较 conversation 期望 vs actual_conversation 实际 - 归因增强：直接读取 comparator 的 category/evidence - 新增 23 个 comparator 单元测试，全量 212 tests 通过 Signed-off-by: popo <18682875253@163.com>
…valset data - 新增 tests/test_gold_verdicts.py：84 条黄金判定表锁定归因精度（≥90%） - 修复 train 数据标注错误：train_reasoning_002_fail / train_tool_002_fail 改为真正失败 - 新增 large_train.evalset.json（50 cases，17 个 _fail） - comparator 增强：货币千分位、数字子集匹配 - 全量 299 tests 通过 Signed-off-by: popo <18682875253@163.com>
…te rejection - 新增 --scenario CLI（fix_attributed/noop/overfit）演示三类验收场景 - validate.py: run_validation_trace 用 TraceMatcher 重评候选 actuals，带 per_case_results - gate.py: 候选在验证集新增失败 → REJECT（过拟合检测真实生效） - optimize.py: SCENARIOS 注册表 + candidate_strategy/fixed_categories - 修复 Windows GBK 控制台 emoji print 崩溃 - 三类场景验证：fix_attributed=ACCEPT, noop=NEEDS_REVIEW, overfit=REJECT(CI 退出码 1) Signed-off-by: popo <18682875253@163.com>
- JSON 报告新增 candidate 块（train/validation 评分 + 逐 case delta） - MD 报告新增 Candidate vs Baseline 逐 case 对比表 - 归因条目补充 evidence 字段（可解释性） - 修复 FailureCategory 枚举序列化 Signed-off-by: popo <18682875253@163.com>
- agent.py: 新增 build_call_agent()（确定性离线 CallAgent） - baseline.py: run_baseline_sdk 变 async，用 AgentEvaluator.evaluate_eval_set； SDK 失败降级到 trace comparator - optimize.py: run_optimize_live 正确 await AgentOptimizer.optimize(call_agent=...) - optimizer.json: 补充 reflection_lm 配置 - run_pipeline.py: live 模式用 asyncio.run 隔离，项目根加入 sys.path - 修复 SDK schema 不兼容时 live 模式崩溃问题 Signed-off-by: popo <18682875253@163.com>
… tests - test_scenarios.py: 三场景端到端（fix_attributed=ACCEPT, noop=NEEDS_REVIEW, overfit=REJECT） - test_attribution_accuracy.py: 归因准确率 ≥90%（验收标准 trpc-group#4） - test_live_mode_import.py: live 模式健壮性 + fake 性能 <3s - 全量 317 tests 通过 Signed-off-by: popo <18682875253@163.com>
…kage entry - pipeline/__init__.py: 统一 re-export 全部核心符号 - 支持 from pipeline import PipelineConfig, run_baseline_fake, ... - 清理 SDK live 运行产生的垃圾文件（baseline_prompts/ 等） - 317 tests 保持全绿，零回归 Signed-off-by: popo <18682875253@163.com>
…gful sample report - README: 三场景演示、工作原理、模块地图、CLI 参数、验收标准对照 - DESIGN: comparator/三场景/6 维度 gate/live 降级说明 - ai-prompts: 补充第 5 轮（trace 回放评测、三场景、过拟合拒绝） - attribution: 修复 by_category 序列化（枚举 .value） - sample_output: 有意义的默认报告（失败+归因+候选+gate ACCEPT） - .gitignore: 忽略 SDK live 运行产物 Signed-off-by: popo <18682875253@163.com>
a4c5a20 to
41e84f8
Compare
AI Code Review我已经掌握了足够的信息。让我来整理最终的审查意见。 发现的问题🚨 Critical
|
| open(evalset_path, encoding="utf-8").read() | ||
| ) | ||
| # trace 模式离线评测：evaluate_eval_set 返回 per-case 结果 | ||
| _, _, _, case_results = await AgentEvaluator.evaluate_eval_set( |
There was a problem hiding this comment.
live baseline 调用 evaluate_eval_set 未传必填 eval_config
SDK 在 eval_config 为 None 时直接抛 ValueError，使 live baseline 必然落入 except 降级分支，真实评测从未执行、永远走 trace 回放兜底。需构造 EvalConfig（或从 optimizer.json 的 evaluate 段加载）后传入。
| for case_id, results in (case_results or {}).items(): | ||
| for cr in results: | ||
| total += 1 | ||
| ok = getattr(cr, "passed", False) |
There was a problem hiding this comment.
读取 EvalCaseResult 用了不存在的属性导致全部误判失败
EvalCaseResult 只有 final_eval_status 和 error_message，没有 passed/failure_reason；getattr 默认值使 passed 恒为 0、所有 case 进 failed_case_ids，pass_rate 恒为 0。应改为 cr.final_eval_status == EvalStatus.PASSED。
| ) | ||
|
|
||
| # Extract results | ||
| result.total_cost = getattr(opt_result, 'total_cost', 0.0) |
There was a problem hiding this comment.
live optimize 读取 OptimizeResult 字段名不匹配被静默清零
SDK OptimizeResult/RoundRecord 字段名与代码读取的几乎全部不匹配，getattr 默认值掩盖错误，使成本、轮次、best_prompt、rounds 评分全部为 0/空，报告失真。应按 SDK 实际字段映射。
Critical fixes: - baseline.py: pass required eval_config to evaluate_eval_set (was always falling back) - baseline.py: use EvalCaseResult.final_eval_status instead of nonexistent 'passed' - optimize.py: map SDK OptimizeResult fields correctly (total_llm_cost/total_rounds/best_prompts/rounds) Other: - config.py: add load_optimize_config() to build EvalConfig from optimizer.json - baseline/optimize/config: extract sys.path setup into named helpers (no silent except) - validate.py: move copy import to module top - run_pipeline.py: load EvalConfig in live mode - tests: add live-mode contract tests (mock SDK, verify field mapping) 319 tests pass Signed-off-by: popo <18682875253@163.com>
AI Code Review我已经收集了所有经过验证的发现，并附上了行号。让我来编写审查报告。 发现的问题
|
…erfit default (AI review round 2) - optimize.py: converged now checks SDK status == 'SUCCEEDED' (not 'accepted') - comparator.py: _compare_tools reads tool_responses (real evalset structure); numeric comparison with rounding tolerance - validate.py: overfit scenario auto-perturbs 2 val cases when --val-regression-cases empty (was mis-ACCEPT) - test_gold_verdicts: train_tool_002_fail now correctly attributed to tool_call_error - tests: mock SDK status updated to 'SUCCEEDED'; tool test cases use real data structure 319 tests pass; three scenarios give ACCEPT/NEEDS_REVIEW/REJECT Signed-off-by: popo <18682875253@163.com>
AI Code Review发现的问题🚨 Critical未发现明确阻塞问题。代码逻辑、过拟合检测、SDK 字段映射均与仓库上下文（
|
… param (AI review round 3) - run_pipeline.py: load and score --holdout-evalset via comparator; write to report audit - validate.py: remove unused fixed_categories param from _apply_scenario - three scenarios still give ACCEPT/NEEDS_REVIEW/REJECT; 319 tests pass Signed-off-by: popo <18682875253@163.com>
AI Code Review关键细节： 此外， 确实， 让我给出最终的审查意见。 发现的问题🚨 Critical无。
|
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
- tracing: keep injected reproduce_command covering all non-default CLI args instead of overwriting with the minimal mode/seed fallback - run_pipeline: CI mode exits 2 on NEEDS_REVIEW (REJECT stays 1) - gate: extend critical-case protection to validation-set regressions and wire --critical-cases CLI arg so the protection is reachable
6c63b87 to
f0eb2fa
Compare
AI Code Review测试方面没有安全问题。我已经充分验证了代码评审结果。现在让我整理最终报告。 发现的问题🚨 Critical
|
…roup#139 - gate: compute all 6 checks before branching so audit detail survives early REJECT paths (no_degradation/critical_cases/new_failures/ overfitting/cost_budget always recorded) - optimize: converged by attribution coverage, not iteration-cap proxy - tracing: make finalize() idempotent so report and terminal duration match - comparator: use _CATEGORY_PRIORITY for MISSING_EXPECTED_OUTPUT instead of hardcoded 99 priority - run_pipeline: explicitly warn when live-mode validation/gate runs on scenario-simulated candidates (honest labeling per review Critical)
AI Code ReviewREADME 声称 "Live mode: 真实 SDK AgentOptimizer (需配置 TRPC_AGENT_API_KEY)"，但 我现在已经有足够的信息来撰写审查报告了。让我整理一下发现的问题。 关键发现总结：
让我再仔细检查一个潜在的真正 Bug：在 让我核实一下是否存在可能导致除以零的 让我检查最后一个真正值得关注的问题： 让我核实一下测试中的 等等 —— 在 让我通过检查测试是否实际能通过导入来验证一下 —— 实际上我无法运行。测试 我现在对我的发现充满信心。让我再仔细检查一下 让我最后检查一下 让我核实一下是否存在 现在让我再核实一个真正的 bug 候选：在 我已经完成了审查。让我来写出审查结论。 发现的问题
|
The live graceful-degradation blocks did 'from pipeline.baseline import run_baseline_fake' / 'from pipeline.optimize import OptimizeResult' inside main(). Python treats a from-import as a local binding for the whole function, so the fake-mode path referenced run_baseline_fake before it was assigned → UnboundLocalError crash. Both names are already imported at module top; removed the inner imports. Verified fake (ACCEPT) and live (degrade) both exit 0.
AI Code Review测试执行需要批准；我将跳过它，并依靠静态审查。我已经彻底检查了代码。现在开始撰写审查结论。 基于我的全面审查，代码质量很高且经过了充分测试。我发现了一些值得标记的微小问题。让我来最终确定审查结论。 发现的问题
|
AI Code Review我现在已经有了所有已确认的行号。让我来撰写最终的审查意见。 发现的问题
|
…ent loop - baseline.run_baseline_sdk: aggregate results per case_id (dedupe) instead of per-run, so total_cases/passed/failed are consistent and failed_case_ids has no duplicates when the SDK returns multiple runs per case (NUM_RUNS>1) - run_pipeline: run train/val live baselines in one asyncio.run(asyncio.gather) instead of two separate event loops (avoids loop-reuse/resource warnings)
asyncio.run only accepts a coroutine; passing the gather future raised 'ValueError: a coroutine was expected', which made live baseline always fall back to fake. Wrapped gather in an inner async function. Verified live mode now degrades only for the genuine SDK schema mismatch. Also adds a num_runs>1 per-case aggregation test.
…urb + narrow except - baseline: run_baseline_sdk now uses the user's --optimizer-config via optimizer_config_path instead of silently falling back to default data/optimizer.json when eval_config is missing - validate: _perturb_case guards final_response=None (was AttributeError crash) - optimize: run_optimize_live narrows its catch to (ValueError/KeyError/TypeError) so pipeline bugs propagate instead of being masked as 'optimization failed' - tests: fix math expected-answer operator detection; make overfit REJECT test actually reject (validation_new_failures=1)
AI Code Review这些是有效的回归测试。很好。 现在让我最后检查一下 让我再检查一下 现在，一个真正值得关注的问题：在 让我最后检查一下 我已经完成了审查。让我整理一下最终的发现。真正实质性的问题：
让我再重新考虑一下第 2 点是否真的可达/令人担忧。 让我再重新考虑一下是否还有其他 Critical 问题。让我检查一下 让我验证一下 现在让我再重新考虑一下 让我再检查一下 我已经完成了审查。让我撰写最终总结。让我再仔细检查一下 MD 报告的一个细节，当 让我再验证一个潜在问题： 现在让我做最后的总结。让我重新考虑一下关于测试有效性的发现是否也适用于其他测试。查看 让我写出审查结果。## 发现的问题
|
AI Code Review很好，没有文件被写入。现在开始进行审查。 发现的问题🚨 Critical未发现 Critical 级别问题。
|
- baseline: ImportError now also falls back to trace comparator (matches the 'fell back to trace comparator' warning instead of returning an empty 0% baseline with no scoring) - baseline: all-NOT_EVALUATED results now carry an error so an empty 0% baseline can't silently drive the gate - run_pipeline: live baseline fallback appends the fallback's own errors instead of overwriting them - run_pipeline: end config stage before early return on load failure so the audit trail stays complete - tests: update SDK-missing test for the fake-fallback behavior
AI Code Review
现在，让我更仔细地查看 让我专注于检查 现在，让我看看一个真正的问题：在 让我检查 让我验证 现在让我寻找真正的 bug。让我重新检查 让我检查归因置信度（attribution confidence）： 让我看看一个值得标记的具体问题：在 让我检查一下，当 SDK 导入成功但 现在，一个真正值得关注的问题是 现在让我写审查报告了。让我仔细重新评估最实质性的发现： 发现 1（警告）： 发现 2： 让我重新考虑一下——实际上这里有一个真正的问题。在 live 模式下的 stage 2 中，如果 让我看看真正有影响的发现： 考虑一个有风险的情况：期望 让我检查另一个： 我觉得代码相当稳健。让我确定要报告的实际问题。 让我重新审视一个具体问题： 现在，让我指出一个具体的兼容性/正确性问题。在 让我看看 让我验证 f"| Train Pass Rate | {baseline_train.pass_rate:.1%} | "
f"{candidate_train_rate:.1%} | {candidate_train_rate - baseline_train.pass_rate:+.1%} |"
if candidate_train_rate is not None
else f"| Train Pass Rate | {baseline_train.pass_rate:.1%} | — | — |",这是一个单一元素（条件表达式），如果 现在让我检查一个测试有效性问题： 现在让我写最终审查报告。让我重新考虑哪些是真正值得标记的。 实际问题：
实际上，更仔细地看：在
让我确定最终的问题集。鉴于没有明显的严重 bug，让我专注于真正有影响的问题： 警告 1： Live 模式下的 gate 降级不完整 —— 非过拟合的 REJECT 路径（退化、关键 case、新的训练失败、成本）比较了不兼容的 SDK 与 trace 比较器的 pass rate，但没有被降级，这可能导致 警告 2： 让我重新考虑是否真的有严重问题。让我重新检查 让我再检查一件事： 让我做最后的定稿。我将标记：
让我重新考虑 让我验证一个可能很重要的细微问题：在 好了，我将写出包含最实质性的、可验证的发现以及几个建议的审查报告。让我重新考虑严重性 —— live 模式的不完整降级是代码库中记录最详尽的顾虑，他们明确地降级了 ACCEPT 和过拟合 REJECT，但没有降级其他路径。这可以说是一个真正的差距。我将其标记为警告。 让我再重新考虑一个具体的正确性问题：在 我准备好写审查报告了。让我最后检查一下 现在写报告。 实际上，让我重新考虑一件事，这可能是测试有效性的真正问题。 让我验证
所以两个基准 val case 都通过了，扰动使它们失败 → new_failures=2。测试有效。很好。 现在，最终报告。 基于我对 diff（一个完整的 发现的问题🚨 Critical（无）
|
…uto-select - run_pipeline: in live mode, downgrade ACCEPT and all pass-rate-based REJECTs (degradation/critical/overfit/new-failures) to NEEDS_REVIEW; only the cost-budget REJECT (a real constraint independent of scoring) is kept - validate: overfit auto-selection now picks val cases that are actually perturbable (have final_response.parts), avoiding a silent ValueError when the first two cases can't be regressed
AI Code Review我已经获取了所有必要的信息。现在我来编写审查报告。 发现的问题
|
AI Code Review基于对 发现的问题
|
…to-select, output-dir hint - run_pipeline: live gate keeps REJECT for cost over budget via numeric check (optimization_cost > max_cost_budget) instead of fragile 'exceeds budget' reason-string matching - validate: overfit auto-selection now prefers baseline-passing perturbable cases so a regression is actually produced (avoids spurious ValueError) - run_pipeline: clarify output-dir rejection message to hint running inside the repo (relative paths resolve against CWD)
…unavailable run_baseline_sdk now separates ImportError (SDK unavailable → graceful comparator fallback) from ValueError/KeyError/TypeError (evalset rejected by SDK, e.g. pydantic ValidationError): the fallback result is explicitly marked 'NOT SDK scoring' so a config/data problem isn't mistaken for healthy SDK scoring. (Live gate downgrade from earlier rounds already covers the other review warning.)
AI Code Review确认：live-mode 门控降级逻辑（ 让我再验证一下 我现在已经掌握了所需信息。让我来撰写审查结论。## 发现的问题 🚨 Critical无。
|
…ricter overfit select - run_pipeline: live gate downgrade logic extracted to live_gate_downgrade() (unit-tested: ACCEPT/REJECT -> NEEDS_REVIEW in live, cost-budget REJECT kept) - validate: overfit auto-selection now requires baseline explicitly knows a case AND it passed (unknown case ids no longer default to passing)
AI Code Review已确认 #1。回退（fallback）的 现在让我检查一下 让我再检查一下是否存在实际的 我现在已经有足够的材料了。我来撰写审查报告。I have completed a thorough review of the PR diff (50 files, all under 发现的问题🚨 Critical（无）
|
…ario-error reason - baseline: SDK fallback paths (ImportError/ValueError) now prepend their message to the fake fallback's own errors instead of overwriting, so an evalset-missing/parse error isn't hidden behind 'SDK not available' - run_pipeline: overfit scenario config error (empty val / non-perturbable regression case) now rewrites the gate reason to 'Validation scenario configuration error' instead of misleadingly reporting real 'Overfitting'
AI Code Review
现在我已经有了全面的了解。让我来撰写最终的审查报告，将智能体发现的结果与我自身的验证结果进行综合，并去重保留最有价值的内容。 发现的问题🚨 Critical
|
AI Code Review基于我对 diff 的审查，以下是我的发现。 发现的问题
|
…rror surfacing
- test_large_scale: 50-case eval now asserts failed_cases>=10 & passed>0
(was tautological), so comparator regressions can't pass silently
- test_attribution: train+val aggregation asserted ==7 instead of >=3
- test_pipeline_fake_mode: complete pipeline asserts gate=ACCEPT,
attr.total_failures>0, opt.total_iterations>0 (not just task_id)
- test_pipeline_overfit: use {} audit-only metrics (removed misleading
val_pass_rate that falsely implied metrics drive the gate)
- report: _baseline_to_dict now includes errors so an empty/errored baseline
shows why instead of a bare 0%/0
- baseline: SDK metric_breakdown gains final_response_avg_score key to match
the fake path
(Deferred with note: overfit early-stop & diverse-categories test rewrites,
comparator overflow-invocation behavior change, SDK fallback test mocking.)
…r from live downgrade - run_pipeline: in live mode a NEEDS_REVIEW gate now exits 0 in --ci, since baseline=SDK vs candidate=comparator scores are incomparable and the gate is always downgraded there (previously --mode live --ci always failed with code 2) - live_gate_downgrade: scenario-config-error REJECT is exempted from the live downgrade (structured reason_code), so its real root cause isn't masked as 'scoring differ' like cost-budget REJECT - tests: cover the scenario-error exemption
AI Code Review已确认：超时测试是空洞的（ 发现的问题🚨 Critical
|
| assert gate.decision == GateDecision.ACCEPT | ||
| assert data["gate"]["decision"] == "accept" | ||
|
|
||
| def test_pipeline_with_overfitting_rejection(self, data_dir): |
There was a problem hiding this comment.
测试未真正覆盖声称的风险路径
test_pipeline_with_overfitting_rejection 调用了 run_baseline_fake 但完全不用其结果，转而用硬编码 (0.8, 0.2) 直接调 evaluate_gate；名为早停的测试手动构造两次独立 evaluate_gate，未跑多轮优化循环。测试会通过却无法捕获 baseline 回归或多轮优化回归，给出虚假安全感。建议从 bl_train 派生候选值或真实跑 run_optimize_fake+run_validation_trace 并断言早停行为。
…assthrough, misc - baseline: a case passes only if ALL SDK runs passed (was any-pass, hiding flaky mixed results and inflating pass_rate); test updated - baseline: SDK fallback now uses the caller's PipelineConfig instead of a fresh default, so user thresholds/scenario survive degradation - validate: fake-mode delta defaults a missing side to False (was True), so a single-sided case can't hide a regression as 'unchanged' - optimize: drop the overfit max_rounds+1 dead logic (regression comes from validate scenario perturbation, not extra rounds) - _paths: use warnings.warn instead of print for path-insert failures
AI Code Review我现在已经完全掌握了上下文，我来总结一下审查意见。 发现的问题
|
…candidate_conversation - validate: run_validation_trace now defaults a baseline-missing case to False like run_validation_fake (was True), so id drift/new cases can't hide a regression as unchanged - validate: overfit val-regression targets bypass the candidate_conversation replay shortcut so they are actually perturbed (hidden-sample replay would otherwise skip the regression and hit the new_failures==0 ValueError)
AI Code Review没有硬编码的密钥。很好。现在让我完成评审。 发现的问题🚨 Critical无。
|
…r, deterministic fallback test - comparator: when actual_conversation has MORE invocations than expected, the overflow is now scored 0.0 and classified (final_response_mismatch) instead of being silently ignored (was only handling the missing-output side) - optimize: live-mode prompt_dir (relative 'data/prompts') is anchored to the example dir so running from another CWD no longer crashes live optimization - test_baseline: force the SDK-fallback branch deterministically via monkeypatch (repo ships trpc_agent_sdk, so it wasn't actually testing the fallback path) and assert the fallback errors
2 participants
Description | 描述
Reproducible Evaluation + Optimization closed-loop pipeline.
Implements: baseline evaluation → failure attribution (10 categories) → optimization → validation set comparison → multi-dimensional gate → JSON/MD report with audit trail.
Related Issue | 关联 Issue
Fix #91
Change Type | 修改类型
Test Coverage | 测试覆盖
python -m pytest examples/optimization/eval_optimize_loop/tests/ -vSelf-test Checklist | 自测清单