[fix][evaluation] write NULL (not 0) for absent evaluator score to fix N/A miscounting in aggregation - #592
Open
xueyizheng wants to merge 1 commit into
Open
[fix][evaluation] write NULL (not 0) for absent evaluator score to fix N/A miscounting in aggregation#592xueyizheng wants to merge 1 commit into
xueyizheng wants to merge 1 commit into
Conversation
## 背景 nil/null 分数是评估器协议里表达"不适用(N/A)"的合法信号(score 为 optional/ 指针, 评估器无法评价输入时返回 nil 分 + reasoning)。这类 record 会带 status=Success: - 异步上报路径 status 由外部 agent 回调传入, 对 Success 不校验是否带分 - 同步 prompt 路径解析成功即 Success, 也不禁止 nil 分 ## 问题 UpdateEvaluatorRecordResult(异步路径, 自 #421 引入)用 `var score float64` 值类型, nil 分被写成 0 落库(而非 NULL)。此前聚合从 output_data 反序列化取分、对 nil 分跳过, 不读 score 列, 故该脏 0 无影响; 但 #590 聚合改为直读 score 列(止 OOM)后, `status=Success AND score IS NOT NULL` 挡不住值为 0 的行, 会把 N/A 误当 0 分算进 均值/分布, 拉低聚合结果。 ## 修复 - repo/DAO 的 score 参数 float64 → *float64, nil 分写 NULL 而非 0, 让 score 列语义诚实 - 与同步 create 路径(ConvertEvaluatorRecordDO2PO 一直写 *float64/NULL)对齐 - 补 DAO 层 NULL/非 NULL 写入单测 + repo 层 nil 用例; 重生成 mock 签名 修复后聚合窄查询 score IS NOT NULL 与老路径"跳过 nil"语义完全一致。 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## main #592 +/- ##
=======================================
Coverage ? 77.51%
=======================================
Files ? 679
Lines ? 80914
Branches ? 0
=======================================
Hits ? 62720
Misses ? 14463
Partials ? 3731
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
A nil/null score is a legitimate signal in the evaluator protocol for expressing "Not Applicable (N/A)": when an evaluator cannot judge an input (e.g. an image with no text), it returns a nil score plus a reasoning. score is optional in the SPI DTO and a pointer in the DO by design. Such records can carry status=Success:
Async report path: status is supplied by the external agent callback (POST /v1/loop/evaluation/evaluators/result); the converter does not require a score when status=SUCCESS, and ReportEvaluatorInvokeResult performs no status/score consistency check.
Sync prompt path: a successful parse yields Success, and a nil score is not rejected.
Problem
UpdateEvaluatorRecordResult (async path, introduced in #421) used a value-typed var score float64, so a nil score was coerced to 0 and persisted as 0 (not NULL).
Previously this was harmless: aggregation deserialized output_data and skipped nil scores (hasScore=false), never reading the score column. After #590 switched aggregation to read the score column directly (to avoid OOM), the filter status=Success AND score IS NOT NULL can no longer exclude a literal 0, so N/A records are miscounted as 0 and drag down the average / distribution.
Fix
Change the score parameter from float64 to *float64 in both the repo impl and the DAO, writing NULL (not 0) when there is no score, so the score column semantics are honest.
This aligns the async update path with the sync create path (ConvertEvaluatorRecordDO2PO, which has always written a *float64 / NULL).
Add DAO-level tests for NULL vs. concrete-value writes, a repo-level nil case, and regenerate the mock signature.
After this fix, the narrow aggregation query's score IS NOT NULL filter is fully consistent with the old in-memory "skip nil" semantics.
━━━━━━━━━━━━━━━━━━━━━━━━━