Name the eval fixture's fields for what they mean

[trask]

Rename fields in the reviewer-feedback evaluation set and its scorer. No case is relabelled and no classification is re-run; the migration is a field rename over the existing data.

substantive and noise described the comment rather than the decision, and noise was unfair to an approval or an automation command, neither of which is noise so much as something the author owes nothing on. They become author_action and no_author_action. The mapping from the shipped classifier's answers survives, but only one of its entries now carries meaning: author and none map to the same concept under a clearer name, leaving unclear -> author_action as the single rule a reader has to learn, which is the fail-safe direction an unclassifiable item takes.

stability previously carried unobserved alongside stable and flaky, which conflated two things: whether a case counts toward the metrics, and whether its runs agreed. Coverage now lives in role, which is scored or context, so the two can be read separately. stability stays null for every context case, because an unanswered run is not an observation and this migration does not relabel data. Recording agreement from a partial run set would give 14 of the 17 context cases a stability, some on as few as two observations; that is a data change and a judgement about how much evidence is enough, so it is left as follow-up rather than folded into a rename.

flaky named both the baseline's inconsistency and the candidate's, on adjacent lines of the same report. The scorer's metric is now inconsistent, leaving flaky to mean only the recorded property.

observed_runs and observed_actions read as though they hold runs rather than answers, so they become run_labels and run_actions. baseline and adjudicated hold the same kind of value and are now recorded_label and adjudicated_label. The scorer's three exclusion buckets were near synonyms and are now no_answer, partial_answer, and tied.

[opentelemetry-pr-dashboard]

Pull request dashboard status

Waiting on reviewers · refreshed 2026-07-30 15:25 UTC

Review the latest changes.

Status above doesn't look right?

• Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
• Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

[Copilot]

Pull request overview

Renames reviewer-feedback evaluation fields and metrics to reflect their semantics without relabelling cases.

Changes:

• Renames fixture labels, roles, run fields, and adjudication fields.
• Renames scorer exclusion and inconsistency metrics.
• Updates fixture and scoring tests for the new schema.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.

File	Description
eval/reviewer_feedback_cases.json	Migrates evaluation data to the renamed schema.
eval/score_reviewer_feedback.py	Updates scoring and reporting terminology.
test_eval_fixture.py	Validates the migrated fixture schema.
test_eval_scoring.py	Updates scorer tests for renamed fields and metrics.

Comments suppressed due to low confidence (1)

.github/scripts/pull-request-dashboard/eval/score_reviewer_feedback.py:254

• When drift is non-empty, this report branch still reads d['baseline'] below, but drift entries now carry recorded_label. Every detected regression therefore raises KeyError instead of printing its details. Update that lookup to the renamed field.

        "Moving a case to author_action keeps the pull request with its author, which "
        "one /dashboard route:reviewers comment corrects. The reverse does not.\n"

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

.github/scripts/pull-request-dashboard/test_eval_fixture.py:58

• The fixture's top-level note still tells future adjudicators to set adjudicated, while this migration makes adjudicated_label the field the validator and scorer read. Following that documented instruction would add an ignored field and leave the case out of accuracy scoring; update the fixture note to name adjudicated_label.

                for field in ("recorded_label", "adjudicated_label"):

.github/scripts/pull-request-dashboard/test_eval_scoring.py:54

• This assertion uses the renamed inconsistent metric, but the test method still calls candidate disagreement “flaky.” That retains the exact adjacent terminology ambiguity this migration is intended to remove; rename the test to use “inconsistent.”

        self.assertEqual(1, len(summary["inconsistent"]))

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

.github/scripts/pull-request-dashboard/test_eval_fixture.py:81

• This still uses role to determine stability, so the new fields remain coupled despite the PR defining them as independent dimensions. A fully observed context case or a partially observed scored case would be rejected even though role controls metric inclusion and run coverage should control stability; branch on run_labels coverage instead.

                if case["role"] == "context":
                    self.assertIsNone(case["stability"])
                    self.assertIsNone(case["recorded_label"])
                    self.assertIn(None, case["run_actions"])
                    continue

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

.github/scripts/pull-request-dashboard/test_eval_fixture.py:80

• This still couples coverage to stability by forcing every context case to have null stability. The PR description says these dimensions are independent and that a partial case can still have agreed runs; for example, reviewer_feedback_cases.json:16050-16061 has four identical author_action labels and only one missing action, yet this assertion requires null. Derive stable/flaky from the available run_labels when there are enough to judge, reserving null only for insufficient observations, while role alone controls scoring.

                    self.assertIsNone(case["stability"])

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated no new comments.