Classify top-level reviewer feedback with the author-action binary - #196
Merged
trask merged 3 commits intoJul 30, 2026
Merged
Conversation
Pull request dashboard statusMerged · refreshed 2026-07-30 15:20 UTC Status above doesn't look right?
|
trask
force-pushed
the
pr-dashboard-reviewer-feedback-binary
branch
from
July 30, 2026 13:39
bf2c60c to
3d2eb75
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Replaces top-level reviewer feedback triage with an author-action binary and aligns evaluation and fallback behavior.
Changes:
- Adds binary reviewer-feedback classification and routing.
- Updates evaluator vocabulary mappings.
- Updates classification and fallback tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
classification.py |
Adds the binary prompt, verdict mapping, and classifier integration. |
eval/score_reviewer_feedback.py |
Maps binary verdicts to evaluation labels. |
test_top_level_actions.py |
Updates tests for binary verdicts and author-safe fallbacks. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
maryliag
approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #151.
A maintainer reviewed opentelemetry-python-genai#147 saying the change papers over a deeper problem rather than fixing it. The dashboard read that as needing nothing from the author, so the pull request stopped being listed as theirs and the author was never told anything was outstanding. It was not a one-off: the evaluation set records that comment with no settled answer at all, because over five runs the shipped classifier never twice agreed with itself about it.
The cause is the question being asked. The classifier is given an open question — who has the next action for this feedback — and answers author, none, or unclear, which leaves it deciding on behalf of people it knows nothing about. It now answers the narrow question the evaluation set in #183 was built to measure: does this comment need something from the pull request author. On that same pull request the maintainer's review now comes out as the author's, while a reviewer's "PR looks good to me" still comes out as needing nothing, so it is discriminating rather than simply routing everything to the author.
Across the 441 pull request corpus, 611 top-level comments split 368 to the author and 243 to nobody.
Feedback the classifier cannot decide, or fails to read, now records that the author holds it instead of recording it as undecided. Both already reached the author, so no pull request changes hands; what changes is that the recorded reason says which way it failed rather than leaving that to a default further down.
One fix travels with this because it only matters once the classifier is live: the evaluation harness was scoring this classifier against answers it does not give, so every result counted as unanswered and the whole run would have read as total failure rather than a scoring bug.