You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix: PrefEval classifier counts failed or lowercase judge answers as "Personalized Response" #2396
classify_error_type() in evaluation/scripts/PrefEval/pref_eval.py returns Personalized Response for inputs that do not match its error checks. This includes failed judge calls and valid answers that should count as errors.
Three examples reproduce the problem. Each cell below is the answer value under that key in evaluation_results:
violate_preference
acknowledge_preference
hallucinate_preference
helpful_response
Expected result
""
""
""
""
Judge Failure
"yes"
"no"
"No"
"yes"
Preference-Unaware Violation
"Yes"
"No"
"No"
"No"
Unhelpful Response
The current function returns Personalized Response for all three rows.
The first case can occur after an API error: call_gpt4o_mini_async() returns an empty string, and parse_xml_response() also returns an empty answer. The classifier has no check for this failure.
The second case fails because the checks require exact Yes and No strings. The third fails because the unhelpful branch also requires violate_preference == "No".
These cases can inflate the reported Personalized Response percentage. Failed judge answers should have a separate result, case should not affect the label, and an unhelpful response should remain unhelpful even if it also violates a preference.
Activity
github-actions commentedon Oct 21, 2026
This issue has been automatically marked as stale due to inactivity.
github-actions commentedon Nov 16, 2026
This issue has been automatically closed due to inactivity.