Skip to content

fix: improve repair feedback for LLM-as-a-Judge validations#1248

Open
akihikokuroda wants to merge 1 commit into
generative-computing:mainfrom
akihikokuroda:repairfeedback
Open

fix: improve repair feedback for LLM-as-a-Judge validations#1248
akihikokuroda wants to merge 1 commit into
generative-computing:mainfrom
akihikokuroda:repairfeedback

Conversation

@akihikokuroda

@akihikokuroda akihikokuroda commented Jun 10, 2026

Copy link
Copy Markdown
Member

Pull Request

Issue

Fix: #1247

Description

When LLM-as-a-Judge validation returns binary yes/no answers, use the requirement
description as the repair reason instead of just 'yes' or 'no'. This applies to all
semantic validations (check(), req() without validation_fn, etc.) regardless of
check_only setting.

Previously, repair feedback would show:

     The following requirements failed before:
     * no

Now it shows:

     The following requirements failed before:
     * The email should have a salutation

This makes repair feedback more actionable and helps models understand what needs
to be fixed during repair iterations in RepairTemplateStrategy and MultiTurnStrategy.

Testing

  • Tests added to the respective file if code was changed
  • New code has 100% coverage if code was added
  • Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Attribution

  • AI coding assistants used: claudecode

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

  • Component
  • Requirement
  • Sampling Strategy
  • Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.

Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>
@github-actions github-actions Bot added the bug Something isn't working label Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: RepairTemplateStrategy includes only yes/no for the failing LLM-as-a-Judge requirement in the repair request

1 participant