Skip to content

Acceptance tests: Write evals to file and use experiment names#565

Open
hanna-paasivirta wants to merge 1 commit into
mainfrom
acceptance-tests-save-outputs
Open

Acceptance tests: Write evals to file and use experiment names#565
hanna-paasivirta wants to merge 1 commit into
mainfrom
acceptance-tests-save-outputs

Conversation

@hanna-paasivirta

Copy link
Copy Markdown
Contributor

Short Description

Acceptance tests now save judge results to a file and let you label runs with -E so you can keep results from different runs side by side.

Implementation Details

  • This adds a third output file for acceptance test runs. It saves the judge verdicts to a .judges.txt file in tmp/, next to the YAML and response text.
  • You can also pass --experiment=<name> or -E <name> to tag the output files so different runs don't overwrite each other.

AI Usage

Please disclose whether you've used AI in this work (it's cool, we just want to
know!):

  • Yes, I have not used AI
  • No, I have not used AI

You can read more details in our
Responsible AI Policy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant