LaunchGate Eval Action

Run LaunchGate eval suites in your GitHub Actions workflow and gate deployments on results.

Usage

- uses: AivaStack/eval-action@v0
  with:
    api-key: ${{ secrets.LAUNCHGATE_API_KEY }}
    suite: my-eval-suite
    output: ${{ steps.my-step.outputs.result }}

Inputs

Input	Required	Default	Description
api-key	Yes		LaunchGate API key
suite	Yes		Eval suite slug to run
output	Yes		Output text to evaluate
input	No	{}	Input payload as JSON string
fail-on-abort	No	true	Fail the action when eval status is "aborted"
comment	No	true	Post results as a PR comment
api-url	No	https://api.launchgate.ai	LaunchGate API URL

Outputs

Output	Description
status	Run status (cleared, aborted, error, skipped)
pass-rate	Pass rate as a decimal (0.0 - 1.0)
run-id	The eval run ID

Example: Gate deployment on eval results

name: Eval Gate
on:
  pull_request:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Generate AI output
        id: ai
        run: echo "output=Your AI response here" >> $GITHUB_OUTPUT

      - name: Run LaunchGate eval
        uses: AivaStack/eval-action@v0
        with:
          api-key: ${{ secrets.LAUNCHGATE_API_KEY }}
          suite: rag-faithfulness
          output: ${{ steps.ai.outputs.output }}
          input: '{"context": "...", "query": "..."}'

When the eval passes, you'll see "Cleared for launch" in the PR comment. When it fails, the action exits with code 1 and posts "Launch aborted" with a detailed breakdown.

License

MIT