feat(appkit): eval suite features — concurrency, retries, reporters (stack 4/5)#480
Open
MarioCadenas wants to merge 8 commits into
Open
feat(appkit): eval suite features — concurrency, retries, reporters (stack 4/5)#480MarioCadenas wants to merge 8 commits into
MarioCadenas wants to merge 8 commits into
Conversation
…al CLI Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
The eval driver captured tool-call names but discarded their arguments. Parse the function-call `arguments` JSON into `DriveResult.toolCallDetails` (the later `done` event's fuller args win over the initial `added`), and expose `t.calledToolWith(name, expected)` — passes when a call to `name` had args that deep-contain `expected` (nested-aware partial match; extra args ignored). Gate by default, like `calledTool`. `toolCalls: string[]` is unchanged. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
A managed-dataset row's `inputs.messages` can be a full conversation, not just one question. Add `userTurns(input)` to extract every user-message content in order; the example dataset eval replays them against one thread so the agent sees the accumulating conversation. Interleaved assistant/system turns are ignored — the agent generates its own responses. Single-user-turn rows are unchanged (one send). Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
The runner drove every eval and dataset row serially. Add `maxConcurrency` (RunEvalsOptions + `--concurrency <n>`, default 1 = serial) so up to N rows run in parallel — a real speedup for dataset sweeps, where each row is an HTTP turn. A `runBounded` worker pool places each result in its input slot, so output order matches discovery/row order regardless of completion order (reportToMlflow and the summary depend on it). Work is expanded serially first to preserve load and dataset-read error handling; each row still gets a fresh driver. This also makes the previously-declared-but-unwired `EvalConfig.maxConcurrency` real. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Add `--min-pass-rate <0..1>` to the agent eval CLI: instead of requiring every eval to pass, exit non-zero only when the aggregate pass rate falls below the threshold. `summarize()` now returns `passRate` (passed / scored, excluding skips; 1 when nothing scored). Without the flag, behavior is unchanged (any gate failure fails the run). Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Wire three eval fields that were declared but inert:
- `timeoutMs` — runEval races the test against a per-eval timeout and records a
clean non-passing result ("eval timed out after Nms") instead of hanging.
Precedence: def.timeoutMs > runner/CLI --timeout > unbounded. Timer is always
cleared.
- `evals.config.ts` (defineEvalConfig) — discovered per-agent and loaded via the
tsx loader; its maxConcurrency/timeoutMs apply as defaults (CLI flag > config >
built-in). Judge model still comes from the CLI (needs creds the config lacks).
- `tags` + `--tag <tag...>` — run only evals whose tags intersect the filter.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Add `--retries <n>` / `RunEvalsOptions.retries`: re-run an eval up to N extra times when it fails on an infrastructure error (a thrown error or timeout — `result.error` set), to absorb transient turn/stream flakiness. Assertion failures are never retried — a wrong reply is real signal, and retrying a flaky judge until it passes would corrupt the result. Each attempt gets a fresh driver. Extracted as `runWithRetries` (unit-tested for attempt counting, stop-on-success, and no-retry-on-assertion-failure). Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Add `--reporter <text|json|junit>` (default text) and `--output <file>` to the
agent eval CLI. `formatResultsJson` emits `{summary, results}`; `formatResultsJUnit`
emits a `<testsuite>` with a `<testcase>` per eval (`<failure>`/`<skipped>` as
appropriate, all values XML-escaped). In json/junit mode the per-eval streaming
is suppressed and human banners go to stderr so stdout stays clean for piping or
`--output`; exit-code and pass-rate gating are unchanged.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 4/5 · targets
pr/agent-evals-3-datasets(review after #3).CI-grade suite features.
t.calledToolWith(name, expected)(deep-partial arg match).userTurns(input)replays a row's fullmessages[]against one thread.--concurrency N/maxConcurrency— bounded, order-preserving pool.--min-pass-rate 0..1— gate on aggregate rate instead of all-must-pass.timeoutMs, load per-direvals.config.ts,--tagfiltering.--retries N— re-run only on infra failures (turn/timeout), never on assertion failures.--reporter json|junit+--outputfor CI.Includes two small dedup refactors (credential resolution; agent-dir listing / int parsing).