feat(appkit): managed eval datasets + turn semantics (stack 3/5)#479
Open
MarioCadenas wants to merge 3 commits into
Open
feat(appkit): managed eval datasets + turn semantics (stack 3/5)#479MarioCadenas wants to merge 3 commits into
MarioCadenas wants to merge 3 commits into
Conversation
Let an agent eval sweep a Databricks managed evaluation dataset instead of a
single hardcoded turn, and make one-shot vs multi-turn conversations explicit:
- defineEval gains `dataset: { table, limit? }`. The runner reads the UC table
(via SQLWarehouseConnector over the SQL Statement Execution API — the Python
datasets API needs Spark), then runs the test once per row with the row bound
to `t.input`/`t.expected`.
- readEvalDataset reuses the SQL connector's result transform, which already
JSON-parses `inputs`/`expectations` columns whether stored as JSON or structs.
- resolveWorkspaceClient builds the WorkspaceClient the connector needs, from
the same profile/host/token as resolveDatabricksAuth. CLI: `--warehouse`.
- Each dataset row gets a fresh driver so rows are independent conversations
(no thread/context bleed between rows).
- Turn semantics: consecutive `t.send`s share one thread (multi-turn); new
`t.reset()` starts a fresh conversation for independent one-shot checks in a
single test.
- Example dataset.eval.ts + unit tests for the reader and reset.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
…al_dataset Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
LLM-judge assertions (`t.judge.*`) now fail the eval on a miss, like the deterministic assertions — previously they were soft-by-default and a failing guideline left the run green. `.atLeast(n)` sets the pass threshold without demoting; chain `.soft()` to keep a judge as a tracked-only metric. Surfaces real signal: e.g. a dataset row whose guidelines the agent can't satisfy now fails the run (non-zero exit) instead of passing silently. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 3/5 · targets
pr/agent-evals-2-framework(review after #2).Managed datasets and explicit conversation semantics.
dataset: { table }sweeps a Databricks managed evaluation dataset (a Unity Catalogcatalog.schema.tableread via the SQL Statement Execution API — the Python datasets API needs Spark). One run per row;t.input/t.expectedbind each row.--warehouse+ auth required.send), multi-turn (consecutivesends share a thread), andt.reset()for an independent turn in one test. Each dataset row gets a fresh driver so rows don't bleed..atLeast(n)sets the threshold,.soft()opts out.