feat(appkit): agent eval framework, judge, mlflow connector (stack 2/5)#478
Open
MarioCadenas wants to merge 3 commits into
Open
feat(appkit): agent eval framework, judge, mlflow connector (stack 2/5)#478MarioCadenas wants to merge 3 commits into
MarioCadenas wants to merge 3 commits into
Conversation
…runs eve-style eval authoring (defineEval + t-context + matchers) discovered from config/agents/<id>/evals/*.eval.ts and run via 'appkit agent eval' against a running app. Streams per-eval progress and gates CI via exit code. When Databricks creds + an experiment are set, it creates a real MLflow evaluation run (mlflow.runType=genai_evaluate): each eval's trace links to the run, pass/fail is written as feedback assessments, and aggregate metrics are logged. All via the MLflow REST API. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Extend the agent eval framework and tighten MLflow output to match the native `mlflow.genai.evaluate` experience: - LLM-as-judge via autoevals (factuality, closedQA, custom), pointed at a Databricks serving endpoint; exposed through `t.judge.*`. - One Feedback assessment per assertion (judges as LLM_JUDGE with score + rationale) plus an overall `appkit_eval`; assessment names sanitized to `[A-Za-z0-9_-]` since the API rejects dots. - Trace-table parity: set Request/Response previews and the `mlflow.traceName` tag (the Trace-name column reads the tag, not the span name). - Eval runs carry `mlflow.source.name`/`type` tags so linked traces show Source and Run name; live chat traces have no run so those stay empty. - Example judge eval under config/agents/query/evals. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Introduce connectors/mlflow as the shared REST + auth layer for MLflow,
so the eval runner (and future callers) stop threading host/token and
hand-rolling fetch/URL logic:
- MlflowClient owns {host, token}: normalizes the host once, exposes
post() (throws) for runs/* and postResult() (structured failure) for
best-effort assessment writes, plus servingEndpointsUrl() for the judge.
- resolveDatabricksAuth() mints an OAuth bearer from a CLI profile via the
SDK WorkspaceClient (the AppKit-native path), so `agent eval` no longer
requires a hand-set DATABRICKS_TOKEN. Adds an `--profile` flag.
- Eval run create/finish, assessment reporting, and the judge take the
client; the agents plugin's host normalization now delegates to the
connector's normalizeHost.
The mlflow-tracing SDK wrapper stays in the agents plugin: it manages a
process-global provider (like TelemetryManager) and has an agent-shaped
API, so it isn't a connector.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 2/5 · targets
pr/agent-evals-1-tracing(review after #1).The core eval framework, plus LLM-as-judge and the MLflow REST connector.
defineEval): drive an agent over HTTP against a running app; assert witht.succeeded(),t.calledTool(),t.check(value, matcher)(includes/equals/matches). Gate-by-default,.soft()to demote.genai_evaluate; each turn's trace links viamlflow.sourceRun; per-assertion feedback written via the assessments REST API.t.judge.factuality/closedQA/custom) via autoevals → a Databricks serving endpoint.connectors/mlflow:MlflowClient(host/token, post/postResult, serving URL) +resolveDatabricksAuth/resolveWorkspaceClient(OAuth from a CLI profile — no hand-set PAT). Extracted so both evals and future callers share the REST/auth layer.appkit agent evalCLI.Squashed history note: contains the framework, judge, and connector-extraction commits.