feat(appkit): agent eval framework, judge, mlflow connector (stack 2/5) - #478
Open
MarioCadenas wants to merge 23 commits into
Open
feat(appkit): agent eval framework, judge, mlflow connector (stack 2/5)#478MarioCadenas wants to merge 23 commits into
MarioCadenas wants to merge 23 commits into
Conversation
MarioCadenas
force-pushed
the
pr/agent-evals-1-tracing
branch
from
August 5, 2026 15:22
412b08d to
f517306
Compare
MarioCadenas
force-pushed
the
pr/agent-evals-2-framework
branch
from
August 18, 2026 12:57
0b55719 to
1431ce7
Compare
Contributor
📦 Bundle size reportCompared against
|
| dist | raw | gzip |
|---|---|---|
| JS (runtime) | 1.1 MB (+49 KB) | 396 KB (+20 KB) |
| Type declarations | 406 KB (+20 KB) | 145 KB (+9.3 KB) |
| Source maps | 2.2 MB (+94 KB) | 742 KB (+37 KB) |
| Other | 11 KB | 3.7 KB |
| Total | 3.7 MB (+164 KB) | 1.3 MB (+66 KB) |
Per-entry composition (own code — deps external (as shipped))
| Entry | Initial (gz) | Lazy (gz) | Total (gz) | node_modules (min) | Own code (min) |
|---|---|---|---|---|---|
. |
95 KB (+338 B) | 2.5 KB | 98 KB (+338 B) | external | 312 KB (+1.3 KB) |
./beta |
81 KB (+5.7 KB) | 455 B (-2 B) | 81 KB (+5.7 KB) | external | 241 KB (+16 KB) |
./testing |
17 KB (+326 B) | 0 B | 17 KB (+326 B) | external | 51 KB (+1.3 KB) |
./tsdown |
520 B | 0 B | 520 B | external | 813 B |
./type-generator |
22 KB | 0 B | 22 KB | external | 65 KB |
Chunks:
| Entry | Chunk | Load | Size (gz) |
|---|---|---|---|
. |
index.js |
initial | 91 KB |
. |
utils.js |
initial | 4.0 KB |
. |
remote-tunnel-manager.js |
lazy | 2.5 KB |
./beta |
beta.js |
initial | 64 KB |
./beta |
stream-manager.js |
initial | 5.8 KB |
./beta |
wide-event-emitter.js |
initial | 3.2 KB |
./beta |
databricks.js |
initial | 3.2 KB |
./beta |
configuration.js |
initial | 2.1 KB |
./beta |
service-context.js |
initial | 1.3 KB |
./beta |
client.js |
initial | 434 B |
./beta |
client-options.js |
initial | 219 B |
./beta |
supervisor-api.js |
lazy | 192 B |
./beta |
databricks.js |
lazy | 141 B |
./beta |
index.js |
lazy | 122 B |
./testing |
index.js |
initial | 17 KB |
./tsdown |
index.js |
initial | 520 B |
./type-generator |
index.js |
initial | 22 KB |
@databricks/appkit-ui
npm tarball (packed): 350 KB — gzipped download (dist + bin; excludes release-only docs/NOTICE).
| dist | raw | gzip |
|---|---|---|
| JS (runtime) | 395 KB | 132 KB |
| Type declarations | 229 KB | 84 KB |
| Source maps | 766 KB | 253 KB |
| CSS | 16 KB | 3.2 KB |
| Total | 1.4 MB | 473 KB |
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
| Entry | Initial (gz) | Lazy (gz) | Total (gz) | node_modules (min) | Own code (min) |
|---|---|---|---|---|---|
./js |
5.3 KB | 49 KB | 55 KB | 208 KB | 14 KB |
./js/beta |
20 B | 0 B | 20 B | 0 B | 0 B |
./react |
432 KB | 49 KB | 481 KB | 1.3 MB | 177 KB |
./react/beta |
1.0 KB | 0 B | 1.0 KB | 0 B | 1.9 KB |
Chunks:
| Entry | Chunk | Load | Size (gz) |
|---|---|---|---|
./js |
index.js |
initial | 5.2 KB |
./js |
chunk |
initial | 120 B |
./js |
apache-arrow |
lazy | 49 KB |
./js/beta |
beta.js |
initial | 20 B |
./react |
index.js |
initial | 430 KB |
./react |
tslib |
initial | 2.1 KB |
./react |
apache-arrow |
lazy | 49 KB |
./react/beta |
beta.js |
initial | 1.0 KB |
⚠️ Over budget: a package's shipped tarball, or a browser entry's consumer bundle (deps included), grew by more than 5% (and >10 KB). This check will fail — reduce the size, or acknowledge the increase by updatingbundle-size-baseline.json.
Contributor
🤖 AppKit PR bot🔬 Run evalsStart an eval for this PR from the evals-monitor app: Go to Evals Monitor → 📦 Try this PR's app templateScaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh run download 33643196657 -R databricks/appkit -n appkit-template-0.70.0-pr.4be0ca0-pr-agent-evals-2-framework-478 -D appkit-pr-478 \
&& unzip -o "appkit-pr-478/appkit-template-0.70.0-pr.4be0ca0-pr-agent-evals-2-framework-478.zip" -d "appkit-pr-478" \
&& databricks apps init --template "appkit-pr-478"The template pins |
MarioCadenas
force-pushed
the
pr/agent-evals-2-framework
branch
2 times, most recently
from
August 21, 2026 10:37
e32afdd to
65de478
Compare
MarioCadenas
force-pushed
the
pr/agent-evals-2-framework
branch
from
August 31, 2026 13:48
5da7ea4 to
1f15f16
Compare
…runs eve-style eval authoring (defineEval + t-context + matchers) discovered from config/agents/<id>/evals/*.eval.ts and run via 'appkit agent eval' against a running app. Streams per-eval progress and gates CI via exit code. When Databricks creds + an experiment are set, it creates a real MLflow evaluation run (mlflow.runType=genai_evaluate): each eval's trace links to the run, pass/fail is written as feedback assessments, and aggregate metrics are logged. All via the MLflow REST API. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Extend the agent eval framework and tighten MLflow output to match the native `mlflow.genai.evaluate` experience: - LLM-as-judge via autoevals (factuality, closedQA, custom), pointed at a Databricks serving endpoint; exposed through `t.judge.*`. - One Feedback assessment per assertion (judges as LLM_JUDGE with score + rationale) plus an overall `appkit_eval`; assessment names sanitized to `[A-Za-z0-9_-]` since the API rejects dots. - Trace-table parity: set Request/Response previews and the `mlflow.traceName` tag (the Trace-name column reads the tag, not the span name). - Eval runs carry `mlflow.source.name`/`type` tags so linked traces show Source and Run name; live chat traces have no run so those stay empty. - Example judge eval under config/agents/query/evals. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Introduce connectors/mlflow as the shared REST + auth layer for MLflow,
so the eval runner (and future callers) stop threading host/token and
hand-rolling fetch/URL logic:
- MlflowClient owns {host, token}: normalizes the host once, exposes
post() (throws) for runs/* and postResult() (structured failure) for
best-effort assessment writes, plus servingEndpointsUrl() for the judge.
- resolveDatabricksAuth() mints an OAuth bearer from a CLI profile via the
SDK WorkspaceClient (the AppKit-native path), so `agent eval` no longer
requires a hand-set DATABRICKS_TOKEN. Adds an `--profile` flag.
- Eval run create/finish, assessment reporting, and the judge take the
client; the agents plugin's host normalization now delegates to the
connector's normalizeHost.
The mlflow-tracing SDK wrapper stays in the agents plugin: it manages a
process-global provider (like TelemetryManager) and has an agent-shaped
API, so it isn't a connector.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Post-rebase integration with main's biome->oxc migration (#538) and the SDK-facade boundary rule (#534): - Route the mlflow connector's auth through createWorkspaceClient instead of importing @databricks/sdk-experimental directly (oxlint no-restricted-imports); behaviour is unchanged. - Apply oxfmt import grouping to the evals + connector files authored before the migration. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Remove dead surface flagged in review:
- EvalConfig + defineEvalConfig: nothing loads evals.config.ts (the runner
never reads it), so the type and helper configured nothing.
- EvalDefinition.tags / .timeoutMs: never read by the runner.
- discover: replace hand-rolled statSync/recursion with
readdirSync({ recursive, withFileTypes }).
- resolveEvalDefault: drop the seen-Set, redundant with the loop bound.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
…er-agent layout Adapts the eval framework to #533's unified agent discovery: agents now live in server/agents/<id>/agent.{md,ts} (config/agents is a deprecated fallback), so evals move next to them. - discover.ts scans server/agents/<id>/evals/, reusing CODE_AGENTS_SOURCE_DIR and agentDirNames so eval discovery follows the same folder-selection policy (symlinked agent folders included) as the agents plugin. - Relocate the dev-playground example evals config/agents/query/evals -> server/agents/query/evals. - Update discover.test fixtures + doc strings. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Regenerated index + sidebar pick up main's defineManifest (#485) alongside the eval-framework entries. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
…assing on errors The eval HTTP driver had two failure modes: - A hung agent (blocked tool / stalled model) never ends the SSE stream; heartbeats keep reader.read() producing bytes, so the read loop spins forever and the sequential suite wedges. Add an AbortSignal.timeout (default 120s, configurable via RunEvalsOptions.timeoutMs) and fail the turn on abort. - A thrown exception is framed by SSEWriter.writeError as `event: error` with a payload that has no `type` field, so applyEvent never sees it and the turn reports succeeded:true (false PASS). Track the SSE `event:` line and treat `error` as a failed turn. Add http-driver.test.ts covering both plus a happy-path guard. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
MarioCadenas
force-pushed
the
pr/agent-evals-2-framework
branch
from
September 1, 2026 13:20
1f15f16 to
274ec9d
Compare
… to 4.3.1) The rebase's git auto-merge left autoevals's transitive js-yaml pinned at 4.2.0 while the top-level entries unified to 4.3.1, leaving 4.2.0 with no package entry. pnpm install --frozen-lockfile (CI) fails this as ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY, blocking every job at install. autoevals declares js-yaml ^4.1.0, which 4.3.1 satisfies. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Behavior-preserving extractions bringing six functions from 10-25 down to <=8 (oxlint complexity, max 8): - http-driver: split send into buildRequestBody + drainSse + handleSseLine; applyEvent delegates to recordToolCall + applyMetadata. The SSE pipeline is now unit-testable end to end instead of only via a live server. - run-evals: extract runOne, maybeConfigureJudge, finalizeMlflow. - eval CLI: extract resolveMlflow, resolveJudge, makeProgressReporter, formatFailureLine, printMlflowOutcome. - mlflow-report: extract assertionAssessment + overallAssessment. - mlflow/auth: extract extractBearer + resolveViaSdk. No behavior change; 35 eval/mlflow tests pass, typecheck + lint clean. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Replace the bespoke SSE line parser (handleSseLine + drainSse) in the eval HTTP driver with the existing readSseEvents reader from stream/sse-reader. Net -29 lines, and gains the reader's maxLineChars/maxBufferChars DoS bounds that the hand-rolled buffer lacked. Behavior preserved: timeouts still yield succeeded:false (explicit signal.aborted check, since the reader cancels cleanly rather than throwing) and stream-level error frames still fail the turn via the event name. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
- http-driver: reply now reads terminal `message` output-item text (replacing deltas), so non-streaming/LangChain adapters and delta-then-corrected turns no longer return an empty/stale reply. - http-driver: set `redirect: "manual"` so custom auth headers are not replayed across a cross-origin redirect. - agents: drop unused `traceTool` import (oxlint no-unused-vars). - deps: exact-pin `autoevals` (0.3.0) to match the repo convention. Adds http-driver tests for the message-only and delta-then-message paths. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Replace the sequential loop in runEvalsInDir with a bounded worker pool (new internal `mapPool` helper) so independent eval turns overlap instead of summing their latencies. Adds a `concurrency` option to RunEvalsOptions (default 4, clamped to [1, total]) and a `--concurrency` CLI flag; the default stays at/below the server's maxConcurrentStreamsPerUser (5) to avoid the 429 guard. Results preserve discovery order. The CLI progress reporter now prints one full line per completion (via formatEvalHeadline) since concurrent starts would otherwise interleave. Regenerates the API reference, which also syncs pre-existing doc drift in the PR's own surface (HttpDriverOptions.timeoutMs, buildAssessments, resolveDatabricksAuth). Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
reportToMlflow issued one serial POST per assessment (Sigma(assertions+1) round trips as a tail phase). Flatten all assessments and write them through the bounded pool instead; written/skipped/failures semantics are unchanged. Extract `mapPool` from run-evals into `evals/pool.ts` so both the eval runner and the reporter can share it without a circular import; move its test to pool.test.ts. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
- judge: snapshot and restore the OPENAI_* env vars in a run-level finally (teardownJudge) so the judge bearer doesn't linger in process.env after the run, readable by any imported eval code. - cli: catch runEvalsInDir setup failures (e.g. a bad --experiment) and print a clean error + non-zero exit instead of an unhandled promise rejection. - mlflow client: doc-comment that post/postResult embed up to 500 chars of the upstream body — safe for the dev CLI, must not be relayed to end-user responses. Regenerates Class.MlflowClient.md. Adds a judge env-lifecycle test. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
The app exports each turn's MLflow trace asynchronously, so the eval runner's assessment writes (issued right after the run) could beat the trace into the store and 404 with "Trace not found". - reportToMlflow: retry an assessment write on 404 with linear backoff (5 attempts) to ride out ingestion lag; non-404 failures don't retry. - agents: flushTrace() after the turn's span ends when an mlflowRunId is present (eval mode), so the trace is exported before the eval attaches assessments. Live chat turns skip the flush. Adds reportToMlflow tests for the 404-retry and non-404 no-retry paths. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
15f2fb1 accidentally included dev-playground generated files (metric-views metadata + appkit-types/*.d.ts) that a background build regenerated without warehouse connectivity, degrading typed query results to `unknown`. Restore them and untrack the newly-added database.d.ts; they're unrelated to the assessment-write fix. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
…ls-2-framework Takes #545's rewritten agents/mlflow.ts + agents.ts (single-provider @mlflow/core UC tracing) over the eval branch's mlflow-tracing versions; package.json is the union (@mlflow/core + sdk-experimental + sdk-trace-node from #545, autoevals from evals). Eval's updateTracePreview + flushTrace are dropped here (superseded by Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com> #545's tracing + mlflow-report retry-on-404); re-port as follow-ups if wanted.
…r' into pr/agent-evals-2-framework Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com> # Conflicts: # pnpm-lock.yaml
MarioCadenas
changed the base branch from
main
to
fix/agents-mlflow-single-provider
September 2, 2026 14:39
UC-backed experiments emit V4 trace ids (`trace:/<location>/<hex>`), which the
Databricks CreateAssessment API addresses as
`/api/4.0/mlflow/traces/{location}/{hex}/assessments` with the location and
bare hex as separate path segments and the bare assessment as the body — not
the V3 `/api/3.0/mlflow/traces/{id}/assessments` + `{ assessment }` wrapper the
eval used, which 400s ("request id must map to a trace") on every UC run.
reportToMlflow now dispatches on the trace-id form: V4 UC ids -> the V4
endpoint/bare body; classic `tr-...` ids keep the V3 path/wrapper. Mirrors
mlflow's parse_trace_id_v4 + databricks_rest_store.create_assessment.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
UC-backed experiments store traces in a Delta table, so writing an assessment to a V4 trace (`trace:/<location>/<hex>`) runs a SQL write and Databricks requires `?sql_warehouse_id=` on the request — without it the V4 assessments endpoint 400s with "Could not resolve a SQL warehouse ID". Resolve the warehouse in the eval process (a separate process from the app, so it needs its own env) from --warehouse-id, else MLFLOW_TRACING_SQL_WAREHOUSE_ID, else DATABRICKS_WAREHOUSE_ID, and append it to the V4 request. Classic V3 writes are unchanged (no warehouse). Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
…uard mlflow finalize
Three review fixes for the agent eval framework:
- pool.ts: coerce a non-finite `--concurrency` (NaN) to 1 before the [1,len]
clamp. Otherwise Math.min(NaN, len) is NaN, Array.from({length: NaN}) is [],
zero workers spawn, and every eval is silently skipped (then crashes).
- mlflow-report.ts: persist a generic "eval errored" marker as the appkit_eval
rationale instead of `result.error`, which was POSTed to MLflow and readable
by anyone with experiment access. Full error stays on the operator console.
- run-evals.ts: wrap reportToMlflow so a throw can't skip finishEvalRun and
leave the run stuck in RUNNING forever.
Adds tests for the NaN clamp and the rationale redaction.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 2/5 · targets
pr/agent-evals-1-tracing(review after #1).The core eval framework, plus LLM-as-judge and the MLflow REST connector.
defineEval): drive an agent over HTTP against a running app; assert witht.succeeded(),t.calledTool(),t.check(value, matcher)(includes/equals/matches). Gate-by-default,.soft()to demote.genai_evaluate; each turn's trace links viamlflow.sourceRun; per-assertion feedback written via the assessments REST API.t.judge.factuality/closedQA/custom) via autoevals → a Databricks serving endpoint.connectors/mlflow:MlflowClient(host/token, post/postResult, serving URL) +resolveDatabricksAuth/resolveWorkspaceClient(OAuth from a CLI profile — no hand-set PAT). Extracted so both evals and future callers share the REST/auth layer.appkit agent evalCLI.Squashed history note: contains the framework, judge, and connector-extraction commits.