feat(appkit): migrate agent mlflow tracing to @mlflow/core with Unity Catalog support by MarioCadenas · Pull Request #545 · databricks/appkit

MarioCadenas · 2026-08-21T13:11:32Z

What

Migrates the agents plugin's MLflow tracing from mlflow-tracing@0.1.3 to @mlflow/core@0.4.0, adding Unity Catalog trace-location support so agent traces land in UC-backed MLflow experiments — while keeping AppKit's single OTel tracer provider (a plugin contributes its span processor via TelemetryManager.registerSpanProcessor(); nothing else stands up a competing provider).

Also includes the registerSpanProcessor telemetry seam this depends on: TelemetryManager builds one global tracer provider from the OTLP processor plus every plugin-contributed processor, replacing the old per-SDK NodeSDK.start() that raced OTel's allowOverride=false global registration.

Why

UC experiments got no traces. mlflow-tracing@0.1.3 only writes to classic MLFLOW_EXPERIMENT storage. UC-backed experiments store traces in UC tables (e.g. main.mario.<expid>_otel_spans) and received nothing. @mlflow/core supports UC natively via DatabricksUCTableSpanProcessor/DatabricksUCTableSpanExporter + a traceLocation config.
Two providers still can't race. @mlflow/core's init() builds its own NodeSDK and registers the global provider — the exact collision the single-provider work fixes. So init() is never used to register a provider here.

How

UC path is fully clean. When a UC trace location is bound, we build new DatabricksUCTableSpanProcessor(new DatabricksUCTableSpanExporter(client), ucLocation) with an explicit MlflowClient (built via the now-public createAuthProvider), wrap it in GatedMlflowSpanProcessor, and contribute it through registerSpanProcessor(). The UC processor/exporter read no global config, so this path never calls init() — no second NodeSDK, no competing registration.
Classic path unchanged in spirit. The classic MlflowSpanProcessor.onStart reads getConfig().experimentId, so classic still calls init() purely to seed that global config; its provider registration harmlessly loses the already-claimed global slot.
UC trace-location resolution: env override (MLFLOW_UC_CATALOG + MLFLOW_UC_SCHEMA + MLFLOW_UC_TABLE_PREFIX), else auto-detect for numeric experiment ids from the experiment's mlflow.experiment.databricksTraceDestinationPath tag via MlflowClient.getExperiment. Any failure falls back to classic — a tracing misconfiguration never breaks the agent. The resolved mode is logged in the boot line (… UC main.mario.<expid>).
Fewer deep imports. createAuthProvider, MlflowClient, InMemoryTraceManager, and SpanAttributeKey are public in @mlflow/core, so only the exporter/processor classes remain deep-imported — pinned to the exact version and guarded by a tripwire test.
Bounded flush. GatedMlflowSpanProcessor.forceFlush/shutdown are time-bounded so a stuck UC export can't hold graceful shutdown (exports are fire-and-forget, so a stuck one can only wedge at shutdown, never a turn).
@databricks/sdk-experimental@0.17.0 is declared directly on appkit (matching shared) so the inlined workspace-client's hoist survives @mlflow/core's 0.15.0 pin; knip's ignoreDependencies covers the runtime-only require.

Verification

pnpm --filter=@databricks/appkit typecheck — clean
mlflow suite — 26 pass (behavior + UC env/tag/fallback paths + GatedMlflowSpanProcessor gating + bounded-flush), including a tripwire that constructs the real public + deep-imported symbols against the installed @mlflow/core
telemetry + full agents + database suites — pass; full appkit suite green (3462)
oxlint + oxfmt — clean
OTLP + MLflow coexistence verified end-to-end against a real OTLP/HTTP receiver: one provider carrying both the OTLP BatchSpanProcessor and the gated UC processor — OTLP received every span from both an agent trace and a plain request; the UC exporter was invoked once (agent trace only), never for the plain request.

Manual (pending, needs a Databricks profile): dev-playground with MLFLOW_EXPERIMENT_ID set to a UC experiment — confirm the boot log reads … UC <catalog>.<schema>.<prefix>, and traces + assessments actually land in the UC experiment's Traces tab (this is the one hop the local coexistence test stubs at the Databricks client boundary); classic experiment still works.

Dependency note

On @mlflow/core@0.4.0. It was briefly pinned to 0.3.0 because 0.4.0 (released 2026-08-27) was inside the internal Artifactory registry's new-version quarantine window and CI pnpm install --frozen-lockfile returned ERR_PNPM_FETCH_403; now past the window. 0.3.0 and 0.4.0 share the exact API surface AppKit uses, so the version is a one-line change.

github-actions · 2026-09-02T14:16:00Z

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

`@databricks/appkit`

npm tarball (packed): 1.0 MB (+13 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	1.1 MB (+15 KB)	382 KB (+5.1 KB)
Type declarations	385 KB (+40 B)	136 KB (+18 B)
Source maps	2.1 MB (+26 KB)	714 KB (+8.4 KB)
Other	11 KB	3.7 KB
Total	3.6 MB (+41 KB)	1.2 MB (+13 KB)

Per-entry composition (own code — deps external (as shipped))

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
`.`	95 KB (+341 B)	2.5 KB	98 KB (+341 B)	external	312 KB (+1.3 KB)
`./beta`	76 KB (+1.4 KB)	457 B	77 KB (+1.4 KB)	external	230 KB (+4.1 KB)
`./testing`	17 KB (+326 B)	0 B	17 KB (+326 B)	external	51 KB (+1.3 KB)
`./tsdown`	520 B	0 B	520 B	external	813 B
`./type-generator`	22 KB	0 B	22 KB	external	65 KB

Chunks:

Entry	Chunk	Load	Size (gz)
`.`	`index.js`	initial	91 KB
`.`	`utils.js`	initial	4.0 KB
`.`	`remote-tunnel-manager.js`	lazy	2.5 KB
`./beta`	`beta.js`	initial	60 KB
`./beta`	`stream-manager.js`	initial	5.8 KB
`./beta`	`wide-event-emitter.js`	initial	3.2 KB
`./beta`	`databricks.js`	initial	3.2 KB
`./beta`	`configuration.js`	initial	2.1 KB
`./beta`	`service-context.js`	initial	1.3 KB
`./beta`	`client.js`	initial	434 B
`./beta`	`client-options.js`	initial	219 B
`./beta`	`supervisor-api.js`	lazy	193 B
`./beta`	`databricks.js`	lazy	142 B
`./beta`	`index.js`	lazy	122 B
`./testing`	`index.js`	initial	17 KB
`./tsdown`	`index.js`	initial	520 B
`./type-generator`	`index.js`	initial	22 KB

`@databricks/appkit-ui`

npm tarball (packed): 350 KB (-4 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	395 KB	132 KB
Type declarations	229 KB	84 KB (-1 B)
Source maps	766 KB	253 KB
CSS	16 KB	3.2 KB
Total	1.4 MB	473 KB (-1 B)

Per-entry composition (consumer bundle — deps bundled, peerDeps external)

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
`./js`	5.3 KB	49 KB	55 KB	208 KB	14 KB
`./js/beta`	20 B	0 B	20 B	0 B	0 B
`./react`	432 KB	49 KB	481 KB	1.3 MB	177 KB
`./react/beta`	1.0 KB	0 B	1.0 KB	0 B	1.9 KB

Chunks:

Entry	Chunk	Load	Size (gz)
`./js`	`index.js`	initial	5.2 KB
`./js`	`chunk`	initial	120 B
`./js`	`apache-arrow`	lazy	49 KB
`./js/beta`	`beta.js`	initial	20 B
`./react`	`index.js`	initial	430 KB
`./react`	`tslib`	initial	2.1 KB
`./react`	`apache-arrow`	lazy	49 KB
`./react/beta`	`beta.js`	initial	1.0 KB

github-actions · 2026-09-02T14:20:25Z

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 33736498966 -R databricks/appkit -n appkit-template-0.70.0-pr.26a9bd4-fix-agents-mlflow-single-provider-545 -D appkit-pr-545 \
  && unzip -o "appkit-pr-545/appkit-template-0.70.0-pr.26a9bd4-fix-agents-mlflow-single-provider-545.zip" -d "appkit-pr-545" \
  && databricks apps init --template "appkit-pr-545"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

…ls-2-framework Takes #545's rewritten agents/mlflow.ts + agents.ts (single-provider @mlflow/core UC tracing) over the eval branch's mlflow-tracing versions; package.json is the union (@mlflow/core + sdk-experimental + sdk-trace-node from #545, autoevals from evals). Eval's updateTracePreview + flushTrace are dropped here (superseded by Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com> #545's tracing + mlflow-report retry-on-404); re-port as follow-ups if wanted.

Split TelemetryManager into two phases so a single global tracer provider can carry both the OTLP exporter and plugin-contributed processors (e.g. MLflow), instead of each SDK racing to register the global provider. - initialize() registers the meter and logger providers eagerly, because OTel's metrics API has no lazy proxy: an instrument bound against the NoOp meter stays NoOp for the process lifetime. - registerSpanProcessor() lets plugins contribute a span processor during setup(); ignored with a warning after start() since a started provider's processors are immutable in OTel JS 2.x. - start() (called after plugin setup) builds the global NodeTracerProvider with the OTLP processor plus every contributed one, and no-ops when nothing needs tracing. Deferring is safe: ProxyTracer rebinds tracers obtained earlier and no span is emitted during setup. Swaps the @opentelemetry/sdk-node dependency for @opentelemetry/sdk-trace-node, since the tracer provider is now built directly instead of via NodeSDK. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

The mlflow-tracing SDK's init() stands up and globally registers its own OpenTelemetry tracer provider. OTel's registerGlobal is allowOverride=false, so when an OTLP endpoint and agent tracing are both active, AppKit's provider and mlflow's race for the global slot and one exporter is silently dropped. Instead of calling init() (which self-registers), the agents plugin now builds mlflow's span processor itself and contributes it to AppKit's single provider via TelemetryManager.registerSpanProcessor() during setup(). mlflow's global config is seeded lazily on first trace (after start()), so init()'s own registration harmlessly loses the already-claimed global slot. A GatedMlflowSpanProcessor keeps the contributed processor inert until config is seeded, so AppKit's own spans created before the first agent turn don't hit mlflow's getConfig() throw. The processor/exporter are deep-imported from mlflow-tracing internals (no public export in 0.1.3), pinned and guarded by a tripwire test. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

…rom turn 1 The mlflow span processor rides AppKit's shared OTel provider, so it saw every span — including the exporters' own outbound HTTP calls. Each trace upload became a new span to trace and upload: a feedback loop that flooded the experiment and could wedge the process. - Drop parentless CLIENT spans in the gated processor. An outgoing request made outside any agent turn (mlflow/OTLP shipping a trace) is exactly the loop's return edge; spans inside a real request tree keep their parent (or are the incoming SERVER root), so agent turns are untouched. - Seed mlflow's config on the "setup:complete" lifecycle event — after TelemetryManager.start(), before the server serves — instead of lazily on the first trace. The first turn's request-root span is then already forwarded, so that turn assembles into a trace instead of being dropped as a cold-start. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

The mlflow span processor rides AppKit's shared OTel provider, which carries every HTTP/DB span, and mlflow roots a trace at every parentless span. So once an experiment was bound, every request (analytics, genie, static assets) became an MLflow trace — a createTrace/uploadTraceData per request, plus non-agent payloads landing in a GenAI experiment. GatedMlflowSpanProcessor now decides at the root span's onEnd: export only if some span in the trace carried mlflow.spanType (an AGENT/TOOL span), otherwise popTrace to discard the in-memory trace mlflow built. Plain HTTP requests no longer reach the MLflow backend. Deep-imports InMemoryTraceManager.popTrace and SpanAttributeKey.SPAN_TYPE (pinned 0.1.3, tripwire-guarded). Hardening from review: - ensureConfigured() guards on gatedProcessor, so a processor-build failure can't let mlflow.init() register its own ungated global provider and win the empty slot (which would re-introduce the over-tracing + exporter loop). - #agentTraceIds is FIFO-bounded so an orphaned entry (root ends before or without its agent child) can't leak over process uptime. - Three telemetry-manager tests that passed against a NoOp meter or asserted only not.toThrow now assert their real invariant. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

…og support Migrate agent tracing from mlflow-tracing@0.1.3 to @mlflow/core@0.4.0 and add Unity Catalog trace-location support, keeping AppKit's single-provider architecture from this branch. - Swap mlflow-tracing -> @mlflow/core@0.4.0 and declare @databricks/sdk-experimental@0.17.0 directly on appkit (matching shared) so the inlined workspace-client's hoist survives @mlflow/core's 0.15.0 pin, avoiding a runtime "Cannot find module '@databricks/sdk-experimental'". knip can't see that runtime require, so the dep is listed in its ignoreDependencies. - Resolve a UC trace location (MLFLOW_UC_CATALOG/SCHEMA/TABLE_PREFIX env override, else the experiment's databricksTraceDestinationPath tag for numeric ids) and build the UC processor/exporter when present, classic otherwise. Any failure falls back to classic so tracing never breaks the agent. The resolved mode is logged in the boot line. - Keep the single global OTel provider: the UC path reads no global config and never calls @mlflow/core's init() (no competing NodeSDK); only the classic processor needs init()'s config seed, whose provider loses the global race to AppKit's already-registered one, as before. - Move the symbols now public in @mlflow/core (createAuthProvider, MlflowClient, InMemoryTraceManager, SpanAttributeKey) off deep imports; only the exporter/processor classes remain deep-imported, guarded by the tripwire test. - Bound the gated processor's flush/shutdown so a stuck UC export can't hold graceful shutdown. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

0.4.0 (released 2026-08-27) is inside the internal Artifactory registry's new-version quarantine window, so CI pnpm install --frozen-lockfile 403s on it. 0.3.0 (2026-07-07) is past the window and carries the same UC trace-location API (DatabricksUCTableSpanProcessor/Exporter, public createAuthProvider/ MlflowClient/InMemoryTraceManager, MlflowClient.getExperiment), so no code changes are needed. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

The gate decided 'agent trace, export it' on whether mlflow.spanType was present. But @mlflow/core's processor stamps mlflow.spanType on EVERY span it processes (createAndRegisterMlflowSpan defaults to SpanType.UNKNOWN), so the SERVER root of every recorded request carried the attribute — and every non-agent API request (e.g. POST /api/analytics/query) was exported as an MLflow trace, the exact over-tracing this branch exists to prevent. Discriminate on the AGENT/TOOL value instead of mere presence; buildMlflowSpanProcessor now returns the accepted JSON-stringified values. Also switch UC auto-detect to mlflow's own ucLocationFromExperimentTags (deep-imported like the exporter classes) instead of a hand-rolled tag split: it preserves backend-populated span/log storage-table names for custom- provisioned UC locations, and its .d.ts is not redacted in 0.3.0. Tests: regression for the UNKNOWN-stamped plain request; resolver branches (no-tag/getExperiment-failure/non-numeric-skip → classic; env + tag → UC with the location asserted on the processor); tripwire now covers the tracing entrypoints and ucLocationFromExperimentTags against the real package. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

0.4.0 has cleared the internal Artifactory registry's new-version quarantine window (released 2026-08-27), so CI can now fetch it. The API surface AppKit uses is unchanged from 0.3.0 (same public exports, deep-import paths, UC processor/exporter, ucLocationFromExperimentTags, getExperiment), so no code changes are needed; verified via typecheck + the mlflow suite tripwire against the real 0.4.0 package. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Rebuilt on the current fix/agents-mlflow-single-provider base (#545 was force-pushed onto v0.70.0, so the branch's merged copies were stale). The eval framework is self-contained — it imports nothing from the agents plugin — so every #545 file is taken from the base untouched; only the framework's own files plus additive seam edits (beta/connectors/cli exports, autoevals dep) are applied. Framework: discover server/agents/<agent>/evals/*.eval.ts, drive a running app over HTTP+SSE, run deterministic assertions + LLM-as-judge (autoevals), and report results + per-trace assessments to MLflow (classic V3 + UC V4). Includes the mlflow REST connector and the `appkit agent eval` CLI. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

MarioCadenas requested a review from a team as a code owner August 21, 2026 13:11

MarioCadenas requested review from calvarjorge and removed request for a team August 21, 2026 13:11

MarioCadenas changed the base branch from telemetry-manager-refactor to main August 21, 2026 16:53

MarioCadenas changed the title ~~fix(agents): route mlflow tracing through AppKit's single otel provider~~ feat(appkit): plugin-contributed otel span processors + single-provider agent mlflow tracing Aug 21, 2026

MarioCadenas mentioned this pull request Aug 21, 2026

refactor(appkit): let plugins contribute otel span processors #536

Closed

MarioCadenas force-pushed the fix/agents-mlflow-single-provider branch from 41cfa86 to 40ae433 Compare September 2, 2026 13:34

MarioCadenas changed the title ~~feat(appkit): plugin-contributed otel span processors + single-provider agent mlflow tracing~~ feat(appkit): migrate agent mlflow tracing to @mlflow/core with Unity Catalog support Sep 2, 2026

MarioCadenas requested review from atilafassina and ditadi September 3, 2026 08:54

MarioCadenas added 9 commits September 3, 2026 10:57

chore(deps): regenerate lockfile after merging main

152db62

Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

MarioCadenas force-pushed the fix/agents-mlflow-single-provider branch from 7efc589 to fd96b77 Compare September 3, 2026 09:00

atilafassina approved these changes Sep 3, 2026

View reviewed changes

MarioCadenas merged commit 4c767f8 into main Sep 3, 2026
10 checks passed

MarioCadenas deleted the fix/agents-mlflow-single-provider branch September 3, 2026 11:59

Navigation Menu

Uh oh!

feat(appkit): migrate agent mlflow tracing to @mlflow/core with Unity Catalog support - #545

feat(appkit): migrate agent mlflow tracing to @mlflow/core with Unity Catalog support#545
MarioCadenas merged 9 commits into
mainfrom
fix/agents-mlflow-single-provider

MarioCadenas commented Aug 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

MarioCadenas commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How

Verification

Dependency note

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Bundle size report

@databricks/appkit

@databricks/appkit-ui

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 AppKit PR bot

🔬 Run evals

📦 Try this PR's app template

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MarioCadenas commented Aug 21, 2026 •

edited

Loading

github-actions Bot commented Sep 2, 2026 •

edited

Loading

`@databricks/appkit`

`@databricks/appkit-ui`

github-actions Bot commented Sep 2, 2026 •

edited

Loading