hud-evals · jdchawla29 · Aug 19, 2026 · Aug 19, 2026
diff --git a/docs/v6/cookbooks/fireworks-rl-training.mdx b/docs/v6/cookbooks/fireworks-rl-training.mdx
@@ -245,23 +245,6 @@ Metrics are written to `runs/fireworks-serverless/metrics.jsonl`. Each row inclu
 within-group reward spread, valid rollout count, retained groups, training datums, whether an update
 was applied, loss, snapshot, and step duration.
 
-## Choose a loss
-
-The cookbook exposes three server-side policy-gradient objectives:
-
-| `--loss-fn` | Description |
-| --- | --- |
-| `importance_sampling` | Importance-sampling objective; default |
-| `ppo` | PPO objective |
-| `cispo` | CISPO objective |
-
-While comparing objectives, append `--loss-fn ppo` or `--loss-fn cispo` to the bounded command
-rather than launching the full 30-step recipe.
-
-All three consume the same target tokens, rollout logprobs, and token-level advantages. Fireworks
-also supports SFT, DPO, gradient accumulation, and client-defined losses; those paths use different
-data or control flow and are not implemented in this example.
-
 ## Save and resume
 
 Sampling and training checkpoints serve different purposes:
@@ -316,51 +299,8 @@ A hosted run executes the environment and grader on HUD while the cookbook's `Fi
 continues to call the Fireworks sampler from the training process.
 
 The included `FireworksAgent` and batch builder support one generated assistant response per Run.
-Tool-using and multi-turn environments require an adapter that executes each tool turn and a batch
-builder that combines every trainable assistant turn while masking user and tool-result tokens.
-
-## Example: BFCL multi-turn
-
-HUD can package agent tasks whose reward depends on the final state of an interaction. In the HUD
-version of the
-[Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html), each
-task starts fresh stateful backends, exposes the entry's functions as MCP tools, advances through
-scripted user turns, and grades the resulting calls with BFCL's own state and response checkers.
-
-| Component | BFCL environment |
-| --- | --- |
-| Task | One multi-turn BFCL entry |
-| State | Fresh file-system, trading, social, or other backends per rollout |
-| Tools | Entry-specific functions exposed through MCP |
-| Reward | Fraction of leading turns that pass the official checks |
-| Evaluation | Strict all-turns-pass result stored separately |
-
-In a separate validation run using HUD's managed training service, a Qwen3.5-4B policy trained on
-the 200-entry `multi_turn_base` taskset with groups of 8 and an importance-sampling loss. Mean
-training reward increased from 0.13 to approximately 0.52 in fewer than 20 optimizer steps. This
-result validates the environment and reward design; it is not a Fireworks serverless benchmark.
-
-BFCL is not a drop-in input to this Fireworks cookbook. `--taskset` can select its hosted tasks and
-runtime, but the included adapter cannot execute the tool loop and the batch builder keeps only one
-assistant turn. Use the
-[HUD RL training cookbook](https://github.com/hud-evals/hud-python/tree/main/cookbooks/rl-training)
-as a runnable reference for multi-turn rollout and batch construction.
-
-## Serverless and dedicated Fireworks training
-
-This example uses Serverless Training. Fireworks also supports dedicated trainers and inference
-deployments:
-
-| | Serverless | Dedicated |
-| --- | --- | --- |
-| Training | LoRA on a shared pool | LoRA or full-parameter on provisioned resources |
-| Sampling | In-session sampler snapshots | Inference deployment with snapshot refresh |
-| Lifecycle | Managed by Fireworks | Explicit trainer and deployment lifecycle |
-| Best fit | Short or variable per-token workloads | Sustained runs or broader model and method support |
-
-The HUD environment and rollout structure can be reused with either path. For dedicated training,
-start from the Fireworks `rl_loop.py` or `async_rl_loop.py` recipes and replace the arithmetic
-rollout source with a HUD taskset.
+Use a different agent adapter and batch builder for tool-using or multi-turn environments so every
+trainable assistant turn is recorded and user and tool-result tokens remain masked.
 
 ## See also
 

diff --git a/docs/v6/experimental/compose.mdx b/docs/v6/experimental/compose.mdx
@@ -5,6 +5,9 @@ icon: "cubes"
 tag: "Experimental"
 ---
 
+Its behavior is part of the SDK contract; its documentation path, naming, and organization may
+change without a major version.
+
 An environment is rarely one process. A task worth grading tends to be a small world - a database
 the agent may write, an application it may only reach over the network, services with their own
 lifecycles and users. A **Compose environment** declares that world as a standard Compose project:

diff --git a/docs/v6/experimental/harbor.mdx b/docs/v6/experimental/harbor.mdx
@@ -5,6 +5,9 @@ icon: "ship"
 tag: "Experimental"
 ---
 
+Its behavior is part of the SDK contract; its documentation path, naming, and organization may
+change without a major version.
+
 The **Harbor adapter** (`hud/integrations/harbor/`) translates between Harbor
 task directories and HUD's native `Environment`, `Task`, and `Taskset`
 contracts. `adapt()` packages Harbor tasks without building or publishing
@@ -34,6 +37,11 @@ Tasks that cannot be adapted appear in `result.failures`, with every detectable
 finding classified by a stable code. Other tasks in the dataset are still
 packaged.
 
+`AdaptResult.taskset` contains the successfully adapted rows. `AdaptResult.failures` is a tuple of
+`AdaptFailure(task, path, findings)` records, where each finding is an
+`AdaptFinding(code, kind, message)` and `kind` is `contract` or `invalid`. `adapt()` raises
+`ValueError` when the source contains no Harbor task directories.
+
 ```text
 .hud-adapt/<environment>/
 ├── build.sh
@@ -167,7 +175,7 @@ covers that review.
 
 | Function | Contract |
 | --- | --- |
-| `harbor.adapt(path, *, hud_requirement="hud") -> AdaptResult` | Package valid tasks as generated Compose environments and native task rows, and return structured failures for the rest. |
+| `harbor.adapt(path, *, hud_requirement="hud") -> AdaptResult` | Package valid tasks as generated Compose environments and native task rows, and return structured failures for the rest. Raises `ValueError` when no task directories exist. |
 | `await harbor.export(source, out_dir, *, answer_file=..., timeout_sec=600) -> list[Path]` | Write HUD task rows as Harbor task directories. |
 
 <CardGroup cols={2}>

diff --git a/docs/v6/experimental/verifier-environments.mdx b/docs/v6/experimental/verifier-environments.mdx
@@ -5,6 +5,9 @@ icon: "shield-check"
 tag: "Experimental"
 ---
 
+Its behavior is part of the SDK contract; its documentation path, naming, and organization may
+change without a major version.
+
 A grader is sometimes an environment of its own - a judge that needs services, golden data, or
 credentials the agent must never see. A **verifier environment** makes that structural: the actor
 task owns the environment the agent changes, and its `verifier` field names an agent-less task
@@ -92,9 +95,9 @@ engine keeps the actor connection and substrate alive and starts the verifier ta
 channel immediately after the actor task completes. A different environment name or verifier
 runtime configuration forces actor cleanup followed by a fresh provider acquisition.
 
-`HostedRuntime` supports the same-environment form without a verifier `runtime_config`, keeping
-both phases inside one hosted acquisition. Verifiers that require another runtime remain
-client-driven through `LocalRuntime`, `DockerRuntime`, or a custom provider.
+`HostedRuntime` accepts a `Task.verifier` row and serializes it with the actor submission, including
+the verifier's environment, arguments, and `runtime_config`. `LocalRuntime`, `DockerRuntime`, and
+custom providers execute verifier acquisitions through the client-side engine described above.
 
 ## What runs where
 
@@ -136,9 +139,17 @@ runtime, the phases remain isolated:
   mode, allowlist, and credentials directory; and
 - verifier output is read from `/logs/verifier/reward.json` or `/logs/verifier/reward.txt`.
 
-`RuntimeConfig.compose.service_access=True` gives the generated `main` service access to the
-runtime's Docker socket solely for declared collection from sibling services. The local and hosted
-socket behavior is described in [Compose environments](/v6/experimental/compose#runtime-behavior).
+`RuntimeConfig.compose.service_access=True` mounts a Docker socket in the generated `main` service.
+
+<Warning>
+Under local `DockerRuntime`, the mounted socket is the **host's own Docker daemon** (remote Docker
+endpoints are rejected). Code in `main` can then control every container on the machine - grant
+this only to environment code you trust. Under hosted execution, the daemon is private to the
+rollout.
+</Warning>
+
+See [Compose environments](/v6/experimental/compose#runtime-behavior) for the complete runtime
+behavior.
 
 ## See also
 

diff --git a/docs/v6/guides/training-agents.mdx b/docs/v6/guides/training-agents.mdx
@@ -176,28 +176,14 @@ hud models head arith-rl --set <id>   # roll back or branch from an earlier poin
 The [training reference](/v6/reference/training) covers the `TrainingClient` API, the built-in and
 custom losses, and the checkpoint tree in full.
 
-## Use HUD environments with Fireworks
+## Train with Fireworks
 
-A training service can update model weights, but it still needs a repeatable task and a reliable
-reward. HUD packages those pieces as an environment: the prompt, tools, state, and grader live
-together, and the same taskset can be used for evaluation or training.
-
-The managed loop above uses `TrainingClient`. In the one-turn Fireworks cookbook, HUD still runs and
-grades each rollout; a Fireworks sampler produces the model tokens, and the Fireworks Training API
-applies the update.
-
-| HUD provides | Fireworks provides |
-| --- | --- |
-| Task definitions and grouped tasksets | LoRA training on the serverless shared pool |
-| Local or hosted environment execution | Sampling from the current adapter |
-| Grading, rewards, and traces | Training and sampler checkpoints |
-| Portable environment and task interfaces | Forward, backward, and optimizer operations |
-
-HUD can also represent browser, coding, and multi-turn tool environments. Those tasks require a
-Fireworks agent adapter and batch builder that record every trainable assistant turn; they are not
-drop-in inputs to the one-turn example.
+Use the Fireworks cookbook to run grouped HUD tasks, convert the graded one-turn rollouts into
+training datums, and update a LoRA adapter with Fireworks Serverless Training. The included agent
+adapter supports one generated assistant response per run. Tool-using and multi-turn tasks require
+an adapter and batch builder that record every trainable assistant turn.
 
 <Card title="Fireworks Serverless RL" icon={<svg className="size-6 m-0! shrink-0 bg-primary dark:bg-primary-light" aria-hidden="true" style={{ maskImage: "url(/logo/fireworks.svg)", maskRepeat: "no-repeat", maskPosition: "center", maskSize: "contain", WebkitMaskImage: "url(/logo/fireworks.svg)", WebkitMaskRepeat: "no-repeat", WebkitMaskPosition: "center", WebkitMaskSize: "contain" }} />} href="/v6/cookbooks/fireworks-rl-training">
-  Train a Fireworks LoRA adapter on HUD-graded rollouts, including reward calibration,
-  group-relative updates, checkpoints, evaluation, and the path to hosted tasksets.
+  Train a Fireworks LoRA adapter on HUD-graded rollouts with reward calibration,
+  group-relative updates, checkpoints, and evaluation.
 </Card>
diff --git a/docs/v6/reference/runtime.mdx b/docs/v6/reference/runtime.mdx
@@ -80,17 +80,21 @@ RuntimeConfig(resources=RuntimeResources(tpu=RuntimeTPU(type="v5", topology="2x2
 Support differs per runtime. Providers reject unsupported requirements except `storage_mb`, which
 is best effort: `DockerRuntime` admits against available disk and `DaytonaRuntime` provisions
 enough whole GiB, while providers without disk sizing proceed with their default capacity.
-Daytona accepts a list of GPU alternatives; Modal accepts one GPU type. Docker ignores `limits`;
-Daytona rejects `run_timeout_s` and resource overrides when booting from an already-built snapshot.
-`LocalRuntime` rejects a per-task `runtime_config`.
+`HUDRuntime` proceeds on the deployed environment's defaults and warns when it ignores CPU, memory,
+or storage requirements; it rejects GPU, OS, TPU, and `limits`. Daytona accepts a list of GPU
+alternatives; Modal accepts one GPU type. Docker ignores `limits`; Daytona rejects `run_timeout_s`
+and resource overrides when booting from an already-built snapshot. `LocalRuntime` rejects a
+per-task `runtime_config`.
 
 `run_timeout_s` bounds the lifetime of the provisioned environment, not the complete rollout.
-The optional `rollout_timeout` passed to `Task.run` or `Taskset.run` bounds submission, queueing,
-provisioning, the agent, and grading together. When both are set, `run_timeout_s` and other known
-phase limits must be less than `rollout_timeout`; an agent timeout must also be less than the
-actor environment's `run_timeout_s`. Contradictory values fail before execution. When
-`rollout_timeout` is omitted, the SDK adds no overall deadline; configured phase limits and limits
-imposed by the selected runtime still apply.
+The optional `rollout_timeout` passed to `Task.run` or `Taskset.run` is the execution deadline
+through grading, including submission, queueing, and provisioning. It is not an upper bound on
+method-return latency: cancellation and trace reporting still run after expiry, and `Taskset.run`
+then flushes telemetry. When both are set, `run_timeout_s` and other known phase limits must be less
+than `rollout_timeout`; an agent timeout must also be less than the actor environment's
+`run_timeout_s`. Contradictory values fail before execution. When `rollout_timeout` is omitted, the
+SDK adds no overall deadline; configured phase limits and limits imposed by the selected runtime
+still apply.
 
 ## Runtime directory
 
@@ -112,8 +116,9 @@ Runs an environment in this process. `source` can be:
   envs), called fresh per rollout with the placed row.
 
 `ready_timeout` bounds `@env.initialize` startup. Paths and constructors create a fresh environment
-per rollout. Env hooks share this process and event loop; use `SubprocessRuntime` or
-`DockerRuntime` when rollouts need process isolation.
+per rollout. A live `Environment` instance is shared and guarded by a lock, so acquisitions through
+that `LocalRuntime` are serialized. Env hooks share this process and event loop; use
+`SubprocessRuntime` or `DockerRuntime` when rollouts need process isolation.
 
 ### `SubprocessRuntime`
 

diff --git a/docs/v6/reference/tasks.mdx b/docs/v6/reference/tasks.mdx
@@ -162,15 +162,17 @@ job = await ts.run(agent, runtime=LocalRuntime("env.py"), group=8, max_concurren
 | `group` | `int \| None` | Repeats each task N times for GRPO. |
 | `max_concurrent` | `int \| None` | Caps how many rollouts run in parallel. |
 | `job` | `Job \| None` | An open [`Job`](/v6/reference/types#job) to accumulate into; defaults to a fresh job per call. |
-| `rollout_timeout` | `float \| None` | Optional hard deadline (seconds) for the full rollout lifecycle in every placement; omitted means no SDK-wide deadline. |
+| `rollout_timeout` | `float \| None` | Optional execution deadline (seconds) through grading in every placement; omitted means no SDK-wide deadline. |
 
 `agent_config.timeout_seconds` bounds only the agent phase. If it expires, the
 run is marked as timed out but grading still runs against any completed work.
-`rollout_timeout` instead bounds startup, the agent, and grading together; runtime cleanup may
-finish in the background after a timeout. When both are set, the agent timeout must be less than
-the rollout timeout and any explicit actor `run_timeout_s`. Limits imposed by the selected runtime
-still apply when `rollout_timeout` is omitted. Grader-specific limits, such as
-`BashGrader.timeout_seconds`, remain local to that grader.
+`rollout_timeout` instead bounds submission, queueing, startup, the agent, and grading together.
+It is not a wall-clock bound on `Task.run` or `Taskset.run` returning: after expiry, local execution
+may wait up to two seconds for cancellation and still reports the trace, and `Taskset.run` can spend
+up to 120 seconds flushing telemetry. Runtime cleanup may finish in the background. When both are
+set, the agent timeout must be less than the rollout timeout and any explicit actor
+`run_timeout_s`. Limits imposed by the selected runtime still apply when `rollout_timeout` is
+omitted. Grader-specific limits, such as `BashGrader.timeout_seconds`, remain local to that grader.
 
 A crashed rollout comes back as a failed [`Run`](/v6/reference/types#run) inside the job rather than
 raising, so one bad rollout never collapses a batch. For the end-to-end workflow see

diff --git a/docs/v6/reference/telemetry.mdx b/docs/v6/reference/telemetry.mdx
@@ -18,15 +18,26 @@ hud set HUD_TELEMETRY_ENABLED=0
 or set `HUD_TELEMETRY_ENABLED=0` in your environment.
 
 When you run evaluations with an API key configured, the SDK uploads trace
-spans (steps, tool calls, timings) to your team's traces on the HUD platform.
-This is the product feature that powers the trace viewer — it is scoped to
-your team and requires your API key.
+spans to the HUD platform. Depending on the agent, task, instrumentation, and
+workspace configuration, span payloads can include:
+
+- prompts and other structured messages, task arguments, and task setup and grading results;
+- model content and reasoning, tool calls, raw provider data, usage, and token-level samples;
+- complete tool requests and results;
+- serialized arguments and results from instrumented functions, plus exception messages; and
+- workspace setup and agent file diffs and final artifacts when workspace file tracking is active.
+
+Spans also include execution metadata such as step order, timings, status, and errors. Trace
+uploads require your API key and power the trace viewer.
 
 - **No API key (the default):** spans are not uploaded anywhere.
-- **`HUD_TELEMETRY_LOCAL_DIR=./spans`:** spans are also written to local
-  files. Works without an API key, and keeps working with telemetry disabled.
-- **Fully local:** `HUD_TELEMETRY_ENABLED=0` plus `HUD_TELEMETRY_LOCAL_DIR`
-  keeps every span on your machine and sends nothing.
+- **`HUD_TELEMETRY_LOCAL_DIR=./spans`:** emitted spans are also written to local
+  files, including without an API key or with backend telemetry disabled. Local writes are
+  best-effort; filesystem errors do not fail the evaluation.
+- **Fully local trace storage:** `HUD_TELEMETRY_ENABLED=0` plus
+  `HUD_TELEMETRY_LOCAL_DIR=./spans` stops backend trace uploads and writes other emitted spans
+  locally on a best-effort basis. Disabling trace telemetry also stops workspace file-tracking
+  spans. CLI analytics are independent; set `HUD_CLI_ANALYTICS_ENABLED=0` to disable them too.
 
 ## Anonymous CLI usage events
 
@@ -38,9 +49,9 @@ hud set HUD_CLI_ANALYTICS_ENABLED=0
 
 or set `HUD_CLI_ANALYTICS_ENABLED=0` in your environment.
 
-Each CLI invocation sends one small event recording which command ran and
-how it exited, so HUD can improve the commands people actually use and fix
-the ones that fail. The payload is a fixed allowlist:
+Except for `hud --version`, each CLI invocation sends one small event recording
+which command ran and how it exited, so HUD can improve the commands people
+actually use and fix the ones that fail. The payload is a fixed allowlist:
 
 | Field | Example |
 | --- | --- |
@@ -58,7 +69,6 @@ task file, a trace id, a typo, is recorded as `other` or omitted, so positional
 input is never transmitted. The install id is a random identifier for one
 machine — it is not tied to your account.
 
-Events are sent in the background and abandoned after about a second, so a
-slow or failed send can never break a command and adds at most that bound to
-its exit. A one-time notice is printed the first time the install id is
-created.
+Events are sent in the background and abandoned after 1.5 seconds, so a slow
+or failed send cannot break a command and adds at most 1.5 seconds to its exit.
+A one-time notice is printed the first time the install id is created.