From b64a6f612197dedb2491340c1c67bd499b3d404d Mon Sep 17 00:00:00 2001 From: Jaideep <67646710+jdchawla29@users.noreply.github.com> Date: Wed, 19 Aug 2026 15:28:26 -0700 Subject: [PATCH] docs: correct v0.6.14 release contracts --- docs/v6/cookbooks/fireworks-rl-training.mdx | 64 +------------------ docs/v6/experimental/compose.mdx | 3 + docs/v6/experimental/harbor.mdx | 10 ++- .../v6/experimental/verifier-environments.mdx | 23 +++++-- docs/v6/guides/training-agents.mdx | 28 ++------ docs/v6/reference/runtime.mdx | 27 ++++---- docs/v6/reference/tasks.mdx | 14 ++-- docs/v6/reference/telemetry.mdx | 38 +++++++---- 8 files changed, 86 insertions(+), 121 deletions(-) diff --git a/docs/v6/cookbooks/fireworks-rl-training.mdx b/docs/v6/cookbooks/fireworks-rl-training.mdx index efbac1b2d..b54d7f724 100644 --- a/docs/v6/cookbooks/fireworks-rl-training.mdx +++ b/docs/v6/cookbooks/fireworks-rl-training.mdx @@ -245,23 +245,6 @@ Metrics are written to `runs/fireworks-serverless/metrics.jsonl`. Each row inclu within-group reward spread, valid rollout count, retained groups, training datums, whether an update was applied, loss, snapshot, and step duration. -## Choose a loss - -The cookbook exposes three server-side policy-gradient objectives: - -| `--loss-fn` | Description | -| --- | --- | -| `importance_sampling` | Importance-sampling objective; default | -| `ppo` | PPO objective | -| `cispo` | CISPO objective | - -While comparing objectives, append `--loss-fn ppo` or `--loss-fn cispo` to the bounded command -rather than launching the full 30-step recipe. - -All three consume the same target tokens, rollout logprobs, and token-level advantages. Fireworks -also supports SFT, DPO, gradient accumulation, and client-defined losses; those paths use different -data or control flow and are not implemented in this example. - ## Save and resume Sampling and training checkpoints serve different purposes: @@ -316,51 +299,8 @@ A hosted run executes the environment and grader on HUD while the cookbook's `Fi continues to call the Fireworks sampler from the training process. The included `FireworksAgent` and batch builder support one generated assistant response per Run. -Tool-using and multi-turn environments require an adapter that executes each tool turn and a batch -builder that combines every trainable assistant turn while masking user and tool-result tokens. - -## Example: BFCL multi-turn - -HUD can package agent tasks whose reward depends on the final state of an interaction. In the HUD -version of the -[Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html), each -task starts fresh stateful backends, exposes the entry's functions as MCP tools, advances through -scripted user turns, and grades the resulting calls with BFCL's own state and response checkers. - -| Component | BFCL environment | -| --- | --- | -| Task | One multi-turn BFCL entry | -| State | Fresh file-system, trading, social, or other backends per rollout | -| Tools | Entry-specific functions exposed through MCP | -| Reward | Fraction of leading turns that pass the official checks | -| Evaluation | Strict all-turns-pass result stored separately | - -In a separate validation run using HUD's managed training service, a Qwen3.5-4B policy trained on -the 200-entry `multi_turn_base` taskset with groups of 8 and an importance-sampling loss. Mean -training reward increased from 0.13 to approximately 0.52 in fewer than 20 optimizer steps. This -result validates the environment and reward design; it is not a Fireworks serverless benchmark. - -BFCL is not a drop-in input to this Fireworks cookbook. `--taskset` can select its hosted tasks and -runtime, but the included adapter cannot execute the tool loop and the batch builder keeps only one -assistant turn. Use the -[HUD RL training cookbook](https://github.com/hud-evals/hud-python/tree/main/cookbooks/rl-training) -as a runnable reference for multi-turn rollout and batch construction. - -## Serverless and dedicated Fireworks training - -This example uses Serverless Training. Fireworks also supports dedicated trainers and inference -deployments: - -| | Serverless | Dedicated | -| --- | --- | --- | -| Training | LoRA on a shared pool | LoRA or full-parameter on provisioned resources | -| Sampling | In-session sampler snapshots | Inference deployment with snapshot refresh | -| Lifecycle | Managed by Fireworks | Explicit trainer and deployment lifecycle | -| Best fit | Short or variable per-token workloads | Sustained runs or broader model and method support | - -The HUD environment and rollout structure can be reused with either path. For dedicated training, -start from the Fireworks `rl_loop.py` or `async_rl_loop.py` recipes and replace the arithmetic -rollout source with a HUD taskset. +Use a different agent adapter and batch builder for tool-using or multi-turn environments so every +trainable assistant turn is recorded and user and tool-result tokens remain masked. ## See also diff --git a/docs/v6/experimental/compose.mdx b/docs/v6/experimental/compose.mdx index d307c5f8e..61090ac54 100644 --- a/docs/v6/experimental/compose.mdx +++ b/docs/v6/experimental/compose.mdx @@ -5,6 +5,9 @@ icon: "cubes" tag: "Experimental" --- +Its behavior is part of the SDK contract; its documentation path, naming, and organization may +change without a major version. + An environment is rarely one process. A task worth grading tends to be a small world - a database the agent may write, an application it may only reach over the network, services with their own lifecycles and users. A **Compose environment** declares that world as a standard Compose project: diff --git a/docs/v6/experimental/harbor.mdx b/docs/v6/experimental/harbor.mdx index 83aa55b94..0dd4cbc24 100644 --- a/docs/v6/experimental/harbor.mdx +++ b/docs/v6/experimental/harbor.mdx @@ -5,6 +5,9 @@ icon: "ship" tag: "Experimental" --- +Its behavior is part of the SDK contract; its documentation path, naming, and organization may +change without a major version. + The **Harbor adapter** (`hud/integrations/harbor/`) translates between Harbor task directories and HUD's native `Environment`, `Task`, and `Taskset` contracts. `adapt()` packages Harbor tasks without building or publishing @@ -34,6 +37,11 @@ Tasks that cannot be adapted appear in `result.failures`, with every detectable finding classified by a stable code. Other tasks in the dataset are still packaged. +`AdaptResult.taskset` contains the successfully adapted rows. `AdaptResult.failures` is a tuple of +`AdaptFailure(task, path, findings)` records, where each finding is an +`AdaptFinding(code, kind, message)` and `kind` is `contract` or `invalid`. `adapt()` raises +`ValueError` when the source contains no Harbor task directories. + ```text .hud-adapt// ├── build.sh @@ -167,7 +175,7 @@ covers that review. | Function | Contract | | --- | --- | -| `harbor.adapt(path, *, hud_requirement="hud") -> AdaptResult` | Package valid tasks as generated Compose environments and native task rows, and return structured failures for the rest. | +| `harbor.adapt(path, *, hud_requirement="hud") -> AdaptResult` | Package valid tasks as generated Compose environments and native task rows, and return structured failures for the rest. Raises `ValueError` when no task directories exist. | | `await harbor.export(source, out_dir, *, answer_file=..., timeout_sec=600) -> list[Path]` | Write HUD task rows as Harbor task directories. | diff --git a/docs/v6/experimental/verifier-environments.mdx b/docs/v6/experimental/verifier-environments.mdx index a7eab4fbe..9fb02d568 100644 --- a/docs/v6/experimental/verifier-environments.mdx +++ b/docs/v6/experimental/verifier-environments.mdx @@ -5,6 +5,9 @@ icon: "shield-check" tag: "Experimental" --- +Its behavior is part of the SDK contract; its documentation path, naming, and organization may +change without a major version. + A grader is sometimes an environment of its own - a judge that needs services, golden data, or credentials the agent must never see. A **verifier environment** makes that structural: the actor task owns the environment the agent changes, and its `verifier` field names an agent-less task @@ -92,9 +95,9 @@ engine keeps the actor connection and substrate alive and starts the verifier ta channel immediately after the actor task completes. A different environment name or verifier runtime configuration forces actor cleanup followed by a fresh provider acquisition. -`HostedRuntime` supports the same-environment form without a verifier `runtime_config`, keeping -both phases inside one hosted acquisition. Verifiers that require another runtime remain -client-driven through `LocalRuntime`, `DockerRuntime`, or a custom provider. +`HostedRuntime` accepts a `Task.verifier` row and serializes it with the actor submission, including +the verifier's environment, arguments, and `runtime_config`. `LocalRuntime`, `DockerRuntime`, and +custom providers execute verifier acquisitions through the client-side engine described above. ## What runs where @@ -136,9 +139,17 @@ runtime, the phases remain isolated: mode, allowlist, and credentials directory; and - verifier output is read from `/logs/verifier/reward.json` or `/logs/verifier/reward.txt`. -`RuntimeConfig.compose.service_access=True` gives the generated `main` service access to the -runtime's Docker socket solely for declared collection from sibling services. The local and hosted -socket behavior is described in [Compose environments](/v6/experimental/compose#runtime-behavior). +`RuntimeConfig.compose.service_access=True` mounts a Docker socket in the generated `main` service. + + +Under local `DockerRuntime`, the mounted socket is the **host's own Docker daemon** (remote Docker +endpoints are rejected). Code in `main` can then control every container on the machine - grant +this only to environment code you trust. Under hosted execution, the daemon is private to the +rollout. + + +See [Compose environments](/v6/experimental/compose#runtime-behavior) for the complete runtime +behavior. ## See also diff --git a/docs/v6/guides/training-agents.mdx b/docs/v6/guides/training-agents.mdx index 8aa0a34d5..1dbfa303b 100644 --- a/docs/v6/guides/training-agents.mdx +++ b/docs/v6/guides/training-agents.mdx @@ -176,28 +176,14 @@ hud models head arith-rl --set # roll back or branch from an earlier poin The [training reference](/v6/reference/training) covers the `TrainingClient` API, the built-in and custom losses, and the checkpoint tree in full. -## Use HUD environments with Fireworks +## Train with Fireworks -A training service can update model weights, but it still needs a repeatable task and a reliable -reward. HUD packages those pieces as an environment: the prompt, tools, state, and grader live -together, and the same taskset can be used for evaluation or training. - -The managed loop above uses `TrainingClient`. In the one-turn Fireworks cookbook, HUD still runs and -grades each rollout; a Fireworks sampler produces the model tokens, and the Fireworks Training API -applies the update. - -| HUD provides | Fireworks provides | -| --- | --- | -| Task definitions and grouped tasksets | LoRA training on the serverless shared pool | -| Local or hosted environment execution | Sampling from the current adapter | -| Grading, rewards, and traces | Training and sampler checkpoints | -| Portable environment and task interfaces | Forward, backward, and optimizer operations | - -HUD can also represent browser, coding, and multi-turn tool environments. Those tasks require a -Fireworks agent adapter and batch builder that record every trainable assistant turn; they are not -drop-in inputs to the one-turn example. +Use the Fireworks cookbook to run grouped HUD tasks, convert the graded one-turn rollouts into +training datums, and update a LoRA adapter with Fireworks Serverless Training. The included agent +adapter supports one generated assistant response per run. Tool-using and multi-turn tasks require +an adapter and batch builder that record every trainable assistant turn. diff --git a/docs/v6/reference/runtime.mdx b/docs/v6/reference/runtime.mdx index 4264a4ed6..70dd359d2 100644 --- a/docs/v6/reference/runtime.mdx +++ b/docs/v6/reference/runtime.mdx @@ -80,17 +80,21 @@ RuntimeConfig(resources=RuntimeResources(tpu=RuntimeTPU(type="v5", topology="2x2 Support differs per runtime. Providers reject unsupported requirements except `storage_mb`, which is best effort: `DockerRuntime` admits against available disk and `DaytonaRuntime` provisions enough whole GiB, while providers without disk sizing proceed with their default capacity. -Daytona accepts a list of GPU alternatives; Modal accepts one GPU type. Docker ignores `limits`; -Daytona rejects `run_timeout_s` and resource overrides when booting from an already-built snapshot. -`LocalRuntime` rejects a per-task `runtime_config`. +`HUDRuntime` proceeds on the deployed environment's defaults and warns when it ignores CPU, memory, +or storage requirements; it rejects GPU, OS, TPU, and `limits`. Daytona accepts a list of GPU +alternatives; Modal accepts one GPU type. Docker ignores `limits`; Daytona rejects `run_timeout_s` +and resource overrides when booting from an already-built snapshot. `LocalRuntime` rejects a +per-task `runtime_config`. `run_timeout_s` bounds the lifetime of the provisioned environment, not the complete rollout. -The optional `rollout_timeout` passed to `Task.run` or `Taskset.run` bounds submission, queueing, -provisioning, the agent, and grading together. When both are set, `run_timeout_s` and other known -phase limits must be less than `rollout_timeout`; an agent timeout must also be less than the -actor environment's `run_timeout_s`. Contradictory values fail before execution. When -`rollout_timeout` is omitted, the SDK adds no overall deadline; configured phase limits and limits -imposed by the selected runtime still apply. +The optional `rollout_timeout` passed to `Task.run` or `Taskset.run` is the execution deadline +through grading, including submission, queueing, and provisioning. It is not an upper bound on +method-return latency: cancellation and trace reporting still run after expiry, and `Taskset.run` +then flushes telemetry. When both are set, `run_timeout_s` and other known phase limits must be less +than `rollout_timeout`; an agent timeout must also be less than the actor environment's +`run_timeout_s`. Contradictory values fail before execution. When `rollout_timeout` is omitted, the +SDK adds no overall deadline; configured phase limits and limits imposed by the selected runtime +still apply. ## Runtime directory @@ -112,8 +116,9 @@ Runs an environment in this process. `source` can be: envs), called fresh per rollout with the placed row. `ready_timeout` bounds `@env.initialize` startup. Paths and constructors create a fresh environment -per rollout. Env hooks share this process and event loop; use `SubprocessRuntime` or -`DockerRuntime` when rollouts need process isolation. +per rollout. A live `Environment` instance is shared and guarded by a lock, so acquisitions through +that `LocalRuntime` are serialized. Env hooks share this process and event loop; use +`SubprocessRuntime` or `DockerRuntime` when rollouts need process isolation. ### `SubprocessRuntime` diff --git a/docs/v6/reference/tasks.mdx b/docs/v6/reference/tasks.mdx index 4db2241eb..7a21ec070 100644 --- a/docs/v6/reference/tasks.mdx +++ b/docs/v6/reference/tasks.mdx @@ -162,15 +162,17 @@ job = await ts.run(agent, runtime=LocalRuntime("env.py"), group=8, max_concurren | `group` | `int \| None` | Repeats each task N times for GRPO. | | `max_concurrent` | `int \| None` | Caps how many rollouts run in parallel. | | `job` | `Job \| None` | An open [`Job`](/v6/reference/types#job) to accumulate into; defaults to a fresh job per call. | -| `rollout_timeout` | `float \| None` | Optional hard deadline (seconds) for the full rollout lifecycle in every placement; omitted means no SDK-wide deadline. | +| `rollout_timeout` | `float \| None` | Optional execution deadline (seconds) through grading in every placement; omitted means no SDK-wide deadline. | `agent_config.timeout_seconds` bounds only the agent phase. If it expires, the run is marked as timed out but grading still runs against any completed work. -`rollout_timeout` instead bounds startup, the agent, and grading together; runtime cleanup may -finish in the background after a timeout. When both are set, the agent timeout must be less than -the rollout timeout and any explicit actor `run_timeout_s`. Limits imposed by the selected runtime -still apply when `rollout_timeout` is omitted. Grader-specific limits, such as -`BashGrader.timeout_seconds`, remain local to that grader. +`rollout_timeout` instead bounds submission, queueing, startup, the agent, and grading together. +It is not a wall-clock bound on `Task.run` or `Taskset.run` returning: after expiry, local execution +may wait up to two seconds for cancellation and still reports the trace, and `Taskset.run` can spend +up to 120 seconds flushing telemetry. Runtime cleanup may finish in the background. When both are +set, the agent timeout must be less than the rollout timeout and any explicit actor +`run_timeout_s`. Limits imposed by the selected runtime still apply when `rollout_timeout` is +omitted. Grader-specific limits, such as `BashGrader.timeout_seconds`, remain local to that grader. A crashed rollout comes back as a failed [`Run`](/v6/reference/types#run) inside the job rather than raising, so one bad rollout never collapses a batch. For the end-to-end workflow see diff --git a/docs/v6/reference/telemetry.mdx b/docs/v6/reference/telemetry.mdx index fb44eb938..c61bf54c2 100644 --- a/docs/v6/reference/telemetry.mdx +++ b/docs/v6/reference/telemetry.mdx @@ -18,15 +18,26 @@ hud set HUD_TELEMETRY_ENABLED=0 or set `HUD_TELEMETRY_ENABLED=0` in your environment. When you run evaluations with an API key configured, the SDK uploads trace -spans (steps, tool calls, timings) to your team's traces on the HUD platform. -This is the product feature that powers the trace viewer — it is scoped to -your team and requires your API key. +spans to the HUD platform. Depending on the agent, task, instrumentation, and +workspace configuration, span payloads can include: + +- prompts and other structured messages, task arguments, and task setup and grading results; +- model content and reasoning, tool calls, raw provider data, usage, and token-level samples; +- complete tool requests and results; +- serialized arguments and results from instrumented functions, plus exception messages; and +- workspace setup and agent file diffs and final artifacts when workspace file tracking is active. + +Spans also include execution metadata such as step order, timings, status, and errors. Trace +uploads require your API key and power the trace viewer. - **No API key (the default):** spans are not uploaded anywhere. -- **`HUD_TELEMETRY_LOCAL_DIR=./spans`:** spans are also written to local - files. Works without an API key, and keeps working with telemetry disabled. -- **Fully local:** `HUD_TELEMETRY_ENABLED=0` plus `HUD_TELEMETRY_LOCAL_DIR` - keeps every span on your machine and sends nothing. +- **`HUD_TELEMETRY_LOCAL_DIR=./spans`:** emitted spans are also written to local + files, including without an API key or with backend telemetry disabled. Local writes are + best-effort; filesystem errors do not fail the evaluation. +- **Fully local trace storage:** `HUD_TELEMETRY_ENABLED=0` plus + `HUD_TELEMETRY_LOCAL_DIR=./spans` stops backend trace uploads and writes other emitted spans + locally on a best-effort basis. Disabling trace telemetry also stops workspace file-tracking + spans. CLI analytics are independent; set `HUD_CLI_ANALYTICS_ENABLED=0` to disable them too. ## Anonymous CLI usage events @@ -38,9 +49,9 @@ hud set HUD_CLI_ANALYTICS_ENABLED=0 or set `HUD_CLI_ANALYTICS_ENABLED=0` in your environment. -Each CLI invocation sends one small event recording which command ran and -how it exited, so HUD can improve the commands people actually use and fix -the ones that fail. The payload is a fixed allowlist: +Except for `hud --version`, each CLI invocation sends one small event recording +which command ran and how it exited, so HUD can improve the commands people +actually use and fix the ones that fail. The payload is a fixed allowlist: | Field | Example | | --- | --- | @@ -58,7 +69,6 @@ task file, a trace id, a typo, is recorded as `other` or omitted, so positional input is never transmitted. The install id is a random identifier for one machine — it is not tied to your account. -Events are sent in the background and abandoned after about a second, so a -slow or failed send can never break a command and adds at most that bound to -its exit. A one-time notice is printed the first time the install id is -created. +Events are sent in the background and abandoned after 1.5 seconds, so a slow +or failed send cannot break a command and adds at most 1.5 seconds to its exit. +A one-time notice is printed the first time the install id is created.