Skip to content

[Presets] Patching framework and --previous support - #4118

Merged
peterschmidt85 merged 13 commits into
masterfrom
preset-trial-folders
Aug 12, 2026
Merged

[Presets] Patching framework and --previous support#4118
peterschmidt85 merged 13 commits into
masterfrom
preset-trial-folders

Conversation

@peterschmidt85

@peterschmidt85 peterschmidt85 commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Patching framework

To reach better performance, the agent can now patch the serving framework's source code, not just tune its configuration. Patches are saved with the preset as unified diffs and applied at startup, so dstack preset apply reproduces them:

type: service

files:
  - local_path: patches/rocm_aiter_mla.py.patch
    path: /patches/rocm_aiter_mla.py.patch
  # ...

commands:
  - for p in /patches/*.patch; do patch -p1 -d / < "$p"; done
  - vllm serve ...

Previous sessions

dstack preset create --previous <ID> (repeatable, or a previous list in the configuration) gives the agent the results of earlier sessions to analyze and improve on, instead of starting from scratch:

$ dstack preset create -f config.yml --previous e8b7e09c

Works with sessions created by this version or later.

Baseline

baseline now defaults to true, so every session has an anchor to measure against: the first trial serves the framework's recommended configuration, or reproduces the previous best when the session builds on --previous.

🤖 Generated with Claude Code

Andrey Cheptsov and others added 5 commits August 9, 2026 12:14
Andrey Cheptsov @claude
Record each preset trial and service attempt in its own directory
48f03fb 
A trial was one appended line in a shared `trials.jsonl`, mirrored into the
session directory by byte offset. An agent rewriting the file moved every
offset, and the mirror then committed a torn fragment forever: one verified
preset still carries an unparseable trial record. Service attempts had the
same shape in `verifications.jsonl`.

Every record is now written exactly once and reading is listing a directory:

- `trials/<n>/` holds the trial's compiled `task.dstack.yml` and its
  `trial.json`; the presence of `trial.json` is what marks the trial
  completed, so in-flight and torn states are visible instead of corrupting.
- `service/<k>/` holds each verification attempt's submitted YAML and a
  `verification.json` written when the attempt ends; an attempt directory
  without a result is one still in progress.
- The byte-offset record mirrors are replaced by a stateless directory
  mirror that re-lists the source and copies changed files whole, scrubbed,
  and atomically. A torn read can never be committed; the next pass
  converges. Trial and attempt directories sort numerically, pinned past 9.

The trial contract in the agent prompt is rewritten around the layout, with
every per-trial file referenced by its full path and the write order stated
where the files are defined.

The listing shows the trial being worked on rather than the completed count
while trialing, so `trialing (2/3)` cannot read as two finished;
`verifying (3/3)` keeps the completed count.

Validated end to end: a session on one RTX PRO 4500 produced three
contract-exact trial records, a recorded verification attempt, and a saved
preset that `dstack preset apply` deploys.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Andrey Cheptsov @claude
Mirror preset record files byte-exactly
49e8120 
The directory mirror decoded each file to text and re-encoded it, which
rewrote newlines (CRLF became LF on Unix, LF would become CRLF on
Windows) and replaced non-UTF-8 bytes. Harmless for the CLI's own JSON
and YAML records, but the mirror now also carries patch files, where the
recorded copy must be exactly the bytes that ran. Copy bytes verbatim
and redact at the byte level.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Andrey Cheptsov @claude
Support patching and failed-trial verification in preset creation
744f800 
The preset agent may now patch the serving framework source code,
generate custom kernels, and patch drivers during trials. Patches are
recorded as unified diffs under `trials/<n>/patches`, referenced from the
trial's `task.dstack.yml` via `files`, and applied with `patch` from its
commands, so the recorded configuration reproduces the trial exactly.
The verified service records its patches the same way under
`service/<k>/patches`, and the saved preset re-roots `files` onto the
mirrored session copies, so `dstack preset apply` reproduces the patched
configuration after the agent workspace is deleted.

When no trial meets the constraints, the best failed trial that has a
benchmark is still verified and saved instead of failing the whole
creation: the hardware's best is worth keeping even when it falls short
of what was asked. To keep that honest, the final report now records the
verified trial, presets store the requested `min_context_length` and
`max_ttft` alongside the verified values, and `dstack preset apply`
warns, instead of refusing, when the preset's verified context is below
the requested one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Andrey Cheptsov @claude
Mark constraint-breaching presets and scale the trial spark from zero
4a4fa65 
A preset saved from a failed trial shows `*` next to its benchmark, the
same mark a running session uses when it has only failed trials to show,
computed from the requested constraints the preset now stores. In the
trial spark, bars scale from zero so their heights compare as the numbers
do, failed trials are red rather than gold, and gold marks the best
result only while no trial meets the constraints. While trialing, the
`(N/M)` progress counts the trial being worked on rather than the
completed ones.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Andrey Cheptsov @claude
Add if/else/end conditionals to the preset agent prompt
6a7a374 
The system prompt rendered conditional blocks with a non-nesting
`<!--?NAME:CONTENT-->` directive. Replace it with `<!--?if NAME-->` /
`<!--?else-->` / `<!--?end-->`: blocks nest, markers may be inline or alone
on a possibly indented line, and a branch body is dedented by the one
indentation shared by all its lines. Both branches are always parsed, so an
unknown variable cannot hide behind a flag combination, and unbalanced or
malformed directives fail loudly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@peterschmidt85 peterschmidt85 changed the title [WIP] Preset patching and per-trial records [WIP] Preset patching, per-trial records, and --previous Aug 11, 2026
Andrey Cheptsov @claude
Support --previous in preset creation
6d56b96 
`dstack preset create --previous <ID>` (repeatable, or a `previous` list in
the configuration) gives the agent the records of earlier creation sessions:
their trials, configurations, patches, and reports are copied into the
workspace under `previous/`, and the prompt tells the agent to analyze them
and improve on them. With `baseline: true`, the first trial reproduces the
best comparable previous result before optimizing further.

The IDs are pinned in the session manifest, so a resumed session keeps the
same context. A still-running previous session is rejected; a chained
session whose parents were not included warns. `constraints.json` is now
written as a session record for every creation, not only under `--debug`,
so a preset's constraints survive after its workspace is deleted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Andrey Cheptsov @claude
Wrap the preset listing gracefully on narrow terminals
26d8fd9 
CONSTRAINTS and BENCHMARK now wrap together to keep their full content,
STATUS and SUBMITTED wrap at their spaces, and BASE and GPU truncate (capped
in the compact view) so a long model name cannot starve the data columns.
Previously only BENCHMARK folded while its neighbours clipped, so a narrow
terminal rendered one tall column beside single-line ones.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Andrey Cheptsov @claude
Document preset patching and limitations
8450775 
Note in the concepts intro that the agent optimizes across the serving
stack and may patch the framework's source, generate kernels, and patch
drivers. Replace the roadmap admonition with a limitations one that also
records the random-dataset and runtime-only-patch limits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Andrey Cheptsov and others added 4 commits August 11, 2026 23:58
Andrey Cheptsov @claude
Clean up preset comments and docstrings
2f75a80 
Cut comments that restate the code or explain what it doesn't do, rewrite
jargon-heavy ones concretely, and add docstrings only where a method's
intent is genuinely non-obvious.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Andrey Cheptsov @claude
Remove archive-on-delete from the preset store
ded8b91 
`dstack preset delete` now removes the preset permanently after its
confirmation. Git covers committed presets, and a user who wants a copy
can move `~/.dstack/presets/<id>/` manually.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Andrey Cheptsov @claude
Default preset baseline to true
9b4df51 
Every session gets an anchor by default: the first trial serves the
framework's recommended configuration, or reproduces the previous best
when the session builds on `--previous`. Set `baseline: false` to spend
every trial on optimization.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Andrey Cheptsov @claude
Save preset patch files with relative paths
d868201 
A preset directory is now portable: `preset.yaml` references its patch
files relative to its own location, resolved at load, so the directory
works after being copied to another path or machine. Re-saving a loaded
preset (e.g. on name reuse) keeps the paths relative. Presets saved with
absolute paths continue to load as before.

Verified end to end by applying a patched preset from a relocated copy:
all patches uploaded and applied, and the service answered.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@peterschmidt85
peterschmidt85 marked this pull request as ready for review August 12, 2026 12:51
@peterschmidt85 peterschmidt85 changed the title [WIP] Preset patching, per-trial records, and --previous Preset patching, best-effort presets, and --previous Aug 12, 2026
@peterschmidt85 peterschmidt85 changed the title Preset patching, best-effort presets, and --previous Preset patching and --previous Aug 12, 2026
@peterschmidt85 peterschmidt85 changed the title Preset patching and --previous [Presets] Patching framework and --previous support Aug 12, 2026
@peterschmidt85
peterschmidt85 merged commit 8fec761 into master Aug 12, 2026
27 checks passed
@peterschmidt85
peterschmidt85 deleted the preset-trial-folders branch August 12, 2026 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@peterschmidt85