feat(v1): GEPA prompt optimization for native v1 environments#1952
Open
Ziems wants to merge 10 commits into
Open
feat(v1): GEPA prompt optimization for native v1 environments#1952Ziems wants to merge 10 commits into
Ziems wants to merge 10 commits into
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review This PR introduces a substantial new feature (GEPA system-prompt optimization for v1) with ~400+ lines of new code across 8 new files, including a new console script, adapter, config, and runner. New features of this scope warrant human review. You can customize Macroscope's approvability policy. Learn more. |
mikasenghaas
reviewed
Jul 8, 2026
xeophon
reviewed
Jul 9, 2026
xeophon
requested changes
Jul 9, 2026
Add a `gepa` CLI command that runs GEPA (Genetic-Pareto) system-prompt optimization against native v1 taskset/harness environments. The async runner holds env.serving() open and runs the blocking gepa.optimize() via asyncio.to_thread; GEPAv1Adapter marshals each rollout batch back onto the loop and injects candidates by rebuilding each Task around a data row with the new system_prompt. Rollouts stream to traces.jsonl like eval. Self-contained in verifiers/v1/gepa/ with a thin cli/gepa.py entrypoint; the v0 path (vf-gepa) is untouched.
Ziems
force-pushed
the
claude/gepa-verifiers-v1-cf6518
branch
from
81bd1e0 to
e88dd64
Compare
xeophon
reviewed
Jul 11, 2026
xeophon
reviewed
Jul 11, 2026
Call the reflection (teacher) LM via `Judge.complete` instead of a hand-constructed OpenAI client, so vf owns the client lifecycle and key/header/Prime-billing resolution. GEPA calls `reflection_lm` synchronously from the `optimize()` worker thread, so each call drives the async `complete` on a private loop.
xeophon
reviewed
Jul 11, 2026
xeophon
left a comment
xeophon
left a comment
Member
There was a problem hiding this comment.
A few remaining places should reuse the existing v1 plumbing instead of carrying parallel behavior.
Address review feedback that gepa carried behavior parallel to the rest of v1: - Extract the fixed-seed shuffle-and-slice into verifiers/v1/utils/sampling.py (`sample`) and migrate eval, server eval, debug, validate, and legacy eval onto it; gepa's split_tasks now layers only its disjoint train/val slicing on top of the shared shuffle. - Resolve the gepa CLI through the shared path (with_positional_taskset, narrow_config, extract_id, references_config_file) like eval/serve, so a leading bare taskset id works, -h renders narrowed taskset/harness help, and a missing taskset fails fast with usage. - Stop re-policing harness prompt delivery in resolve_gepa_seed_prompt; Harness.resolve_prompt already owns fold-vs-emit and warns, so the function only chooses the seed prompt now.
…e v1 CLIs Rename interrupt.install -> install_interrupt (it isn't eval-specific) and generalize its docstring, then route every entrypoint that hand-rolled the SIGTERM->KeyboardInterrupt snippet through it: gepa, validate, debug, and replay now get the same graceful shutdown eval already has -- the first Ctrl-C/SIGTERM raises to unwind teardown `finally`, and a second is swallowed so it can't orphan containers/sandboxes/tool-server runtimes mid-cleanup.
The output_dir docstring promised system_prompt.txt + metadata.json, left over from the removed save_gepa_results path; nothing writes those. Describe what the run dir actually holds — config.toml + the streamed traces.jsonl, alongside GEPA's own candidates.json / run_log.json — and fix three comments that still named the trace file results.jsonl (it's traces.jsonl). Docs only; no behavior change.
Replace the worker-thread bridge (asyncio.to_thread(optimize) + run_coroutine_threadsafe) with the v0 vf-gepa shape: optimize() runs on the main thread and the runner owns one event loop -- env.serving() is entered on it once, each rollout batch runs via loop.run_until_complete (GEPAv1Adapter.evaluate), and it's torn down in finally. This removes the cancellation hazards that bridge created: no orphaned non-daemon thread to hang shutdown, and serving() is no longer torn down while a worker thread still marshals rollouts into it. gepa.py runs run_gepa synchronously and maps KeyboardInterrupt to exit 130, matching eval. Verified end-to-end across many optimize iterations -- the interception pool serves each batch fine with the loop idle between them.
GEPA scores one rollout per task (n=1) for the per-task scalar its pareto frontier needs, but @group_reward compares >=2 rollouts of a task -- so Environment.episode raises on the first batch of a group-reward taskset, crashing the run mid-optimization with a cryptic per-task error. Detect @group_reward at startup (as run_eval does) and raise a clear, actionable error before optimizing instead.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2f98cd4. Configure here.
run_gepa enters env.serving() by hand and its finally called __aexit__ unconditionally, so a failure in __aenter__ (a tool server or the interception pool not coming up) ran teardown against a context that never entered -- violating the async CM contract and risking masking the original startup error with a follow-on one. Nest the optimize block so __aexit__ pairs only with a successful __aenter__; keep the always-needed client/loop teardown in an outer finally.
resolve_client opens an httpx pool at EvalClient construction, but the client was built before the try/finally that closes it -- so an exception while constructing ModelContext or the reflection Judge would skip client.close() and leak the pool. Build client/ctx/reflection_lm inside that try and guard the close on `client is not None` (so a resolve_client failure doesn't turn into a NameError in finally).
xeophon
approved these changes
Jul 11, 2026
Member
There was a problem hiding this comment.
#1969 puts this logic into Taskset.select but fine to keep for now
| uuid: str = Field(default_factory=lambda: str(uuid4()), exclude=True) | ||
| """Auto-generated run id — the leaf of the output dir, so runs never overwrite. | ||
| Excluded from the saved config so re-running `@ config.toml` lands in a fresh dir.""" | ||
| model: str = Field(..., validation_alias=AliasChoices("model", "m")) |
| Excluded from the saved config so re-running `@ config.toml` lands in a fresh dir.""" | ||
| model: str = Field(..., validation_alias=AliasChoices("model", "m")) | ||
| """Model id for rollouts under optimization.""" | ||
| client: ClientConfig = EvalClientConfig() |
Member
There was a problem hiding this comment.
this should prob be typed as eval client config? assume we dont need tokens/logprobs here?
| sampling: SamplingConfig = SamplingConfig() | ||
| reflection_model: str | None = None | ||
| """Teacher model that proposes new system prompts. None = reuse `model`.""" | ||
| reflection_client: ClientConfig | None = None |
| """Train tasks sampled per reflection step.""" | ||
| perfect_score: float | None = None | ||
| """Skip reflecting on a minibatch that already scores this well.""" | ||
| state_columns: list[str] = Field(default_factory=list) |
Member
There was a problem hiding this comment.
hm not sure i like this name
| """Seed system prompt. None = the first loaded task's `Task.system_prompt`, if any task | ||
| sets one (see `resolve_gepa_seed_prompt`).""" | ||
|
|
||
| max_concurrent: int | None = Field( |
Member
There was a problem hiding this comment.
default eval concurrency is 128, should we match this or is 32 deliberate?
|
|
||
|
|
||
| @dataclass | ||
| class GEPAv1Adapter: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds GEPA (Genetic-Pareto) system-prompt optimization for native v1 environments as a new
gepaCLI command. The existing v0 path (vf-gepa/verifiers/gepa/) is untouched.Why
GEPA only worked against the legacy v0
Environment.generate()API. v1 composesTaskset+Harness+Environmentwith nogenerate()and no dataset split, so there was no way to optimize a native v1 taskset. This wires GEPA into the v1 execution path (Environment.episode()/Episode.run()/Trace).How
verifiers/v1/gepa/— a self-contained package (adapter,config,dataset,reflection,runner); the CLI is a thincli/gepa.pyentrypoint. Imports nothing from v0.run_gepais synchronous, mirroring how v0vf-gepabridged sync GEPA to async rollouts: the third-partygepa.optimize()is blocking, so it runs on the main thread and the runner manages one event loop by hand —env.serving()is entered on it once, each rollout batch runs vialoop.run_until_complete(GEPAv1Adapter.evaluate), and everything is torn down infinally. No worker thread, so a Ctrl-C raises straight throughoptimize()into that teardown. Candidates are injected viaTask.model_copy(update={"system_prompt": ...}).GEPAConfiginheritsEnvConfig(likeEvalConfig). CLI resolution reuses the sharedcli/resolve.pypath (with_positional_taskset/narrow_config/extract_id/references_config_file) exactly likeevalandserve, so a leading bare taskset id works,-hrenders narrowed taskset/harness help, and a missing taskset fails fast.Judge(Judge.complete), so it reuses the v1 client config — resolved API key, extra headers (e.g. Prime team billing) — instead of a hand-constructed client.Tracestreams totraces.jsonlvia the sameon_complete→append_tracehookrun_evaluses (each trace records its candidate prompt); per-iteration progress logs through v1 logging; the best prompt prints to stdout. GEPA's own artifacts (candidates.json,run_log.json, …) land in the run dir too.verifiers/v1/utils/sampling.py::sample(the fixed-seed shuffle/slice, also adopted byrun_eval/run_eval_server/validate/debug/ the v0 bridge);split_taskslayers two disjoint train/val slices on top.verifiers/v1/utils/interrupt.py::install_interrupt(renamed frominstall), now used byeval,gepa,validate,debug, andreplayin place of one-off SIGTERM handlers.Scope / non-goals
--idenv is rejected (usevf-gepa).EnvGrouphas no v1 equivalent); single taskset per run.--serverworker pool — GEPA runs in-process.n=1) for the per-task scalar its pareto frontier needs, but@group_rewardcompares ≥2 rollouts of a task, so those tasksets can't be optimized this way.APPENDS_SYSTEM_PROMPT=False, e.g. coding agents): the harness folds the candidate into the user prompt and warns (Harness.resolve_promptowns that policy) rather than blocking. Tasksets that bake instructions intopromptinstead ofTask.system_prompt(e.g.gsm8k-v1) need--initial-prompt.Testing
echo-v1andreverse-text-v1across multiple optimize iterations: rollouts stream totraces.jsonl, all traces scored with no errors, and GEPA proposes/optimizes the system prompt.ruff+pre-commitclean.Files
verifiers/v1/gepa/—adapter,config,dataset,reflection,runnerverifiers/v1/cli/gepa.py— thegepaentrypointverifiers/v1/utils/sampling.py— sharedsamplehelper (also adopted by eval / server eval / validate / debug / legacy bridge)verifiers/v1/utils/interrupt.py—install_interruptshared graceful-shutdown handler (renamed; adopted by eval / validate / debug / replay / gepa)verifiers/v1/cli/{eval/main,eval/runner,validate,debug,replay}.py,verifiers/v1/legacy.py— migrated onto the sharedsample/install_interrupthelperspyproject.toml— register thegepaconsole scriptOpen question: docs & skill (feedback wanted)
Bugbot flagged that this doesn't update the user-facing docs (
docs/) or the GEPA skill (skills/optimize-with-environments/SKILL.md), both of which currently describe the v0prime gepa run/vf-gepaflow. I deliberately left those out of this PR and want a maintainer's call on where they should go, because I'm not sure of the intended structure:docs/training.mdhas a "Prompt Optimization withprime gepa run" section. Should the v1gepacommand be (a) a new subsection there, (b) folded into the existing section as a v0/v1 split, or (c) documented wherever the other v1 CLIs (eval,serve,replay) are described? I couldn't find an existing v1-CLI doc home to match.skills/optimize-with-environments/SKILL.mdis v0-centric (prime gepa run, v0 env config). Should it gain a parallel v1 path, be rewritten aroundgepa+GEPAConfig, or stay v0-only for now?Happy to do the doc/skill update in this PR or a follow-up once we agree on placement — just let me know the preferred structure.
🤖 Generated with Claude Code
Note
Medium Risk
Runs many paid model rollouts and bridges blocking GEPA to async v1 execution on one event loop; mitigations include upfront rejection of incompatible tasksets and shared eval-style teardown/interrupt handling.
Overview
Adds a native v1 GEPA path: new
gepaconsole script andverifiers/v1/gepa/(GEPAConfig,GEPAv1Adapter, train/val split, reflection viaJudge, synchronousrun_gepadrivinggepa.api.optimizeon the main thread with a hand-managed event loop andenv.serving()). Candidates are applied by rebuilding tasks with an updatedsystem_prompt; rollouts stream totraces.jsonllike eval. Legacy v0 envs and@group_rewardtasksets are rejected at the CLI/dataset layer.Refactors:
verifiers/v1/utils/sampling.samplecentralizes fixed-seed shuffle + limit (used by eval, server eval, debug, validate, legacy eval, and GEPA).install_interruptreplaces per-CLI SIGTERM hacks in eval, debug, validate, and replay (eval already used the shared helper; import renamed frominstall).Reviewed by Cursor Bugbot for commit f64fc94. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
What
Adds GEPA (Genetic-Pareto) system-prompt optimization for native v1 environments as a new
gepaCLI command. The existing v0 path (vf-gepa/verifiers/gepa/) is untouched.Why
GEPA only worked against the legacy v0
Environment.generate()API. v1 composesTaskset+Harness+Environmentwith nogenerate()and no dataset split, so there was no way to optimize a native v1 taskset. This wires GEPA into the v1 execution path (Environment.episode()/Episode.run()/Trace).How
verifiers/v1/gepa/— a self-contained package (adapter,config,dataset,reflection,runner); the CLI is a thincli/gepa.pyentrypoint. Imports nothing from v0.run_gepais synchronous, mirroring how v0vf-gepabridged sync GEPA to async rollouts: the third-partygepa.optimize()is blocking, so it runs on the main thread and the runner manages one event loop by hand —env.serving()is entered on it once, each rollout batch runs vialoop.run_until_complete(GEPAv1Adapter.evaluate), and everything is torn down infinally. No worker thread, so a Ctrl-C raises straight throughoptimize()into that teardown. Candidates are injected viaTask.model_copy(update={"system_prompt": ...}).GEPAConfiginheritsEnvConfig(likeEvalConfig). CLI resolution reuses the sharedcli/resolve.pypath (with_positional_taskset/narrow_config/extract_id/references_config_file) exactly likeevalandserve, so a leading bare taskset id works,-hrenders narrowed taskset/harness help, and a missing taskset fails fast.Judge(Judge.complete), so it reuses the v1 client config — resolved API key, extra headers (e.g. Prime team billing) — instead of a hand-constructed client.Tracestreams totraces.jsonlvia the sameon_complete→append_tracehookrun_evaluses (each trace records its candidate prompt); per-iteration progress logs through v1 logging; the best prompt prints to stdout. GEPA's own artifacts (candidates.json,run_log.json, …) land in the run dir too.verifiers/v1/utils/sampling.py::sample(the fixed-seed shuffle/slice, also adopted byrun_eval/run_eval_server/validate/debug/ the v0 bridge);split_taskslayers two disjoint train/val slices on top.verifiers/v1/utils/interrupt.py::install_interrupt(renamed frominstall), now used byeval,gepa,validate,debug, andreplayin place of one-off SIGTERM handlers.Scope / non-goals
--idenv is rejected (usevf-gepa).EnvGrouphas no v1 equivalent); single taskset per run.--serverworker pool — GEPA runs in-process.n=1) for the per-task scalar its pareto frontier needs, but@group_rewardcompares ≥2 rollouts of a task, so those tasksets can't be optimized this way.APPENDS_SYSTEM_PROMPT=False, e.g. coding agents): the harness folds the candidate into the user prompt and warns (Harness.resolve_promptowns that policy) rather than blocking. Tasksets that bake instructions intopromptinstead ofTask.system_prompt(e.g.gsm8k-v1) need--initial-prompt.Testing
echo-v1andreverse-text-v1across multiple optimize iterations: rollouts stream totraces.jsonl, all traces scored with no errors, and GEPA proposes/optimizes the system prompt.ruff+pre-commitclean.Files
verifiers/v1/gepa/—adapter,config,dataset,reflection,runnerverifiers/v1/cli/gepa.py— thegepaentrypointverifiers/v1/utils/sampling.py— sharedsamplehelper (also adopted by eval / server eval / validate / debug / legacy bridge)verifiers/v1/utils/interrupt.py—install_interruptshared graceful-shutdown handler (renamed; adopted by eval / validate / debug / replay / gepa)verifiers/v1/cli/{eval/main,eval/runner,validate,debug,replay}.py,verifiers/v1/legacy.py— migrated onto the sharedsample/install_interrupthelperspyproject.toml— register thegepaconsole scriptOpen question: docs & skill (feedback wanted)
Bugbot flagged that this doesn't update the user-facing docs (
docs/) or the GEPA skill (skills/optimize-with-environments/SKILL.md), both of which currently describe the v0prime gepa run/vf-gepaflow. I deliberately left those out of this PR and want a maintainer's call on where they should go, because I'm not sure of the intended structure:docs/training.mdhas a "Prompt Optimization withprime gepa run" section. Should the v1gepacommand be (a) a new subsection there, (b) folded into the existing section as a v0/v1 split, or (c) documented wherever the other v1 CLIs (eval,serve,replay) are described? I couldn't find an existing v1-CLI doc home to match.skills/optimize-with-environments/SKILL.mdis v0-centric (prime gepa run, v0 env config). Should it gain a parallel v1 path, be rewritten aroundgepa+GEPAConfig, or stay v0-only for now?Happy to do the doc/skill update in this PR or a follow-up once we agree on placement — just let me know the preferred structure.
🤖 Generated with Claude Code
Changes since #1952 opened
verifiers.v1.gepa.runner.run_gepafunction [f64fc94]run_gepafunction to initialize HTTP client, model context, and reflection LM inside a try block with the client variable initialized to None beforehand, added conditional guard to close the client only if successfully created, and ensured cleanup of both client and event loop occurs in the outer finally block regardless of initialization failures [18ee54e]