Multi-harness CI eval rig: harness adapters + local-Docker backend + with/without-Lightcone A/B (LCR-131)

[cailmdaley]

What this is

LCR-131 — extend the per-PR eval (lc eval, the supernova/snae rig) so a skill/plugin change gets graded across multiple agent harnesses, and so we can measure whether the Lightcone layer itself helps an agent. This PR delivers the rig and a first end-to-end run.

It's a local-Docker demo, not yet wired into CI (GitHub Actions + org-owned keys deferred — see open questions).

What's built

The one place Claude Code was hardwired (EvalSandbox.exec_claude) is now a clean seam:

• harnesses/ — a Harness ABC (install · credentials · prepare · invoke) + a normalized AgentResult, with claude / codex / pi adapters and a registry. Grading is untouched — it reads the materialized filesystem (lc status, astra validate), never the agent — so it stays harness-agnostic.
• backends/ — a Sandbox ABC + LocalDockerSandbox: one throwaway container per trial from a cached per-harness image (python 3.12 + Node 22 + claude/codex/pi). Auth is copied in from the host (claude via CLAUDE_CODE_OAUTH_TOKEN, codex via ~/.codex, pi via ~/.pi). The existing Daytona backend stays for the eventual CI path.
• Matrix run + report — run_eval schedules tasks × harnesses × skill_variants × trials; the report grows a task × harness × {with, without}-Lightcone scorecard with a Δ-lift column (the with−without delta). Per-harness model is configurable.
• Fixes: a Daytona auto_stop_interval backstop (the budget-leak bug named in the issue — a skipped teardown could otherwise leak a forever-running sandbox); and the snae seed astra.yaml conformed to astra-spec 0.0.10 (it had drifted to fields the current schema rejects and failed astra validate).

49/49 eval unit tests green.

First results

{claude, codex, pi} × {with, without skills} on snae (fit Union2.1 SNe Ia to flat ΛCDM via MAP), real agents in local Docker, on a deliberately comparable, cheap model tier — claude haiku, codex gpt-5.4-mini, pi gpt-5.4-mini (via GitHub Copilot):

        Eval Matrix: score by harness × Lightcone layer
  Task   Harness   with skills        without skills     Δ lift
  snae   claude    1.00  pass 100%    1.00  pass 100%    +0.00   ($0.42 / $0.45)
  snae   codex     1.00  pass 100%    1.00  pass 100%    +0.00
  snae   pi        1.00  pass*  0%    1.00  pass*  0%    +0.00

The rig works end to end across all three harnesses — 6/6 trials, both arms, fully built the analysis (wrote the fit + plot scripts, ran them through lc run, materialized all three outputs) and passed spec_valid. claude authenticates via an OAuth token, codex via ChatGPT-account auth, pi via a GitHub Copilot subscription — all inside throwaway local containers.

The score-lift is +0.00 — snae saturates, even on this weaker/cheaper tier. Every arm hits 1.00. And the efficiency signal we hoped might carry the lift does not robustly replicate: durations are close and mixed (claude with/without 328 / 370 s, codex 363 / 442 s, pi 596 / 559 s — pi reversed) — noise at n=1, not a clean with-beats-without ordering. (An earlier run on the harnesses' default models showed a dramatic pi efficiency gap, but that was a weak default model floundering without guidance, not a stable effect.)

Conclusion: snae can't discriminate the Lightcone layer's value — it's too easy. The rig is sound; the task set is the work.

* pi reads 1.00 / pass 0% because it builds everything correctly but doesn't emit the BUILD_COMPLETE marker the way Claude Code does — its result is real; the completion flag is a per-harness detection nuance.

Open questions (this is where feedback helps most)

1. Designing discriminating tasks — now the central question. What analyses are hard enough that the Lightcone layer changes the outcome, not just the wall-clock? (more outputs, trickier spec features, deliberate failure-diagnosis paths.)
2. Trials per cell + what to report — n=1 is too noisy; we likely want pass@k over several trials, and possibly cost/turns alongside score.
3. Model tier — this run standardized on haiku / gpt-5.4-mini / gpt-5.4-mini (comparable, cheap). Is that the tier we certify on, or a spread?
4. CI surface — local Docker today; GitHub Actions vs Daytona for the actual per-PR check, and org-owned credentials rather than personal ones.
5. pi completion detection — small adapter refinement so a correct pi build registers as complete.

🤖 Generated with Claude Code

— Claude, on behalf of Cail

[github-actions]

❌ Eval Results

Metric	Value
Score	0.00
Build complete	❌
Cost	$0.00
Turns	0
Duration	0s
lightcone-cli	0.3.7.dev47+g9d1c15fa5 (9d1c15fa)
Results	Download

Graders

No grader results

Full output

01:18:17 lightcone.eval.build Building lightcone-cli wheel from /home/runner/work/lightcone-cli/lightcone-cli ...
01:18:23 lightcone.eval.build Built lightcone_cli-0.3.7.dev47+g9d1c15fa5-py3-none-any.whl (commit 9d1c15fa)
01:18:25 lightcone.eval.harness Trial build-snae-claude-skills-0 failed: Failed to create sandbox: Invalid credentials
Traceback (most recent call last):
  File "/home/runner/work/lightcone-cli/lightcone-cli/src/lightcone/eval/harness.py", line 149, in run_trial
    sandbox.create()
  File "/home/runner/work/lightcone-cli/lightcone-cli/src/lightcone/eval/sandbox.py", line 158, in create
    self._sandbox = self._daytona.create(
                    ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/lightcone-cli/lightcone-cli/.venv/lib/python3.12/site-packages/daytona_sdk/_utils/errors.py", line 206, in sync_wrapper
    process_n_raise_exception(e)
  File "/home/runner/work/lightcone-cli/lightcone-cli/.venv/lib/python3.12/site-packages/daytona_sdk/_utils/errors.py", line 139, in process_n_raise_exception
    raise create_daytona_error(
daytona_sdk.common.errors.DaytonaAuthenticationError: Failed to create sandbox: Invalid credentials
  snae trial 0: score=0.00 error: Failed to create sandbox: Invalid credentials

lightcone-cli: 0.3.7.dev47+g9d1c15fa5 (HEAD 9d1c15fa)
ASTRA: 0.2.9

  Eval Results: Scores  
┏━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Task ┃     Score     ┃
┡━━━━━━╇━━━━━━━━━━━━━━━┩
│ snae │ 0.00 +/- 0.00 │
│      │  pass@k: 0%   │
│      │   1 errors    │
└──────┴───────────────┘

   Eval Results: Cost &   
         Duration         
┏━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Task ┃ Cost / Duration ┃
┡━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ snae │      $0.00      │
│      │       0s        │
└──────┴─────────────────┘

Total: 1 trials, $0.00, 0s

   Eval Matrix: score by harness × Lightcone layer   
┏━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Task ┃ Harness ┃   with skills   ┃ without skills ┃
┡━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ snae │ claude  │ 0.00  pass@k 0% │       —        │
│      │         │      1 err      │                │
└──────┴─────────┴─────────────────┴────────────────┘

Results saved to: eval-results/build-9d1c15fa/results.json

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	lightcone-cli	b7fd9e4	Commit Preview URL Branch Preview URL	Jun 14 2026, 01:19 AM