Loop speed: fail-fast check order, keep failing output, no duplicate runs, concurrent views + 2 verification slots, stall watchdog #264

Loop speed: get to "what needs fixing" in minutes, not 40 (operator request, 2026-10-27)

Measured today

• UI loop status for any loop: a Lab-wide loop marker (crash-safe), one loop per Lab, pill shows CLI-started loops #254, one round of about 50 min:
  • planning: 4 attempts, about 5 min;
  • coding: 23 min;
  • an unexplained 10 min gap between coding finishing and checks starting;
  • checks: 16 min, run serially:
    • candidate targeted, 3.5 min;
    • protected targeted, 3.5 min again;
    • the full protected suite, 9 min. It ran even though the targeted check had already failed.
  • The failing check kept no output (604 bytes captured, stdout ""), so the cause needed a manual rerun.
• Redesign 3/5: Molecule view — plan in plain words, one primary Approve, Request changes, Retire behind a confirmation #251:
  • waited behind "another Molecule in this Lab holds the verification slot" (one verification slot per Lab), so parallel Molecules queue for checks;
  • its first checks ran 9 h overnight with no failure signal.

Changes (engine only; nothing under lab/internal/ui/web/)

1. Fail fast, cheapest first. Run checks in cost order: gofmt and vet, then build (go build ./... / go vet compile errors), then targeted tests, then the full suite. When a required check fails, the remaining required checks are recorded as not run (an earlier required check failed), never as passed. The Molecule fails at once with that evidence. A failing round should end in minutes, not after the 9-minute full suite.
2. Never drop failing output. A failing check always keeps a bounded, redacted head and tail of stdout and stderr (plus the failed test names when the check is a direct go test), in both views. An empty stdout with nonzero stdoutBytes must not happen.
3. No duplicate targeted runs. In the protected view, skip a targeted direct-go-test check when a protected baseline check covers a superset of its packages in the same view. Record it as "covered by ", never as passed on its own.
4. Candidate and protected views run concurrently when resources allow. Each view gets its own workspace copy and GOTMPDIR; the shared GOCACHE stays read-safe.
5. More than one verification slot per Lab. The resource-guarded limit defaults to 2, set in the Lab config, is refused when disk or memory is low (the existing guard), and is shown in doctor. Queued Molecules show their position.
6. Explain the gap. Record timed phase events between coding capture and the first check (tree capture, workspace prepare, cache prepare, waiting for a slot) in the ledger. Fix whatever takes most of the 10 minutes.
7. Stuck-check watchdog. A check whose process tree produced no output and no CPU for longer than its timeout, or a wall clock far past its budget (for example after sleep), ends as interrupted: stalled with a plain reason instead of running for hours.

Done when

• Tests are deterministic, using fake commands and seams with no wall-clock budgets. They cover:
  • fail-fast order and the "not run" records;
  • failing output kept in both views;
  • the "covered by" skip, which must never apply to a check outside the baseline's packages;
  • two Molecules verifying concurrently, plus refusal under low resources;
  • the phase-timing events;
  • stall detection via an injected clock.
• On the v3 Lab, a Molecule with a broken test reaches "checks failed" with the failing test's output in under 5 minutes after coding.
• Targeted checks: go test ./internal/verification/... ./internal/molecule/... ./internal/resources/....

Review checklist (read before coding)

The adversarial review checks these every time. Recent slices failed review on all of them, so build them in from the start:

• Bounds: sizes, counts, time and total work are bounded, and hitting a bound fails safe (incomplete or truncated, never silently accepted).
• Filesystem safety: paths cannot traverse. Use Lstat and O_NOFOLLOW, and check symlinks and ownership. Watch for TOCTOU races on shared dirs (/tmp). Only ever delete Forgecell-owned, marked paths.
• Untrusted input: issue text, repository files, model and tool output, and ledgers can be malformed, huge or hostile. Validate them, never interpolate them into HTML or a shell, and never let them trigger retries or approvals.
• No local leaks: no absolute local paths, environment variables or credentials in output, logs, the UI or ledgers.
• Contract changes: a changed adapter invocation bumps the protocol, and approved digests still bind exactly what was shown.
• Tests: deterministic. Use no wall-clock budgets (use readiness markers or seams), write nothing into the user's real temp or home, and cover the failure path as well as the happy path.

Plan checks (for intake)

Check category must be one of regression, candidate or setup. Use targeted package tests only as plain go test commands (for example go test ./internal/<pkg>/...); the protected view adds -json itself when protected deltas are declared. The Formula baseline already runs the full suite once, so don't add another full go test ./... check, and don't repeat its go vet or gofmt checks. A plan may hold at most 16 checks in total, counting the 5 Formula baseline checks, so prefer one combined targeted command over one check per package.