Deflake CI: TUF warn mode, readiness diagnostics, test races

[JAORMX]

Summary

The merge queue was being blocked by recurring CI flakes. Analysis of three weeks of Main build runs on main (146 runs, 75 failed) ranked the group-workload readiness family as the top three flakes, with a conformance gating gap and an envtest watch-test race behind them. This PR fixes one proven product bug (TUF outages hard-failing thv run in warn mode), closes the conformance gating gap, fixes one racy test baseline, and adds the startup diagnosability that turned the group-readiness failures from mysteries into named causes.

Correction — this PR body previously claimed a different mechanism for the group-readiness failures, and that claim is withdrawn. An earlier revision asserted that #6043's workload-name lengthening introduced a deterministic >63-char name-length failure, and that "every PR based on ≥be0e0c34 fails the core shard deterministically". Both statements were wrong. The instrumented run on this very PR refuted them: a 56-character workload failed the same way, and the 62-vs-66-char split observed across seven earlier runs was just which participant lost a 4-way startup race. What the diagnostics in this PR actually captured, on this PR's own first core-shard run:

• status "error" mode — the (previously clobbered, now visible) status context read: failed to create ingress container: ... Bind for 127.0.0.1:8081 failed: port is already allocated. The ingress proxy port was derived as UpstreamPort + 1, so four concurrently starting workloads all contended for 8081. Root-cause fixed on main by Stop deriving the ingress proxy port from a fixed upstream port #6069 ("Stop deriving the ingress proxy port from a fixed upstream port").
• status "starting" mode — the new probe-progress lines showed last_observed: "HTTP 502" at attempts 20/35/50/65 across the full 121s while the workload's own container had been listening within ~1s: the ingress proxy never recovered its route to the backend. Root-cause fixed on main by Fix concurrent isolated-startup races (network create + ingress DNS latch) #6071 ("Fix concurrent isolated-startup races (network create + ingress DNS latch)").

Credit for both product fixes goes to #6069/#6071; this PR's contribution to that story is the instrumentation that exposed the mechanisms (and will name the next one). Relatedly: issue #6040 tracked exactly these failures and was closed against #6043, which did not fix them — the failures had a genuine product cause that #6069/#6071 have now addressed.

What remains in this PR after rebasing onto main (57e681f1):

1. Sigstore TUF outages hard-failed thv run in warn mode (product bug, root-caused here). In run 30297766499, five specs across THVIgnore, List Group, and the group suites failed in one job while tuf-repo-cdn.sigstore.dev was resetting connections: thv run fetch exited 1 with failed to create TUF client ... connection reset by peer. The cause is an inconsistency in VerifyImage: in the default warn mode, an actual failed verification only warns, but a verifier construction failure (TUF unreachable) and any non-signature verification error were returned as fatal. Warn mode now warns and continues on every verification failure; enabled still fails closed. The new unit test forces the same construction-failure path deterministically (a SigstoreURL with no embedded TUF root) with no network involved — it fails against the old code and passes with the fix.

2. Workload-startup diagnosability (proven useful by this PR's own CI run). Three gaps had made the group-readiness failures undiagnosable from CI output:

• startWorkload and the detached-spawn failure path set status error with an empty context, clobbering the failure reason RunWorkload's retry loop had just recorded. They now record err.Error() — this is what surfaced the 8081 bind race.
• The readiness probe (waitForInitializeSuccess) logged per-attempt outcomes at DEBUG only. It now logs an INFO progress line every 30s with the last observed outcome and includes it in the timeout error — this is what surfaced the persistent ingress 502.
• DebugServerState dumped only container logs, which show a healthy server precisely in these failures; the supervisor's own output goes to the proxy log file. It now dumps thv logs --proxy too.

3. Conformance quarantine list was incomplete. Suite v0.1.16 has exactly four scenarios that register client-side handlers for server→client requests and are therefore exposed to the upstream ordering race (modelcontextprotocol/conformance#407): tools-call-sampling, tools-call-elicitation, elicitation-sep1034-defaults, elicitation-sep1330-enums — verified from the suite source (src/scenarios/server/tools.ts, elicitation-{defaults,enums}.ts), not by name-matching. The gate only ignored the first two; on #6054 the race hit tools-call-sampling and elicitation-sep1034-defaults (both timing out at exactly 60s), so the job hard-failed. The intermittency was directly observed, not just inferred: the same commit failed on elicitation-sep1034-defaults on one run and passed it on the re-run. The alternation now covers all four; anything else still fails the job, and the block still documents its removal condition (#407 fixed upstream + CONFORMANCE_VERSION bump).

4. CompositeToolDefinition watch test races controller settle (3 main failures since 2026-07-06). The "should NOT trigger reconciliation" spec captures a Ready-condition-timestamp baseline right after a BeforeAll gate that only proves ObservedGeneration > 0, so a still-settling controller flips the condition inside the 2s observation window and fails the spec (confirmed in run 30250838615: timestamp changed within the 2.78s spec). The spec now waits for the Ready condition's transition time to hold stable across ~2s of polls before taking the baseline. This is unrelated to the #6069/#6071 fixes and still needed.

Dropped during rebase: the dual_era_k8s_test.go fix that was previously in this PR (bare Expect on reconcile-lagging MCPServer.Status) — fully superseded by #6057, which shipped the same Eventually-wrap plus stricter replica-count gates.

References: #6040 (closed), #6043, #6069, #6071, #6057, #5886, #5888, modelcontextprotocol/conformance#407.

Type of change

• Bug fix
• New feature
• Refactoring (no behavior change)
• Dependency update
• Documentation
• Other (describe):

Test plan

• Unit tests

• E2E tests (task test-e2e)

• Linting (task lint-fix)

• Manual testing (describe below)

• After the rebase onto 57e681f1: go build ./..., go vet, and go test on every touched package (pkg/runner/retriever, pkg/runner, pkg/workloads) are green; task lint-fix reports 0 issues. task test was not used — it panics locally on the known gotestfmt v2.5.0 bug — so the underlying go test was run directly; CI runs the real thing.

• The new TestVerifyImage case warn with unavailable verifier continues was proven to be a real regression guard: it fails against the pre-fix retriever.go (verified by stashing the fix) and passes after.

• The conformance gate change was exercised with a scripted reproduction feeding synthetic suite output through the exact sed/grep pipeline, old regex vs new: the Let Cedar read user claims the upstream asserts only in its ID token #6054 combination (sampling + sep1034) fails old / passes new; a quarantined-only failure passes both; a genuine regression — alone, alongside a quarantined flake, or as a prefix-lookalike scenario id (tools-call-sampling-extended) — fails both.

• The diagnostics commit was validated by this PR's own pre-rebase CI run (job 90109089450): it captured the 8081 bind-race error context and the ingress 502 probe history described above. A maintainer's merge of main (with Stop deriving the ingress proxy port from a fixed upstream port #6069/Fix concurrent isolated-startup races (network create + ingress DNS latch) #6071) into this branch then turned E2E Tests Core (core) green.

• Not run locally (no cluster/docker in this environment): the k8s operator suites and the CLI e2e suites. The virtualmcpserver_compositetool_watch_test.go change is compile-verified (go vet) and runs in Operator CI on this PR.

API Compatibility

• This PR does not break the v1beta1 API, OR the api-break-allowed label is applied and the migration guidance is described above.

Changes

File	Change
pkg/runner/retriever/retriever.go	Warn-mode VerifyImage warns and continues on verifier construction and verification errors; enabled unchanged (fails closed)
pkg/runner/retriever/retriever_test.go	Table-driven TestVerifyImage: warn vs enabled vs disabled vs invalid, incl. a deterministic offline verifier-construction failure
pkg/workloads/manager.go	Record err.Error() as status context in startWorkload and the detached-spawn failure path instead of clobbering it with ""
pkg/runner/runner.go	Readiness probe: 30s INFO progress line with last observed outcome; timeout error includes it
test/e2e/helpers.go	DebugServerState also dumps thv logs --proxy
test/conformance/run-conformance.sh	Quarantine alternation extended to all four server→client scenarios; comment records the source-verified set
cmd/thv-operator/test-integration/virtualmcp/virtualmcpserver_compositetool_watch_test.go	Wait for Ready-condition transition time to settle before capturing the no-reconcile baseline

Does this introduce a user-facing change?

Yes. With the default --image-verification=warn, thv run no longer fails when the sigstore TUF repository is unreachable (e.g. offline or a CDN outage) — it logs a warning and continues, consistent with how a failed verification already behaved in warn mode. --image-verification=enabled still fails closed. Additionally, a workload that fails to start now records the failure reason in its status context, and a slow-starting workload logs readiness-probe progress every 30s.

Special notes for reviewers

• What is a root-cause fix vs not: items 1 and 4 are root-cause fixes. Item 3 is a quarantine extension of a documented upstream bug, same reasoning as the existing block (Ignore flaky tools-call-sampling in conformance CI #5888). Item 2 is diagnosability only — the underlying startup races were fixed by Stop deriving the ingress proxy port from a fixed upstream port #6069/Fix concurrent isolated-startup races (network create + ingress DNS latch) #6071, not here.
• The warn-mode semantics change in retriever.go is the one behavioral change worth scrutiny: previously warn mode failed closed on infrastructure errors while failing open on verification errors. If anyone relies on warn mode blocking runs when sigstore is unreachable, this changes that — but that behavior is what broke CI during a CDN outage, and it contradicts the mode's documented intent ("prints a warning when image validation fails").
• The branch was rebased onto 57e681f1 and force-pushed (dropping three interim "merge main" commits whose content is included in the new base, and the superseded dual_era commit); history before that reflects the withdrawn name-length analysis and should not be relied on.
• Verified no file overlap with in-flight PRs Complete Modern client-facing dispatch: listen + pagination #6050, Replace vMCP Modern kill-switch with a capability gate #6033, Make forwarding tests' Legacy dependency explicit #6051, Source Cedar claims from the upstream ID token for opaque access tokens #6052, Let Cedar read user claims the upstream asserts only in its ID token #6054, Emit conformant JSON-RPC on denial and error paths #6055.

Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 55.55556% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.25%. Comparing base (57e681f) to head (1f14126).

Files with missing lines	Patch %	Lines
pkg/runner/runner.go	58.33%	4 Missing and 1 partial ⚠️
pkg/workloads/manager.go	0.00%	2 Missing ⚠️
pkg/runner/retriever/retriever.go	75.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6063   +/-   ##
=======================================
  Coverage   72.24%   72.25%           
=======================================
  Files         722      722           
  Lines       75125    75138   +13     
=======================================
+ Hits        54277    54291   +14     
+ Misses      16985    16983    -2     
- Partials     3863     3864    +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ChrisJBurns]

/retest

[jhrozek]

Using the proxy-log dump this PR adds to `DebugServerState`, I pulled the actual CI log for this PR's own `E2E Tests Core (core)` failure and found the real error underneath the "stuck in starting" / bare "error" symptom:

```
Bind for 127.0.0.1:8081 failed: port is already allocated
```

Root cause: the ingress proxy's host port was derived from the container's fixed internal listen port (`UpstreamPort + 1`), which is identical across every workload of the same image — so concurrently-started workloads of the same image (as these specs do) all raced for the same port, and whichever lost got exactly this Docker error.

Opened #6069 with the fix (pick the ingress port at random instead). It's not a fixup of this PR — no file overlap, and this PR's diagnostics are what made finding it possible — so they can merge independently in either order. Flagging here since it's the missing piece for the readiness-stall this PR documents but doesn't claim to fix.