Guardian: chunked review path has never run — routing is flag-based, needs bench evidence before enabling

[zaebee]

The chunked review path has never run in production

Split out of #275, where it was originally (and wrongly) blamed for a timeout. It is not a crash cause — it is dead code in practice.

src/cgis/guardian/chunked.py:207:

chunked = "chunked" in collector.features

Routing is purely opt-in, with no size threshold anywhere. GUARDIAN_FEATURES is unset as a repo variable, so the flag is never on.

Evidence from guardian_metrics.jsonl on data/guardian-metrics: 41 recorded runs, 41 with chunk_count: null. The connected-subgraph chunker built in #154 / #185 — union-find over IMPORTS/CALLS, per-chunk context collection, honest per-chunk degradation, MAX_CHUNKS capping, dedup and merge — has never executed on a real review.

Why it still matters

Single-pass puts the entire diff in one request, so one failed call loses the whole review. The chunked path fails per chunk: a chunk whose finder call raises contributes zero findings and a ⚠ bullet, and the rest of the review survives. That failure isolation is worth having independently of prompt size.

On #274's diff the chunker produces 6 chunks, largest ~11k tokens against 32.8k for the single prompt — roughly a 3× reduction in per-call size.

Why it is not a quick fix

Turning chunking on changes the default review path for every PR, and with it the finder's context: collect_for_chunk assembles per-chunk files and graph context rather than one whole-PR prompt. This project's recall behaviour is prompt-sensitive — see the recall work on #246 / #270, where finder precision-bias was traced to the prompt rather than the model.

So this needs bench evidence, not a threshold. scripts/guardian_bench.py replays past PRs against curated ground truth; the question to answer is whether chunked review finds the same defects as single-pass on the same PRs, and what it costs in API calls.

Suggested shape

1. Bench both paths over the existing ground-truth set; report recall and call count per path.
2. Only then decide routing: always-chunk-when-a-graph-DB-exists, or size-escalation with a threshold justified by the bench.
3. Whatever is chosen, record chunk_count behaviour in the metrics so the next question has data.

One observation for step 1: on #274 the largest chunk was a 1182-line implementation-plan markdown file. Docs-only chunks consume a finder call and cannot contain code defects — worth checking whether the chunker should skip non-source files entirely.

Related: #275 (timeout/retry — the actual cause of the #274 failure), #154 and #185 (chunker lineage).

🤖 Generated with Claude Code

[zaebee]

The cost side, measured offline — and it changes the order of work

build_chunks is pure logic, so what chunking would cost can be measured with no API calls. Ran it over three merged PRs from this repo against a real graph DB:

PR	chunks	source	non-source	non-source share of the diff
#274	6	3	3	53%
#278	7	2	5	67%
#280	7	2	5	65%

Chunking would cost 6–7 API calls where single-pass costs 1, and the majority of those calls would review documentation — spec markdown, uv.lock, pyproject.toml, .pre-commit-config.yaml.

Example, #278:

DOCS     467 chars   .pre-commit-config.yaml
DOCS  10,687 chars   docs/specs/...-timeout-retry-design.md
DOCS  27,178 chars   docs/specs/...-timeout-retry-plan.md
DOCS     332 chars   pyproject.toml
src   13,741 chars   src/cgis/guardian/{metrics,providers/base,...}.py
src    4,849 chars   tests/unit/test_guardian_providers.py
DOCS     728 chars   uv.lock

Good news first: the chunker's core works. Graph connectivity correctly grouped mcp_server.py + cli.py + fractal.py into one chunk on #274, and the test-pairing put test_guardian_providers.py beside its implementation. The waste is entirely in which files reach it.

Docs chunks are degenerate, not merely extra

collect_for_chunk already narrows context to Python before assembling a prompt:

py_files = [f for f in chunk.files if f.endswith(".py")]

So a markdown-only chunk gets no full_files and no graph context. That call asks a code reviewer to review a spec document blind, with CONTRIBUTING.md for company. It is not a weak review — it is a call that cannot succeed at what guardian is for.

Consequence for this issue's plan

Step 1 of the suggested shape ("bench both paths") would, as things stand, measure an arm where two thirds of the calls review documentation. That is a configuration nobody would enable, so the number would be real and useless.

So the order flips: filter first, then bench. Filtering is safe to do now precisely because of this issue's other finding — the chunked path has never run in production (41/41 recorded runs single-pass), so changing it cannot affect anyone today.

One correction to this issue

The suggested shape implies the comparison needs building. It does not. GUARDIAN_FEATURES=chunked already reaches the collector (guardian_bench.py:121 → 141), run_review_routed reads it, and results.jsonl records the features string, so the two arms are already distinguishable. The experiment needs no new code — only a provider key.

Cost of that experiment, for whoever runs it: 7 benchmark PRs, MAX_CHUNKS = 8 → up to 56 finder calls on the chunked arm against 7 on single-pass, plus skeptic calls.

Next

Spec for the filter: docs/specs/2026-07-30-chunk-source-filter-design.md. Routing stays flag-based and untouched — that is still gated on the recall evidence this issue asks for.

🤖 Generated with Claude Code