Decision: the cgis GitHub App is a credential, not a product — revisit after #302 and #246

[zaebee]

Recording a call made on 2026-07-31 so it doesn't get relitigated from scratch.

The question that came up — should cgis become its own GitHub App, a review agent you install on a repo? — is really two questions with opposite answers.

Decided: App as credential and identity — yes, done

Shipped in #304. A GitHub App token replaces both GITHUB_TOKEN (whose PRs park their checks in action_required forever) and the PAT that was the first instinct. It is scoped to the repo, lives an hour, rotates itself, and belongs to the App rather than to a person.

Cheap: no hosting, the token is minted inside the workflow. The identity is a real side benefit — reviews posting as cgis[bot] rather than github-actions[bot] matters now that an architecture audit is a paid offering (docs/AUDIT.md).

Setup is a repo variable CGIS_APP_ID and secret CGIS_APP_PRIVATE_KEY; see #304.

Not yet: App as an installable review agent

The blocker is not infrastructure. It is that guardian's output is not yet measurable, let alone good.

Evidence from the first production runs after guardian was restored (#300):

• Precision is poor. 25 findings on drift trust batch: shallow-depth false violation, tricode cross-validation, tolerance ratchet #297, of which the majority were verifiably wrong — "unvalidated dictionary access" on code with an isinstance guard one line above, "floating-point comparison may be flaky" on a line carrying an explicit + 1e-9, "prohibited use of Any" for a pattern used throughout drift.py under green strict mypy.
• Variance dwarfs the signal. The same fix(cli): escape data interpolated into Rich console output #298 diff produced 6 → 46 → 36 findings across three consecutive runs. A reviewer whose output swings 8× on an unchanged input cannot be sold, and cannot be debugged.
• We cannot measure it. CI has no frozen-finder replay (guardian: replay a recorded finder pass in CI so skeptic A/B is measurable #302), so every skeptic comparison is swamped by finder variance. The cross-family skeptic switch (guardian: cross-model skeptic to cut recall-lean noise without killing recall #246) cut noise 39–80%, but those numbers cannot be attributed to the skeptic with any confidence.
• The recall work is still open: guardian: recall blind classes — pr-142 cosine masking + float-equality-in-tests #247, guardian: recall-lean robustness — long-output truncation/timeout + self-consistency union #248, guardian: expand finder bug-class focus-areas (mine open-code-review taxonomy; security/concurrency gaps) #258, guardian: does impact_score earn its cost? (collinear with verdict, +69% tokens) #271, Guardian: chunked review path has never run — routing is flag-based, needs bench evidence before enabling #277.

There is also an unanswered business question: today's runs cost 13–20K tokens each. On someone else's repository, somebody pays for that inference. That is pricing and rate limiting, not engineering, and it needs an answer before installation is offered to anyone.

Re-evaluation trigger

Revisit when #302 (frozen-finder replay, so results are reproducible) and #246 (a settled finder+skeptic operating point) are both closed. Until there is a repeatable measurement, a product discussion is an argument about taste.

Related ambition already parked in Vision (parked): #45 (SaaS autodoc), #51 (autonomous background agent), #67 (semantic intelligence infrastructure). This decision does not block them — it sequences them behind a working measurement.

What would change the answer early

• Finder variance drops to something defensible on a fixed input.
• A paying user asks for multi-repo installation specifically (distribution pull, rather than a technical push).