Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/claude-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,15 @@
# (the action accepts that only in block mode). Block mode is revisited
# before org-wide enforcement (build plan, phase 5).

name: claude-review
name: Claude of Duty

on: pull_request_target

permissions: {}

jobs:
collect:
name: Recon
runs-on: ubuntu-24.04
timeout-minutes: 10
permissions:
Expand Down Expand Up @@ -94,6 +95,7 @@ jobs:
retention-days: 3

analyze:
name: Analysis
needs: collect
runs-on: ubuntu-24.04
timeout-minutes: 10
Expand Down Expand Up @@ -198,6 +200,7 @@ jobs:
retention-days: 3

dependency-review:
name: Supply line check
needs: collect
runs-on: ubuntu-24.04
timeout-minutes: 10
Expand Down Expand Up @@ -226,6 +229,7 @@ jobs:
show-openssf-scorecard: false

oc-claude-review:
name: Mission report
needs: [collect, analyze, dependency-review]
if: always()
runs-on: ubuntu-24.04
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Before and after JSON of owner-run releases (review/tools/release.py)
release-records/
22 changes: 22 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Owner-run release targets. Not used by any workflow or CI job: the release
# tool holds an Anthropic org:admin login and GitHub write at once, which no
# job may hold. See review/tools/release.py and docs/claude-review-design.md,
# "Each release" in "Auth and cost".
#
# make release-plan SHA=<sha> [NO_CONSOLE=1] read-only plan
# make release SHA=<sha> plan, then apply at once (no prompt), read back
# make release-rule-test RULE=<id> SHA=<sha> Console-only edit of a scratch rule
# rollback: make release SHA=<previous release sha>

TOOL = cd review && uv run --frozen python tools/release.py

.PHONY: release-plan release release-rule-test

release-plan:
$(TOOL) plan --sha "$(SHA)" $(if $(NO_CONSOLE),--no-console)

release:
$(TOOL) release --sha "$(SHA)"

release-rule-test:
$(TOOL) rule-test --rule "$(RULE)" --sha "$(SHA)"
15 changes: 15 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,18 @@ Every PR in every OperationCode repo gets an automatic review from Claude: a cod
Design: [docs/claude-review-design.md](docs/claude-review-design.md)

Build plan: [docs/build-plan.md](docs/build-plan.md)

## Releases (owner only)

A release points the production federation rule and every ruleset pin at one merge SHA on main. The owner runs it from a clone of this repo. No workflow runs it. See "Each release" in the [design](docs/claude-review-design.md) and the [build plan](docs/build-plan.md).

One-time setup: install the [`ant` CLI](https://platform.claude.com/docs/en/cli-sdks-libraries/cli/authentication) with `brew install anthropics/tap/ant`, and log in to `gh` as an org owner. You do not log in to `ant` yourself and you keep no `ant` profile. The tool runs `ant auth login --profile admin --scope "org:admin"` in a private temp dir (your browser opens), and runs `ant auth logout --all` and deletes the dir when it exits. The federation rule endpoints refuse Admin API keys, and an `org:admin` login stored on disk is readable by any process that runs as you, so no key or token is kept.

```sh
make release-plan SHA=<sha> # read-only; NO_CONSOLE=1 skips the Console reads
make release SHA=<sha> # plan, then apply at once (no prompt), read back; `make release-plan` is the read-only dry run
make release SHA=<previous sha> # rollback
make release-rule-test RULE=<id> SHA=<sha> # Console-only edit of a scratch rule (refuses the production rule)
```

Before and after JSON goes to `release-records/` (git-ignored, no secrets).
26 changes: 14 additions & 12 deletions docs/build-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,10 @@ Exit criteria:
- [ ] Right after enabling the pilot ruleset, list the open PRs in the pilot repos again (do not reuse a list from earlier recon) and start a run on each (close and reopen, or a new push). A required workflow starts only on a new PR event, so a PR opened between the recon and the enable gets no review run: front-end PR 2027 (opened 2026-10-29T08:06Z) was missed this way.
- [ ] Each week, record false positives, overrides, bypasses, failed runs by reason, egress blocks and cost per run. Keep cost numbers (any currency amount) in the owner's private notes and the Anthropic Console, never in this public repo. Public text may give ratios and timing only.
- [ ] Fix false positives in prompts or code. Each fix goes through phase 2 again, then ships as in "Each release" below.
- [x] Release cf19216 (PR #21, spec K-18, K-19): npm registry responses capped at 128 MiB decoded, gzip requested with a bounded inflate, and visible failure reasons for a failed lookup. Found by the pilot: a false hard block on front-end PR 2027.
- [x] Release 96f4966 (PR #22, spec K-20, K-21): the byte budget counts received bytes with no up-front reservation, the budget is 8 GiB, script text is cut to 1000 chars, and the local harness `review/tools/dep_repro.py` reproduces the dependency stage. Rule: a dependency-check change is reproduced and validated locally with the harness before release.
- [x] Pilot result, 2026-10-31: the first real PR (`OperationCode/front-end` PR 2027, a large dependency update) was falsely blocked three times at earlier releases (the pnpm two-document lockfile parse, the lockfile diff size, then the npm cap and the byte-budget race). At 96f4966 it passes with 0 hard blocks and 0 findings, matching the local harness. `operationcode_infra` PR 191 passed with low, non-blocking findings. The test PRs were closed 2026-10-31. The pilot continues; the exit judgment is pending the owner.
- [ ] Release item (pending): "Claude of Duty" branding of the PR comment and check names (spec P4-8). Display text only; no change to markers, job ids, file names or the `test / unit-tests` check.
- [x] Keep the fixture eval results as an artifact: `eval.yml` uploads `results.jsonl` and each case's `summary.md` as `eval-results-a<N>` (14 days, `if: always()`). Before this, a release eval's per-case results existed only in the job log (phase 2 run 40445885087 and phase 3 run 40528717338 were read from logs). Ships as in "Each release".
- [ ] Plan publication before writing: bounded continuation comments, pointed to from the sticky summary, hold every summary-only finding (changed keys, unverified, outside the diff) when one comment cannot. Then no explanation is cut from the PR comments and summary capacity no longer fails a run whose computed result is `pass` (spec "Publishing (code)", size limit residual; review r-p3-cut-final Important 1). Keep the fail-closed check until this path is complete. Ships as in "Each release". Plan (planned, not built): [docs/continuation-comments-plan.md](continuation-comments-plan.md).

Expand Down Expand Up @@ -247,15 +251,13 @@ Exit criteria:

## Each release (from phase 4 on)

One federation rule allows one SHA, and every release is validated at its final SHA before a pin moves (spec "Auth and cost", "Each release"):

- [ ] Owner: create a new federation rule R2 as a copy of the current rule R1, with the placeholder SHA `0000000000000000000000000000000000000000` in `job_workflow_ref`. Read it back (V26).
- [ ] Put the R2 ID in the central code. That commit is the release candidate S2.
- [ ] Owner: set `job_workflow_ref` of R2 to S2 and read it back.
- [ ] Validate S2, for every release: unit tests pass, the owner-approved fixture eval passes, and a run under the scratch test ruleset pinned to S2 gets a token through R2.
- [ ] Merge. If the merge makes a new SHA, check that its tree equals S2 (`git diff` is empty, so the content and rule ID are the same). If the trees differ, treat the merge SHA as a new candidate and go back to the step that sets R2. That SHA is the release SHA. Phase 3 learned: every squash merge makes a new SHA, so one release takes two Console edits per rule, the candidate SHA and then the merge SHA.
- [ ] Before the second Console edit, pin the scratch test ruleset to the merge SHA and run once: the exchange must be denied, because the rule still names the candidate SHA (phase 3: run 40583290922). Then the owner sets R2 to the merge SHA and reads it back.
- [ ] Pin the scratch test ruleset to the release SHA and do one token exchange through R2. Any new candidate, for example a fix commit after a failed validation, repeats from the step that sets R2 to the candidate SHA.
- [ ] Owner: move the pin of every ruleset that requires the workflow (the pilot ruleset, the org ruleset, or both) to the release SHA, then read each back: `repository_id`, path, `sha`, no `ref`.
- [ ] Start a new run on each open PR (push, or close and reopen). Do not re-run old runs: after R1 is deleted, a re-run at the old SHA fails auth. Do this after every pin move, also a move back: the box counts only runs at the pinned SHA, so a move back revives older passes at that SHA (spec "Pinned SHA only").
- [ ] Owner: delete R1 only when each new run has completed at the release SHA (`job.workflow_sha`); a queued run does not count (V22).
Each release edits rule R1 in place, once, at the merge SHA, with the owner-run release tool (spec "Auth and cost", "Each release"; decision P4-9):

- [ ] Merge the release PR. The merge SHA is the release SHA.
- [ ] Validate at the release SHA: `test.yml` (the push run on main) and the owner-approved fixture eval (`eval.yml`, a `workflow_dispatch` run on main) both conclude success. A pull_request run at the same head SHA does not count.
- [ ] Owner: `make release-plan SHA=<release SHA>`. Read the plan: every precondition `ok`, the R1 `job_workflow_ref` change, and the pin change of each ruleset.
- [ ] Owner: `make release SHA=<release SHA>`. There is no prompt: it prints the plan and applies at once, so read `make release-plan` first. The tool edits R1 first, then moves the pin of every ruleset that requires the workflow (the pilot ruleset and the scratch test ruleset), and reads all three back. A required run started between the rule edit and the last pin move fails closed. A non-zero exit after a write prints which object is at which SHA, the records path and the recovery (the same command again, or the rollback command it prints): follow it before anything else.
- [ ] Start a new run on each open PR (push, or close and reopen). Do not re-run old runs: a run at the old SHA fails auth. Do this after every pin move, also a move back: the box counts only runs at the pinned SHA (spec "Pinned SHA only").
- [ ] Check one run under the scratch test ruleset (claude-review-test) gets a token at the release SHA.
- Rollback: `make release SHA=<previous release SHA>`, then new runs on each open PR.
- After a change to the release tool or to `ant`: `make release-rule-test RULE=fdrl_01AfACzupW3U4Bx361ZUbsyR SHA=<sha>` on the scratch rule R2 first. Done live on 2026-10-31 17:21Z (R2 `claude-review-p3-r2` moved `1cefe42` to `96f4966`, read-back showed only `job_workflow_ref` changed, logout and temp dir removed); repeat only after a later change to the tool or to `ant`.
Loading
Loading