fleet-playbook-curator: key citations on node_id, not the mutable repo name

[JRichlen]

Split out of #134, which is docs-only again. The first commit is a cherry-pick onto main. The two follow-ups address review findings on this PR.

The defect

The plugin's thesis is "membership is joined on node_id, never full_name". That holds for the manifest and for the diff. It breaks at the claim ledger.

Stage	Keys on
list-fleet-members.sh	node_id	✅
diff-fleet.sh	node_id — every bucket carries it	✅
gather-context.sh:31	reduces each member to full_name and drops node_id	❌
validate-citations.sh	can therefore only match full_name	❌
index.schema.json	no node_id field, and additionalProperties: false forbids adding one	❌
PROMPT.md:42-43 (the only instructions the deployed curator reads)	lists repo, path, sha, curated_at as the ledger fields	❌

What that does in practice — measured against the real validator

A plain rename already failed closed. The old name is missing from context.json, so the claim reads as fabricated and the validator exits 1. The finding as I first recorded it in #134 said the ledger "cannot survive a rename". That was wrong. It breaks safely.

What actually goes wrong is a reused name. The original repo leaves the fleet and a different repository takes its old name. The stale citation then matches the impostor's gathered tree and resolves to the wrong repository with exit 0. Nothing in the output says so. GitHub's rename documentation warns that reusing an old name retargets the redirect.

The fix

• gather-context.sh writes node_id into every context entry.
• index.schema.json accepts node_id as an optional claim field.
• validate-citations.sh: a claim that carries node_id is matched on node_id only. A rename resolves with a note. A reused name fails closed, including against a context that has no node_ids (3e8d366, from a Copilot finding). A claim without node_id falls back to full_name, and the validator says so on every run.
• PROMPT.md requires node_id on every ledger entry, copied verbatim from context.json (ceb325d, from a Codex finding). Without this, the deployed curator would never have written a keyed claim.

Prose claims corrected

Both were checked against GitHub's own documentation and neither is supported. They are now fixed in SKILL.md, templates/fleet.example.yaml and scripts/list-fleet-members.sh:

• "GitHub's stable node_id" → opaque. GitHub documents it only as the value to persist "across API versions", and states that the legacy global-ID format "will be closing down and replaced with a new format".
• "strongly-consistent" → authoritative. GitHub publishes no consistency guarantee for REST list endpoints.

Gates

• Cheap tier: 1302 passed, 0 failed. New checks cover: the hijack is rejected, including against a legacy context; gather-context carries node_id; the schema admits it; PROMPT.md's field list includes it. Each check was confirmed to go red with its fix reverted.
• Deep tier (pier): the seed context carries node_id, the oracle writes keyed claims, and the verifier fails any file citation without one. After the ANTHROPIC_API_KEY account was fixed, the re-run on 3e8d366 passed with a real agent: claude-code OK (reward=1), oracle OK (reward=1), nop OK (reward=0). So the agent read the new PROMPT.md and wrote keyed claims unprompted.
• Behavioral tier: the same re-run passed all six scenarios 3/3 under the Sonnet grader. That includes the new one, where the user asks for "just repo/path/sha" and the subject still copies node_id.
• Demonstration comments: posted below for each skill change.

What this does not fix

• Existing ledgers, and a curator that ignores PROMPT.md. node_id stays optional for backward compatibility, so validate-citations.sh only warns on an unkeyed claim. Only the pier verifier fails one. A strict mode for new claims would need a way to tell them from old ones. It has not been built.
• The legacy global-ID format migration. diff-fleet.sh joins manifests across passes on node_id. If one manifest was captured before GitHub's format change and the other after, every member looks like removed + added instead of renamed. It is item 6 on the knowledge-management corpus roadmap in Research: where the Harness Knowledge Graph lands, and what structure fits instead #134.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-23T02:09:02.124146Z	853c817	PR opened
🔒 Security Review	✅ Completed 2026-09-23T02:20:30.005949Z	853c817	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[JRichlen]

Demonstration — fleet-playbook-curator on real input

This PR edits a SKILL.md and two files under a skill's scripts/, so the repo's demonstration discipline applies. Everything below was run on this PR's branch (853c817), and the before comes from main (9ace33a), this PR's actual base. I re-ran all of it here instead of relying on the earlier run on #134, because a cherry-pick is a new commit.

The skill's deterministic core is list-fleet-members.sh → diff-fleet.sh → gather-context.sh → validate-citations.sh. I exercised the last stage with context bundles shaped exactly like gather-context.sh output, modelling two behaviours GitHub documents: a rename, and an old name reused by a different repository.

Input: a claim curated before anything moved, citing acme/billing-svc@aaaaaaa:README.md.

The case that bites: the old name taken by a different repository

The original repo left the fleet, and a new repo (different node_id) now holds acme/billing-svc.

Before (main):

validate-citations: all 1 citations traceable to a read surface or the manifest.
-> exit=0

The check is green, but the citation resolved against the impostor's tree. Anyone following it lands on an unrelated repository and nothing tells them. That happens even though the claim carries a node_id, because main's validator never reads that field.

After (this PR):

FABRICATED CITATION: claim cites acme/billing-svc@…:README.md (node_id R_kgDO_ORIGINAL)
but that repository was not read this pass (no such node_id in context.json).
A renamed member keeps its node_id; a REUSED name does not.
-> exit=1

The rename case, which was never the danger

The same repo, renamed to acme/payments-svc and still in the fleet.

Claim form	Before (main)	After
legacy (no node_id)	exit 1, fails closed	exit 1, unchanged
node_id-keyed	exit 1, fails closed	exit 0 with note: …renamed to acme/payments-svc…the citation still resolves because it is keyed on node_id

A plain rename was always fail-closed. The difference is that a keyed citation now follows the rename instead of being thrown away as fabricated.

The rule that produced each change

gather-context.sh:31 reduced each member to full_name and dropped node_id, one line before the validator needed it. The manifest and the diff are keyed on node_id and the ledger was not. There was exactly one seam.

Proof the new check can go red

I disabled the node_id preference in the validator and re-ran the cheap tier:

[FAIL] validate-citations ACCEPTED a node_id-keyed citation whose repo left the fleet
       while its NAME was reused — the citation resolved to the wrong repository

With the preference restored, the cheap tier on this branch is 1299 passed, 0 failed.

The misses

• My first attempt at the reused-name fixture was wrong (on Research: where the Harness Knowledge Graph lands, and what structure fits instead #134, before the split). I left the renamed original in the fleet next to the impostor, predicted FAIL, and got a pass. The code was correct: the node_id followed the rename. My fixture just didn't model the threat. I rebuilt it before drawing any conclusion.
• Legacy ledgers are flagged, not fixed. node_id is optional for backward compatibility, so any claim curated before this change still matches on the mutable name and still has the hole. You can see this in the legacy + reused name row above: exit 0, now with a warning printed, but not a failure.
• This is unit-level, not end-to-end. The context bundles are synthetic. I never called the real gh api, so this proves the validator's logic, not that a live fleet pass fills in node_id on every code path.
• Neither prose correction is demonstrated here. Changing "stable node_id" to "opaque" and "strongly-consistent" to "authoritative" fixes unsupported claims, checked against GitHub's docs. No run here exercises them; the behavioral tier covers prose.
• The cross-format-migration case is not covered. If manifests straddle GitHub's legacy→new global-ID migration, every member reads as removed+added. That is out of scope and listed in the PR body.

---

Generated by Claude Code

[JRichlen]

CI red — not this PR's. confirm grader model resolves fails, and behavioral tier (promptfoo) fails only because of it (GRADER: failure, RUN: skipped). One root cause:

##[error]grader model 'claude-sonnet-5' did not resolve (HTTP 400).

Why it isn't this PR's

• This PR touches no promptfoo config and no workflow file. All 12 packs use anthropic:messages:claude-sonnet-5, inherited unchanged from main.
• Research: where the Harness Knowledge Graph lands, and what structure fits instead #134 is now docs-only and fails identically. It's the control.
• The same config passed on Research: where the Harness Knowledge Graph lands, and what structure fits instead #134 at 23:03Z 2026-09-22, and started failing about 3h later. Nothing in the repo changed in between.

What's actually wrong can't be read from the log. claude-sonnet-5 is a current, valid model ID. An unknown model returns 404; this is 400 (invalid_request_error). So the check's advice to "fix the slug" is very likely wrong. My best guess is account-level: Anthropic returns 400 when the credit balance is too low. I can't confirm that, though, because the check runs curl -o /dev/null and throws the API's error body away.

Fix: there's nothing I can port in-repo. Whoever holds the ANTHROPIC_API_KEY secret needs to look at the account (credit balance, key status, model access).

Proposed patch, not pushed here, so the next failure explains itself (.github/workflows/evals.yml, grader ping loop):

-            code=$(curl -sS -o /dev/null -w '%{http_code}' https://api.anthropic.com/v1/messages \
+            resp=$(mktemp)
+            code=$(curl -sS -o "$resp" -w '%{http_code}' https://api.anthropic.com/v1/messages \
               ...
             if [ "$code" = "200" ]; then
               echo "grader model '$grader' resolved OK."
             else
-              echo "::error::grader model '$grader' did not resolve (HTTP $code). Fix the slug in that plugin's promptfooconfig.yaml."
+              why=$(jq -r '"\(.error.type // "unknown"): \(.error.message // "")"' "$resp" 2>/dev/null || head -c 300 "$resp")
+              case "$code" in
+                404)     hint="Fix the slug in that plugin's promptfooconfig.yaml." ;;
+                401|403) hint="The ANTHROPIC_API_KEY secret is invalid or lacks access." ;;
+                *)       hint="Not a slug problem - read the API error above." ;;
+              esac
+              echo "::error::grader model '$grader' failed (HTTP $code) - $why. $hint"
               fail=1
             fi
+            rm -f "$resp"

⚠ Specific to this PR: because the grader never resolved, the promptfoo run was skipped, not passed. This PR changes SKILL.md prose, which is exactly what the behavioral tier exists to check, so that change currently has no behavioral coverage. The cheap tier (1299) and the pier run still hold. But this shouldn't merge on the strength of a behavioral tier that never executed. Once the account is fixed, it needs a real run.

Re-running the failed jobs once to confirm it reproduces.

---

Generated by Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 853c817f57

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Copilot review overview

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 1 · 1

Open (3)

• Use claim presence to select node_id versus legacy fallback · New
• Require node_id in curator prompts and copied instructions · New
• Update shipped guidance for node_id and endpoint semantics · New

What changed in this PR

This PR hardens fleet citation identity by carrying GitHub node_id through context gathering and validation, while preserving legacy ledger compatibility.

Changes:

• Adds optional node_id claim support and identity-aware validation.
• Updates skill prose and adds reused-name hijack coverage.
• Retains legacy full_name fallback with diagnostic output.

File	Description
plugins/​fleet-playbook-curator/​skills/​fleet-playbook-curator/​templates/​fleet-playbook/​index.schema.json	Updated as part of this pull request.
plugins/​fleet-playbook-curator/​skills/​fleet-playbook-curator/​SKILL.md	Updated as part of this pull request.
plugins/​fleet-playbook-curator/​skills/​fleet-playbook-curator/​scripts/​validate-citations.sh	Updated as part of this pull request.
plugins/​fleet-playbook-curator/​skills/​fleet-playbook-curator/​scripts/​gather-context.sh	Updated as part of this pull request.
plugins/​fleet-playbook-curator/​evals/​cheap/​fixtures/​cite-index-hijack.json	Updated as part of this pull request.
plugins/​fleet-playbook-curator/​evals/​cheap/​fixtures/​cite-context.json	Updated as part of this pull request.
plugins/​fleet-playbook-curator/​evals/​cheap/​checks.sh	Updated as part of this pull request.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JRichlen]

Demonstration: ceb325d (PROMPT.md requires node_id)

Input. The PR's existing name-reuse fixture, evals/cheap/fixtures/cite-index-hijack.json. One claim cites o/new-svc:README.md and was curated when that name belonged to repository R_ORIGINAL. This pass, cite-context.json shows the name o/new-svc now belongs to R_NEW.

Before. This is the ledger the old PROMPT.md:42-43 field list produced (repo, path, sha, curated_at, no node_id):

validate-citations: all 1 citations traceable to a read surface or the manifest.
validate-citations: 1 claim(s) carried no node_id and were matched on the MUTABLE full_name. …
exit=0

The citation resolves to the wrong repository and CI passes.

After. The same claim written under the new field list (node_id copied from context):

FABRICATED CITATION: claim cites o/new-svc@…:README.md (node_id R_ORIGINAL) but that repository was not read this pass (no such node_id in context.json). A renamed member keeps its node_id; a REUSED name does not.
validate-citations: 1 non-traceable citation(s) — failing.
exit=1

Rule that produced the change: PROMPT.md step 2: "every claim has an entry with node_id, repo, path, sha, curated_at … Copy node_id verbatim from the repo's entry in context.json."

Pier verifier, run locally against the task's real seed. Docker was not used. The paths were rewritten to a scratch dir.

oracle ledger (keyed)            … PASS every file citation carries node_id      reward=1
same ledger, node_id stripped    … FAIL 1 file citation(s) carry no node_id      reward=0

Misses

• No model has run the new prose yet. Both runs above are deterministic: they show what the rule enforces once a model follows it, not whether a model does. The behavioral scenario and the claude-code pier agent could show that. Both are currently blocked by the ANTHROPIC_API_KEY account problem already reported on this PR. This container has no key either.
• The production gate still accepts unkeyed claims. validate-citations.sh only warns on them, so that older ledgers keep validating. A curator that ignores PROMPT.md still passes fleet-sync.yml CI. Only the pier verifier fails it. One option is a strict mode (fail on unkeyed new claims), which would need a way to tell new claims from old ones. I have not built it.
• fleet.example.yaml still said "stable node_id" after the last commit. My earlier sweep for "stable" missed it. It is fixed here.

---

Generated by Claude Code

[JRichlen]

Demonstration: 3e8d366 (a keyed claim fails closed regardless of context format)

Input. cite-index-hijack.json, a claim keyed R_ORIGINAL that cites o/new-svc:README.md. It is checked against cite-context.json with the node_id removed from every context[] entry. That is the context shape from before gather-context emitted node_ids, and it is now committed as cite-context-legacy.json.

Before (the validator at ceb325d):

validate-citations: all 1 citations traceable to a read surface or the manifest.
validate-citations: 1 claim(s) carried no node_id and were matched on the MUTABLE full_name. …
exit=0

It accepts the reused name, and the summary is also wrong: the claim did carry a node_id.

After:

FABRICATED CITATION: claim cites o/new-svc@…:README.md (node_id R_ORIGINAL) but that repository was not read this pass (no such node_id in context.json). …
exit=1

Rule: if [ -n "$node_id" ]. The branch now depends on the claim alone, and an empty read_node_ids fails the lookup instead of routing to the full_name fallback. cite-index-good.json still passes against the normal context (exit 0, 3 citations).

Misses

• The new cheap fixture strips node_ids from context[] only. removed[] still carries R_GONE, as it did before this PR, because the diff always had node_ids. I did not test a context with no removed[] node_ids, since the validator never reads removed[].
• This makes a keyed claim strictly less forgiving. If someone runs a new ledger against an old gather-context.sh, every keyed file claim now fails. I think that is correct: it is the hijack case. But it is a behavior change for a mixed-version install, and no test here covers an upgrade sequence.

---

Generated by Claude Code

[JRichlen]

deep tier — pier run (fleet-playbook-curator) is red on 3e8d366, and the cause is outside this PR. The part of the check that tests this PR's change passes.

claude-code  ERROR (ERR:NonZeroAgentExitCodeError)
oracle       OK (reward=1, expected=1)
nop          OK (reward=0, expected=0)

• Oracle and nop both pass. This is the first CI run with the new verifier check ("every file citation carries node_id"). The oracle's keyed ledger scores 1 and nop scores 0. So the seed, the oracle and the verifier all behave correctly with this PR's change.
• claude-code fails. The agent exits nonzero before it produces any output, and it did the same on 853c817. That run uses the ANTHROPIC_API_KEY secret, the same key whose grader ping returns HTTP 400 on this PR and on Research: where the Harness Knowledge Graph lands, and what structure fits instead #134. Research: where the Harness Knowledge Graph lands, and what structure fits instead #134 does not touch fleet-playbook-curator and fails identically on the grader. No fix exists that I can port, because the fix is at the account level.
• Re-run: I am holding this PR's one re-run until the key is fixed. A re-run now would only reproduce the same failure. Once the key works, that re-run is the real test of whether a model follows the new PROMPT.md.

The deep tier (pier) failure on ceb325d is not a signal at all. That run was cancelled, because 3e8d366 superseded it.

---

Generated by Claude Code

[JRichlen]

Resolved: the key is fixed, and the held re-run on 3e8d366 passed all 52 jobs. This also closes the first miss in the ceb325d demonstration ("No model has run the new prose yet"):

• Behavioral (Sonnet grader, 3 repeats per scenario): all six scenarios passed 3/3. That includes the new one, keys a new ledger entry on node_id, not only the mutable repo name. The subject model copied R_kgDOM0n1 into the entry even though the user asked for "just repo/path/sha".
• Deep (pier): claude-code OK (reward=1), oracle OK (reward=1), nop OK (reward=0). The claude-code agent read the new PROMPT.md and produced a ledger that passes verifier check 4, "every file citation carries node_id". So a real harness wrote keyed claims unprompted.

Since then I've merged main in (f4cea99), which brings #137. On this head the paid legs skip unless the PR carries paid-evals. That merge changed only .github/workflows/evals.yml, and the cheap tier passes 1302/1302.

---

Generated by Claude Code

[JRichlen]

Demonstration: the no-placeholder-sha rule in SKILL.md (15708d7)

Input. The behavioral scenario "keys a new ledger entry on node_id". The user supplies context.json for acme/ansible-homelab-monitoring (node_id R_kgDOM0n1), a manifest head_sha of mmm2222, and asks for a short ledger entry. SKILL.md is injected. The subject is qwen/qwen3.8-flash and the grader is Haiku. Both are real CI runs.

Before: fc2294c, 1/3, below the 0.6 floor

One failing row, verbatim from the job log:

{
  "node_id": "R_kgDOM0n1",
  "repo": "acme/ansible-homelab-monitor",
  "claim": "check `site.yml` — as of <head_sha> it is the monitoring playbook entry point",
  "citation": "acme/ansible-homelab-monitor@<head_sha>:site.yml",
  ...
}

The subject kept node_id, but wrote the placeholder <head_sha> instead of mmm2222 and truncated the repo name. The grader failed it on the sha, correctly.

Why. SKILL.md's own directive example is "check hosts.yml — as of <sha> it held the inventory." The rule that a placeholder sha is invalid lived only in PROMPT.md. The deployed curator reads that file, but this tier doesn't inject it.

The rule added

A ledger entry holds real values, copied verbatim. node_id and repo (the full_name, unabridged) come from the repo's context.json entry; sha is that repo's manifest head_sha. The <sha> in the examples here marks where the real sha goes — it is never a value to write. An entry with a placeholder or empty sha is invalid; if you do not have the sha, you do not have a citation.

The rubric is unchanged. A cheap check pins the sentence: it fails with the sentence removed (1321/1322) and passes with it in (1322/1322).

After: 15708d7, 3/3, all six scenarios 3/3

Two of the three rows, from the job log's results table (the table truncates each cell):

{ "node_id": "R_kgDOM0n1", "repo": "acme/ansible-homelab-monitoring", "sha": "mmm2222",
  "path": "site.yml", "claim": "site.yml installs prometheus/grafana", "as_of": "2026-07-11" }

"repo": "acme/ansible-homelab-monitoring", "sha": "mmm2222", "path": "site.yml", "as_of": "2026-07-11",
"claim": "check site.yml — as of mmm2222 it configures prometheus/grafana"

The third row opens by flagging that the subject has the structural pieces but has not read site.yml's contents this pass. That's the "traceable ≠ supported" distinction the skill draws. All three show the real sha and the full repo name. Deep tier on the same commit: pier fleet-playbook-curator passed with Haiku as the agent against the verifier's "every file citation carries node_id" check.

Misses and limits

• The other before-failure is not something this rule addresses. One fc2294c row never wrote an entry. It explained an invented {"node_id": "R_kgDOM0n1", "full_name": "acme/homelab"} object instead. That is a subject task-following failure, and no sentence in SKILL.md targets it. It didn't recur in these 3 samples, but 3 samples is no evidence it's gone.
• n = 3 on each side. This shows the placeholder failure went from present to absent across one run each. It is not a measured rate.
• The truncated table hides one row's node_id. For the first after-row, the node_id line is above the visible cells in the log. The grader passed it, and the rubric fails any entry that omits or alters node_id.
• A placeholder sha is forbidden but not enforced at runtime. index.schema.json already rejects one: sha has pattern ^[0-9a-f]{7,40}$, described as "no placeholders". But nothing in fleet-sync.yml validates the ledger against the schema, and validate-citations.sh checks only that the repo was read and that the path is in its tree, not the sha. A deployed curator that wrote <head_sha> would pass today's pipeline. This rule makes that less likely; it does not make it impossible. Wiring schema validation into fleet-sync.yml is a separate change, not made here.

---

Generated by Claude Code