diff --git a/skills/council/SKILL.md b/skills/council/SKILL.md index 7f01e00..1338c40 100644 --- a/skills/council/SKILL.md +++ b/skills/council/SKILL.md @@ -315,7 +315,7 @@ Put full Stage 5 ballots here, after the outcome and vote tables. - `Agent` tool: every sub-agent in every stage is an `Agent` call when that primitive is available. - `karpathy-guidelines`: the council criteria above operationalize Think-Before, Simplicity-First, Surgical-Changes, and Goal-Driven. -- `example-led-instructions`: brittle outputs only; 0/1/few-shot gate, max 1-3 examples, skip if obvious. +- For brittle outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious. - `worklog plan`: only when the user has not opted out of worklog tracking. Feed the council's kept list into `/worklog plan ` for planning artifacts. ## Examples diff --git a/skills/ship-hygiene/SKILL.md b/skills/ship-hygiene/SKILL.md index dc53e10..c9640a4 100644 --- a/skills/ship-hygiene/SKILL.md +++ b/skills/ship-hygiene/SKILL.md @@ -85,7 +85,7 @@ Skip if: only one PR open, body is short, no recent worklog activity. Overhead n - `budget-mode` — apply during the worklog compress step. Terse prose, code untouched. - `karpathy-guidelines` — apply during the PR title/body audit step. "Don't refactor what isn't broken" — most PRs need nothing. - `systematic-debugging` — apply when CI red is per-PR (not systemic) to actually root-cause each failure. -- `example-led-instructions` — brittle outputs only; 0/1/few-shot gate, max 1-3 examples, skip if obvious. +- For brittle outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious. ## Examples diff --git a/skills/tightening-a-pr/SKILL.md b/skills/tightening-a-pr/SKILL.md index 551d674..b88fc16 100644 --- a/skills/tightening-a-pr/SKILL.md +++ b/skills/tightening-a-pr/SKILL.md @@ -56,6 +56,8 @@ Council gates whether a *learning* is kept — not its *destination*. Before wri Apply `ship-hygiene`'s PR-title/body audit and **internal-reference purge** to this one PR, then the same purge on the tracking worklog task's public-facing fields. `ship-hygiene` **owns** the authoritative leak-token list and the two leak greps (title/body + the diff's added comments) — run them from there rather than keeping a second copy in sync here. Rewrite product-first (what changed for users + why) unless it's pure engineering/infra; skill command names are a leak *unless* the PR changes skill files, where they're product surface. +Before calling the PR ready, record the current head SHA, the focused validation commands with pass/fail results, and the green CI run or check set in the PR body or a final PR comment. Keep "evidence complete" distinct from "ready for review": a draft PR is not ready until it is explicitly marked ready after those facts are current. + ### 4. Compress + checkpoint the worklog Compress the decided iteration drama out of the task body (drop ToT/Reflexion scaffolding, verified "Assumptions", multi-row iteration tables — git log is the audit trail; keep the decision rationale, lessons, re-runnable commands, `next_action`), then checkpoint it **on its own**: `"$WORKLOG_BIN/checkpoint.sh" `. Use the plain command — its staged-scope guard *refuses* any staged path outside the task file, which is exactly what enforces the single-concern commit; do **not** pass `WORKLOG_CHECKPOINT_FORCE=1` (that bypasses the guard) unless you have a stated reason. Keep this separate from the step-2 codify commits — up to three concerns, up to three commits, never one bundle. @@ -85,7 +87,7 @@ Compress the decided iteration drama out of the task body (drop ToT/Reflexion sc - `ship-hygiene` — step 3 deslop + internal-ref purge (delegated, not duplicated). - `worklog` — step 4 checkpoint (`checkpoint.sh`); follow-up learnings become `next_action` items. - `karpathy-guidelines` — "don't refactor what isn't broken": most kept learnings are one-off tasks or drops, not new infra. -- `example-led-instructions` — brittle outputs only; 0/1/few-shot gate, max 1-3 examples, skip if obvious. +- For brittle outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious. ## Output format diff --git a/skills/worklog/SKILL.md b/skills/worklog/SKILL.md index 838c0bc..186c030 100644 --- a/skills/worklog/SKILL.md +++ b/skills/worklog/SKILL.md @@ -162,7 +162,9 @@ For per-task detail, use `"$WORKLOG_BIN/context.sh" ` (its output ends in ## Skill maintenance opt-in -For brittle mode/template outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious. Do not invoke it for normal `/worklog` runtime. +For brittle outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious. + +Do not invoke it for normal `/worklog` runtime. ## Codex / Cursor / other agents diff --git a/tests/run.sh b/tests/run.sh index 54b91e2..44721fa 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -27,6 +27,7 @@ test_static() { say SKIP "shellcheck not installed" fi if ./tools/check-manifest.sh >/dev/null 2>&1; then ok "check-manifest.sh"; else fail "check-manifest.sh"; fi + if python3 ./tools/check-skill-opt-ins.py >/dev/null 2>&1; then ok "check-skill-opt-ins.py"; else fail "check-skill-opt-ins.py"; fi if python3 - <<'PY' import pathlib import re @@ -315,6 +316,107 @@ if missing: PY then ok "ship-hygiene checkpoint guard contract"; else fail "ship-hygiene checkpoint guard contract"; fi + local optin_tmp optin_output + optin_tmp=$(mktemp -d) + mkdir -p \ + "$optin_tmp/skills/example-led-instructions" \ + "$optin_tmp/skills/bad" \ + "$optin_tmp/skills/duplicate" \ + "$optin_tmp/skills/fenced" \ + "$optin_tmp/skills/long-fence" \ + "$optin_tmp/skills/mixed-indent" \ + "$optin_tmp/skills/tilde" \ + "$optin_tmp/skills/worklog" + cp skills/example-led-instructions/SKILL.md "$optin_tmp/skills/example-led-instructions/SKILL.md" + cat >"$optin_tmp/skills/bad/SKILL.md" <<'EOF' +--- +name: bad +description: bad fixture +--- + +Always invoke $example-led-instructions for every output. +EOF + cat >"$optin_tmp/skills/duplicate/SKILL.md" <<'EOF' +--- +name: duplicate +description: bad fixture +--- + +For brittle outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious. +For brittle outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious. +EOF + cat >"$optin_tmp/skills/fenced/SKILL.md" <<'EOF' +--- +name: fenced +description: bad fixture +--- + +```text +For brittle outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious. +``` +EOF + cat >"$optin_tmp/skills/tilde/SKILL.md" <<'EOF' +--- +name: tilde +description: bad fixture +--- + +~~~text +For brittle outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious. +~~~ +EOF + cat >"$optin_tmp/skills/long-fence/SKILL.md" <<'EOF' +--- +name: long-fence +description: bad fixture +--- + +````text +``` +For brittle outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious. +```` +EOF + cat >"$optin_tmp/skills/mixed-indent/SKILL.md" <<'EOF' +--- +name: mixed-indent +description: bad fixture +--- + +EOF + printf ' \tFor brittle outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious.\n' >>"$optin_tmp/skills/mixed-indent/SKILL.md" + cat >"$optin_tmp/skills/worklog/SKILL.md" <<'EOF' +--- +name: worklog +description: bad fixture +--- + +For brittle outputs, invoke `$example-led-instructions`: 0/1/few-shot gate, max 1-3 examples, skip if obvious. + +~~~text +Do not invoke it for normal `/worklog` runtime. +~~~ + + Do not invoke it for normal `/worklog` runtime. +EOF + printf ' \tDo not invoke it for normal `/worklog` runtime.\n' >>"$optin_tmp/skills/worklog/SKILL.md" + set +e + optin_output=$(python3 tools/check-skill-opt-ins.py --root "$optin_tmp" 2>&1) + rc=$? + set -e + if [[ $rc -eq 1 \ + && "$optin_output" == *"must use the exact compact opt-in preamble"* \ + && "$optin_output" == *"duplicate \$example-led-instructions opt-in"* \ + && "$optin_output" == *"opt-in is inside a code fence"* \ + && "$optin_output" == *"skills/tilde/SKILL.md"* \ + && "$optin_output" == *"skills/long-fence/SKILL.md"* \ + && "$optin_output" == *"skills/mixed-indent/SKILL.md"* \ + && "$optin_output" == *"needs unfenced runtime guard"* ]]; then + ok "check-skill-opt-ins rejects sloppy opt-ins" + else + fail "check-skill-opt-ins red path drifted (exit=$rc, output=${optin_output@Q})" + fi + rm -rf "$optin_tmp" + local catalog_home catalog_fixture catalog_json unknown_provider_output detection_home detected_path ancestry_dir local -a clean_env=( env diff --git a/tools/check-skill-opt-ins.py b/tools/check-skill-opt-ins.py new file mode 100755 index 0000000..4ecc7a6 --- /dev/null +++ b/tools/check-skill-opt-ins.py @@ -0,0 +1,195 @@ +#!/usr/bin/env python3 +"""Validate compact cross-skill opt-ins.""" + +from __future__ import annotations + +import argparse +import pathlib +import sys + +SKILL_NAME = "example-led-instructions" +CANONICAL_PREAMBLE = ( + "For brittle outputs, invoke $example-led-instructions: " + "0/1/few-shot gate, max 1-3 examples, skip if obvious." +) +CONSUMER_PREAMBLE = ( + "For brittle outputs, invoke `$example-led-instructions`: " + "0/1/few-shot gate, max 1-3 examples, skip if obvious." +) +OUTPUT_FIELDS = ( + "shot_count:", + "format:", + "examples_or_skip_reason:", + "risk_check:", + "acceptance_test:", +) +WORKLOG_RUNTIME_GUARD = "Do not invoke it for normal `/worklog` runtime." +FENCE_CHARS = ("`", "~") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--root", + default=pathlib.Path(__file__).resolve().parents[1], + type=pathlib.Path, + help="repository root to validate", + ) + return parser.parse_args() + + +def skill_files(root: pathlib.Path) -> list[pathlib.Path]: + return sorted((root / "skills").glob("*/SKILL.md")) + + +def relative(path: pathlib.Path, root: pathlib.Path) -> str: + return str(path.relative_to(root)) + + +def validate_home_skill(root: pathlib.Path, problems: list[str]) -> None: + skill_md = root / "skills" / SKILL_NAME / "SKILL.md" + if not skill_md.is_file(): + problems.append(f"skills/{SKILL_NAME}/SKILL.md: missing") + return + + text = skill_md.read_text() + for term in (CANONICAL_PREAMBLE, *OUTPUT_FIELDS): + if term not in text: + problems.append(f"{relative(skill_md, root)}: missing {term!r}") + + +def strip_bullet(line: str) -> str: + stripped = line.strip() + return stripped[2:].strip() if stripped.startswith("- ") else stripped + + +def leading_indent_columns(line: str) -> int: + columns = 0 + for char in line: + if char == " ": + columns += 1 + elif char == "\t": + columns += 4 - (columns % 4) + else: + break + return columns + + +def fence_marker(line: str) -> tuple[str, int, str] | None: + if leading_indent_columns(line) > 3: + return None + stripped = line.lstrip(" \t") + if not stripped: + return None + + marker = stripped[0] + if marker not in FENCE_CHARS: + return None + + marker_len = len(stripped) - len(stripped.lstrip(marker)) + if marker_len < 3: + return None + return marker, marker_len, stripped[marker_len:] + + +def is_indented_code(line: str) -> bool: + return leading_indent_columns(line) >= 4 + + +def find_mentions(lines: list[str], term: str) -> list[tuple[int, str, bool]]: + mentions = [] + active_fence: tuple[str, int] | None = None + for index, line in enumerate(lines, start=1): + marker = fence_marker(line) + line_in_fence = active_fence is not None or marker is not None or is_indented_code(line) + if term in line: + mentions.append((index, line, line_in_fence)) + if marker is None: + continue + marker_char, marker_len, marker_suffix = marker + if active_fence is None: + active_fence = (marker_char, marker_len) + elif ( + marker_char == active_fence[0] + and marker_len >= active_fence[1] + and not marker_suffix.strip() + ): + active_fence = None + return mentions + + +def validate_reference_line( + root: pathlib.Path, + path: pathlib.Path, + line_number: int, + line: str, + in_fence: bool, + problems: list[str], +) -> None: + rel = relative(path, root) + stripped = line.strip() + reference = strip_bullet(line) + + if in_fence: + problems.append(f"{rel}:{line_number}: $example-led-instructions opt-in is inside a code fence") + if reference != CONSUMER_PREAMBLE: + problems.append( + f"{rel}:{line_number}: $example-led-instructions must use the exact compact opt-in preamble" + ) + + limit = 180 if rel == "skills/worklog/SKILL.md" else 140 + if len(stripped) > limit: + problems.append( + f"{rel}:{line_number}: example-led opt-in is {len(stripped)} chars; limit {limit}" + ) + + +def validate_consumers(root: pathlib.Path, problems: list[str]) -> None: + home_skill = root / "skills" / SKILL_NAME / "SKILL.md" + for path in skill_files(root): + if path == home_skill: + continue + + lines = path.read_text().splitlines() + references = find_mentions(lines, SKILL_NAME) + + if len(references) > 1: + problems.append( + f"{relative(path, root)}: duplicate $example-led-instructions opt-in; expected at most one" + ) + for line_number, line, line_in_fence in references: + validate_reference_line(root, path, line_number, line, line_in_fence, problems) + + if path == root / "skills" / "worklog" / "SKILL.md" and references: + runtime_guards = find_mentions(lines, WORKLOG_RUNTIME_GUARD) + has_unfenced_runtime_guard = any( + not in_fence and line.strip() == WORKLOG_RUNTIME_GUARD + for _, line, in_fence in runtime_guards + ) + if not has_unfenced_runtime_guard: + problems.append( + f"{relative(path, root)}: example-led opt-in needs unfenced runtime guard: " + f"{WORKLOG_RUNTIME_GUARD}" + ) + + +def main() -> int: + args = parse_args() + root = args.root.resolve() + problems: list[str] = [] + + validate_home_skill(root, problems) + validate_consumers(root, problems) + + if problems: + print("check-skill-opt-ins: FAIL") + for problem in problems: + print(f" - {problem}") + return 1 + + print("check-skill-opt-ins: OK") + return 0 + + +if __name__ == "__main__": + sys.exit(main())