From 09c09ad4210fd4d991ec5d4a4f53389c84791254 Mon Sep 17 00:00:00 2001 From: Brock Wilcox Date: Sun, 6 Sep 2026 09:56:36 -0400 Subject: [PATCH 1/2] docs: describe the agent-driven user guide sweep and add screenshot utilities Adds docs/agentic_update.md, a write-up of the process used in September 2026 to document a batch of merged PRs and refresh every page of the bank user guide, plus the reusable pieces under docs/utils/: - utils/screenshots/shot.tmpl.js + mkshot.py: spec-driven Playwright helper that logs in, stages the page, grows the viewport, draws the red boxes and numbered labels after the resize, and crops. - utils/screenshots/example_spec.json and README.md: spec format, conventions and the gotchas hit along the way (no fullPage, modals, select2, sidebar dropdown headers, seed state). - utils/check_guide_images.sh: reports image references with no file and image files nothing references. Also ignores .playwright-mcp/, the Playwright MCP plugin's scratch directory. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01LMAt57FU2qBq45yR7YCsEZ --- .gitignore | 3 + docs/agentic_update.md | 217 +++++++++++++++++++++++ docs/readme.md | 1 + docs/utils/check_guide_images.sh | 34 ++++ docs/utils/screenshots/README.md | 105 +++++++++++ docs/utils/screenshots/example_spec.json | 37 ++++ docs/utils/screenshots/mkshot.py | 62 +++++++ docs/utils/screenshots/shot.tmpl.js | 112 ++++++++++++ 8 files changed, 571 insertions(+) create mode 100644 docs/agentic_update.md create mode 100755 docs/utils/check_guide_images.sh create mode 100644 docs/utils/screenshots/README.md create mode 100644 docs/utils/screenshots/example_spec.json create mode 100755 docs/utils/screenshots/mkshot.py create mode 100644 docs/utils/screenshots/shot.tmpl.js diff --git a/.gitignore b/.gitignore index 2fa6f630b6..f96db505a9 100644 --- a/.gitignore +++ b/.gitignore @@ -90,3 +90,6 @@ out/ .vscode/ .aider* .claude + +# Playwright MCP plugin scratch space (screenshot specs, console logs) +.playwright-mcp/ diff --git a/docs/agentic_update.md b/docs/agentic_update.md new file mode 100644 index 0000000000..1187af844c --- /dev/null +++ b/docs/agentic_update.md @@ -0,0 +1,217 @@ +# Sweeping user-guide updates with an AI agent + +This describes the process used in September 2026 to bring the bank user guide +(`docs/user_guide/bank/`) up to date: documenting a batch of merged feature +PRs, then walking every page of the guide to refresh text, screenshots and +annotations, and finally splitting the result into reviewable pull requests. +It was done with Claude Code driving a local checkout, the running dev app and +the Playwright MCP plugin, but the steps are the same for a person or any other +agent. The goal of writing it down is that the next sweep is faster and lands +in the same style. + +The reusable pieces live in [`docs/utils/`](utils/): + +- [`utils/screenshots/`](utils/screenshots/README.md): spec-driven Playwright helper for annotated screenshots. +- [`utils/check_guide_images.sh`](utils/check_guide_images.sh): missing and orphaned image check. + +## 1. Set up + +- Check out a working branch from `main`, e.g. `guide-updates`. Everything is + committed here first; PR branches are carved out at the end (section 6). +- Reset the local database to seed data (`bin/rails db:reset` or `bin/setup`) + so screenshots show the Pawnee Diaper Bank sample data that the rest of the + guide uses. Start the app with `bin/start`. +- Log-ins used throughout: `org_admin1@example.com` (bank admin), + `user_1@example.com` (bank user), `verified@example.com` (partner). All + passwords are `password!`. +- Preview the guide as readers see it: `cd docs && bundle exec jekyll serve + --livereload`, then open http://localhost:4000/user_guide/bank/. Jekyll picks + up markdown edits immediately, which makes it easy to check relative links + and image paths. +- If the agent is Claude Code, the Playwright MCP plugin is what takes the + screenshots. It can only read files inside the repository; `.playwright-mcp/` + (untracked) is a convenient place for specs and generated snippets. + +## 2. Scope the feature work + +For each merged app PR that needs documenting: + +1. Read the PR description and diff (`gh pr view N`, `gh pr diff N`). Note the + user-visible change: new fields, renamed statuses, new buttons, new emails, + new settings. +2. Find where the guide already talks about that area: + `grep -rn -i "audit" docs/user_guide/bank/*.md`. Features usually touch more + than one page (a setting on the organization page plus the form it affects + plus the partner-side view). List all of them. +3. Open the feature in the running app and click through it before writing + anything. Take quick unannotated exploratory captures to look at; do not + spend time annotating until the text is settled. +4. Write the text, then the annotated screenshots (section 4), then commit + with a message naming the PR: `User guide: document request limits on + Items (#5386)`. One commit per feature keeps the later PR split trivial. + +Things that came up and how they were handled: + +- If the app's behaviour looks wrong or a label is confusing, that is an app + issue, not a docs issue. Document what the app does and raise the app + problem separately rather than describing what it "should" do. +- Behaviour that only applies to migrated data ("banks that existed before X + have both switches on") tends to get deleted in review. Prefer describing + the current behaviour. +- Emails: the mailer preview pages (`/rails/mailers`) are the easiest way to + screenshot an email consistently. + +## 3. Build the burn-down list + +Before the sweep, inventory the guide so progress is visible and nothing is +skipped: + +```sh +ls docs/user_guide/bank/*.md | wc -l # pages +grep -rhoE '\]\(images/[^)]+' docs/user_guide/bank/*.md | sort -u | wc -l # image references +docs/utils/check_guide_images.sh # broken refs, orphans +``` + +Keep the list somewhere both you and the agent can see (a markdown table in +the conversation, a scratch file, a dashboard). Each row is a page with a +status and a one-line note. The order that worked: Getting Started, Everyday +Essentials, Partners, Inventory, Community, Reports, User and Account +Management. Getting Started overlaps with everything else, so doing it first +means later pages can link to it. + +Also list the merged PRs from section 2 as rows so the two kinds of work are +tracked in the same place. + +## 4. The per-page review loop + +For every page: + +1. **Read the page** end to end, and open each screen it describes in the + running app side by side. +2. **Compare text to UI.** Button labels, menu names, column headers, tab + names, status words and field names must match the app exactly, including + capitalisation and quotes. Numbered steps must match the numbered + annotations in the image below them. Watch for features that have been + removed or renamed since the page was written. +3. **Fix links.** Relative links between pages (`[Partners](partners.md)`) and + anchors. The Jekyll preview catches most of these. +4. **Re-shoot every annotated screenshot** on the page rather than deciding + image by image whether it is stale. Uniform width, crop and annotation + style across a page matters more than saving a few captures. Keep the same + filename so the markdown does not need to change; retire an image only by + deleting it and its reference together. +5. **Look at each image** after it is written. The helper reports where it + drew boxes, which catches a selector that matched nothing, but only your + eyes catch a box on the wrong button, a stale flash message or an + unexpected dev toolbar. +6. **Commit per page or per small group of pages** with a message like + `User guide: refresh Partners screenshots and fix invite steps`. Frequent + small commits made the PR split at the end cheap. + +Screenshot conventions and the spec format are in +[`utils/screenshots/README.md`](utils/screenshots/README.md). In short: +1400px wide, content cropped from x=250 unless the sidebar is the subject, +red 3px boxes with red numerals matching the text, nothing boxed that the app +already highlights. + +State that had to be toggled for screenshots (put it back afterwards): + +```sh +bin/rails runner 'Organization.find_by!(name: "Pawnee Diaper Bank").update!(bank_is_set_up: false)' # Getting Started prompt +bin/rails runner 'Partner.find_by!(name: "Pawnee Middle School").update!(status: :awaiting_review)' # approval buttons +``` + +## 5. Verify before splitting + +- `docs/utils/check_guide_images.sh` reports nothing missing. +- Jekyll preview: click through every page once, looking for broken images + and links. +- `git diff --stat main` shows only `docs/` (and any gem changes you meant to + make for the docs site). +- Spot check a handful of right-aligned annotations (buttons in card headers, + table action columns). Those are the ones that drift when something is wrong + with the capture flow. + +## 6. Split into pull requests + +Reviewers cannot usefully review one PR with several hundred image changes. +The split that worked, as a stacked chain where each PR's base is the previous +branch: + +| # | Contents | Why separate | +| --- | --- | --- | +| 1 | Tooling fixes (e.g. gems so Jekyll runs on the current Ruby) | Zero-risk, merge first | +| 2 | Typos and broken links | Mechanical, easy to approve | +| 3..8 | One PR per documented feature PR | Reviewer is often the feature's author | +| 9 | Previously undocumented features found during the sweep | Needs a real read | +| 10..13 | Screenshot refresh, one PR per guide section | Big but skimmable | + +Rules that were requested for these PRs: + +- Cross-link: each feature-doc PR says "Documents #NNNN" and links back, and + a comment on the app PR links to the docs PR. +- Anything with significant generated prose is opened as a **draft** so the + maintainer can rewrite before asking others to review. Mechanical PRs + (gems, typos, links) can be opened ready. +- Each PR body lists the pages touched and notes what a reviewer should look + at (e.g. "check the numbered steps against the image"). + +Mechanics. Local branches were named `pr/NN-topic` and pushed as +`guide/NN-topic` so the remote names sort together: + +```sh +git checkout -b pr/03-request-limits main +git cherry-pick +git push -u origin pr/03-request-limits:refs/heads/guide/03-request-limits +gh pr create --draft --base guide/02-typos-links --head guide/03-request-limits --title "..." --body-file body.md +``` + +Screenshot-section PRs are made the same way with `git checkout +guide-updates -- docs/user_guide/bank/images/partners` on a branch based on +the previous PR branch, then one commit. Check at the end that the tip of the +last PR branch has the same tree as the working branch: +`git diff --stat pr/13-... guide-updates` should be empty. + +## 7. Respond to review + +Review comments on screenshot PRs are mostly "this box is off" or "this image +does not show what the text says". The loop: + +1. Reproduce the problem locally (open the image, compare with the app). + Often one comment reveals a systematic cause; in September 2026 a single + ordering bug in the capture helper had shifted every right-aligned box, so + every annotated image was re-taken, not only the flagged ones. +2. Fix on the working branch, commit, then add a commit to the affected PR + branch (`git checkout guide-updates -- `), and rebase the branches + above it: `git rebase --onto pr/12-... pr/13-...`. +3. Push with a lease against the SHA you fetched, never with a bare force: + + ```sh + git fetch origin + git ls-remote origin refs/heads/guide/12-inventory # note the sha + git push origin --force-with-lease=refs/heads/guide/12-inventory: pr/12-inventory:refs/heads/guide/12-inventory + ``` + + The maintainer may apply suggestions on GitHub directly; a bare `--force` + once overwrote one of those commits and it had to be recovered from the + reflog. +4. Reply on every thread with what changed and the commit, or with the reason + for not changing it (for example, not boxing something the app already + highlights in red). Use `gh api repos/OWNER/REPO/pulls/N/comments/ID/replies -f body=...` + to answer inline threads. + +## 8. Checklist for the agent prompt + +When kicking off the next sweep, the prompt that worked contained: + +- the list of merged PR numbers to document; +- that the local environment is reset and may be used freely (including + restarting services); +- "make a burn-down list and walk the whole guide, including fresh screenshots + and annotations"; +- "commit locally as you go so we can split into PRs later"; +- pointers to this document and `docs/utils/`. + +Things to say up front if you care about them, because they were asked for +mid-way last time: draft status for generated PRs, cross-links to the app +PRs, and any formatting rule for commit messages. diff --git a/docs/readme.md b/docs/readme.md index 58c93cac07..904157b629 100644 --- a/docs/readme.md +++ b/docs/readme.md @@ -1,5 +1,6 @@ * [User Guide](user_guide/bank/) * [Developer Architecture Overview](architecture/overview) +* [Updating the user guide with an AI agent](agentic_update) ## Developer Notes diff --git a/docs/utils/check_guide_images.sh b/docs/utils/check_guide_images.sh new file mode 100755 index 0000000000..7cea1a0a99 --- /dev/null +++ b/docs/utils/check_guide_images.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# Check the user guide's images: report markdown references to files that do +# not exist, and image files that no markdown page references (orphans). +# +# Usage: docs/utils/check_guide_images.sh [docs/user_guide/bank] +# Exit status is non-zero if anything is missing (orphans are a warning only). +set -euo pipefail +GUIDE="${1:-docs/user_guide/bank}" +cd "$(git rev-parse --show-toplevel)" + +refs() { + # Matches both ![alt](images/x.png) and + grep -rhoE '(\]\(|src=")images/[^)" ]+' "$GUIDE"/*.md | sed -E 's/^(\]\(|src=")//' | sort -u +} + +echo "== Missing images (referenced in markdown, file not found)" +missing=0 +while read -r ref; do + if [ ! -f "$GUIDE/$ref" ]; then echo " MISSING $ref"; missing=$((missing+1)); fi +done < <(refs) + +echo "== Orphan images (file exists, no markdown page references it)" +find "$GUIDE/images" -type f \( -iname '*.png' -o -iname '*.jpg' -o -iname '*.jpeg' -o -iname '*.gif' \) | sort | while read -r f; do + rel="${f#"$GUIDE"/}" + if ! grep -rqF "$rel" "$GUIDE"/*.md; then echo " ORPHAN $rel"; fi +done + +echo "== Case mismatches (reference exists only with different capitalisation)" +while read -r ref; do + if [ ! -f "$GUIDE/$ref" ] && find "$GUIDE/$(dirname "$ref")" -maxdepth 1 -iname "$(basename "$ref")" 2>/dev/null | grep -q .; then + echo " CASE $ref" + fi +done < <(refs) +[ "$missing" -eq 0 ] diff --git a/docs/utils/screenshots/README.md b/docs/utils/screenshots/README.md new file mode 100644 index 0000000000..82258a1a71 --- /dev/null +++ b/docs/utils/screenshots/README.md @@ -0,0 +1,105 @@ +# Annotated screenshot helper + +Tools for taking the user guide's screenshots in a repeatable way: a 1400px-wide +capture of the running dev app, cropped to the area the text is talking about, +with red boxes and red numbers/letters that match the numbered steps on the page. + +Files: + +| File | Purpose | +| --- | --- | +| `shot.tmpl.js` | Playwright snippet template. Logs in, sets up the page, grows the viewport, draws the boxes, crops, saves. | +| `mkshot.py` | Inlines a JSON spec into the template and writes `.playwright-mcp/run.js`. | +| `example_spec.json` | A spec showing the common shapes: sidebar navigation, a button on a list page, cropping to a dashboard card, a modal. | +| `../check_guide_images.sh` | Finds markdown image references with no file, and image files no page references. | + +## Running + +1. Start the app (`bin/start`) with a freshly seeded database so the sample data matches what other pages show. +2. Write a spec (see below), then generate the snippet: + + ```sh + docs/utils/screenshots/mkshot.py my_spec.json + ``` + +3. Run it. From Claude Code with the Playwright MCP plugin, call `browser_run_code_unsafe` with `filename` set to the generated `.playwright-mcp/run.js`. The tool only reads files under the repository, and `.playwright-mcp/` is where the plugin keeps its own logs, so it is a good untracked home for specs and generated snippets. + + Without the MCP tool, the same snippet body can be pasted into any Playwright script as `await (SNIPPET)(page)`. + +4. Read the returned `boxes` list. Every mark should have `x/y/w/h`; a `missing` entry means the selector matched nothing, and a 4x4 box at `-2,-2` means it matched a hidden element (typically a sidebar dropdown header, see gotchas). +5. Open each image and look at it. The box list tells you a target was found, not that it was the right one. + +## Spec format + +```json +{ + "base": "http://localhost:3000", + "password": "password!", + "shots": [ { ...shot }, { ...shot } ] +} +``` + +Shots run in order in the same browser tab, so a shot without `url` or `pre` reuses the state left by the previous shot (handy for several crops of one page). + +| Key | Meaning | +| --- | --- | +| `out` | Output path. Relative paths are resolved from the repo root, e.g. `docs/user_guide/bank/images/partners/partners_add.png`. | +| `url` | Page to open first (absolute URL). | +| `pre` | List of steps run before capturing (below). | +| `width`, `height` | Viewport size. Default 1400 wide; height is only meaningful with `viewport`. | +| `viewport` | `true` keeps a fixed-height viewport instead of growing to the document height. Use for modals and other `position: fixed` content. | +| `clip` | `{x, y, width, height}` crop in page pixels. The content area starts at `x: 250` (the sidebar is 250px). | +| `clipTo` | CSS selector of an element to crop to, padded by `clipPad` (default 12). | +| `full` | Capture the whole (grown) viewport with no crop. | +| `css` | Extra CSS injected before capture (hide something, force a state). | +| `marks` | List of things to box (below). | + +Pre-steps (each step is an object with one of these keys): + +| Step | Meaning | +| --- | --- | +| `{"login": "org_admin1@example.com"}` | Sign in via `/users/sign_in`. Optional `password`. | +| `{"goto": "/partners"}` | Navigate; relative paths use `base`. | +| `{"click": "selector", "wait": 400}` | Click, then wait ms. Playwright selectors, so `a.btn:has-text('Filter')` works here. | +| `{"fill": "selector", "value": "..."}`, `{"select": ...}`, `{"check": ...}`, `{"uncheck": ...}` | Form interaction. | +| `{"hover": "selector"}`, `{"mouse": [x, y]}` | Pointer moves, for tooltips and hover menus. | +| `{"scroll": "selector"}` | Scroll an element into view. | +| `{"sleep": 800}` | Wait for animations/turbo frames. | +| `{"eval": "js"}` | Run arbitrary JS in the page. Used to tag an element (`el.classList.add('__target')`) so `clipTo`/`marks` can reach something with no stable selector. | + +Marks: + +| Key | Meaning | +| --- | --- | +| `sel` | CSS selector (`document.querySelector`, so no `:has-text`). | +| `text` | Exact visible text; combined with `sel` as the candidate set. | +| `contains` | Substring of visible text; candidates default to `a,button`. | +| `nth` | Pick the nth match of `sel`. | +| `pad` | Pixels of padding around the element (default 4; 6 for buttons, 2 for menu items). | +| `label` | Text for the red label. Defaults to the 1-based index; `false` for a box with no label. | +| `side` | Where the label sits: `left` (default), `right`, `above`, `below`, `inside`. Use `right` for things at the left edge of the crop. | + +## Conventions + +- 1400px wide. Crop to the content (`x: 250`) unless the text is about the left-hand menu, in which case include the sidebar from `x: 0`. +- Boxes are 3px `#e01b24` with a 4px radius; labels are bold 26px red with a white glow. Numbers when the text has numbered steps, letters when it is naming things rather than sequencing them. +- Do not box something the app already highlights (red shortage numbers, warning banners). The box would be redundant and could be mistaken for part of the UI. +- Name files after the page and the thing being shown, and keep the name when re-shooting so the markdown does not change. +- Composite images (before/after, two tabs) are made with ImageMagick after capture: + + ```sh + magick top.png bottom.png -background '#dddddd' -splice 0x6 -append out.png # stack with a grey separator + magick wide.png -crop 1145x225+72+0 +repage out.png # sub-crop an existing capture + ``` + +## Gotchas + +- **Draw after resizing.** The template grows the viewport to the document height and only then draws boxes. Doing it the other way round leaves right-aligned controls (buttons, tabs, action columns) with boxes that are visibly off. +- **No `fullPage`.** AdminLTE's `layout-fixed` body reflows under Playwright's full-page mode and the sidebar is painted over the content. Grow the viewport instead (the template does). +- **Sidebar dropdown headers** (Donations, Purchases, Inventory, Community...) are `a.nav-link[href="#"]`. Match them with `contains`, not by href. +- **Modals** are `position: fixed`. Use `viewport: true`, `clipTo: ".modal-content"` (not `.modal-dialog`, which is the full overlay), and rely on the template's fade-transition kill. +- **select2** hides the real `