[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-11

[github-actions[bot]]

Summary

• Date of test: 2026-09-11
• Pages visited:
  • Home — `(localhost/redacted)
  • Quick Start Guide — `(localhost/redacted)
  • CLI Commands — `(localhost/redacted)
• Overall impression: As a first-time user, the site felt professional and the top navigation made it obvious where to start ("Quick Start" is the first nav item). The Quick Start page itself is well-structured with clear steps, prerequisites, and copy-to-clipboard commands, though a couple of jargon terms and one internal-only link could trip up a true beginner.

🔴 Critical Issues Found

None. All 3 pages loaded successfully (HTTP 200), and I did not encounter any broken links, 404s, or non-functional code blocks in the sections I reviewed.

🟡 Confusing Areas

1. "frontmatter" jargon introduced before it's defined — On the Quick Start page, the very first paragraph says workflows are defined "in Markdown with YAML frontmatter (the configuration block between the --- markers...)". This is a good inline definition, but the term "frontmatter" is then reused later ("Step 4") without re-explaining it, so a reader who skimmed past the first mention may be lost. Consider a short glossary link or tooltip the first time it's used, and reinforcing it with a link every time.
2. Two competing ways to add the sample workflow with unclear guidance on when to use which — The Quick Start page instructs the reader to run gh aw add-wizard githubnext/agentics/repo-status, but the CLI Commands page lists both gh aw add-wizard (interactive) and gh aw add (non-interactive) as "Day-one commands" with only a one-line distinction ("guided prompts" vs. "scripted/CI"). A beginner following Quick Start might later browse CLI Commands and wonder which one they actually used, or why there are two near-identical commands.
3. Authentication step assumes prior familiarity with PATs — The "Configuring authentication" tab panel for Copilot mentions creating a "fine-grained Personal Access Token (PAT)" and setting "Copilot Requests" permission — for a total beginner unfamiliar with GitHub PAT scopes, this is a conceptual leap with no link explaining what a PAT is in general (only a link to create one). A one-line explainer or link to GitHub's PAT docs would help.
4. Gallery link behaves like an anchor jump, not a page — The top nav "Gallery" link points to /gh-aw/index.html#gallery while "Docs" and other links point to clean routes like /gh-aw/introduction/overview/. This inconsistency (.html fragment vs. clean URL) is a minor confusion point but not a broken link.

🟢 What Worked Well

• Clear top-level navigation: "Quick Start" is the first item in the primary nav and also has a prominent "Get Started with CLI" button on the homepage — easy to find as a new user.
• Prerequisites are explicit and complete: The Quick Start page lists AI account, repository access, GitHub Actions enabled, GitHub CLI version, and login state — with direct links and copy-pasteable verification commands (gh --version, gh auth status).
• Copy-to-clipboard code blocks: Every command is in a code block with a one-click copy button, reducing typo risk.
• Numbered, sequential steps (Step 1 → Step 4) with a realistic time estimate ("Estimated time: 10 minutes") set good expectations.
• CLI Commands page has a "Day-one commands" table up front with Command / Description / When to use columns — this immediately orients a new user to the ~10 commands that matter most, before diving into advanced/enterprise options further down the page.
• Helpful troubleshooting tip box: The install step includes a fallback curl-based installer script for users who hit gh extension install auth issues — a thoughtful anticipation of a common failure mode.
• Example screenshot of expected output: The Quick Start page includes an image of what the generated "Daily Repo Status Report" issue looks like, which helps set expectations for what success looks like.

Recommendations

Quick wins:

1. Add a one-sentence explainer (or tooltip) linking "frontmatter" the second time it appears, not just the first.
2. Add a short note near gh aw add-wizard vs gh aw add in the CLI Commands table clarifying "use add-wizard if you're new; use add once you know what you want" (a "for beginners, prefer this" cue).
3. Normalize the Gallery nav link to a clean route rather than an index.html#anchor fragment for URL consistency with the rest of the nav.

Longer-term:

1. Consider a lightweight glossary page (frontmatter, lock file, safe outputs, MCP, etc.) linked from the Quick Start intro, since several beginner-unfriendly terms are introduced in the first few steps.
2. Add a one-line "What is a Personal Access Token?" aside or link in the authentication section for engines that require manual PAT creation, since this is the step most likely to stump a true newcomer.

Screenshots

📎 home.png — Home page — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/20743b893ae7b3d68987db0c006d1535e901224f643aa73b4db787e95b68d836.png?raw=true

📎 quickstart-top.png — Quick Start guide — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/da89e909db850b3cae8246a8ef33300bb1ba0d4ff35d5e835980519c76803ac2.png?raw=true

📎 cli-commands.png — CLI Commands page — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/71e04a6f50bc0b4ae492cc778f20e15c00fe63452e0190c28fdbff0428b21809.png?raw=true

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · copilot · auto · 48.1 AIC · ⌖ 6.93 AIC · ⊞ 11.1K · ◷

• expires on Sep 11, 2026, 7:53 PM UTC-08:00