Han Feedback — 2026-08-26
Skills used: han-coding:manual-test-planning Agents used: none visible (the plan was produced in an earlier
session; this session executed the plan, hit the defect, and traced its cause) Context: Running a nine-test manual
plan, produced by the skill for a behavior change in a Rails monorepo's consumer web app, where each test says which
persona to pick and what to do in the app by hand. Outcome: Test 1's setup instruction told the tester to perform a
UI flow that does not exist in the application, and the tester lost time attempting it before a code trace proved it
impossible.
The defect: the plan invented a UI capability instead of verifying it
Test 1 needed an application state no persona ships: a savings goal funded only by one specific account type. The plan
noticed the fixture gap — its own text says no existing persona has this state — and bridged it by instructing the
tester to "create the goal yourself in the app" and pick that account type to fund it.
That flow does not exist. A code trace this session showed the web frontend has no path that puts this account type
into a goal: the add-goal flow drops the goal reference when this product type is chosen (a hardcoded redirect discards
it), the account-creation confirmation never sends the goal argument the backend mutation accepts, the
assign-account-to-goal mutation rejects this account type outright, and the two "add to a goal" entry points in
settings and on the account page are typed to other account kinds. The backend supports the state; no shipped UI
reaches it. So the plan's setup step was unrunnable by anyone, and the failure surfaced only when a person tried to
follow it.
The plan's own premise makes this worse: it promises "anyone who can open the local test apps and pick a persona can
run this plan." Eight of nine tests honor that. The ninth silently required a capability the author never checked.
Suggested fixes.
- Evidence-gate every action step. The skill verifies personas against their source files (and that held up: every
persona claim spot-checked this session was accurate). It applies no equivalent check to action steps. Any step that
tells the tester to do something in the app — create, attach, configure — should require the same evidence standard:
trace the route, component, or mutation that performs it before the step is written. "I could not find the flow" must
block the step, not soften into an instruction.
- A fixture gap is a finding, not a bridge to improvise over. When no persona ships the needed state and no
verified UI path creates it, the skill should say the test is not manually runnable as written and hand the gap back
as options (add a persona variant, cover it in automated tests, or confirm the state is reachable by some client at
all) instead of inventing setup steps. The skill's own contract — "when nothing in the supplied context can be
manually tested, it says so" — should apply per-test, not only to the whole plan.
- Name flag prerequisites per persona. The state under test sat behind a feature flag that one suggested persona
variant sets and another does not. The plan never mentioned the flag, so even a partially-successful setup would have
rendered the wrong UI. Setup preconditions that live in flags belong in the persona line.
What worked well
- Persona selection was evidence-based and accurate. Eight of nine tests key to real personas and variants;
spot-checks of the persona source files this session matched every claim the plan made about what those personas
contain, including a fallback persona that really does own the unusual account type.
- The plan was honest about the fixture gap. It flagged that no persona ships test 1's state rather than
hand-waving a persona that does not exist. The failure was in how it bridged the gap, not in hiding it.
- The plain-language format worked for its reader. Each test names its persona, steps, and observable expected
outcomes; the areas grouping and per-test pass/fail affordances made the plan genuinely runnable by hand — for the
eight tests whose setup exists.
What didn't work
- The defect above: one of nine tests instructed a UI flow that does not exist, discovered only at execution time.
- The recommended persona was the wrong starting point even in principle. Test 1 recommended a persona that owns
none of the required account type, on the theory the tester would create everything in-app; the fallback persona that
already owns the account was listed second. With the evidence gate in place, the ordering inverts or the test gets
blocked outright.
Overall
The skill produced a plan whose persona facts were verified and whose format served a hand-running tester well, but it
holds action steps to a lower evidence standard than persona claims. Where a needed state had no fixture, the plan
asserted an application capability it never traced, and the cost landed on the person following the steps. The missing
control is a per-step evidence gate: a manual test plan's instructions are claims about what the app can do, and each
one needs the same source-level verification the skill already applies to personas — with "not manually testable as
written" as the honest output when verification fails.
Rating
| Dimension |
Score |
| Output accuracy |
3/5 |
| Evidence discipline |
2/5 |
| Finding signal-to-noise |
4/5 |
| Output length vs. decision count |
4/5 |
| Turn efficiency |
3/5 |
Accuracy 3/5: persona facts and eight tests' steps held up; the ninth test's central instruction was false. Evidence
discipline 2/5: the failure the report exists for — an unverified capability claim shipped as a step, in a plan that
demonstrably knew how to verify its other claims. Signal-to-noise 4/5: the plan's content is real and lean; no
agent-finding noise visible from this session. Length 4/5: proportionate to nine tests. Turn efficiency 3/5: the
planning session itself is not visible here, but the unrunnable step cost this session a full investigate-and-workaround
cycle that per-step verification would have spent once, at authoring time.
Han Feedback — 2026-08-26
Skills used:
han-coding:manual-test-planningAgents used: none visible (the plan was produced in an earliersession; this session executed the plan, hit the defect, and traced its cause) Context: Running a nine-test manual
plan, produced by the skill for a behavior change in a Rails monorepo's consumer web app, where each test says which
persona to pick and what to do in the app by hand. Outcome: Test 1's setup instruction told the tester to perform a
UI flow that does not exist in the application, and the tester lost time attempting it before a code trace proved it
impossible.
The defect: the plan invented a UI capability instead of verifying it
Test 1 needed an application state no persona ships: a savings goal funded only by one specific account type. The plan
noticed the fixture gap — its own text says no existing persona has this state — and bridged it by instructing the
tester to "create the goal yourself in the app" and pick that account type to fund it.
That flow does not exist. A code trace this session showed the web frontend has no path that puts this account type
into a goal: the add-goal flow drops the goal reference when this product type is chosen (a hardcoded redirect discards
it), the account-creation confirmation never sends the goal argument the backend mutation accepts, the
assign-account-to-goal mutation rejects this account type outright, and the two "add to a goal" entry points in
settings and on the account page are typed to other account kinds. The backend supports the state; no shipped UI
reaches it. So the plan's setup step was unrunnable by anyone, and the failure surfaced only when a person tried to
follow it.
The plan's own premise makes this worse: it promises "anyone who can open the local test apps and pick a persona can
run this plan." Eight of nine tests honor that. The ninth silently required a capability the author never checked.
Suggested fixes.
persona claim spot-checked this session was accurate). It applies no equivalent check to action steps. Any step that
tells the tester to do something in the app — create, attach, configure — should require the same evidence standard:
trace the route, component, or mutation that performs it before the step is written. "I could not find the flow" must
block the step, not soften into an instruction.
verified UI path creates it, the skill should say the test is not manually runnable as written and hand the gap back
as options (add a persona variant, cover it in automated tests, or confirm the state is reachable by some client at
all) instead of inventing setup steps. The skill's own contract — "when nothing in the supplied context can be
manually tested, it says so" — should apply per-test, not only to the whole plan.
variant sets and another does not. The plan never mentioned the flag, so even a partially-successful setup would have
rendered the wrong UI. Setup preconditions that live in flags belong in the persona line.
What worked well
spot-checks of the persona source files this session matched every claim the plan made about what those personas
contain, including a fallback persona that really does own the unusual account type.
hand-waving a persona that does not exist. The failure was in how it bridged the gap, not in hiding it.
outcomes; the areas grouping and per-test pass/fail affordances made the plan genuinely runnable by hand — for the
eight tests whose setup exists.
What didn't work
none of the required account type, on the theory the tester would create everything in-app; the fallback persona that
already owns the account was listed second. With the evidence gate in place, the ordering inverts or the test gets
blocked outright.
Overall
The skill produced a plan whose persona facts were verified and whose format served a hand-running tester well, but it
holds action steps to a lower evidence standard than persona claims. Where a needed state had no fixture, the plan
asserted an application capability it never traced, and the cost landed on the person following the steps. The missing
control is a per-step evidence gate: a manual test plan's instructions are claims about what the app can do, and each
one needs the same source-level verification the skill already applies to personas — with "not manually testable as
written" as the honest output when verification fails.
Rating
Accuracy 3/5: persona facts and eight tests' steps held up; the ninth test's central instruction was false. Evidence
discipline 2/5: the failure the report exists for — an unverified capability claim shipped as a step, in a plan that
demonstrably knew how to verify its other claims. Signal-to-noise 4/5: the plan's content is real and lean; no
agent-finding noise visible from this session. Length 4/5: proportionate to nine tests. Turn efficiency 3/5: the
planning session itself is not visible here, but the unrunnable step cost this session a full investigate-and-workaround
cycle that per-step verification would have spent once, at authoring time.