test(generation): add structural reliability battery by anandpant · Pull Request #314 · shpitdev/sketchi

anandpant · 2026-09-01T03:29:50Z

Scope

Register eight reliability scenarios spanning production regressions, interacting loops, near-limit flowcharts, and deep 25-35 topic mindmaps.
Expose them through the eval-harness generation registry.
Add an Effect/jiti endpoint battery that sends fresh generation requests, repeats every scenario at least three times, supports targeted scenario runs, and derives its 302.25-second outer timeout from the complete generation-policy budget plus a 30-second margin.
Keep scenario prompts organic by removing generic self-loop policy text; system prompts and prioritized repair now own that invariant. The manuscript scenario only adds the business requirement that author revision is followed by a separate resubmission process.
Model required path waypoints as alternative-label groups and require every group on the same qualifying path. Expense rejection must cycle through both expense submission and resubmission; manuscript revision must cycle through both author revision and resubmission; manuscript ethics must cycle through both ethics investigation and editorial triage.
Bind every required branch label to an expected source-node label group. In particular, only finance approval/audit can satisfy the expense approved terminal assertion; a manager's approved branch cannot.
Stop a required-cycle search the first time it returns to its branch source without collecting every waypoint group, preventing adjacent cycles from being stitched into one false qualifying cycle.
Extended renderer scope: add local exterior routing lanes so same-rank branch arrows do not cross nodes in intervening ranks during Excalidraw export.

Proof

pnpm test:tools - 93/93 passed at fa0d38d.
pnpm nx run-many -t typecheck,test,build --skip-nx-cache - passed at b4d1487.
pnpm nx build-storybook diagram-ui --skip-nx-cache - passed at b4d1487.
Added regressions proving that a cycle missing expense resubmission fails, adjacent short and resubmission cycles cannot be stitched into one qualifying path, an ethics cycle missing Editorial Triage fails, and a manager-sourced approved branch cannot satisfy the finance terminal requirement, alongside inverse-semantics, one-cycle/two-decision, timeout-budget, organic-scenario, registry, selection, structural-fidelity, and reproduced ecommerce export-route coverage.
Corrected the structural-fidelity fixture so every padding edge references an existing node while preserving its intended cycle/path/count assertions; refreshed required CI passed at fa0d38d in workflow run 33560282033.
Runtime preview deployment passed at b4d14878428c in workflow run 33558720216; the only subsequent change at fa0d38d is the test-fixture correction.
Ecommerce returns 18-step scenario: 8/8 passed (100%) against the real preview with the scenario-level self-loop instruction removed.
Expense resubmission scenario: 8/8 passed (100%) against the real preview.
Full fresh preview battery: 24/24 passed (100%, three runs across all eight scenarios) against https://sketchi-studio-pr-314.dimethyl.workers.dev/api/v1/generate.

This is PR 6 of 6 in the generation-reliability stack.

anandpant · 2026-09-01T03:29:59Z

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

merge - adds this PR to the back of the merge queue
fast - for urgent changes, fast-track this PR to the front of the merge queue

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2026-09-01T03:31:04Z