fix(generation): repair invalid model output by anandpant · Pull Request #313 · shpitdev/sketchi

anandpant · 2026-09-01T03:29:48Z

Scope

Add a configurable semantic repair budget with maxRepairAttempts: 2 by default, allowing one more semantic repair when the first repair introduces a different violation class.
Repair invalid JSON/IR with the prior output plus all validator diagnostics; regenerate truncated output fresh.
Promote flowchart.start_has_incoming and flowchart.self_loop, including their validator hints, at the top of repair requests. Start-node failures reroute to the first process after start; self-loops reroute to an earlier distinct process or decision node.
Preserve the original malformed candidate and its text/raw/error/usage when a repair call fails or returns invalid output; enrich it with both attempts' diagnostics.
Catch typed repair-call failures while allowing interruption to propagate, and always record sketchi_generation_repairs{outcome="failed"} for failed repair calls.
Classify truncation from only the latest model response so a truncated regeneration that is still invalid proceeds to semantic repair.
Preserve every existing node and label except the minimal edit required by reported diagnostics, and restate the original diagram type, scenario, parsed minimum node count, required node labels, and required branch labels on every repair call.
Keep transport retries and timeouts around each model call, normalize empty optional edge labels, and extract the first complete JSON object when a model appends trailing output.
Bypass Cloudflare AI Gateway cache in fresh mode with cf-aig-skip-cache: true.

Expected latency tradeoff

Each provider attempt has a 30-second timeout and up to two transient retries with 250 ms and 500 ms backoff, so one exhausted model call can take up to 90.75 seconds.
A malformed initial response can trigger two complete repair calls, making the configured generation-policy ceiling 272.25 seconds before outer request overhead. Requests above 120 seconds are therefore expected in the worst repair/retry cases; the added latency is the tradeoff for recovering when a repair introduces a new violation such as dropping below a requested minimum node count.
The deployed probe permits 302.25 seconds: the full 272.25-second policy budget plus a 30-second margin.

Proof

pnpm nx test diagram-generation --skip-nx-cache - 37/37 passed at 237076a.
pnpm nx run-many -t typecheck,test,build --skip-nx-cache - passed at 237076a.
pnpm nx build-storybook diagram-ui --skip-nx-cache - passed at 237076a.
Added typed repair-failure preservation, invalid-repair original-candidate preservation, interruption propagation, failed repair metric, latest-response truncation classification, priority start-node and self-loop correction, preserve-content/hard-requirement repair instructions, repair-introduced minimum violation recovery, cache bypass, requested-cycle, minimum-count, diagnostic-composition, and repair-success fixtures.

This is PR 5 of 6 in the generation-reliability stack.

anandpant · 2026-09-01T03:29:58Z

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

merge - adds this PR to the back of the merge queue
fast - for urgent changes, fast-track this PR to the front of the merge queue

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

graphite-app · 2026-09-01T21:25:06Z

Merge activity

Sep 1, 9:25 PM UTC: anandpant added this pull request to the Graphite merge queue.
Sep 1, 9:31 PM UTC: Merged by the Graphite merge queue.

## Scope - Add a configurable semantic repair budget with `maxRepairAttempts: 2` by default, allowing one more semantic repair when the first repair introduces a different violation class. - Repair invalid JSON/IR with the prior output plus all validator diagnostics; regenerate truncated output fresh. - Promote `flowchart.start_has_incoming` and `flowchart.self_loop`, including their validator hints, at the top of repair requests. Start-node failures reroute to the first process after start; self-loops reroute to an earlier distinct process or decision node. - Preserve the original malformed candidate and its text/raw/error/usage when a repair call fails or returns invalid output; enrich it with both attempts' diagnostics. - Catch typed repair-call failures while allowing interruption to propagate, and always record `sketchi_generation_repairs{outcome="failed"}` for failed repair calls. - Classify truncation from only the latest model response so a truncated regeneration that is still invalid proceeds to semantic repair. - Preserve every existing node and label except the minimal edit required by reported diagnostics, and restate the original diagram type, scenario, parsed minimum node count, required node labels, and required branch labels on every repair call. - Keep transport retries and timeouts around each model call, normalize empty optional edge labels, and extract the first complete JSON object when a model appends trailing output. - Bypass Cloudflare AI Gateway cache in fresh mode with `cf-aig-skip-cache: true`. ## Expected latency tradeoff - Each provider attempt has a 30-second timeout and up to two transient retries with 250 ms and 500 ms backoff, so one exhausted model call can take up to 90.75 seconds. - A malformed initial response can trigger two complete repair calls, making the configured generation-policy ceiling 272.25 seconds before outer request overhead. Requests above 120 seconds are therefore expected in the worst repair/retry cases; the added latency is the tradeoff for recovering when a repair introduces a new violation such as dropping below a requested minimum node count. - The deployed probe permits 302.25 seconds: the full 272.25-second policy budget plus a 30-second margin. ## Proof - `pnpm nx test diagram-generation --skip-nx-cache` - 37/37 passed at `237076a`. - `pnpm nx run-many -t typecheck,test,build --skip-nx-cache` - passed at `237076a`. - `pnpm nx build-storybook diagram-ui --skip-nx-cache` - passed at `237076a`. - Added typed repair-failure preservation, invalid-repair original-candidate preservation, interruption propagation, failed repair metric, latest-response truncation classification, priority start-node and self-loop correction, preserve-content/hard-requirement repair instructions, repair-introduced minimum violation recovery, cache bypass, requested-cycle, minimum-count, diagnostic-composition, and repair-success fixtures. This is PR 5 of 6 in the generation-reliability stack.

github-actions · 2026-09-01T21:32:11Z