feat(providers): repair double-encoded structured output from LLMs - #104
Conversation
- Add state-machine repair layer for NoObjectGeneratedError in VercelAIProvider, covering both runPromptStructured and runWithTools - Handle models that emit nested fields as stringified JSON with unescaped internal quotes (observed with Anthropic via Bedrock on complex schemas) - Extract handleNoObjectGenerated to shared private method so both call paths get identical repair logic with distinct error framing - Add 6 tests covering happy-path repair through both entrypoints, usage preservation, schema-fail rethrow, unparseable JSON rethrow, and trailing-backslash fail-safe
|
Warning Review limit reached
Next review available in: 51 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe Vercel AI provider now repairs recoverable JSON output, validates repaired values against generated Zod schemas, and preserves usage metrics for structured-output and tool-calling failures. Tests cover successful recovery and unrecoverable error paths. ChangesVercel AI structured-output repair
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant AI_SDK
participant VercelAIProvider
participant JSONRepair
participant GeneratedZodSchema
AI_SDK->>VercelAIProvider: return NoObjectGeneratedError
VercelAIProvider->>JSONRepair: parse and repair raw JSON
JSONRepair-->>VercelAIProvider: repaired value
VercelAIProvider->>GeneratedZodSchema: validate repaired value
GeneratedZodSchema-->>VercelAIProvider: validated output or failure
VercelAIProvider-->>AI_SDK: repaired result or original error
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
tests/vercel-ai-provider-repair.test.ts (1)
68-89: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the duplicated schema fixture.
This 22-line schema literal is repeated in all six tests, at lines 68-89, 127-148, 175-196, 220-241, 270-291, and 321-342. It never varies. Extracting it to a module-level constant removes about 130 duplicated lines and makes each test's distinguishing input visible.
♻️ Proposed shared fixture
Define the constant once, after
MOCK_MODEL:const REVIEW_SCHEMA = { name: 'submit_review', schema: { properties: { score: { type: 'number' }, violations: { type: 'array', items: { type: 'object', properties: { line: { type: 'number' }, rule_quote: { type: 'string' }, severity: { type: 'string' }, }, required: ['line', 'rule_quote', 'severity'], }, }, }, required: ['score', 'violations'], type: 'object', }, } as const;Then replace each local declaration:
const provider = new VercelAIProvider(config); - const schema = { - name: 'submit_review', - schema: { - properties: { - score: { type: 'number' }, - violations: { - type: 'array', - items: { - type: 'object', - properties: { - line: { type: 'number' }, - rule_quote: { type: 'string' }, - severity: { type: 'string' }, - }, - required: ['line', 'rule_quote', 'severity'], - }, - }, - }, - required: ['score', 'violations'], - type: 'object', - }, - }; const result = await provider.runWithTools({ systemPrompt: 'system', prompt: 'prompt', tools: {}, - schema, + schema: REVIEW_SCHEMA, });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/vercel-ai-provider-repair.test.ts` around lines 68 - 89, Extract the duplicated schema fixture into a module-level REVIEW_SCHEMA constant immediately after MOCK_MODEL, preserving its structure and marking it as immutable. Replace all six identical local schema declarations in the tests with references to REVIEW_SCHEMA so each test retains only its distinguishing input.src/providers/vercel-ai-provider.ts (3)
33-33: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSuppress
no-control-regexinline.ESLint reports an error on this regular expression. The control-character range is intentional here, so the lint run fails on correct code. Add a targeted disable comment.
♻️ Proposed lint suppression
function escapeControlChars(value: string): string { + // eslint-disable-next-line no-control-regex -- control characters are the intended match return value.replace(/[\x00-\x1f]/g, char => {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/providers/vercel-ai-provider.ts` at line 33, Add a targeted ESLint suppression for no-control-regex directly on the intentional control-character regular expression in the value.replace callback, preserving the existing sanitization behavior and avoiding broader lint-rule disables.Source: Linters/SAST tools
100-111: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSimplify the return type to
unknown.
unknown | undefinedcollapses tounknown, becauseundefinedis already assignable tounknown. The annotation suggests a discriminated result that the type system does not provide. The same pattern appears ontryRepairStructuredOutputat line 426.Consider a discriminated result instead, so callers cannot confuse a repaired
undefinedwith a failure.♻️ Proposed type change
-function repairJsonString(value: string): unknown | undefined { +function repairJsonString(value: string): { ok: true; value: unknown } | { ok: false } { const attempts = [value, escapeControlChars(value), escapeInternalQuotes(value)]; for (const candidate of attempts) { try { - return JSON.parse(candidate); + return { ok: true, value: JSON.parse(candidate) }; } catch { // try next strategy } } - return undefined; + return { ok: false }; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/providers/vercel-ai-provider.ts` around lines 100 - 111, Update repairJsonString and tryRepairStructuredOutput to return unknown rather than the redundant unknown | undefined annotation. Preserve their existing runtime behavior, and if callers must distinguish a successfully parsed undefined value from repair failure, introduce an explicit discriminated result instead of relying on the return type.
400-417: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueThrow a domain error here.
Reuse the repository error hierarchy for this failure instead of
throw new Error(...). AVectorlintError/ConfigError-based domain error fits better than a bareError.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/providers/vercel-ai-provider.ts` around lines 400 - 417, Update handleNoObjectGenerated to throw the repository’s domain-specific VectorlintError or ConfigError instead of a bare Error when structured-output repair fails. Preserve the existing descriptive message and rawText details, and follow the established error hierarchy and constructor usage elsewhere in the provider.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/providers/vercel-ai-provider.ts`:
- Around line 414-416: Update the structured-output failure throw near the
rawText interpolation to truncate rawText to 500 characters, matching the
existing truncation used by the sibling failure paths in this file. Keep the
error context and message otherwise unchanged.
In `@tests/vercel-ai-provider-repair.test.ts`:
- Around line 110-124: Rename the test around MOCK_GENERATE_TEXT to describe
that usage remains undefined after repair, matching the result. Remove the
unused Error intersection cast and usage property declaration from the
NoObjectGeneratedError setup. Strengthen the assertions to verify the repair
path actually executed, while preserving the existing undefined result.
---
Nitpick comments:
In `@src/providers/vercel-ai-provider.ts`:
- Line 33: Add a targeted ESLint suppression for no-control-regex directly on
the intentional control-character regular expression in the value.replace
callback, preserving the existing sanitization behavior and avoiding broader
lint-rule disables.
- Around line 100-111: Update repairJsonString and tryRepairStructuredOutput to
return unknown rather than the redundant unknown | undefined annotation.
Preserve their existing runtime behavior, and if callers must distinguish a
successfully parsed undefined value from repair failure, introduce an explicit
discriminated result instead of relying on the return type.
- Around line 400-417: Update handleNoObjectGenerated to throw the repository’s
domain-specific VectorlintError or ConfigError instead of a bare Error when
structured-output repair fails. Preserve the existing descriptive message and
rawText details, and follow the established error hierarchy and constructor
usage elsewhere in the provider.
In `@tests/vercel-ai-provider-repair.test.ts`:
- Around line 68-89: Extract the duplicated schema fixture into a module-level
REVIEW_SCHEMA constant immediately after MOCK_MODEL, preserving its structure
and marking it as immutable. Replace all six identical local schema declarations
in the tests with references to REVIEW_SCHEMA so each test retains only its
distinguishing input.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: fd205771-f9c4-474c-aa2e-1001742fe76d
📒 Files selected for processing (2)
src/providers/vercel-ai-provider.tstests/vercel-ai-provider-repair.test.ts
- Move JSON repair functions to src/providers/utils.ts, keeping vercel-ai-provider.ts focused on provider logic - Truncate rawText in error message to 500 chars, matching sibling failure paths - Simplify return types from unknown | undefined to unknown - Add eslint-disable for intentional control-char regex - Extract duplicated schema fixture to module-level REVIEW_SCHEMA - Fix misleading test name and remove dead Error cast - Strengthen usage test with data assertion
1 participant
Why
When reviewing content against complex schemas, some LLMs (observed with Anthropic models via Bedrock) emit nested fields — like
violations— as stringified JSON with unescaped internal quotes. The Vercel AI SDK's structured output parser throwsNoObjectGeneratedError, and VectorLint surfaces this as an unrecoverable crash. Reviews against affected models cannot complete.What this PR covers
runPromptStructuredandrunWithTools) recover fromNoObjectGeneratedErrorwith identical repair logic.Scope
In scope
src/providers/vercel-ai-provider.ts— repair helpers, sharedhandleNoObjectGeneratedmethod, wiring in both catch blocks.tests/vercel-ai-provider-repair.test.ts— 6 tests covering repair paths and fail-safe edge cases.Out of scope
repairTextintegration (requires investigation against SDK version compatibility).Behavior Impact
NoObjectGeneratedErroron affected models now complete successfully. No configuration changes required.Risk and Mitigations
[vectorlint] Structured output repaired after NoObjectGeneratedError) makes every repair observable.NoObjectGeneratedError.API / Contract / Schema Changes
How to test / verify
Checks run
npx vitest run→ 353 passed (55 files)npx tsc --noEmit→ cleanManual verification
us.anthropic.claude-haiku-4-5-20251001-v1:0) in~/.vectorlint/config.toml.npx tsx src/index.ts <test-doc.md> --model-call singleagainst a document that triggers a complex schema response.NoObjectGeneratedError. After: completes review with findings, quality scores, and token usage.Future improvements
repairTextoption can replace the custom repair layer.Summary by CodeRabbit