diff --git a/CHANGELOG.md b/CHANGELOG.md index 74fd08f..8bb3b1a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,12 @@ # Security and behavior changes +## Unreleased — Responses output schema compatibility + +- Project group/continuation/synthesis output schemas onto the documented Responses subset. Keep `uniqueItems` and string-length constraints in unchanged strict host validation, rather than sending them to the provider. +- Check every group schema and the actual pinned Codex wire schema against the same contract; duplicate reasons remain invalid locally. Retain only exact `invalid_json_schema` / `text.format.schema` diagnostics from bounded error envelopes. +- Track successful tool inspection across the execution instead of demanding a redundant command in every later group/synthesis turn. Every group's input is still independently hash-verified/provided, an execution with no inspection is still rejected, and incomplete group/context/synthesis checks still deny approval. +- Restore the first verified native image inputs after Codex compaction drops all image history. Retain their labels, fingerprints and original request/byte limits; partial replacement or substitution remains denied. Native acceptance forces real pinned-Codex compaction across a 93-file/four-distinct-image review. + ## Unreleased — bounded review orchestration - Harden required visual-input retention against duplicate/substituted images; native multi-turn acceptance now uses four distinct PNGs across groups, continuation and synthesis. diff --git a/docs/validation.md b/docs/validation.md index 1485c62..52726cb 100644 --- a/docs/validation.md +++ b/docs/validation.md @@ -1,7 +1,17 @@ # Validation record and release gates +## Responses schema regression + +Output schemas are checked recursively against the [documented Structured Outputs subset](https://developers.openai.com/api/docs/guides/structured-outputs#supported-schemas): required closed objects, enums, nullable types, patterns, numeric bounds and array `maxItems`. `uniqueItems` is not part of that subset; string-length bounds are also absent from its documented supported string properties. These constraints remain in host Ajv validation and are omitted only from the provider projection. All 32 possible group IDs and synthesis are checked; continuation uses the same group factory. Host-only provenance and coverage fields are never supplied to the model. + +The native Docker fixture now validates `text.format.schema` on every request instead of accepting arbitrary schemas. It uses pinned Codex with synthetic responses across groups, continuation and synthesis, including first-request HTTP 400 and duplicate reasons that the host must reject. The old schema reproduces a first-request rejection under this contract. This proves the transmitted schema defect and failure path, not the real provider's private rejection reason or successful live inference. The production HTTP 400 lacked a usable allowlisted provider code/parameter; its exact cause remains unconfirmed without authorized live acceptance. Future bounded JSON envelopes may expose only the approved `invalid_json_schema` code and `text.format.schema` parameter, never messages or arbitrary schema paths. + ## Bounded orchestration (synthetic acceptance) +PR50 isolated real ChatGPT testing completed a clean two-file review and a 33-file, three-group review with the expected authorization finding. The first 93-file run completed four groups before `proxy_image_inputs_missing` at request 20. A native pinned-Codex test then reproduced the same failure by forcing real compaction with synthetic token accounting: the compaction request succeeded, but subsequent history contained no images. The proxy now restores only its first validated native image parts and generated labels when all images are absent; it still rejects partial loss, duplicate-as-replacement and substitution, and rechecks the unchanged byte/request/model limits. A fresh proxy cannot reuse another execution's image cache. The native `large-compaction` case retains all four distinct images on every upstream request, completes six groups, continuation and synthesis in 17 scripted provider requests, and leaves production compaction thresholds unchanged. This proves protocol recovery, not semantic interpretation of complex screenshots. + +Isolated live acceptance also exposed `codex_inspection_not_performed` in synthesis and later groups whose verified contents were already supplied in the model input. The legacy filesystem-only review guard had incorrectly become a fresh-command quota per turn. Native `synthesis-no-reread` and `provided-group-evidence` regressions reproduce both failures. Successful tool inspection is now tracked across the execution; every subsequent unit still goes through its independent immutable-file/hash verification before its contents are provided. Missing/tampered later-unit evidence still fails, and all group/context/synthesis/output checks remain unchanged. Existing no-inspection, incomplete-group, missing-context and invalid-output cases remain non-approving; neither a bare model completion claim nor a previous inspection can waive required evidence. + Final hardening replaces the identical-image fixture with four distinct valid PNG pixel patterns, verifies four Git identities and retention of all four encoded image inputs on every native request across six groups, one continuation and synthesis (16 scripted provider requests). The pinned first-turn-only attachment flow passed; actual image loss was not reproduced. A separate deterministic regression did reproduce the proxy accepting four copies of one image in place of four originals; required input fingerprints now reject loss, duplicate-as-replacement and substitution. One-group native completion takes one turn/two scripted inspection/report requests, while multi-group synthesis and global budget denial remain enforced. Incomplete GitHub reports expose only bounded counts, synthesis state and fixed reason enums, preserve summary/inline findings and cannot use metadata to authorize approval. Metadata is included in the existing size/DLP normalization before submission; a 59,950-byte review that overflows when annotated remains a failed, non-approving result with explicit output-loss flag. Real-model latency/token use and compaction/re-encoding at large production sizes remain unmeasured. No real inference or rollout was performed. Deterministic tests cover host-owned immutable grouping, evidence tampering, exact unit/turn scope, global continuation limits, unresolved mandatory/visual evidence, valid partial output, lost execution after partial findings, duplicate findings and independent host coverage validation. Publication tests require COMMENT with preserved summary and inline findings for incomplete analysis, and failed-job tests retain nonzero review-job outcome. The actual pinned native Codex/Docker harness uses scripted Responses for small clean/findings/incomplete reviews, a 93-file / four-PNG case, changed callers across groups, targeted continuation, incomplete synthesis, changed evidence and a shared proxy budget exhausted between turns. Every scenario retains the existing write/network/credential denial checks. This is protocol, coverage and isolation evidence, not a claim of real-model semantic quality or measured production cost improvement. No live inference or production rollout belongs to this validation pass. diff --git a/src/core/execution.ts b/src/core/execution.ts index 7f230d3..27660d8 100644 --- a/src/core/execution.ts +++ b/src/core/execution.ts @@ -16,8 +16,8 @@ const PROVIDER_CODES = new Set(`subscription_sharing_user_not_eligible subscript subscription_sharing_usage_unavailable subscription_sharing_unsupported_capability subscription_sharing_route_not_supported subscription_sharing_invalid_user chatpass_v2_scope_not_authorized chatpass_v2_invalid_authorization_context subscription_sharing_user_unavailable invalid_api_key insufficient_quota rate_limit_exceeded model_not_found -unsupported_parameter unsupported_value invalid_request_error`.split(/\s+/)); -const PROVIDER_PARAMS = new Set(['model', 'reasoning.effort', 'tools', 'tool_choice', 'input', 'text.format', 'store', 'stream', +unsupported_parameter unsupported_value invalid_request_error invalid_json_schema`.split(/\s+/)); +const PROVIDER_PARAMS = new Set(['model', 'reasoning.effort', 'tools', 'tool_choice', 'input', 'text.format', 'text.format.schema', 'store', 'stream', 'max_output_tokens', 'previous_response_id', 'temperature', 'prompt_cache_retention', 'metadata']); const METHODS = new Set(['initialize', 'model/list', 'thread/start', 'turn/start']); const SIGNALS = new Set(['SIGKILL', 'SIGTERM', 'SIGABRT', 'SIGSEGV', 'SIGINT']); diff --git a/src/runner/engine.ts b/src/runner/engine.ts index e6fbcfb..5ddb182 100644 --- a/src/runner/engine.ts +++ b/src/runner/engine.ts @@ -69,9 +69,10 @@ export async function runCodex(snapshot: RequestSnapshot, manifest: string, plan developerInstructions: SYSTEM_REVIEW, config: { web_search: 'disabled' } })); const thread = record(started.thread); requireCondition(typeof thread.id === 'string', 'codex_thread_id_missing'); - let attachedImages=false; + // Unit contents are independently verified/provided; tool inspection is execution-wide, not a per-turn quota. + let attachedImages=false,inspectionCompleted=false; const turn = async (unitId:string, evidence:string, schema:unknown):Promise => { - let finalText='',inspectionCompleted=false,turnId:string|undefined; + let finalText='',turnId:string|undefined; const seen: {method:string;params:unknown}[]=[];let queuedBytes=0; let receive=(method:string,params:unknown)=>{seen.push({method,params});}; const completed=new Promise((resolve,reject)=>{ diff --git a/src/runner/orchestration.ts b/src/runner/orchestration.ts index b86927d..5b662b1 100644 --- a/src/runner/orchestration.ts +++ b/src/runner/orchestration.ts @@ -40,7 +40,15 @@ export interface ReviewExecution { const hash = (value: string | Buffer) => createHash('sha256').update(value).digest('hex'); const turnSchema = { type: 'object', additionalProperties: false, required: ['unit_id', 'review', 'unfinished_reasons'], properties: { unit_id: { type: 'string', pattern: '^(group-[0-9]{2}|synthesis)$' }, review: REVIEW_SCHEMA, unfinished_reasons: { type: 'array', maxItems: UNFINISHED_REASONS.length, uniqueItems: true, items: { enum: UNFINISHED_REASONS } } } }; const checkTurn = new Ajv().compile(turnSchema); -export function reviewTurnSchema(id: string) { return { ...turnSchema, properties: { ...turnSchema.properties, unit_id: { enum: [id] } } }; } +// Keep uniqueness/string bounds in Ajv; send only documented Responses constraints. +function providerSchema(schema: Record): Record { + const projected = { ...schema }; + for (const key of ['uniqueItems', 'minLength', 'maxLength']) delete projected[key]; + if (projected.properties) projected.properties = Object.fromEntries(Object.entries(record(projected.properties)).map(([key, child]) => [key, providerSchema(record(child))])); + if (projected.items) projected.items = providerSchema(record(projected.items)); + return projected; +} +export function reviewTurnSchema(id: string) { return providerSchema({ ...turnSchema, properties: { ...turnSchema.properties, unit_id: { enum: [id] } } }); } /** The credential-owning host writes immutable evidence inputs; the model cannot choose coverage. */ export async function buildReviewPlan(directory: string, head: string, base: string, binaries: BinaryChange[]): Promise { const files = JSON.parse(await readFile(join(directory, 'diff.json'), 'utf8')) as ChangedFile[]; diff --git a/src/runner/proxy.ts b/src/runner/proxy.ts index 37cd12e..a6c062e 100644 --- a/src/runner/proxy.ts +++ b/src/runner/proxy.ts @@ -66,13 +66,14 @@ async function hasEventStreamPrefix(upstream: Response): Promise { finally { void reader.cancel().catch(() => {}); } } -export function inspectMedia(value:unknown,depth=0,images?:Set):number { +export function inspectMedia(value:unknown,depth=0,images?:Set,retained?:unknown[]):number { requireCondition(depth<=32,'proxy_unsupported_media_input');if(value===null||typeof value!=='object')return 0; - if(Array.isArray(value))return value.reduce((count,item)=>count+inspectMedia(item,depth+1,images),0); + if(Array.isArray(value))return value.reduce((count,item)=>count+inspectMedia(item,depth+1,images,retained),0); const item=value as Record; requireCondition(!['input_audio','input_video','input_file'].includes(String(item.type)),'proxy_unsupported_media_input'); - if(item.type==='input_image'){requireCondition(typeof item.image_url==='string'&&/^data:image\/(?:png|jpeg|webp|gif);base64,[A-Za-z0-9+/]+={0,2}$/.test(item.image_url),'proxy_image_source_denied');images?.add(createHash('sha256').update(item.image_url).digest('hex'));return 1;} - return Object.values(item).reduce((count,item)=>count+inspectMedia(item,depth+1,images),0); + if(item.type==='input_text'&&typeof item.text==='string'&&item.text.startsWith('Verified PR image: '))retained?.push({type:'input_text',text:item.text}); + if(item.type==='input_image'){requireCondition(typeof item.image_url==='string'&&/^data:image\/(?:png|jpeg|webp|gif);base64,[A-Za-z0-9+/]+={0,2}$/.test(item.image_url),'proxy_image_source_denied');images?.add(createHash('sha256').update(item.image_url).digest('hex'));retained?.push({type:'input_image',image_url:item.image_url,...(item.detail===undefined?{}:{detail:item.detail})});return 1;} + return Object.values(item).reduce((count,item)=>count+inspectMedia(item,depth+1,images,retained),0); } export async function inferenceProxy(request: IncomingMessage, response: ServerResponse, token: string, fetcher: typeof fetch = fetch, enforceSelection: (body: Record) => void = () => {}, report: (failure: ExecutionFailure) => void = () => {}, onRequest?: (value: ProxyRequestDiagnostic) => void, requestIndex = 1): Promise { @@ -165,8 +166,11 @@ export function createInferenceServer(token: string, limits: InferenceLimits = { let active = false; let selection: string | null = null; let requiredImages:Set|null=null; + let retainedImages:unknown[]=[]; const enforceSelection = (body: Record) => { - const images=new Set(),count=inspectMedia(body.input,0,images); + const images=new Set(),retained:unknown[]=[];let count=inspectMedia(body.input,0,images,retained); + // Codex 0.169 compaction drops images; restore only the first verified inputs, never partial replacements. + if(count===0&&requiredImages!==null){(body.input as unknown[]).push({role:'user',content:retainedImages});count=inspectMedia(retainedImages,0,images);} requireCondition(count>=(limits.minimumImages??0)&&(!requiredImages||[...requiredImages].every(image=>images.has(image))),'proxy_image_inputs_missing'); requireCondition(Buffer.byteLength(JSON.stringify(body)) <= (limits.maxInputBytes ?? 24_000_000), 'proxy_acceptance_input_budget'); // Only the trusted API-key acceptance harness sets this. OAuth never adds unsupported fields. @@ -176,7 +180,7 @@ export function createInferenceServer(token: string, limits: InferenceLimits = { if (selection === null) selection = next; requireCondition(selection === next, 'proxy_model_or_effort_change_denied'); // Lock the first trusted native image inputs; repetition cannot replace missing visual evidence. - if((limits.minimumImages??0)>0&&requiredImages===null)requiredImages=images; + if((limits.minimumImages??0)>0&&requiredImages===null){requiredImages=images;retainedImages=retained;} }; return createServer((request, response) => { const index=Math.min(++requests,101); diff --git a/tests/inspection-check.mjs b/tests/inspection-check.mjs index de05c6c..074ce20 100644 --- a/tests/inspection-check.mjs +++ b/tests/inspection-check.mjs @@ -14,40 +14,41 @@ import { ExecutionDiagnostics } from '../dist/src/core/execution.js'; import { cleanupReviewContainers } from '../dist/src/runner/docker.js'; import { writeInspectionFixture } from './inspection-fixture.mjs'; import {buildReviewPlan,validateExecution} from '../dist/src/runner/orchestration.js'; +import { assertProviderSchema } from './provider-schema-fixture.mjs'; const hash = value => createHash('sha256').update(value).digest('hex'); const dockerConfig = resolve('.local/docker'); const uid = process.getuid?.() ?? 1000, gid = process.getgid?.() ?? 1000; assert.notEqual(uid, 0, 'Use a nonroot Linux runner for the inspection test.'); function docker(args) { return execFileSync('docker', ['--config', dockerConfig, ...args], { encoding: 'utf8', stdio: 'pipe', timeout: 120_000 }); } -function events(items) { +function events(items,inputTokens=1) { return [{ type: 'response.created', response: { id: 'resp_fixture', status: 'in_progress', output: [] } }, ...items.flatMap((item, index) => [{ type: 'response.output_item.added', output_index: index, item }, { type: 'response.output_item.done', output_index: index, item }]), - { type: 'response.completed', response: { id: 'resp_fixture', status: 'completed', output: items, usage: { input_tokens: 1, output_tokens: 1, total_tokens: 2 } } } + { type: 'response.completed', response: { id: 'resp_fixture', status: 'completed', output: items, usage: { input_tokens: inputTokens, output_tokens: 1, total_tokens: inputTokens+1 } } } ].map((value, sequence_number) => `data: ${JSON.stringify({ ...value, sequence_number })}\n\n`).join(''); } // Scripted Responses only; real app-server policy, shell execution, mounts and output validation. -for (const scenario of ['untrusted', 'findings', 'clean', 'incomplete','vision','vision-invalid','schema-invalid','json-invalid','large','continuation','synthesis-incomplete','evidence-tampered','global-budget','cross-group']) { +for (const scenario of ['untrusted', 'findings', 'clean', 'incomplete','vision','vision-invalid','schema-invalid','json-invalid','large','continuation','synthesis-incomplete','evidence-tampered','global-budget','cross-group','provider-schema-rejected','reasons-duplicate','synthesis-no-reread','provided-group-evidence','large-compaction']) { const id = randomUUID(), proxy = `codexbot-inspection-proxy-${id}`, review = `codexbot-inspection-review-${id}`, volume = `codexbot-inspection-${id}`; const root = await mkdtemp(resolve('.local/inspection-')), directory = join(root, 'context'), head = 'a'.repeat(40); const config = globalThis.structuredClone(DEFAULT_CONFIG), request = { id, head, base: 'b'.repeat(40), config, command: { merge: false, effort: null, instructions: '' } }; const fixture = await writeInspectionFixture(directory, request, scenario !== 'clean'); - if(['cross-group','synthesis-incomplete','global-budget'].includes(scenario)){ + if(['cross-group','synthesis-incomplete','global-budget','synthesis-no-reread','provided-group-evidence'].includes(scenario)){ for(let i=1;i<=15;i++){const filename='src/padding-'+String(i).padStart(2,'0')+'.js';fixture.files.push({filename,status:'added',patch:'@@ -0,0 +1 @@\n+export const padding = true;'});await writeFile(join(directory,'source/head',filename),'export const padding = true;');} await mkdir(join(directory,'source/head/z-consumer'),{recursive:true}); const filename='z-consumer/handler.js',consumer="import { canAccess } from '../src/access.js';\nexport const serve = (role, record) => canAccess(role) ? record : null;\n"; fixture.files.push({filename,status:'added',patch:'@@ -0,0 +1,2 @@\n+'+consumer.trimEnd().split('\n').join('\n+')});await writeFile(join(directory,'source/head',filename),consumer);await writeFile(join(directory,'diff.json'),JSON.stringify(fixture.files)); } - if(scenario.startsWith('vision')||scenario==='large'){ + if(scenario.startsWith('vision')||scenario.startsWith('large')){ const chunk=(type,data)=>{const body=Buffer.concat([Buffer.from(type),data]),out=Buffer.alloc(data.length+12);out.writeUInt32BE(data.length);body.copy(out,4);out.writeUInt32BE(crc32(body),out.length-4);return out;}; const png=index=>scenario==='vision-invalid'?Buffer.from('invalid image'):Buffer.concat([Buffer.from([137,80,78,71,13,10,26,10]),chunk('IHDR',Buffer.from([0,0,0,1,0,0,0,1,8,6,0,0,0])),chunk('IDAT',deflateSync(Buffer.from([0,...[[220,20,20],[20,220,20],[20,20,220],[220,220,20]][index],255]))),chunk('IEND',Buffer.alloc(0))]); await mkdir(join(directory,'media/head'),{recursive:true}); - if(scenario==='large')for(let i=1;i<89;i++){ + if(scenario.startsWith('large'))for(let i=1;i<89;i++){ const filename='src/module-'+String(i).padStart(3,'0')+'.js'; fixture.files.push({filename,status:'modified',patch:'@@ -1 +1 @@\n-old\n+new'}); await writeFile(join(directory,'source/base',filename),'old');await writeFile(join(directory,'source/head',filename),'new'); } - const images=scenario==='large'?['image.png','second.png','third.png','fourth.png']:['image.png']; + const images=scenario.startsWith('large')?['image.png','second.png','third.png','fourth.png']:['image.png']; const imageBytes=images.map((_,index)=>png(index)); for(const [index,filename]of images.entries()){fixture.files.push({filename,status:'added'});await writeFile(join(directory,'media/head',filename),imageBytes[index]);} await writeFile(join(directory,'diff.json'),JSON.stringify(fixture.files)); @@ -77,18 +78,21 @@ for (const scenario of ['untrusted', 'findings', 'clean', 'incomplete','vision', console.log('fixture_isolation_verified');`; await writeFile(join(directory, 'inspect.mjs'), probe); await writeFile(join(root, 'credential'), 'fixture-only', { mode: 0o600 }); - // Negative control changes only the actual native thread's approval setting. + // Native controls change approval policy or force compaction using synthetic token accounting only. const preload = `import cp from 'node:child_process';import {syncBuiltinESMExports} from 'node:module'; - const spawn=cp.spawn;cp.spawn=(...args)=>{const child=spawn(...args);const write=child.stdin.write.bind(child.stdin);child.stdin.write=(chunk,...rest)=>{const msg=JSON.parse(String(chunk));if(msg.method==='thread/start')msg.params.approvalPolicy='untrusted';return write(JSON.stringify(msg)+'\\n',...rest)};return child};syncBuiltinESMExports();`; + const spawn=cp.spawn;cp.spawn=(...args)=>{const child=spawn(...args);const write=child.stdin.write.bind(child.stdin);child.stdin.write=(chunk,...rest)=>{const msg=JSON.parse(String(chunk));if(msg.method==='thread/start'){if(${JSON.stringify(scenario)}==='large-compaction')msg.params.config={...msg.params.config,model_context_window:131072,model_auto_compact_token_limit:50000};else msg.params.approvalPolicy='untrusted';}return write(JSON.stringify(msg)+'\\n',...rest)};return child};syncBuiltinESMExports();`; await writeFile(join(directory, 'untrusted-control.mjs'), preload); - const proxyScript = `import {createHash} from 'node:crypto';import {createInferenceServer} from '/app/dist/src/runner/proxy.js';import {contextServer} from '/app/dist/src/runner/context-server.js';import {executionEnvelope,proxyRequestEnvelope} from '/app/dist/src/core/execution.js';import {readFile} from 'node:fs/promises'; + const proxyScript = `import assert from 'node:assert/strict';import {createHash} from 'node:crypto';import {createInferenceServer} from '/app/dist/src/runner/proxy.js';import {contextServer} from '/app/dist/src/runner/context-server.js';import {executionEnvelope,proxyRequestEnvelope} from '/app/dist/src/core/execution.js';import {readFile} from 'node:fs/promises'; + const assertProviderSchema=${assertProviderSchema.toString()}; await contextServer('/run/proxy/context.sock',async(snapshot,path)=>snapshot==='head'&&path==='src/caller.js'); const token=(await readFile('/run/credential','utf8')).trim(); const evidence=${JSON.stringify(evidence)};let calls=0;const unitCalls=new Map();let firstImages=null; const events=${events.toString()}; globalThis.fetch=async(url,init)=>{ if(url!=='https://api.openai.com/v1/responses'||init.headers.Authorization!=='Bearer fixture-only')throw new Error('Fixture boundary failed'); - const body=JSON.parse(init.body);const media=[];const texts=[];const visit=v=>{if(v&&typeof v==='object'){if(v.type==='input_image'&&typeof v.image_url==='string')media.push(createHash('sha256').update(v.image_url).digest('hex'));if(typeof v.text==='string'){try{const item=JSON.parse(v.text);if(item.head_sha&&typeof item.unit_id==='string')texts.push(item)}catch{}}for(const x of Object.values(v))visit(x)}};visit(body.input);const requiredImages=${manifest.binaryFiles?.length??0};if(requiredImages){const unique=[...new Set(media)];if(!firstImages){if(unique.length!==requiredImages)throw new Error('Initial distinct image count incorrect');firstImages=unique;}if(firstImages.some(image=>!unique.includes(image)))throw new Error('A distinct required image was lost across turns');}const assigned=texts.at(-1);const unitId=assigned?.unit_id;if(unitId&&unitId!=='synthesis'){const payload=JSON.parse(assigned.evidence);const expected=${JSON.stringify(plan.units)}.find(u=>u.id===unitId);if(!expected||createHash('sha256').update(payload.evidence).digest('hex')!==expected.sha256)throw new Error('Unit evidence receipt missing');}if(unitId==='synthesis'&&${JSON.stringify(plan.units)}.some(u=>!unitCalls.has(u.id)))throw new Error('Synthesis skipped a required group');if(!unitId)throw new Error('No host assigned unit');const round=(unitCalls.get(unitId)??0)+1;unitCalls.set(unitId,round);calls++;if(${JSON.stringify(scenario.startsWith('vision'))}&&!JSON.stringify(body.input).includes('data:image/'))throw new Error('Native Codex did not supply image data');if(round%2===1){const tools=[...(body.tools??[]),...body.input.filter(item=>item.type==='additional_tools').flatMap(item=>item.tools??[])]; + try{const format=JSON.parse(init.body).text?.format;if(format?.schema)assert.equal(format.strict,true);if(${JSON.stringify(scenario)}==='provider-schema-rejected')format.schema.properties.unfinished_reasons.uniqueItems=true;if(format?.schema)assertProviderSchema(format.schema);else assert.equal(${JSON.stringify(scenario)},'large-compaction');}catch{return Response.json({error:{code:'invalid_json_schema',param:'text.format.schema',message:'Synthetic schema contract rejection'}},{status:400});} + if(${JSON.stringify(scenario)}==='synthesis-no-reread'&&JSON.parse(init.body).text.format.schema.properties.unit_id.enum[0]==='synthesis')return new Response(events([{id:'msg_synthesis',type:'message',role:'assistant',status:'completed',content:[{type:'output_text',text:JSON.stringify({unit_id:'synthesis',review:${encodedResult},unfinished_reasons:[]}),annotations:[]}]}]),{headers:{'content-type':'text/event-stream'}}); + const body=JSON.parse(init.body);const media=[];const texts=[];const visit=v=>{if(v&&typeof v==='object'){if(v.type==='input_image'&&typeof v.image_url==='string')media.push(createHash('sha256').update(v.image_url).digest('hex'));if(typeof v.text==='string'){try{const item=JSON.parse(v.text);if(item.head_sha&&typeof item.unit_id==='string')texts.push(item)}catch{}}for(const x of Object.values(v))visit(x)}};visit(body.input);const requiredImages=${manifest.binaryFiles?.length??0};if(requiredImages){const unique=[...new Set(media)];if(!firstImages){if(unique.length!==requiredImages)throw new Error('Initial distinct image count incorrect');firstImages=unique;}if(firstImages.some(image=>!unique.includes(image)))throw new Error('A distinct required image was lost across turns');}if(!body.text?.format?.schema){console.log('fixture_compaction_verified');return new Response(events([{id:'msg_compact',type:'message',role:'assistant',status:'completed',content:[{type:'output_text',text:'Synthetic compaction checkpoint. Continue the next host-assigned group using its verified evidence; preserve the validated finding.',annotations:[]}]}]),{headers:{'content-type':'text/event-stream'}});}const assigned=texts.at(-1);const unitId=assigned?.unit_id;if(unitId&&unitId!=='synthesis'){const payload=JSON.parse(assigned.evidence);const expected=${JSON.stringify(plan.units)}.find(u=>u.id===unitId);if(!expected||createHash('sha256').update(payload.evidence).digest('hex')!==expected.sha256)throw new Error('Unit evidence receipt missing');}if(unitId==='synthesis'&&${JSON.stringify(plan.units)}.some(u=>!unitCalls.has(u.id)))throw new Error('Synthesis skipped a required group');if(!unitId)throw new Error('No host assigned unit');const round=(unitCalls.get(unitId)??0)+1;unitCalls.set(unitId,round);calls++;if(${JSON.stringify(scenario.startsWith('vision'))}&&!JSON.stringify(body.input).includes('data:image/'))throw new Error('Native Codex did not supply image data');if(${JSON.stringify(scenario)}==='provided-group-evidence'&&unitId!=='group-00')return new Response(events([{id:'msg_reuse',type:'message',role:'assistant',status:'completed',content:[{type:'output_text',text:JSON.stringify({unit_id:unitId,review:${encodedResult},unfinished_reasons:[]}),annotations:[]}]}]),{headers:{'content-type':'text/event-stream'}});if(round%2===1){const tools=[...(body.tools??[]),...body.input.filter(item=>item.type==='additional_tools').flatMap(item=>item.tools??[])]; if(!tools.find(t=>t.type==='namespace'&&t.name==='functions')?.tools.some(t=>t.type==='custom'&&t.name==='exec'))throw new Error('Pinned code-mode tool missing');console.log('fixture_tool_selected'); return new Response(events([{id:'fc_fixture_'+calls,call_id:'call_fixture_'+calls,namespace:'functions',name:'exec',type:'custom_tool_call',input:'const r=await tools.exec_command({cmd:"node /workspace/inspect.mjs",yield_time_ms:10000,max_output_tokens:2000});text(r);'}]),{headers:{'content-type':'text/event-stream'}})} const output=JSON.stringify(body.input.filter(item=>['function_call_output','custom_tool_call_output'].includes(item.type))); @@ -96,11 +100,11 @@ for (const scenario of ['untrusted', 'findings', 'clean', 'incomplete','vision', console.log('fixture_inspection_verified'); const value=${JSON.stringify(scenario==='json-invalid'?'not-json':JSON.parse(encodedResult))};let finished=value; if(${JSON.stringify(scenario)}!=='json-invalid'){ - const incomplete=(${JSON.stringify(['continuation','large'])}.includes(${JSON.stringify(scenario)})&&unitId==='group-00'&&round===2)||(${JSON.stringify(scenario)}==='synthesis-incomplete'&&unitId==='synthesis')||${JSON.stringify(scenario)}==='incomplete'; + const incomplete=(${JSON.stringify(['continuation','large','large-compaction'])}.includes(${JSON.stringify(scenario)})&&unitId==='group-00'&&round===2)||(${JSON.stringify(scenario)}==='synthesis-incomplete'&&unitId==='synthesis')||${JSON.stringify(scenario)}==='incomplete'; if(incomplete&&value&&value.head_sha)value.complete=false;if(${JSON.stringify(scenario)}==='cross-group'&&unitId!=='synthesis'){value.findings=[];value.result='clean';} - finished=JSON.stringify({unit_id:unitId,review:value,unfinished_reasons:incomplete?['unreviewed_changes']:[]}); + finished=JSON.stringify({unit_id:unitId,review:value,unfinished_reasons:${JSON.stringify(scenario)}==='reasons-duplicate'?['unresolved_analysis','unresolved_analysis']:incomplete?['unreviewed_changes']:[]}); } - return new Response(events([{id:'msg_fixture_'+calls,type:'message',role:'assistant',status:'completed',content:[{type:'output_text',text:finished,annotations:[]}]}]),{headers:{'content-type':'text/event-stream'}}); + return new Response(events([{id:'msg_fixture_'+calls,type:'message',role:'assistant',status:'completed',content:[{type:'output_text',text:finished,annotations:[]}]}],${JSON.stringify(scenario)}==='large-compaction'&&unitId==='group-00'&&round===4?60000:1),{headers:{'content-type':'text/event-stream'}}); };const finite=globalThis.fetch;globalThis.fetch=async(url,init)=>{const value=await finite(url,init);if(calls%2!==0||!${JSON.stringify(['clean','incomplete'])}.includes(${JSON.stringify(scenario)}))return value;const bytes=new Uint8Array(await value.arrayBuffer());return new Response(new ReadableStream({start(controller){controller.enqueue(bytes);init.signal.addEventListener('abort',()=>controller.error(init.signal.reason),{once:true});}}),{status:value.status,headers:value.headers});};createInferenceServer(token,{minimumImages:${manifest.binaryFiles?.length??0},maxRequests:${scenario==='global-budget'?3:100}},detail=>process.stderr.write(executionEnvelope(detail)+'\\n'),value=>process.stderr.write(proxyRequestEnvelope(value)+'\\n')).listen('/run/proxy/inference.sock',()=>console.log('ready'));`; const common = ['--label', `codexbot.run=${id}`, '--network=none', '--read-only', '--cap-drop=ALL', '--security-opt=no-new-privileges', '--pids-limit=128', '--memory=2g', '--cpus=2', `--user=${uid}:${gid}`, `--tmpfs=/tmp:rw,noexec,nosuid,size=128m,uid=${uid}`, `--tmpfs=/home/codex:rw,nosuid,size=128m,uid=${uid}`]; try { @@ -116,25 +120,27 @@ for (const scenario of ['untrusted', 'findings', 'clean', 'incomplete','vision', logs.on('error', () => { clearTimeout(timer); reject(new Error('Inspection fixture logs unavailable')); }); logs.on('close', () => { clearTimeout(timer); reject(new Error('Inspection fixture proxy closed')); }); }); - const run = spawnSync('docker', ['--config', dockerConfig, 'run', '--name', review, ...common, '--mount', `type=bind,src=${directory},dst=/workspace,readonly`, '--mount', `type=volume,src=${volume},dst=/run/proxy,readonly`, 'codexbot-review:local', 'node', ...(scenario === 'untrusted' ? ['--import', '/workspace/untrusted-control.mjs'] : []), 'dist/src/runner/engine-main.js'], { encoding: 'utf8', stdio: 'pipe', timeout: 60_000, maxBuffer: 100_000 }); + const run = spawnSync('docker', ['--config', dockerConfig, 'run', '--name', review, ...common, '--mount', `type=bind,src=${directory},dst=/workspace,readonly`, '--mount', `type=volume,src=${volume},dst=/run/proxy,readonly`, 'codexbot-review:local', 'node', ...(['untrusted','large-compaction'].includes(scenario) ? ['--import', '/workspace/untrusted-control.mjs'] : []), 'dist/src/runner/engine-main.js'], { encoding: 'utf8', stdio: 'pipe', timeout: 60_000, maxBuffer: 100_000 }); const observed = new ExecutionDiagnostics(); observed.write(run.stderr ?? ''); const logs = spawnSync('docker', ['--config', dockerConfig, 'logs', proxy], { encoding: 'utf8', stdio: 'pipe', timeout: 10_000 }); const inspected = (logs.stdout ?? '').includes('fixture_inspection_verified'); const proxyFailure = new ExecutionDiagnostics(); proxyFailure.write(logs.stderr ?? ''); - if (run.status !== (['untrusted','vision-invalid','schema-invalid','json-invalid','evidence-tampered'].includes(scenario) ? 1 : 0) || (scenario === 'untrusted' && observed.last?.code !== 'codex_inspection_denied')) console.error(JSON.stringify({ scenario, failure: observed.last, proxy: proxyFailure.last, selected: (logs.stdout ?? '').includes('fixture_tool_selected'), inspected })); - if (['untrusted','vision-invalid','schema-invalid','json-invalid','evidence-tampered'].includes(scenario)) { + if(scenario==='provider-schema-rejected'){ + assert.equal(run.status,1);assert.equal(run.stdout.trim(),'');assert.equal(observed.last?.code,'codex_turn_failed');assert.equal(inspected,false);assert.equal(proxyFailure.last?.code,'provider_http_failed');assert.equal(proxyFailure.last?.upstream_status,400);assert.equal(proxyFailure.last?.request_index,1);assert.equal(proxyFailure.last?.provider_code,'invalid_json_schema');assert.equal(proxyFailure.last?.provider_param,'text.format.schema'); + } else if (['untrusted','vision-invalid','schema-invalid','json-invalid','evidence-tampered','reasons-duplicate'].includes(scenario)) { assert.equal(run.status, 1); assert.equal(run.stdout.trim(), ''); - if(scenario!=='vision-invalid')assert.equal(observed.last?.code, scenario === 'untrusted' ? 'codex_inspection_denied' : scenario==='json-invalid'?'codex_output_not_json':scenario==='evidence-tampered'?'review_plan_unverifiable':'review_incomplete'); + if(scenario!=='vision-invalid')assert.equal(observed.last?.code, scenario==='untrusted' ? 'codex_inspection_denied' : scenario==='json-invalid'?'codex_output_not_json':scenario==='evidence-tampered'?'review_plan_unverifiable':'review_incomplete'); else assert.equal((logs.stdout??'').includes('fixture_tool_selected'),false,'Invalid images must not reach upstream inference'); - if(scenario==='schema-invalid')assert.equal(observed.last?.rejection_reason,'schema_invalid'); - assert.equal(inspected, ['incomplete','schema-invalid','json-invalid'].includes(scenario)); + if(['schema-invalid','reasons-duplicate'].includes(scenario))assert.equal(observed.last?.rejection_reason,'schema_invalid'); + assert.equal(inspected, ['incomplete','schema-invalid','json-invalid','reasons-duplicate'].includes(scenario)); } else { assert.equal(run.status, 0, 'Actual Codex inspection failed; raw logs withheld'); assert.equal(inspected, true); - const execution=validateExecution(JSON.parse(run.stdout),plan);assert.equal(execution.synthesis,!['incomplete','synthesis-incomplete','global-budget'].includes(scenario));assert.equal(execution.coverage.length,plan.units.length);const output = validateReview(execution.review, head, fixture.files); assert.deepEqual(output, ['synthesis-incomplete','global-budget'].includes(scenario)?{...result,complete:false}:result);if(scenario==='cross-group'){assert.equal(execution.coverage.length,2);assert.equal(execution.turns,3);}if(scenario==='large'){assert.equal(plan.files.length,93);assert.equal(plan.binaries.length,4);assert.equal(execution.coverage.length,6);assert.equal(new Set(plan.binaries.flatMap(b=>b.snapshots.map(s=>s.sha))).size,4);assert.equal(execution.coverage[0].attempts,2);assert.equal(execution.turns,8);assert.equal(proxyFailure.request?.request_index,16);}if(scenario==='continuation'){assert.equal(execution.coverage[0].attempts,2);assert.equal(execution.turns,2);}if(scenario==='global-budget'){assert.equal(execution.failure.code,'codex_turn_failed');assert.equal(proxyFailure.last.code,'proxy_request_limit');} + const execution=validateExecution(JSON.parse(run.stdout),plan);assert.equal(execution.synthesis,!['incomplete','synthesis-incomplete','global-budget'].includes(scenario));assert.equal(execution.coverage.length,plan.units.length);const output = validateReview(execution.review, head, fixture.files); assert.deepEqual(output, ['synthesis-incomplete','global-budget'].includes(scenario)?{...result,complete:false}:result);if(scenario==='cross-group'){assert.equal(execution.coverage.length,2);assert.equal(execution.turns,3);}if(scenario.startsWith('large')){assert.equal(plan.files.length,93);assert.equal(plan.binaries.length,4);assert.equal(execution.coverage.length,6);assert.equal(new Set(plan.binaries.flatMap(b=>b.snapshots.map(s=>s.sha))).size,4);assert.equal(execution.coverage[0].attempts,2);assert.equal(execution.turns,8);assert.equal(proxyFailure.request?.request_index,scenario==='large-compaction'?17:16);if(scenario==='large-compaction')assert.equal((logs.stdout??'').split('\n').filter(line=>line==='fixture_compaction_verified').length,1);}if(scenario==='continuation'){assert.equal(execution.coverage[0].attempts,2);assert.equal(execution.turns,2);}if(scenario==='global-budget'){assert.equal(execution.failure.code,'codex_turn_failed');assert.equal(proxyFailure.last.code,'proxy_request_limit');} assert.equal(decision(output, config, true), scenario === 'clean' ? 'APPROVE' : ['incomplete','synthesis-incomplete','global-budget'].includes(scenario)?'COMMENT':'REQUEST_CHANGES'); } if(['clean','incomplete'].includes(scenario)){assert.equal(proxyFailure.request?.request_index,scenario==='clean'?2:6);assert.equal(proxyFailure.request.primary,null);assert.equal(proxyFailure.request.cleanup,null);assert.equal(proxyFailure.request.terminal_event,'completed');assert.equal(proxyFailure.last,null);} + if(['synthesis-no-reread','provided-group-evidence'].includes(scenario)){const execution=validateExecution(JSON.parse(run.stdout),plan);assert.equal(execution.coverage.length,2);assert.equal(execution.synthesis,true);assert.equal(execution.turns,3);assert.equal(execution.review.complete,true);assert.equal(proxyFailure.request?.request_index,scenario==='synthesis-no-reread'?5:4);} console.log(`Actual pinned Codex inspection/isolation scenario passed: ${scenario} (scripted Responses, no live AI inference).`); } finally { await cleanupReviewContainers(id); docker(['volume','rm',volume]); assert.equal(dirname(root), resolve('.local')); await rm(root, { recursive:true, force:true }); } } diff --git a/tests/orchestration.test.ts b/tests/orchestration.test.ts index 0910918..9613c09 100644 --- a/tests/orchestration.test.ts +++ b/tests/orchestration.test.ts @@ -47,6 +47,9 @@ await test('host planner supplies every changed base/head file, shared instructi assert(data.available_source.includes('source/base/src/caller.ts')); assert(data.trusted_instructions.includes('Inspect callers')); } + await writeFile(join(directory, 'source/head', changed[20]!.filename), 'tampered'); + await unitEvidence(directory,actual.units[0]!); + await assert.rejects(unitEvidence(directory,actual.units[1]!),/review_plan_unverifiable/); await writeFile(join(directory, 'source/head', changed[0]!.filename), 'tampered'); await assert.rejects(unitEvidence(directory, actual.units[0]!), /review_plan_unverifiable/); await rm(join(directory, 'source/head', changed[0]!.filename)); @@ -157,3 +160,12 @@ await test('a genuinely complete clean analysis is distinguishable from inconclu await test('a sole group retains the global continuation cap and cannot skip an unfinished integrated final check',async()=>{ const p=plan(1);let calls=0;const result=await orchestrateReview(p,async()=>'',async id=>{calls++;return response(id,false,[finding]);});assert.equal(calls,3);assert.equal(result.turns,3);assert.equal(result.synthesis,false);assert.equal(result.review.complete,false);assert.equal(decision(result.review,DEFAULT_CONFIG,true),'COMMENT');validateExecution(result,p); }); +await test('provider schema projection never weakens host uniqueness or review string bounds',async()=>{ + for(const value of [ + {...response('group-00',false),unfinished_reasons:['unresolved_analysis','unresolved_analysis']}, + {...response('group-00'),review:{...clean(),summary:'short'}}, + {...response('group-00'),review:{...clean(),summary:'s'.repeat(8001)}} + ])await assert.rejects(orchestrateReview(plan(1),async()=>'',async()=>value),error=>error instanceof ExecutionError&&error.detail.rejection_reason==='schema_invalid'); + const result=await orchestrateReview(plan(2),async()=>'',async id=>id==='synthesis'?{...response(id,false),unfinished_reasons:['unresolved_analysis','unresolved_analysis']}:response(id,true,[finding])); + assert.equal(result.failure?.rejection_reason,'schema_invalid');assert.equal(result.review.complete,false);assert.deepEqual(result.review.findings,[finding]);assert.equal(decision(result.review,DEFAULT_CONFIG,true),'COMMENT'); +}); diff --git a/tests/provider-schema-fixture.mjs b/tests/provider-schema-fixture.mjs new file mode 100644 index 0000000..e2c218c --- /dev/null +++ b/tests/provider-schema-fixture.mjs @@ -0,0 +1,17 @@ +import assert from 'node:assert/strict'; + +// Deliberately only the documented Responses subset used by our output schemas. +// https://developers.openai.com/api/docs/guides/structured-outputs#supported-schemas +export function assertProviderSchema(schema, depth = 0) { + assert(schema && typeof schema === 'object' && !Array.isArray(schema)); + assert(depth <= 10); + const keywords = new Set(['type', 'enum', 'properties', 'required', 'additionalProperties', 'items', 'pattern', 'minimum', 'maximum', 'maxItems']); + for (const key of Object.keys(schema)) assert(keywords.has(key), `Unsupported output-schema keyword: ${key}`); + assert(schema.type || Array.isArray(schema.enum)); + if (schema.type === 'object') { + assert.equal(schema.additionalProperties, false); + assert.deepEqual([...schema.required].sort(), Object.keys(schema.properties).sort()); + for (const child of Object.values(schema.properties)) assertProviderSchema(child, depth + 1); + } + if (schema.type === 'array') assertProviderSchema(schema.items, depth + 1); +} diff --git a/tests/provider-schema.test.mjs b/tests/provider-schema.test.mjs new file mode 100644 index 0000000..6cf01da --- /dev/null +++ b/tests/provider-schema.test.mjs @@ -0,0 +1,23 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; +import { reviewTurnSchema } from '../src/runner/orchestration.ts'; +import { assertProviderSchema } from './provider-schema-fixture.mjs'; + +test('all model-facing group and synthesis schemas use the documented Structured Outputs subset', () => { + for (const id of [...Array.from({ length: 32 }, (_, i) => 'group-' + String(i).padStart(2, '0')), 'synthesis']) { + const schema = reviewTurnSchema(id); + assertProviderSchema(schema); + assert.deepEqual(schema.properties.unit_id.enum, [id]); + assert.equal(schema.properties.unfinished_reasons.maxItems, 9); + assert.equal(schema.properties.review.additionalProperties, false); + } +}); + +test('schema contract rejects unsupported nested keywords, open objects and optional properties', () => { + for (const key of ['uniqueItems', 'minLength', 'maxLength', 'contains', 'allOf', 'default']) { + assert.throws(() => assertProviderSchema({ type: 'array', items: { type: 'string', [key]: true } }), /Unsupported output-schema keyword/); + } + for (const schema of [{ type: 'object', properties: {}, required: [], additionalProperties: true }, { type: 'object', properties: { value: { type: 'string' } }, required: [], additionalProperties: false }]) { + assert.throws(() => assertProviderSchema(schema)); + } +}); diff --git a/tests/proxy.test.ts b/tests/proxy.test.ts index f888ddb..60d63e6 100644 --- a/tests/proxy.test.ts +++ b/tests/proxy.test.ts @@ -18,6 +18,20 @@ await test('identical native encodings are not mistaken for missing Git images w const localFetch:typeof fetch=globalThis.fetch;const intercepted=mock.method(globalThis,'fetch',async()=>new Response('data: {"type":"response.completed"}\n\n',{headers:{'content-type':'text/event-stream'}}));const server=createInferenceServer('fixture-only',{minimumImages:4});server.listen(0,'127.0.0.1');await once(server,'listening');const address=server.address();assert(address&&typeof address!=='string'); try{for(const count of [4,4,3]){const response:Response=await localFetch(`http://127.0.0.1:${address.port}/v1/responses`,{method:'POST',body:JSON.stringify({model:'fixture',store:false,stream:true,input:Array.from({length:count},()=>({type:'input_image',image_url:'data:image/jpeg;base64,AAAA'}))})});assert.equal(response.status,count===4?200:403);await response.text();}}finally{server.closeAllConnections();server.close();intercepted.mock.restore();} }); +await test('compaction restores the original labeled images, with budgets and per-execution isolation intact',async()=>{ + const localFetch=globalThis.fetch,upstream:Record[]=[]; + const intercepted=mock.method(globalThis,'fetch',async(_url:Parameters[0],init?:RequestInit)=>{upstream.push(JSON.parse(String(init?.body)) as Record);return new Response('data: {"type":"response.completed"}\n\n',{headers:{'content-type':'text/event-stream'}});}); + const images=['AAAA','BBBB','CCCC','DDDD'].flatMap((value,index)=>[{type:'input_text',text:'Verified PR image: head/image-'+index+'.png'},{type:'input_image',image_url:'data:image/png;base64,'+value}]); + const server=createInferenceServer('fixture-only',{minimumImages:4,maxInputBytes:1800}),other=createInferenceServer('other-fixture',{minimumImages:4}); + server.listen(0,'127.0.0.1');other.listen(0,'127.0.0.1');await Promise.all([once(server,'listening'),once(other,'listening')]); + const post=async(target:typeof server,input:unknown[])=>{const address=target.address();assert(address&&typeof address!=='string');const response=await localFetch(`http://127.0.0.1:${address.port}/v1/responses`,{method:'POST',body:JSON.stringify({model:'fixture',store:false,stream:true,input})});await response.text();return response.status;}; + try{ + assert.equal(await post(server,[]),403);assert.equal(await post(server,[{role:'user',content:images}]),200); + for(let i=0;i<2;i++){assert.equal(await post(server,[{role:'user',content:[{type:'input_text',text:'compacted checkpoint'}]}]),200);assert.deepEqual((upstream.at(-1)?.input as unknown[]).at(-1),{role:'user',content:images});} + assert.equal(await post(server,[{role:'user',content:[{type:'input_text',text:'x'.repeat(1200)}]}]),403); + assert.equal(await post(other,[]),403);assert.equal(upstream.length,3); + }finally{for(const value of [server,other]){value.closeAllConnections();value.close();}intercepted.mock.restore();} +}); await test('API-key and ChatGPT adapters inject only their own bearer at the isolated inference boundary', async () => { for (const credential of ['fixture-personal-api-key', 'fixture-personal-oauth-token']) { const requests: { url: string; auth: string; body: string }[] = []; @@ -52,7 +66,7 @@ await test('proxy diagnostics retain only proven HTTP status and exact provider const secret = 'Bearer fixture-secret OAuth-code source prompt private-provider-message'; for (const [status, code, param] of [[401, 'subscription_sharing_invalid_user', null], [403, 'subscription_sharing_user_not_eligible', null], [403, 'chatpass_v2_scope_not_authorized', null], [429, 'subscription_sharing_usage_limit_exceeded', null], - [400, 'subscription_sharing_unsupported_capability', 'tools'], [404, 'model_not_found', 'model'], [503, secret, secret], [502, 'oversized', null]] as const) { + [400, 'subscription_sharing_unsupported_capability', 'tools'], [400, 'invalid_json_schema', 'text.format.schema'], [404, 'model_not_found', 'model'], [503, secret, secret], [502, 'oversized', null]] as const) { const diagnostics: ExecutionFailure[] = []; let calls = 0; const upstream: typeof fetch = async () => { calls++; return new Response(code === 'oversized' ? secret.repeat(1000) : JSON.stringify({ error: { code, param, message: secret, body: secret }, detail: secret }), { status }); diff --git a/tests/sandbox-check.mjs b/tests/sandbox-check.mjs index 06ec5bb..ac6f1a2 100644 --- a/tests/sandbox-check.mjs +++ b/tests/sandbox-check.mjs @@ -9,6 +9,7 @@ import { DEFAULT_CONFIG } from '../dist/src/core/config.js'; import { cleanupReviewContainers } from '../dist/src/runner/docker.js'; import { ExecutionDiagnostics } from '../dist/src/core/execution.js'; import {buildReviewPlan} from '../dist/src/runner/orchestration.js'; +import { assertProviderSchema } from './provider-schema-fixture.mjs'; // Runs real pinned Codex and real container isolation with synthetic Responses SSE, no credentials/quota. const id = randomUUID(), volume = `codexbot-test-${id}`, proxy = `codexbot-test-proxy-${id}`, review = `codexbot-test-review-${id}`; @@ -35,12 +36,14 @@ const events = [ { type: 'response.output_item.done', output_index: 0, item }, { type: 'response.completed', response: { id: 'resp_fixture', status: 'completed', output: [item], usage: { input_tokens: 1, output_tokens: 1, total_tokens: 2 } } } ].map((event, sequence_number) => `event: ${event.type}\ndata: ${JSON.stringify({ ...event, sequence_number })}\n\n`).join(''); -const proxyScript = `import {createInferenceServer} from '/app/dist/src/runner/proxy.js'; +const proxyScript = `import assert from 'node:assert/strict';import {createInferenceServer} from '/app/dist/src/runner/proxy.js'; import {executionEnvelope,proxyRequestEnvelope} from '/app/dist/src/core/execution.js'; +const assertProviderSchema=${assertProviderSchema.toString()}; let calls=0; globalThis.fetch = async (url, init) => { if(url !== 'https://api.openai.com/v1/responses' || init.headers.Authorization !== 'Bearer fixture-only') throw new Error('Invalid inference boundary'); const request=JSON.parse(init.body); + assert.equal(request.text?.format?.strict,true);assertProviderSchema(request.text.format.schema); if(request.store!==false || request.stream!==true || !Array.isArray(request.input) || request.input.some(item=>item.role==='system')) throw new Error('Invalid SIWC request contract'); if(++calls > 1) return Response.json({error:{code:calls===2?'subscription_sharing_unsupported_capability':'model_not_found',param:calls===2?'tools':'model',message:'private-fixture-provider-body'}},{status:calls===2?400:200}); return new Response(${JSON.stringify(events)}, {headers:{'content-type':'text/event-stream'}});