Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# Security and behavior changes

## Unreleased — Responses output schema compatibility

- Project group/continuation/synthesis output schemas onto the documented Responses subset. Keep `uniqueItems` and string-length constraints in unchanged strict host validation, rather than sending them to the provider.
- Check every group schema and the actual pinned Codex wire schema against the same contract; duplicate reasons remain invalid locally. Retain only exact `invalid_json_schema` / `text.format.schema` diagnostics from bounded error envelopes.
- Track successful tool inspection across the execution instead of demanding a redundant command in every later group/synthesis turn. Every group's input is still independently hash-verified/provided, an execution with no inspection is still rejected, and incomplete group/context/synthesis checks still deny approval.
- Restore the first verified native image inputs after Codex compaction drops all image history. Retain their labels, fingerprints and original request/byte limits; partial replacement or substitution remains denied. Native acceptance forces real pinned-Codex compaction across a 93-file/four-distinct-image review.

## Unreleased — bounded review orchestration

- Harden required visual-input retention against duplicate/substituted images; native multi-turn acceptance now uses four distinct PNGs across groups, continuation and synthesis.
Expand Down
10 changes: 10 additions & 0 deletions docs/validation.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,17 @@
# Validation record and release gates

## Responses schema regression

Output schemas are checked recursively against the [documented Structured Outputs subset](https://developers.openai.com/api/docs/guides/structured-outputs#supported-schemas): required closed objects, enums, nullable types, patterns, numeric bounds and array `maxItems`. `uniqueItems` is not part of that subset; string-length bounds are also absent from its documented supported string properties. These constraints remain in host Ajv validation and are omitted only from the provider projection. All 32 possible group IDs and synthesis are checked; continuation uses the same group factory. Host-only provenance and coverage fields are never supplied to the model.

The native Docker fixture now validates `text.format.schema` on every request instead of accepting arbitrary schemas. It uses pinned Codex with synthetic responses across groups, continuation and synthesis, including first-request HTTP 400 and duplicate reasons that the host must reject. The old schema reproduces a first-request rejection under this contract. This proves the transmitted schema defect and failure path, not the real provider's private rejection reason or successful live inference. The production HTTP 400 lacked a usable allowlisted provider code/parameter; its exact cause remains unconfirmed without authorized live acceptance. Future bounded JSON envelopes may expose only the approved `invalid_json_schema` code and `text.format.schema` parameter, never messages or arbitrary schema paths.

## Bounded orchestration (synthetic acceptance)

PR50 isolated real ChatGPT testing completed a clean two-file review and a 33-file, three-group review with the expected authorization finding. The first 93-file run completed four groups before `proxy_image_inputs_missing` at request 20. A native pinned-Codex test then reproduced the same failure by forcing real compaction with synthetic token accounting: the compaction request succeeded, but subsequent history contained no images. The proxy now restores only its first validated native image parts and generated labels when all images are absent; it still rejects partial loss, duplicate-as-replacement and substitution, and rechecks the unchanged byte/request/model limits. A fresh proxy cannot reuse another execution's image cache. The native `large-compaction` case retains all four distinct images on every upstream request, completes six groups, continuation and synthesis in 17 scripted provider requests, and leaves production compaction thresholds unchanged. This proves protocol recovery, not semantic interpretation of complex screenshots.

Isolated live acceptance also exposed `codex_inspection_not_performed` in synthesis and later groups whose verified contents were already supplied in the model input. The legacy filesystem-only review guard had incorrectly become a fresh-command quota per turn. Native `synthesis-no-reread` and `provided-group-evidence` regressions reproduce both failures. Successful tool inspection is now tracked across the execution; every subsequent unit still goes through its independent immutable-file/hash verification before its contents are provided. Missing/tampered later-unit evidence still fails, and all group/context/synthesis/output checks remain unchanged. Existing no-inspection, incomplete-group, missing-context and invalid-output cases remain non-approving; neither a bare model completion claim nor a previous inspection can waive required evidence.

Final hardening replaces the identical-image fixture with four distinct valid PNG pixel patterns, verifies four Git identities and retention of all four encoded image inputs on every native request across six groups, one continuation and synthesis (16 scripted provider requests). The pinned first-turn-only attachment flow passed; actual image loss was not reproduced. A separate deterministic regression did reproduce the proxy accepting four copies of one image in place of four originals; required input fingerprints now reject loss, duplicate-as-replacement and substitution. One-group native completion takes one turn/two scripted inspection/report requests, while multi-group synthesis and global budget denial remain enforced. Incomplete GitHub reports expose only bounded counts, synthesis state and fixed reason enums, preserve summary/inline findings and cannot use metadata to authorize approval. Metadata is included in the existing size/DLP normalization before submission; a 59,950-byte review that overflows when annotated remains a failed, non-approving result with explicit output-loss flag. Real-model latency/token use and compaction/re-encoding at large production sizes remain unmeasured. No real inference or rollout was performed.

Deterministic tests cover host-owned immutable grouping, evidence tampering, exact unit/turn scope, global continuation limits, unresolved mandatory/visual evidence, valid partial output, lost execution after partial findings, duplicate findings and independent host coverage validation. Publication tests require COMMENT with preserved summary and inline findings for incomplete analysis, and failed-job tests retain nonzero review-job outcome. The actual pinned native Codex/Docker harness uses scripted Responses for small clean/findings/incomplete reviews, a 93-file / four-PNG case, changed callers across groups, targeted continuation, incomplete synthesis, changed evidence and a shared proxy budget exhausted between turns. Every scenario retains the existing write/network/credential denial checks. This is protocol, coverage and isolation evidence, not a claim of real-model semantic quality or measured production cost improvement. No live inference or production rollout belongs to this validation pass.
Expand Down
4 changes: 2 additions & 2 deletions src/core/execution.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ const PROVIDER_CODES = new Set(`subscription_sharing_user_not_eligible subscript
subscription_sharing_usage_unavailable subscription_sharing_unsupported_capability subscription_sharing_route_not_supported
subscription_sharing_invalid_user chatpass_v2_scope_not_authorized chatpass_v2_invalid_authorization_context
subscription_sharing_user_unavailable invalid_api_key insufficient_quota rate_limit_exceeded model_not_found
unsupported_parameter unsupported_value invalid_request_error`.split(/\s+/));
const PROVIDER_PARAMS = new Set(['model', 'reasoning.effort', 'tools', 'tool_choice', 'input', 'text.format', 'store', 'stream',
unsupported_parameter unsupported_value invalid_request_error invalid_json_schema`.split(/\s+/));
const PROVIDER_PARAMS = new Set(['model', 'reasoning.effort', 'tools', 'tool_choice', 'input', 'text.format', 'text.format.schema', 'store', 'stream',
'max_output_tokens', 'previous_response_id', 'temperature', 'prompt_cache_retention', 'metadata']);
const METHODS = new Set(['initialize', 'model/list', 'thread/start', 'turn/start']);
const SIGNALS = new Set(['SIGKILL', 'SIGTERM', 'SIGABRT', 'SIGSEGV', 'SIGINT']);
Expand Down
5 changes: 3 additions & 2 deletions src/runner/engine.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,10 @@ export async function runCodex(snapshot: RequestSnapshot, manifest: string, plan
developerInstructions: SYSTEM_REVIEW, config: { web_search: 'disabled' }
}));
const thread = record(started.thread); requireCondition(typeof thread.id === 'string', 'codex_thread_id_missing');
let attachedImages=false;
// Unit contents are independently verified/provided; tool inspection is execution-wide, not a per-turn quota.
let attachedImages=false,inspectionCompleted=false;
const turn = async (unitId:string, evidence:string, schema:unknown):Promise<unknown> => {
let finalText='',inspectionCompleted=false,turnId:string|undefined;
let finalText='',turnId:string|undefined;
const seen: {method:string;params:unknown}[]=[];let queuedBytes=0;
let receive=(method:string,params:unknown)=>{seen.push({method,params});};
const completed=new Promise<unknown>((resolve,reject)=>{
Expand Down
10 changes: 9 additions & 1 deletion src/runner/orchestration.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,15 @@ export interface ReviewExecution {
const hash = (value: string | Buffer) => createHash('sha256').update(value).digest('hex');
const turnSchema = { type: 'object', additionalProperties: false, required: ['unit_id', 'review', 'unfinished_reasons'], properties: { unit_id: { type: 'string', pattern: '^(group-[0-9]{2}|synthesis)$' }, review: REVIEW_SCHEMA, unfinished_reasons: { type: 'array', maxItems: UNFINISHED_REASONS.length, uniqueItems: true, items: { enum: UNFINISHED_REASONS } } } };
const checkTurn = new Ajv().compile(turnSchema);
export function reviewTurnSchema(id: string) { return { ...turnSchema, properties: { ...turnSchema.properties, unit_id: { enum: [id] } } }; }
// Keep uniqueness/string bounds in Ajv; send only documented Responses constraints.
function providerSchema(schema: Record<string, unknown>): Record<string, unknown> {
const projected = { ...schema };
for (const key of ['uniqueItems', 'minLength', 'maxLength']) delete projected[key];
if (projected.properties) projected.properties = Object.fromEntries(Object.entries(record(projected.properties)).map(([key, child]) => [key, providerSchema(record(child))]));
if (projected.items) projected.items = providerSchema(record(projected.items));
return projected;
}
export function reviewTurnSchema(id: string) { return providerSchema({ ...turnSchema, properties: { ...turnSchema.properties, unit_id: { enum: [id] } } }); }
/** The credential-owning host writes immutable evidence inputs; the model cannot choose coverage. */
export async function buildReviewPlan(directory: string, head: string, base: string, binaries: BinaryChange[]): Promise<ReviewPlan> {
const files = JSON.parse(await readFile(join(directory, 'diff.json'), 'utf8')) as ChangedFile[];
Expand Down
16 changes: 10 additions & 6 deletions src/runner/proxy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -66,13 +66,14 @@ async function hasEventStreamPrefix(upstream: Response): Promise<boolean> {
finally { void reader.cancel().catch(() => {}); }
}

export function inspectMedia(value:unknown,depth=0,images?:Set<string>):number {
export function inspectMedia(value:unknown,depth=0,images?:Set<string>,retained?:unknown[]):number {
requireCondition(depth<=32,'proxy_unsupported_media_input');if(value===null||typeof value!=='object')return 0;
if(Array.isArray(value))return value.reduce((count,item)=>count+inspectMedia(item,depth+1,images),0);
if(Array.isArray(value))return value.reduce((count,item)=>count+inspectMedia(item,depth+1,images,retained),0);
const item=value as Record<string,unknown>;
requireCondition(!['input_audio','input_video','input_file'].includes(String(item.type)),'proxy_unsupported_media_input');
if(item.type==='input_image'){requireCondition(typeof item.image_url==='string'&&/^data:image\/(?:png|jpeg|webp|gif);base64,[A-Za-z0-9+/]+={0,2}$/.test(item.image_url),'proxy_image_source_denied');images?.add(createHash('sha256').update(item.image_url).digest('hex'));return 1;}
return Object.values(item).reduce<number>((count,item)=>count+inspectMedia(item,depth+1,images),0);
if(item.type==='input_text'&&typeof item.text==='string'&&item.text.startsWith('Verified PR image: '))retained?.push({type:'input_text',text:item.text});
if(item.type==='input_image'){requireCondition(typeof item.image_url==='string'&&/^data:image\/(?:png|jpeg|webp|gif);base64,[A-Za-z0-9+/]+={0,2}$/.test(item.image_url),'proxy_image_source_denied');images?.add(createHash('sha256').update(item.image_url).digest('hex'));retained?.push({type:'input_image',image_url:item.image_url,...(item.detail===undefined?{}:{detail:item.detail})});return 1;}
return Object.values(item).reduce<number>((count,item)=>count+inspectMedia(item,depth+1,images,retained),0);
}

export async function inferenceProxy(request: IncomingMessage, response: ServerResponse, token: string, fetcher: typeof fetch = fetch, enforceSelection: (body: Record<string, unknown>) => void = () => {}, report: (failure: ExecutionFailure) => void = () => {}, onRequest?: (value: ProxyRequestDiagnostic) => void, requestIndex = 1): Promise<void> {
Expand Down Expand Up @@ -165,8 +166,11 @@ export function createInferenceServer(token: string, limits: InferenceLimits = {
let active = false;
let selection: string | null = null;
let requiredImages:Set<string>|null=null;
let retainedImages:unknown[]=[];
const enforceSelection = (body: Record<string, unknown>) => {
const images=new Set<string>(),count=inspectMedia(body.input,0,images);
const images=new Set<string>(),retained:unknown[]=[];let count=inspectMedia(body.input,0,images,retained);
// Codex 0.169 compaction drops images; restore only the first verified inputs, never partial replacements.
if(count===0&&requiredImages!==null){(body.input as unknown[]).push({role:'user',content:retainedImages});count=inspectMedia(retainedImages,0,images);}
requireCondition(count>=(limits.minimumImages??0)&&(!requiredImages||[...requiredImages].every(image=>images.has(image))),'proxy_image_inputs_missing');
requireCondition(Buffer.byteLength(JSON.stringify(body)) <= (limits.maxInputBytes ?? 24_000_000), 'proxy_acceptance_input_budget');
// Only the trusted API-key acceptance harness sets this. OAuth never adds unsupported fields.
Expand All @@ -176,7 +180,7 @@ export function createInferenceServer(token: string, limits: InferenceLimits = {
if (selection === null) selection = next;
requireCondition(selection === next, 'proxy_model_or_effort_change_denied');
// Lock the first trusted native image inputs; repetition cannot replace missing visual evidence.
if((limits.minimumImages??0)>0&&requiredImages===null)requiredImages=images;
if((limits.minimumImages??0)>0&&requiredImages===null){requiredImages=images;retainedImages=retained;}
};
return createServer((request, response) => {
const index=Math.min(++requests,101);
Expand Down
Loading