diff --git a/CHANGELOG.md b/CHANGELOG.md index 756e930..498a3e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Security and behavior changes +## Unreleased — bounded context reuse and review efficiency + +- Reuse verified source without spending another distinct-path lookup; separate 24 uncached snapshot/path lookups from a hard 100 total context-request cap. Preserve all GitHub API, byte, integrity, authorization and inference limits. Missing evidence and resource exhaustion remain non-approving, with fixed error reasons. +- Avoid retransmitting full modified old files alongside complete patches/current source; all base/head files remain independently hash-verified and available read-only. +- Stop subsequent groups/synthesis after unrecoverable required/visual-context or output-loss reports, preserving validated findings without additional inference. Soft continuations and complete cross-group synthesis retain their existing checks. +- Preserve host context failure flags/counters alongside the primary execution failure. Identify the unchanged 20-minute timeout as execution-wide; never log paths, source, prompts or credentials in these diagnostics. + + ## 0.1.1 — review execution and bounded orchestration ### Responses output schema compatibility diff --git a/docs/adaptive-context.md b/docs/adaptive-context.md index fae809f..e64160b 100644 --- a/docs/adaptive-context.md +++ b/docs/adaptive-context.md @@ -2,7 +2,7 @@ ## Bounded review orchestration -After collection passes, the trusted Actions host writes a private `review-plan.json` and immutable `review-units/` inputs. Changed paths are sorted to keep nearby code together, then grouped within 16 files / 2 MB per input, at most 32 groups. Inputs contain the actual textual patches and complete collected base/head changed-file text, or SHA-bound verified PNG metadata, plus applicable trusted instructions, the whole-PR path overview and an inventory of shared collected source. Every group keeps access to all original source, callers, dependencies, the file index and the same bounded context socket; grouping never removes cross-file context. Trusted instruction input is bounded to 1 MB; unrepresentable essential evidence fails before inference. Original Git/snapshot and binary verification remain required. +After collection passes, the trusted Actions host writes a private `review-plan.json` and immutable `review-units/` inputs. Changed paths are sorted to keep nearby code together, then grouped within 16 files / 2 MB per input, at most 32 groups. Inputs contain the actual textual patches and complete current/deleted changed-file text and hash receipts for every collected base/head file; modified old source remains read-only at its verified path instead of being retransmitted, or SHA-bound verified PNG metadata, plus applicable trusted instructions, the whole-PR path overview and an inventory of shared collected source. Every group keeps access to all original source, callers, dependencies, the file index and the same bounded context socket; grouping never removes cross-file context. Trusted instruction input is bounded to 1 MB; unrepresentable essential evidence fails before inference. Original Git/snapshot and binary verification remain required. The orchestrator is immutable application code, not a model-provided plan. It checks hashes of host-prepared inputs and referenced text/image files before delivery. The actual pinned app-server processes groups sequentially in one ephemeral thread, with one required successful inspection command per turn and matching thread/turn identity. It allows at most two targeted continuations **across the whole review**, including synthesis. Explicit missing mandatory or visual evidence cannot be cleared by a later model claim. For multiple groups, a separate final synthesis reconciles all unit results and cross-group behavior; it cannot waive unfinished units. A sole group instead receives an explicit final whole-PR/cross-file check in its own response, avoiding a redundant turn. Its verified evidence, inspection and completeness gates still apply. No model field such as “all groups inspected” controls scheduling or coverage. @@ -16,7 +16,7 @@ The existing immutable collection and isolated Codex execution remain in place. The collector writes `/workspace/file-index.json`: at most 12 nonrecursive directory walks and 5,000 regular-file paths. A partial supplementary index does not invalidate a small PR. It does not imply exhaustive semantic coverage. No full recursive tree or unrelated blob download is required. -A separate trusted host Unix socket at `/run/proxy/context.sock` accepts POST `/file` for required contents or `/probe` for optional existence, with JSON containing `snapshot: base|head` and a relative `path`. Successful retrieval materializes a verified file under the existing read-only reviewer mount. Paths, immutable commit/tree/blob identity, object modes, UTF-8, size and Git blob SHA are validated by the same collector. It cannot request another repository/ref, fetch URLs, follow symlinks or execute commands. There are at most 24 lookups, sharing the 768-call/16-MB-source/512-KB-file limits and SHA cache. Missing `/file` evidence makes the result incomplete and fails the review job. A `/probe` returns only `{exists: true|false}` from verified immutable tree metadata and never materializes source or satisfies content evidence. Verified absence is neutral; API/tree verification failures are errors and remain fail-closed, not false absence. +A separate trusted host Unix socket at `/run/proxy/context.sock` accepts POST `/file` for required contents or `/probe` for optional existence, with JSON containing `snapshot: base|head` and a relative `path`. Successful retrieval materializes a verified file under the existing read-only reviewer mount. Paths, immutable commit/tree/blob identity, object modes, UTF-8, size and Git blob SHA are validated by the same collector. It cannot request another repository/ref, fetch URLs, follow symlinks or execute commands. There are at most 24 distinct uncached snapshot/path lookups and 100 total socket requests, including cache hits and invalid input, sharing the 768-call/16-MB-source/512-KB-file limits and SHA cache. Missing `/file` evidence makes the result incomplete and fails the review job. A `/probe` returns only `{exists: true|false}` from verified immutable tree metadata and never materializes source or satisfies content evidence. Verified absence is neutral; API/tree verification failures are errors and remain fail-closed, not false absence. The GitHub read token remains on the trusted host through bounded exploration, rather than being revoked before inference. It is never mounted into the reviewer or credential proxy. On completion/failure, the host aborts/drains outstanding lookups, closes the context service, revokes the token and removes temporary source. Token cleanup failure fails the review job; it does not authorize approval. The separate publisher still uses the job conclusion and exact commit to decide whether a safe COMMENT is appropriate. @@ -52,7 +52,7 @@ The Docker acceptance uses the actual pinned Codex binary to prove PNG conversio Final completion is checked again by the trusted host after the isolated engine returns and the context service is closed/drained. A missing file requested through `/file` remains essential missing evidence even if the model returns complete findings or has inspected every attached PNG. The host preserves validated findings in the non-approving submission, fails the review job and logs only completion booleans/result enum plus an approved context failure code. On-demand failures enter required coverage diagnostics rather than being labeled supplementary omissions. Paths, model text and credentials are not diagnostic fields. Neither a later successful lookup nor a complete model result clears a previous essential-evidence failure. -Optional discovery never uses a model-supplied required/optional flag. The existing `/file` endpoint stays mandatory and rejects extra fields such as `required: false`; the separate `/probe` operation supplies existence only, not readable or verified contents. A probe can report an object exists even when a later content read rejects its Git type, size or blob hash. A successful probe cannot waive initial changed-file/instruction/dependency requirements or clear a previous required failure. Both operations share the unchanged 24-request and GitHub API budgets. Exhausted optional probes fail without invalidating otherwise complete evidence; a denied required fetch remains fatal. There is no extra inference pass or model/budget change. Host checks do not claim to prove exhaustive model reasoning about every possible dependency. +Optional discovery never uses a model-supplied required/optional flag. The existing `/file` endpoint stays mandatory and rejects extra fields such as `required: false`; the separate `/probe` operation supplies existence only, not readable or verified contents. A probe can report an object exists even when a later content read rejects its Git type, size or blob hash. A successful probe cannot waive initial changed-file/instruction/dependency requirements or clear a previous required failure. Both operations share 24 distinct uncached snapshot/path lookups and 100 total requests; cached verified contents remain reusable after the distinct-path allowance is consumed. The original GitHub API and byte limits are unchanged. Exhausted optional probes fail without invalidating otherwise complete evidence; a denied required fetch remains fatal. There is no extra inference pass or model/budget change. Host checks do not claim to prove exhaustive model reasoning about every possible dependency. ## Runtime envelope and image retention @@ -68,3 +68,11 @@ For G planned groups, successful baseline scheduling uses T = G + (G > 1 ? 1 : 0 The maximum plan leaves only 30 requests beyond that two-step/35-turn pattern and an arithmetic average of roughly 34 seconds per turn within 20 minutes, before startup overhead. There is no guaranteed large-PR completion or wall-clock/token cost estimate. Real models may need more inspection, compaction or reasoning; exhaustion produces a failed, non-approving partial review. Do not increase limits to hide this. A controlled rollout should first observe representative advisory/no-merge reviews and split work that does not fit the envelope. The native acceptance now creates four different PNG pixel patterns and verifies four distinct Git blob identities and four distinct encoded image inputs on every request across groups, continuation and final synthesis (16 requests in the large/one-continuation fixture). The first-turn-only attachment behavior is supported by the tested pinned app-server history; image loss was not reproduced. The proxy additionally binds fingerprints of the first trusted required visual inputs and rejects subsequent loss, repetition-as-replacement or substitution. Distinct Git images can have identical native encodings (for example after lossy conversion); the guard retains the minimum attachment count and the first trusted fingerprint set rather than inventing a one-to-one Git-to-encoding identity. Fingerprints and image bytes never enter public diagnostics. No images are blindly reattached per turn, which would inflate history. Native context compaction/re-encoding behavior at real large input sizes remains unmeasured; loss of required evidence fails closed rather than authorizing approval. + +## Context reuse and terminal failures + +Read already supplied source directly before using the context socket. A probe and subsequent content request for the same immutable snapshot/path consume one distinct-path slot. Repeated verified content requests use the existing mount and hash-verified cache, never another repository, ref or credential. Total requests stay separately bounded, including invalid bodies; crossing that bound cancels context retrieval and makes the host result non-approving. New essential paths cannot bypass the distinct-path or API/byte/type/integrity limits. + +Missing required files return a fixed context_path_missing reason; infrastructure, integrity, object-type and size failures are distinct fixed errors, never invented absence or raw exception text. Primary execution failures retain safe host completion flags and bounded lookup counts/code. The 20-minute deadline is explicitly identified as execution-wide; it and the 100-provider-request budget are unchanged. + +A unit reporting unavailable required/visual evidence or truncated output ends further inference immediately, preserving validated partial summaries/findings as COMMENT with a failed job. It cannot be repaired by a later model completeness claim. Soft unfinished analysis still uses the existing shared continuation allowance. Complete multi-group reviews still require all groups and final synthesis. diff --git a/docs/architecture.md b/docs/architecture.md index 556e4c0..d6d4fbd 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -59,7 +59,7 @@ See [operations](operations.md), [security](security.md) and [acceptance](accept Review execution uses a host-generated immutable coverage plan, bounded sequential app-server turns in one isolated thread and final cross-group synthesis. The host independently validates evidence receipts, coverage and final context/output completeness. Valid partial reviews survive as failed-job COMMENT results; neither partial findings nor a model assertion can authorize approval or merge. See [orchestration limits and acceptance](adaptive-context.md#bounded-review-orchestration). -The collector also writes a supplementary regular-file index (12 directories / 5,000 paths, no recursive tree dump). Explicit Python relative imports and Rust sibling modules join JS/TS dependencies; nearby Python/Go/Java test names are considered. A separate host Unix socket accepts bounded base/head `/file` requests for required contents and `/probe` requests for optional existence for this authorized immutable repository. It never forwards arbitrary URLs, executes project code or exposes credentials. At most 24 requests share the existing byte/blob/API limits and per-SHA cache. Required files that cannot be obtained invalidate completeness. A metadata-only probe may establish absence without fetching source; it cannot waive required evidence. Probe API/tree failures remain fail-closed rather than being mislabeled as missing paths. The reviewer mount stays read-only and network=none. The read token now lasts through bounded context exploration; shutdown cancels/drains lookups and revokes it before temporary source cleanup. Failed token cleanup fails the review job. +The collector also writes a supplementary regular-file index (12 directories / 5,000 paths, no recursive tree dump). Explicit Python relative imports and Rust sibling modules join JS/TS dependencies; nearby Python/Go/Java test names are considered. A separate host Unix socket accepts bounded base/head `/file` requests for required contents and `/probe` requests for optional existence for this authorized immutable repository. It never forwards arbitrary URLs, executes project code or exposes credentials. At most 24 distinct uncached snapshot/path lookups and 100 total requests share the existing byte/blob/API limits and per-SHA cache. Repeated verified files reuse existing evidence; total-request exhaustion cancels further retrieval and prevents approval. Required files that cannot be obtained invalidate completeness. A metadata-only probe may establish absence without fetching source; it cannot waive required evidence. Probe API/tree failures remain fail-closed rather than being mislabeled as missing paths. The reviewer mount stays read-only and network=none. The read token now lasts through bounded context exploration; shutdown cancels/drains lookups and revokes it before temporary source cleanup. Failed token cleanup fails the review job. The system prompt asks for introduced, evidence-backed bugs traced through unchanged callers and dependencies, with trusted default-branch guidance subordinate to execution/output safety. Additional comment context is advisory. The model/effort CLI precedence is unchanged; a generated pinned-native catalog powers dashboard validation and distinguishes catalog membership from account entitlement. diff --git a/src/core/diagnostics.ts b/src/core/diagnostics.ts index bcbc19d..7b426e0 100644 --- a/src/core/diagnostics.ts +++ b/src/core/diagnostics.ts @@ -1,10 +1,11 @@ import { AppError } from './errors.js'; import { ExecutionError, executionFailure, proxyRequestDiagnostic, UNFINISHED_REASONS, type ProxyRequestDiagnostic, type ExecutionFailure } from './execution.js'; +import { MAX_CONTEXT_PATHS, MAX_CONTEXT_REQUESTS } from './limits.js'; export const REVIEW_STAGES = ['environment','prepare','context','credentials','analyzing','result','publish','release'] as const; export const FAILURE_CODES = new Set(['missing_environment_credentials','ambiguous_environment_credentials','credential_owner_mismatch','missing_read_token', 'api_key_missing_or_ambiguous_credentials','vault_key_missing_or_ambiguous_credentials','invalid_envelope','invalid_vault_key','invalid_saved_session', - 'review_plan_unverifiable','review_plan_budget','review_coverage_invalid','context_path_missing','context_special_object','context_github_api_failed','context_lookup_rejected','context_api_budget','context_collection_timeout','context_diff_size_limit','context_text_patch_truncated','context_unsupported_binary','context_binary_unverifiable','context_binary_size_limit','context_non_text_source','context_missing_dependency','context_file_size_limit','context_byte_budget','context_required_evidence_unavailable','required_file_missing_or_special','blob_integrity_failure','required_tree_unverifiable','review_incomplete','review_context_incomplete','review_output_invalid','read_token_cleanup_failed', + 'review_plan_unverifiable','review_plan_budget','review_coverage_invalid','context_path_missing','context_special_object','context_github_api_failed','context_lookup_rejected','context_lookup_limit','context_request_limit','context_api_budget','context_collection_timeout','context_diff_size_limit','context_text_patch_truncated','context_unsupported_binary','context_binary_unverifiable','context_binary_size_limit','context_non_text_source','context_missing_dependency','context_file_size_limit','context_byte_budget','context_required_evidence_unavailable','required_file_missing_or_special','blob_integrity_failure','required_tree_unverifiable','review_incomplete','review_context_incomplete','review_output_invalid','read_token_cleanup_failed', 'oauth_refresh_too_early','oauth_refresh_window_unavailable', 'session_too_short_reconnect_required','review_image_missing','isolated_review_failed','proxy_start_failed','docker_unavailable','docker_cleanup_failed_check_runner', 'oauth_grant_invalid_reconnect_required','chatgpt_plan_usage_permission_missing','oauth_token_endpoint_failed','oauth_token_exchange_uncertain_reconnect_required', @@ -30,10 +31,12 @@ github_actions_oidc_unavailable invalid_oidc_endpoint github_oidc_request_failed export class BrokerError extends AppError { constructor(public readonly operation: string, status: number, public readonly remoteCode: string | null) { super(`broker_${operation}_failed`, status); } } +export interface ContextLookupStats { requests:number; paths:number; cache_hits:number; last_failure?:string } export interface ReviewCompletion { context_complete: boolean; diff_complete: boolean; required_context_complete: boolean; output_complete: boolean; output_result: 'clean' | 'findings' | 'inconclusive'; quarantined: boolean; output_truncated: boolean; context_failure_code?: string; + context_lookups?: ContextLookupStats; orchestration?: { planned_units:number; completed_units:number; synthesis_complete:boolean; unfinished_reasons: typeof UNFINISHED_REASONS[number][] }; } /** Only fixed completion conditions leave the trusted runner, never paths or model text. */ @@ -44,13 +47,15 @@ export class ReviewCompletionError extends AppError { this.completion={context_complete:value.context_complete===true,diff_complete:value.diff_complete===true,required_context_complete:value.required_context_complete===true, output_complete:value.output_complete===true,output_result:value.output_result==='clean'?'clean':value.output_result==='findings'?'findings':'inconclusive',quarantined:value.quarantined===true,output_truncated:value.output_truncated===true, ...(typeof value.context_failure_code==='string'&&FAILURE_CODES.has(value.context_failure_code)?{context_failure_code:value.context_failure_code}:{})}; + const lookups=value.context_lookups; + if(lookups&&Number.isInteger(lookups.requests)&&lookups.requests>=0&&lookups.requests<=MAX_CONTEXT_REQUESTS+1&&Number.isInteger(lookups.paths)&&lookups.paths>=0&&lookups.paths<=MAX_CONTEXT_PATHS&&Number.isInteger(lookups.cache_hits)&&lookups.cache_hits>=0&&lookups.cache_hits<=MAX_CONTEXT_REQUESTS)this.completion.context_lookups={requests:lookups.requests,paths:lookups.paths,cache_hits:lookups.cache_hits,...(typeof lookups.last_failure==='string'&&FAILURE_CODES.has(lookups.last_failure)?{last_failure:lookups.last_failure}:{})}; const coverage=value.orchestration; if(coverage&&Number.isInteger(coverage.planned_units)&&coverage.planned_units>=1&&coverage.planned_units<=32&&Number.isInteger(coverage.completed_units)&&coverage.completed_units>=0&&coverage.completed_units<=coverage.planned_units)this.completion.orchestration={planned_units:coverage.planned_units,completed_units:coverage.completed_units,synthesis_complete:coverage.synthesis_complete===true,unfinished_reasons:Array.isArray(coverage.unfinished_reasons)?[...new Set(coverage.unfinished_reasons.filter(r=>UNFINISHED_REASONS.includes(r)))].slice(0,UNFINISHED_REASONS.length):[]}; } } -export function diagnostic(error: unknown, stage: string): { event: string; stage: string; code: string; status?: number; broker_code?: string; execution?: ExecutionFailure; proxy_failure?: ExecutionFailure; proxy_cleanup?: ExecutionFailure; proxy_request?: Omit; completion?: ReviewCompletion } { +export function diagnostic(error: unknown, stage: string, completion?: ReviewCompletion): { event: string; stage: string; code: string; status?: number; broker_code?: string; execution?: ExecutionFailure; proxy_failure?: ExecutionFailure; proxy_cleanup?: ExecutionFailure; proxy_request?: Omit; completion?: ReviewCompletion } { const result: ReturnType = { - event: 'review_failed', stage: REVIEW_STAGES.includes(stage as typeof REVIEW_STAGES[number]) ? stage : 'environment', code: 'external_or_execution_failure' + event: 'review_failed', stage: REVIEW_STAGES.includes(stage as typeof REVIEW_STAGES[number]) ? stage : 'environment', code: 'external_or_execution_failure',...(completion?{completion:new ReviewCompletionError(completion).completion}:{}) }; if(error instanceof ReviewCompletionError){result.code=error.code;result.completion=error.completion;return result;} if (error instanceof ExecutionError) { diff --git a/src/core/execution.ts b/src/core/execution.ts index 27660d8..413ba09 100644 --- a/src/core/execution.ts +++ b/src/core/execution.ts @@ -37,6 +37,7 @@ export interface ProxyRequestDiagnostic extends RequestMetadata { request_index: export interface ExecutionFailure extends RequestMetadata { component: string; code: string; exit_code?: number; signal?: string; rpc_code?: number; rpc_method?: string; provider_code?: string; provider_param?: string; origin?: string; transport_code?: string; rejection_reason?: EngineRejectionReason; + timeout_scope?:'execution'; } function object(value: unknown): Record { return value !== null && typeof value === 'object' && !Array.isArray(value) ? value as Record : {}; @@ -70,6 +71,7 @@ export function executionFailure(value: unknown): ExecutionFailure | null { const { component, code } = source; if (typeof component !== 'string' || !COMPONENTS.has(component) || typeof code !== 'string' || !CODES.has(code)) return null; const result: ExecutionFailure = { component, code }; + if((component==='turn'||component==='engine')&&code==='codex_turn_timeout'&&source.timeout_scope==='execution')result.timeout_scope='execution'; if (Number.isInteger(source.exit_code) && Number(source.exit_code) >= 0 && Number(source.exit_code) <= 255) result.exit_code = Number(source.exit_code); if (typeof source.signal === 'string' && SIGNALS.has(source.signal)) result.signal = source.signal; if (typeof source.rpc_method === 'string' && METHODS.has(source.rpc_method)) result.rpc_method = source.rpc_method; diff --git a/src/core/limits.ts b/src/core/limits.ts index 3684df6..b5f3529 100644 --- a/src/core/limits.ts +++ b/src/core/limits.ts @@ -5,3 +5,5 @@ export const MAX_RESULT_BODY_BYTES = MAX_REVIEW_BYTES + 6_000; export const REQUEST_LIFETIME_MS = 60 * 60_000; export const RECOVERY_LEASE_MS = 120_000; export const MAX_RECOVERY_ATTEMPTS = 5; +export const MAX_CONTEXT_REQUESTS = 100; +export const MAX_CONTEXT_PATHS = 24; diff --git a/src/runner/context-server.ts b/src/runner/context-server.ts index 87e9ae4..ae7ff1e 100644 --- a/src/runner/context-server.ts +++ b/src/runner/context-server.ts @@ -1,14 +1,18 @@ import { createServer } from 'node:http'; import { once } from 'node:events'; import { chmod } from 'node:fs/promises'; -import { record, requireCondition } from '../core/errors.js'; +import { AppError, record, requireCondition } from '../core/errors.js'; +import { FAILURE_CODES } from '../core/diagnostics.js'; +import { MAX_CONTEXT_REQUESTS } from '../core/limits.js'; import type { ContextLookup } from './context.js'; /** Only immutable, authorized file lookups; this socket never forwards URLs or credentials. */ -export async function contextServer(socket: string, lookup: ContextLookup, abort: () => void = () => {}): Promise<() => Promise> { +export async function contextServer(socket: string, lookup: ContextLookup, abort: (reason?:'context_request_limit') => void = () => {}): Promise<() => Promise> { const pending = new Set>(); + let requests=0; const server = createServer((request, response) => { const task = (async () => { try { + if(++requests>MAX_CONTEXT_REQUESTS){if(requests===MAX_CONTEXT_REQUESTS+1)abort('context_request_limit');throw new AppError('context_request_limit');} requireCondition(request.method === 'POST' && (request.url === '/file'||request.url === '/probe'), 'context_request_denied'); let body = ''; for await (const chunk of request) { body += String(chunk); requireCondition(Buffer.byteLength(body) <= 1024, 'context_request_too_large'); } @@ -17,7 +21,11 @@ export async function contextServer(socket: string, lookup: ContextLookup, abort const probe=request.url==='/probe'; const available = probe ? await lookup(input.snapshot,input.path,'probe') : await lookup(input.snapshot,input.path); response.writeHead(200, { 'Content-Type': 'application/json', 'Cache-Control': 'no-store' }); response.end(JSON.stringify(probe?{exists:available}:{available})); - } catch { response.writeHead(400, { 'Content-Type': 'application/json' }); response.end('{"error":"context_lookup_unavailable"}'); } + } catch(error) { + const code=error instanceof AppError&&FAILURE_CODES.has(error.code)&&(error.code.startsWith('context_')||['blob_integrity_failure','required_tree_unverifiable','required_file_missing_or_special'].includes(error.code))?error.code:'context_lookup_unavailable'; + response.writeHead(code==='context_path_missing'?200:code==='context_request_limit'||code==='context_lookup_limit'?429:400,{'Content-Type':'application/json','Cache-Control':'no-store'}); + response.end(JSON.stringify(code==='context_path_missing'?{available:false,reason:code}:{error:code})); + } })().catch(() => { response.destroy(); }).finally(() => { pending.delete(task); }); pending.add(task); }); server.requestTimeout = 20_000; server.headersTimeout = 5000; server.maxConnections = 2; server.listen(socket); await once(server, 'listening'); if (process.platform !== 'win32') await chmod(socket, 0o600); diff --git a/src/runner/context.ts b/src/runner/context.ts index bf98800..c564861 100644 --- a/src/runner/context.ts +++ b/src/runner/context.ts @@ -9,11 +9,14 @@ import { safePath, completePatch } from '../core/review.js'; import { splitRepo } from '../worker/github.js'; import type { RequestSnapshot } from '../worker/store.js'; import { requireCondition } from '../core/errors.js'; +import { MAX_CONTEXT_PATHS, MAX_CONTEXT_REQUESTS } from '../core/limits.js'; +import type { ContextLookupStats } from '../core/diagnostics.js'; const MAX_BYTES = 16_000_000, MAX_DIFF_BYTES = 6_000_000, MAX_FILE_BYTES = 512_000, MAX_CALLS = 768, COLLECTION_MS = 120_000; export interface ContextManifest { complete: boolean; diffComplete: boolean; requiredContextComplete: boolean; supplementary: { strategy: 'bounded-heuristics'; selected: number; omitted: number }; files: number; bytes: number; apiCalls: number; limitations: string[]; failureCodes:string[]; binaryFiles:BinaryChange[]; + lookups?:ContextLookupStats; } export type ContextLookup = (snapshot: 'base' | 'head', path: string, operation?: 'probe') => Promise; export async function collectContext(github: Octokit, snapshot: RequestSnapshot, directory: string, register?: (lookup: ContextLookup) => void, signal?: AbortSignal): Promise { @@ -21,10 +24,12 @@ export async function collectContext(github: Octokit, snapshot: RequestSnapshot, const manifest: ContextManifest = { complete: false, diffComplete: true, requiredContextComplete: true, supplementary: { strategy: 'bounded-heuristics', selected: 0, omitted: 0 }, files: 0, bytes: 0, apiCalls: 0, limitations: [], failureCodes:[], binaryFiles:[] }; const limited = (required: boolean, error?:unknown) => { - if(required){const code=error instanceof AppError?error.code:error&&typeof error==='object'&&'status' in error&&typeof error.status==='number'&&error.status>=400&&error.status<=599?'context_github_api_failed':'context_required_evidence_unavailable';if(!manifest.failureCodes.includes(code))manifest.failureCodes.push(code);} + const code=error instanceof AppError?error.code:error&&typeof error==='object'&&'status' in error&&typeof error.status==='number'&&error.status>=400&&error.status<=599?'context_github_api_failed':'context_required_evidence_unavailable'; + if(required&&!manifest.failureCodes.includes(code))manifest.failureCodes.push(code); if (required) manifest.requiredContextComplete = false; else manifest.supplementary.omitted++; const reason = required ? 'Essential source or trusted instructions unavailable.' : 'Supplementary context bounded or unavailable.'; if (!manifest.limitations.includes(reason)) manifest.limitations.push(reason); + return code; }; const deadline=Date.now()+COLLECTION_MS;let collected=false; const call = (final = false) => { requireCondition(!signal?.aborted && (collected||Date.now()Boolean(file.patch))) { @@ -145,26 +150,32 @@ export async function collectContext(github: Octokit, snapshot: RequestSnapshot, } await writeFile(join(directory, 'file-index.json'), JSON.stringify({ paths: index, complete: indexComplete && queue.length === 0 && index.length < 5000 }), { mode: 0o600 }); if (register) { - let requests = 0, serial = Promise.resolve(); + let serial = Promise.resolve();const paths=new Set(); + const stats=manifest.lookups={requests:0,paths:0,cache_hits:0} as ContextLookupStats; register(async (label, path, operation) => { const required=operation!=='probe'; - try {requireCondition(++requests<=24&&(operation===undefined||operation==='probe')&&(label === 'base' || label === 'head') && safePath(path), 'context_lookup_rejected');} - catch(error){limited(required,error);if(required)manifest.complete=false;const work=serial.then(()=>writeFile(join(directory,'manifest.json'),JSON.stringify(manifest),{mode:0o600}));serial=work.catch(()=>{});await work;throw error;} + const request=stats.requests=Math.min(stats.requests+1,MAX_CONTEXT_REQUESTS+1); let available = false; const work = serial.then(async () => { - if(operation==='probe') { - try {available=Boolean(await find(label==='head'?snapshot.head:snapshot.base,path));if(!available)manifest.supplementary.omitted++;} - catch(error){limited(true,error);manifest.complete=false;await writeFile(join(directory,'manifest.json'),JSON.stringify(manifest),{mode:0o600});throw error;} - } else { - await collect(label, label === 'head' ? snapshot.head : snapshot.base, path, true); - available = written.has(`${label}/${path}`); - if (!available) { manifest.requiredContextComplete = false; manifest.complete = false; } - } + let invalidate=required; + try{ + requireCondition(request<=MAX_CONTEXT_REQUESTS,'context_request_limit'); + requireCondition((operation===undefined||operation==='probe')&&(label==='base'||label==='head')&&safePath(path),'context_lookup_rejected'); + const key=`${label}/${path}`; + if(written.has(key)){stats.cache_hits++;available=true;} + else{ + if(!paths.has(key)){requireCondition(paths.size {}); await work; return available; }); - manifest.limitations.push('Context socket /run/proxy/context.sock: POST {"snapshot":"base"|"head","path":"relative/file"} to /file for REQUIRED contents; then read /workspace/source//. For OPTIONAL existence exploration use /probe: {exists:false} means verified absence. Probes supply no contents or completeness waiver. Both share 24 requests and existing byte/API budgets. API/verification errors are not absence. A failed /file permanently invalidates completeness; /probe cannot clear it. No URLs or execution.'); + manifest.limitations.push('Read already supplied source directly under /workspace/source//; do not request it again. Context socket /run/proxy/context.sock: POST {"snapshot":"base"|"head","path":"relative/file"} to /file for REQUIRED contents; then read the supplied file. For OPTIONAL existence exploration use /probe: {exists:false} means verified absence. Both share 24 distinct snapshot/path lookups and 100 total requests; cached contents need no new repository lookup. Existing byte/API budgets apply. API/verification errors are not absence. A failed essential retrieval permanently invalidates completeness; /probe cannot clear it. No URLs or execution.'); } call(true); const { data: current } = await github.rest.pulls.get({ ...repo, pull_number: snapshot.pr }); requireCondition(current.head.sha === snapshot.head && current.base.sha === snapshot.base, 'pr_changed_during_context_collection', 409); diff --git a/src/runner/engine.ts b/src/runner/engine.ts index c9f25fa..04ffa7b 100644 --- a/src/runner/engine.ts +++ b/src/runner/engine.ts @@ -110,8 +110,8 @@ export async function runCodex(snapshot: RequestSnapshot, manifest: string, plan } finally {off();} }; const result=await orchestrateReview(plan,unit=>unitEvidence('/workspace',unit),turn); - if(timedOut){result.failure={component:'turn',code:'codex_turn_timeout'};result.review.complete=false;if(result.review.result==='clean')result.review.result='inconclusive';if(!result.reasons.includes('execution_failed'))result.reasons.push('execution_failed');} + if(timedOut){result.failure={component:'turn',code:'codex_turn_timeout',timeout_scope:'execution'};result.review.complete=false;if(result.review.result==='clean')result.review.result='inconclusive';if(!result.reasons.includes('execution_failed'))result.reasons.push('execution_failed');} return result; - } catch (error) { throw timedOut ? new ExecutionError({ component: 'turn', code: 'codex_turn_timeout' }) : executionError(error, 'codex_rpc', 'invalid_codex_protocol'); } + } catch (error) { throw timedOut ? new ExecutionError({ component: 'turn', code: 'codex_turn_timeout',timeout_scope:'execution' }) : executionError(error, 'codex_rpc', 'invalid_codex_protocol'); } finally { clearTimeout(timer); child.stdin.end(); child.kill('SIGKILL'); } } diff --git a/src/runner/main.ts b/src/runner/main.ts index 06c417e..1db2f2b 100644 --- a/src/runner/main.ts +++ b/src/runner/main.ts @@ -6,7 +6,7 @@ import type { RequestSnapshot } from '../worker/store.js'; import type { Envelope } from '../core/crypto.js'; import { unseal, seal, sessionBinding } from '../core/crypto.js'; import { record, requireCondition, AppError } from '../core/errors.js'; -import { diagnostic, FAILURE_CODES, BROKER_CODES, ReviewCompletionError } from '../core/diagnostics.js'; +import { diagnostic, FAILURE_CODES, BROKER_CODES, ReviewCompletionError, type ReviewCompletion } from '../core/diagnostics.js'; import { OAuth, validateSession } from '../auth/oauth.js'; import { broker, backendUrl, checkWorkload } from './broker.js'; import { collectContext } from './context.js'; @@ -31,6 +31,7 @@ export async function reviewJob(): Promise { let backend = '', temporary: string | undefined; let snapshot: RequestSnapshot | null = null; let submission: { output: unknown; context_complete: boolean; failure?: ReturnType } | null = null; + let completion:ReviewCompletion|undefined; let stage = 'environment'; let closeContext: (() => Promise) | undefined, revokeContext: (() => Promise) | undefined; try { @@ -84,7 +85,7 @@ export async function reviewJob(): Promise { const image = process.env.CODEXBOT_IMAGE; requireCondition(image, 'review_image_missing'); const sockets = join(temporary, 'sockets'); await mkdir(sockets, { mode: 0o700 }); requireCondition(lookup, 'review_context_incomplete'); - closeContext = await contextServer(process.platform === 'win32' ? `\\\\.\\pipe\\codexbot-${crypto.randomUUID()}` : join(sockets, 'context.sock'), lookup, () => contextCancellation.abort()); + closeContext = await contextServer(process.platform === 'win32' ? `\\\\.\\pipe\\codexbot-${crypto.randomUUID()}` : join(sockets, 'context.sock'), lookup, reason => {contextCancellation.abort();if(reason){manifest.complete=false;manifest.requiredContextComplete=false;if(!manifest.failureCodes.includes(reason))manifest.failureCodes.push(reason);}}); stage = 'analyzing'; await broker(backend, id, 'stage', { stage: 'analyzing' }); let partialFailure:ExecutionError|undefined; @@ -101,15 +102,16 @@ export async function reviewJob(): Promise { output=normalizeReview({...output,orchestration:coverage},snapshot.head); if(output.output_truncated&&!coverage.unfinished_reasons.includes('output_truncated')){coverage.unfinished_reasons.push('output_truncated');output.orchestration=coverage;} submission.output=output; - if(partialFailure||execution.failure)throw partialFailure??new ExecutionError(execution.failure!); - if(!(manifest.complete&&manifest.diffComplete&&manifest.requiredContextComplete&&output.complete&&output.result!=='inconclusive'&&!output.quarantined&&!output.output_truncated))throw new ReviewCompletionError({ + completion={ context_complete:manifest.complete,diff_complete:manifest.diffComplete,required_context_complete:manifest.requiredContextComplete, output_complete:output.complete,output_result:output.result,quarantined:output.quarantined===true,output_truncated:output.output_truncated===true, orchestration:coverage, - ...(manifest.failureCodes[0]?{context_failure_code:manifest.failureCodes[0]}:{})}); + ...(manifest.failureCodes[0]?{context_failure_code:manifest.failureCodes[0]}:{}),...(manifest.lookups?{context_lookups:manifest.lookups}:{})}; + if(partialFailure||execution.failure)throw partialFailure??new ExecutionError(execution.failure!); + if(!(manifest.complete&&manifest.diffComplete&&manifest.requiredContextComplete&&output.complete&&output.result!=='inconclusive'&&!output.quarantined&&!output.output_truncated))throw new ReviewCompletionError(completion); await broker(backend, id, 'result', submission); } catch (error) { - const report = diagnostic(error, stage); console.error(JSON.stringify(report)); + const report = diagnostic(error, stage,completion); console.error(JSON.stringify(report)); // Always leave a deterministic non-approving result when preflight succeeded. const failure = { ...report, code: report.broker_code ?? (FAILURE_CODES.has(report.code) || BROKER_CODES.has(report.code) ? report.code : 'external_or_execution_failure') }; try { diff --git a/src/runner/orchestration.ts b/src/runner/orchestration.ts index 5b662b1..77c000a 100644 --- a/src/runner/orchestration.ts +++ b/src/runner/orchestration.ts @@ -8,6 +8,7 @@ import { REVIEW_SCHEMA, normalizeReview, validateReview, safePath, type Review, import { scanSecrets } from '../core/dlp.js'; import { ExecutionError, executionError, executionFailure, UNFINISHED_REASONS, type ExecutionFailure } from '../core/execution.js'; const GROUP_FILES = 16, GROUP_BYTES = 2_000_000, MAX_GROUPS = 32, MAX_CONTINUATIONS = 2; +const TERMINAL_REASONS=new Set(['missing_required_context','visual_evidence_unavailable','output_truncated']); type Reason = typeof UNFINISHED_REASONS[number]; export interface ReviewUnit { id: string; @@ -79,7 +80,7 @@ export async function buildReviewPlan(directory: string, head: string, base: str sha256: string; text?: string; }[] = []; - const encode = () => JSON.stringify({ changes, evidence, trusted_instructions: trusted, available_source: available, whole_pr: files.map(f => ({ path: f.filename, status: f.status })), instructions: 'Inspect the supplied immutable evidence and relevant shared source/callers. This is a scoped unit, not permission to omit the other units. All source remains available read-only. Use /probe only for optional existence; /file for essential missing contents.' }); + const encode = () => JSON.stringify({ changes, evidence, trusted_instructions: trusted, available_source: available, whole_pr: files.map(f => ({ path: f.filename, status: f.status })), instructions: 'Analyze the supplied current/deleted source and complete patches first. Full modified base files remain hash-verified at their evidence paths; read them and shared callers where needed. Do not repeat completed unit analysis. This is a scoped unit, not permission to omit other units or relevant dependencies. All source remains available read-only. Read already supplied paths directly; use /probe only for optional existence and /file for essential missing contents.' }); await mkdir(join(directory, 'review-units'), { recursive: true, mode: 0o700 }); const flush = async () => { if (!changes.length && units.length) return; @@ -99,7 +100,9 @@ export async function buildReviewPlan(directory: string, head: string, base: str requireCondition(available.includes(path), 'review_plan_unverifiable'); const bytes = await readFile(join(directory, path)); requireCondition(bytes.length <= 512_000 && !bytes.includes(0), 'review_plan_unverifiable'); - additions.push({ path, sha256: hash(bytes), text: new TextDecoder('utf8', { fatal: true }).decode(bytes) }); + const text=new TextDecoder('utf8',{fatal:true}).decode(bytes); + // Modified old contents remain hash-verified on disk; the complete patch already supplies removed lines. + additions.push({path,sha256:hash(bytes),...(side==='head'||file.status==='removed'?{text}:{})}); } else { const binary = binaries.find(b => b.filename === file.filename); @@ -178,7 +181,7 @@ function turnReview(value: unknown, id: string, plan: ReviewPlan): { /** One execution/thread/proxy owns all turns and the unchanged global inference/time budget. */ export async function orchestrateReview(plan: ReviewPlan, evidence: (unit: ReviewUnit) => Promise, turn: (id: string, input: string, schema: unknown) => Promise): Promise { const coverage: Coverage[] = [], reviews: Review[] = []; - let continuations = 0, synthesis = false, turns = 0, failure: ExecutionFailure | null = null, final: Review | undefined; + let continuations = 0, synthesis = false, turns = 0, blocked=false, failure: ExecutionFailure | null = null, final: Review | undefined; const reasons = new Set(); try { for (const unit of plan.units) { @@ -191,7 +194,7 @@ export async function orchestrateReview(plan: ReviewPlan, evidence: (unit: Revie reviews.push(result.review); state.reasons = result.reasons; state.complete = result.reasons.length === 0; - if (state.complete || state.reasons.some(r => r === 'missing_required_context' || r === 'visual_evidence_unavailable' || r === 'output_truncated')) + if (state.complete || state.reasons.some(r => TERMINAL_REASONS.has(r))) break; if (continuations >= MAX_CONTINUATIONS) { reasons.add('continuation_limit'); @@ -199,15 +202,16 @@ export async function orchestrateReview(plan: ReviewPlan, evidence: (unit: Revie } continuations++; } + if(state.reasons.some(r=>TERMINAL_REASONS.has(r))){blocked=true;break;} } if(plan.units.length===1){final=reviews.at(-1);synthesis=coverage[0]?.complete===true;} - else for (;;) { + else if(!blocked) for (;;) { turns++; const result = turnReview(await turn('synthesis', JSON.stringify({ unit_id: 'synthesis', instructions: 'Reconcile cross-group dependencies, interfaces, security and compatibility across the entire PR. Inspect shared source as needed. Preserve evidence-backed findings; do not claim completion if a unit or consequential question remains unresolved. Missing runtime test execution and a partial supplementary file index alone are expected, not missing analysis.', coverage, unit_reviews: reviews, whole_pr: plan.files.map(f => ({ path: f.filename, status: f.status })) }), reviewTurnSchema('synthesis')), 'synthesis', plan); final = result.review; reviews.push(final); synthesis = result.reasons.length === 0; - if (synthesis || result.reasons.some(r => r === 'missing_required_context' || r === 'visual_evidence_unavailable' || r === 'output_truncated') || continuations >= MAX_CONTINUATIONS) { + if (synthesis || result.reasons.some(r => TERMINAL_REASONS.has(r)) || continuations >= MAX_CONTINUATIONS) { for (const reason of result.reasons) reasons.add(reason); break; @@ -223,6 +227,7 @@ export async function orchestrateReview(plan: ReviewPlan, evidence: (unit: Revie } if (!synthesis) reasons.add('synthesis_incomplete'); + if(coverage.length { const root = await mkdtemp(join(tmpdir(),'cb-context-')), socket = process.platform === 'win32' ? `\\\\.\\pipe\\cb-test-${crypto.randomUUID()}` : join(root,'context.sock'); const calls: unknown[] = []; - const close = await contextServer(socket, async (snapshot,path,operation) => { calls.push(operation?[snapshot,path,operation]:[snapshot,path]); return path === 'src/caller.ts'; }); + const close = await contextServer(socket, async (snapshot,path,operation) => {if(path==='missing.ts')throw new AppError('context_path_missing');if(path==='api.ts')throw new AppError('context_github_api_failed');if(path==='unsafe.ts')throw new Error('Bearer private-source-code');calls.push(operation?[snapshot,path,operation]:[snapshot,path]); return path === 'src/caller.ts'; }); const send = (body: string, path = '/file', method = 'POST') => new Promise<{ status: number; body: string }>((resolve,reject) => { const req = request({ socketPath:socket, path, method, headers:{'Content-Type':'application/json'} }, response => { let body = ''; response.on('data',chunk => { body += String(chunk); }); response.on('end',() => resolve({ status:response.statusCode!, body })); @@ -18,6 +19,9 @@ await test('context socket exposes only validated immutable file requests and ge assert.deepEqual(await send('{"snapshot":"head","path":"src/caller.ts"}'), {status:200,body:'{"available":true}'}); assert.deepEqual(await send('{"snapshot":"head","path":"optional.ts"}','/probe'),{status:200,body:'{"exists":false}'}); assert.deepEqual(await send('{"snapshot":"head","path":"src/caller.ts"}','/probe'),{status:200,body:'{"exists":true}'}); + assert.deepEqual(await send('{"snapshot":"head","path":"missing.ts"}'),{status:200,body:'{"available":false,"reason":"context_path_missing"}'}); + assert.deepEqual(await send('{"snapshot":"head","path":"api.ts"}'),{status:400,body:'{"error":"context_github_api_failed"}'}); + assert.deepEqual(await send('{"snapshot":"head","path":"unsafe.ts"}'),{status:400,body:'{"error":"context_lookup_unavailable"}'}); for (const body of ['{"snapshot":"other","path":"file"}','{"snapshot":"head","path":"file","url":"https://evil.invalid"}','{"snapshot":"head"}','sensitive fixture error','{"snapshot":"head","path":"file","required":false}','{"snapshot":"head","path":"file","operation":"probe"}']) { assert.deepEqual(await send(body),{status:400,body:'{"error":"context_lookup_unavailable"}'}); } @@ -25,6 +29,13 @@ await test('context socket exposes only validated immutable file requests and ge assert.equal((await send('{}','/url')).status,400); assert.equal((await send('{}','/file','GET')).status,400); assert.deepEqual(calls,[['head','src/caller.ts'],['head','optional.ts','probe'],['head','src/caller.ts','probe']]); } finally { await close(); await rm(root,{recursive:true,force:true}); } }); +await test('the socket bounds all requests, including cache hits and invalid input, and reports cancellation once',async()=>{ + const root=await mkdtemp(join(tmpdir(),'cb-context-cap-')),socket=process.platform==='win32'?`\\\\.\\pipe\\cb-cap-${crypto.randomUUID()}`:join(root,'context.sock'),aborted:unknown[]=[];let calls=0; + const close=await contextServer(socket,async()=>{calls++;return true;},reason=>aborted.push(reason)); + const send=(body='{"snapshot":"head","path":"cached.ts"}')=>new Promise((resolve,reject)=>{const req=request({socketPath:socket,path:'/file',method:'POST'},response=>{response.resume();response.on('end',()=>resolve(response.statusCode!));});req.on('error',reject);req.end(body);}); + try{for(let i=0;i<99;i++)assert.equal(await send(),200);assert.equal(await send('{}'),400);assert.equal(await send(),429);assert.equal(await send(),429);assert.equal(calls,99);assert.deepEqual(aborted,['context_request_limit']);} + finally{await close();await rm(root,{recursive:true,force:true});} +}); await test('context shutdown cancels and drains active lookups before temporary source cleanup', async () => { const root=await mkdtemp(join(tmpdir(),'cb-context-close-')), socket=process.platform==='win32'?`\\\\.\\pipe\\cb-close-${crypto.randomUUID()}`:join(root,'context.sock'); let begin!:()=>void, finish!:()=>void, drained=false; diff --git a/tests/context.test.ts b/tests/context.test.ts index c815a39..9137121 100644 --- a/tests/context.test.ts +++ b/tests/context.test.ts @@ -69,15 +69,15 @@ await test('bounded file index and on-demand lookups reach unchanged immutable f assert.ok(index.paths.includes('src/old.ts')); assert.ok(index.paths.length <= 5000); assert.equal(await lookup('head','src/old.ts'), true); assert.equal(await readFile(join(directory,'source/head/src/old.ts'),'utf8'), 'fixture context text'); - assert.equal(await lookup('base','src/missing.ts'), false); + await assert.rejects(lookup('base','src/missing.ts'),/context_path_missing/); for (const path of ['../secret','src/../../secret','/etc/passwd','src\\old.ts','https://evil.invalid']) await assert.rejects(lookup('head',path)); await assert.rejects(lookup('trusted' as 'head','AGENTS.md')); - const remaining = await Promise.allSettled(Array.from({ length: 25 }, () => lookup('head','src/old.ts'))); + const remaining = await Promise.allSettled(Array.from({ length: 101 }, () => lookup('head','src/old.ts'))); assert.ok(remaining.some(item => item.status === 'rejected')); }); assert.equal(result.complete,false); assert.equal(result.requiredContextComplete,false); assert.equal(requested.filter(path => path.includes('/git/blobs/')).length,1); - await fixture({ mode:'120000' }, async lookup => { assert.equal(await lookup('head','src/old.ts'),false); }); + await fixture({ mode:'120000' }, async lookup => { await assert.rejects(lookup('head','src/old.ts'),/context_special_object/); }); }); await test('direct local dependencies on base and head are essential, with TypeScript .js import resolution', async () => { const found = await fixture({ dependency: 'present' }); assert.equal(found.result.complete, true); assert.equal(found.result.files, 5); @@ -123,7 +123,7 @@ export const fixture = {generated,template};`; await test('initially complete context becomes incomplete after an essential on-demand miss, with a safe durable reason',async()=>{ const {result}=await fixture({},async(lookup,directory,manifest)=>{ assert.equal(manifest.complete,true);assert.equal(manifest.diffComplete,true); - assert.equal(await lookup('head','src/private-missing-name.ts'),false); + await assert.rejects(lookup('head','src/private-missing-name.ts'),/context_path_missing/); assert.equal(manifest.complete,false);assert.equal(manifest.requiredContextComplete,false);assert(manifest.failureCodes.includes('context_path_missing')); const persisted=JSON.parse(await readFile(join(directory,'manifest.json'),'utf8')); assert.equal(persisted.complete,false);assert.equal(persisted.requiredContextComplete,false);assert(!JSON.stringify(persisted.failureCodes).includes('private-missing-name')); @@ -131,14 +131,22 @@ await test('initially complete context becomes incomplete after an essential on- });assert.equal(result.complete,false); }); await test('exhausting on-demand lookup allowance cannot leave an approving context flag',async()=>{ - await fixture({},async(lookup,_directory,manifest)=>{for(let n=0;n<24;n++)assert.equal(await lookup('head','src/old.ts'),true);assert.equal(manifest.complete,true);await assert.rejects(lookup('head','src/old.ts'),/context_lookup_rejected/);assert.equal(manifest.complete,false);assert.equal(manifest.requiredContextComplete,false);assert(manifest.failureCodes.includes('context_lookup_rejected'));}); + await fixture({},async(lookup,_directory,manifest)=>{for(let n=0;n<100;n++)assert.equal(await lookup('head','src/old.ts'),true);assert.equal(manifest.complete,true);await assert.rejects(lookup('head','src/old.ts'),/context_request_limit/);assert.equal(manifest.complete,false);assert.equal(manifest.requiredContextComplete,false);assert(manifest.failureCodes.includes('context_request_limit'));assert.deepEqual(manifest.lookups,{requests:101,paths:1,cache_hits:99,last_failure:'context_request_limit'});}); +}); +await test('verified cached contents and repeated probes do not consume distinct repository lookups',async()=>{ + await fixture({candidate:true},async(lookup,_directory,manifest)=>{ + assert.equal(await lookup('head','candidate.ts','probe'),true);assert.equal(await lookup('head','candidate.ts'),true); + const calls=manifest.apiCalls;assert.deepEqual(await Promise.all(Array.from({length:30},()=>lookup('head','candidate.ts'))),Array(30).fill(true)); + assert.equal(manifest.apiCalls,calls);assert.equal(manifest.complete,true); + assert.deepEqual(manifest.lookups,{requests:32,paths:1,cache_hits:30}); + }); }); await test('optional probes verify absence without source reads or completeness loss; required misses stay sticky',async()=>{ await fixture({},async(lookup,directory,manifest)=>{ assert.equal(await lookup('head','src/speculative-name.ts','probe'),false);assert.equal(manifest.complete,true);assert.equal(manifest.requiredContextComplete,true);assert.equal(manifest.failureCodes.length,0); assert.equal(await lookup('head','src/old.ts','probe'),true);await assert.rejects(readFile(join(directory,'source/head/src/old.ts'))); - assert.equal(await lookup('head','src/speculative-name.ts'),false);assert.equal(manifest.complete,false);assert(manifest.failureCodes.includes('context_path_missing')); + await assert.rejects(lookup('head','src/speculative-name.ts'),/context_path_missing/);assert.equal(manifest.complete,false);assert(manifest.failureCodes.includes('context_path_missing')); assert.equal(await lookup('head','src/speculative-name.ts','probe'),false);assert.equal(await lookup('head','src/old.ts'),true);assert.equal(manifest.complete,false); assert.equal(JSON.parse(await readFile(join(directory,'manifest.json'),'utf8')).complete,false); }); @@ -152,14 +160,14 @@ await test('API failures, truncated and inconsistent trees are never converted i }); await test('mandatory content distinguishes missing paths, special objects, size limits, API errors and blob verification',async()=>{ const cases=[{options:{candidate:true,candidateMode:'120000'},code:'context_special_object'},{options:{candidate:true,candidateMode:'160000'},code:'context_special_object'},{options:{candidate:true,candidateMode:'100664'},code:'context_special_object'},{options:{candidate:true,candidateOversized:true},code:'context_file_size_limit'},{options:{candidate:true,candidateCorrupt:true},code:'blob_integrity_failure'},{options:{candidate:true,candidateApiStatus:404},code:'context_github_api_failed'},{options:{candidate:true,candidateApiStatus:429},code:'context_github_api_failed'}]; - for(const {options,code}of cases)await fixture(options,async(lookup,_directory,manifest)=>{assert.equal(manifest.complete,true);assert.equal(await lookup('head','candidate.ts','probe'),true,'Probe means existence only, never readable/verified contents');assert.equal(manifest.complete,true);assert.equal(await lookup('head','candidate.ts'),false);assert.equal(manifest.complete,false);assert(manifest.failureCodes.includes(code),JSON.stringify(manifest.failureCodes));}); + for(const {options,code}of cases)await fixture(options,async(lookup,_directory,manifest)=>{assert.equal(manifest.complete,true);assert.equal(await lookup('head','candidate.ts','probe'),true,'Probe means existence only, never readable/verified contents');assert.equal(manifest.complete,true);await assert.rejects(lookup('head','candidate.ts'),new RegExp(code));assert.equal(manifest.complete,false);assert(manifest.failureCodes.includes(code),JSON.stringify(manifest.failureCodes));}); }); await test('optional probes share the original lookup cap without invalidating coverage; required calls cannot bypass it',async()=>{ - await fixture({},async(lookup,directory,manifest)=>{for(let n=0;n<24;n++)assert.equal(await lookup('head','src/missing-'+n+'.ts','probe'),false);await assert.rejects(lookup('head','src/no-more.ts','probe'),/context_lookup_rejected/);assert.equal(manifest.complete,true);assert.equal(manifest.failureCodes.length,0);await assert.rejects(lookup('head','src/old.ts'),/context_lookup_rejected/);assert.equal(manifest.complete,false);assert(manifest.failureCodes.includes('context_lookup_rejected'));assert.equal(JSON.parse(await readFile(join(directory,'manifest.json'),'utf8')).complete,false);}); + await fixture({},async(lookup,directory,manifest)=>{for(let n=0;n<24;n++)assert.equal(await lookup('head','src/missing-'+n+'.ts','probe'),false);await assert.rejects(lookup('head','src/no-more.ts','probe'),/context_lookup_limit/);assert.equal(manifest.complete,true);assert.equal(manifest.failureCodes.length,0);assert.equal(await lookup('head','src/auth.ts'),true);await assert.rejects(lookup('head','src/old.ts'),/context_lookup_limit/);assert.equal(manifest.complete,false);assert(manifest.failureCodes.includes('context_lookup_limit'));assert.equal(JSON.parse(await readFile(join(directory,'manifest.json'),'utf8')).complete,false);}); }); await test('concurrent optional exploration and required reads serialize immutable state and never erase a required failure',async()=>{ const {requested}=await fixture({candidate:true},async(lookup,directory,manifest)=>{ const optional=await Promise.all([lookup('head','missing-a.ts','probe'),lookup('base','missing-b.ts','probe'),lookup('head','candidate.ts','probe')]);assert.deepEqual(optional,[false,false,true]);assert.equal(manifest.complete,true); - const mixed=await Promise.all([lookup('head','candidate.ts'),lookup('head','required-missing.ts'),lookup('base','missing-c.ts','probe'),lookup('head','candidate.ts')]);assert.deepEqual(mixed,[true,false,false,true]);assert.equal(manifest.complete,false);assert(manifest.failureCodes.includes('context_path_missing'));assert.equal(JSON.parse(await readFile(join(directory,'manifest.json'),'utf8')).complete,false); + const mixed=await Promise.allSettled([lookup('head','candidate.ts'),lookup('head','required-missing.ts'),lookup('base','missing-c.ts','probe'),lookup('head','candidate.ts')]);assert.deepEqual(mixed.map(item=>item.status==='fulfilled'?item.value:'rejected'),[true,'rejected',false,true]);assert.equal((mixed[1] as PromiseRejectedResult).reason.code,'context_path_missing');assert.equal(manifest.complete,false);assert(manifest.failureCodes.includes('context_path_missing'));assert.equal(JSON.parse(await readFile(join(directory,'manifest.json'),'utf8')).complete,false); });assert.equal(requested.filter(path=>path.includes('/git/blobs/')).length,2,'Concurrent content requests must reuse verified blobs'); }); diff --git a/tests/execution.test.ts b/tests/execution.test.ts index 9f3988c..2bcb2fa 100644 --- a/tests/execution.test.ts +++ b/tests/execution.test.ts @@ -20,6 +20,13 @@ import { decision } from '../src/core/review.js'; import { snapshot, clean } from './fixtures.js'; const sensitive = 'Bearer fixture-private-credential OAuth-code private-source prompt-provider-message'; +await test('primary timeouts retain bounded context completion without private diagnostics or invented HTTP status',()=>{ + const completion:ReviewCompletion={context_complete:false,diff_complete:true,required_context_complete:false,output_complete:false,output_result:'inconclusive',quarantined:false,output_truncated:false,context_failure_code:'context_lookup_limit',context_lookups:{requests:31,paths:24,cache_hits:6,last_failure:'context_lookup_limit'}}; + const report=diagnostic(new ExecutionError({component:'turn',code:'codex_turn_timeout',timeout_scope:'execution'}),'result',completion);assert.equal(report.execution!.timeout_scope,'execution');assert.equal(report.status,undefined);assert.deepEqual(report.completion,completion); + const unsafe={...completion,context_failure_code:sensitive,context_lookups:{requests:10,paths:1,cache_hits:9,last_failure:sensitive,path:sensitive}};const safe=diagnostic(new ExecutionError({component:'turn',code:'codex_turn_timeout'}),'result',unsafe);assert.equal(safe.completion!.context_failure_code,undefined);assert(!JSON.stringify(safe).includes(sensitive));assert.equal(safe.completion!.context_lookups!.last_failure,undefined); + assert.equal(diagnostic(new ExecutionError({component:'turn',code:'codex_turn_timeout'}),'result',{...completion,context_lookups:{requests:102,paths:25,cache_hits:101}}).completion!.context_lookups,undefined); + assert.equal(executionFailure({component:'provider',code:'codex_turn_timeout',timeout_scope:'execution'})!.timeout_scope,undefined); +}); function child(onWrite: (value: string) => void = () => {}) { const result = Object.assign(new EventEmitter(), { stdout: new PassThrough(), stderr: new PassThrough(), @@ -137,7 +144,7 @@ await test('actual review job records incomplete fallback, releases acquired lea const submissions: Record[] = [], operations: string[] = [], logs: unknown[][] = []; let rejectReporting = false, rejectReadTokenCleanup = false; let containerOutput: unknown = null, unavailableContext = false, runs = 0, onDemandMissing = false, probeOnly=false, partialExecutionFailure=false; - let contextSocket='',contextListener:http.Server|undefined; + let timeoutDuringContext=false;let contextSocket='',contextListener:http.Server|undefined; const realServer=http.createServer; const serverMock=mock.method(http,'createServer',(...args:Parameters)=>{const server=realServer(...args);contextListener=server;server.once('listening',()=>{const address=server.address();if(typeof address==='string')contextSocket=address;});return server;}); const blob=Buffer.from('context\nold\n'),blobSha=createHash('sha1').update('blob '+blob.length+'\0').update(blob).digest('hex'); @@ -152,12 +159,12 @@ await test('actual review job records incomplete fallback, releases acquired lea if (args[0] === 'logs' && args.includes('--follow')) { process.stdout.write('codexbot-proxy-ready\n'); return; } if (args[0] === 'run') { runs++; - if(onDemandMissing){await new Promise((resolve,reject)=>{const req=http.request({socketPath:contextSocket,path:probeOnly?'/probe':'/file',method:'POST'},res=>{let body='';res.on('data',x=>{body+=String(x);});res.on('end',()=>{try{assert.deepEqual(JSON.parse(body),probeOnly?{exists:false}:{available:false});resolve();}catch(error){reject(error);}});});req.on('error',reject);req.end(JSON.stringify({snapshot:'head',path:'private-missing-name.ts'}));});} + if(onDemandMissing){await new Promise((resolve,reject)=>{const req=http.request({socketPath:contextSocket,path:probeOnly?'/probe':'/file',method:'POST'},res=>{let body='';res.on('data',x=>{body+=String(x);});res.on('end',()=>{try{assert.deepEqual(JSON.parse(body),probeOnly?{exists:false}:{available:false,reason:'context_path_missing'});resolve();}catch(error){reject(error);}});});req.on('error',reject);req.end(JSON.stringify({snapshot:'head',path:'private-missing-name.ts'}));});} if (containerOutput !== null) { const mount=args.find(arg=>arg.startsWith('type=bind,src=')&&arg.endsWith(',dst=/workspace,readonly'))!; const directory=mount.slice('type=bind,src='.length,-',dst=/workspace,readonly'.length),plan=JSON.parse(await readFile(join(directory,'review-plan.json'),'utf8')) as ReviewPlan; const review=containerOutput as {complete:boolean;result:string};const complete=review.complete&&review.result!=='inconclusive'; - process.stdout.write(JSON.stringify({review:containerOutput,coverage:plan.units.map(u=>({id:u.id,sha256:u.sha256,complete,attempts:1,reasons:complete?[]:['model_incomplete']})),synthesis:complete,turns:plan.units.length+(plan.units.length===1?0:1),reasons:complete?[]:['model_incomplete'],failure:partialExecutionFailure?{component:'turn',code:'codex_turn_failed'}:null})); + process.stdout.write(JSON.stringify({review:containerOutput,coverage:plan.units.map(u=>({id:u.id,sha256:u.sha256,complete,attempts:1,reasons:complete?[]:['model_incomplete']})),synthesis:complete,turns:plan.units.length+(plan.units.length===1?0:1),reasons:complete?[]:['model_incomplete'],failure:partialExecutionFailure?{component:'turn',code:timeoutDuringContext?'codex_turn_timeout':'codex_turn_failed',...(timeoutDuringContext?{timeout_scope:'execution'}:{})}:null})); } else process.stderr.write(executionEnvelope({ component: 'turn', code: 'codex_turn_failed' }) + '\n'); } @@ -211,6 +218,10 @@ await test('actual review job records incomplete fallback, releases acquired lea containerOutput={...clean(),result:'findings',findings:[{severity:'high',confidence:0.99,path:'auth.ts',line:2,side:'RIGHT',title:'Synthetic race condition',explanation:'A stale value can overwrite a newer result.',remediation:'Guard the update.'}]}; await assert.rejects(reviewJob(),error=>{assert(error instanceof ReviewCompletionError);assert.equal(error.completion.context_complete,false);assert.equal(error.completion.required_context_complete,false);assert.equal(error.completion.diff_complete,true);assert.equal(error.completion.output_complete,true);assert.equal(error.completion.output_result,'findings');assert.equal(error.completion.context_failure_code,'context_path_missing');return true;}); assert.deepEqual(withoutStatus(submissions.at(-1)!.output),containerOutput);assert.equal(submissions.at(-1)!.context_complete,false);assert.equal(decision(submissions.at(-1)!.output as ReturnType,request.config,false),'COMMENT');assert.equal((submissions.at(-1)!.failure as {stage:string}).stage,'result');assert(!JSON.stringify(logs).includes('private-missing-name'));assert.equal(operations.at(-1),'release'); + timeoutDuringContext=true;partialExecutionFailure=true;containerOutput={...(containerOutput as object),complete:false}; + await assert.rejects(reviewJob(),error=>{assert(error instanceof ExecutionError);assert.equal(error.detail.code,'codex_turn_timeout');assert.equal(error.detail.timeout_scope,'execution');return true;}); + const combined=submissions.at(-1)!.failure as ReturnType;assert.equal(combined.execution!.code,'codex_turn_timeout');assert.equal(combined.completion!.context_failure_code,'context_path_missing');assert.equal(combined.completion!.context_complete,false);assert.deepEqual(combined.completion!.context_lookups,{requests:1,paths:1,cache_hits:0,last_failure:'context_path_missing'});assert.equal((submissions.at(-1)!.output as {findings:unknown[]}).findings.length,1);assert(!JSON.stringify(logs).includes('private-missing-name')); + timeoutDuringContext=false;partialExecutionFailure=false; onDemandMissing=false;containerOutput=clean(); rejectReadTokenCleanup = true; await assert.rejects(reviewJob(), /read_token_cleanup_failed/); assert.equal(operations.at(-1), 'release'); @@ -220,7 +231,7 @@ await test('actual review job records incomplete fallback, releases acquired lea assert.equal(submissions.at(-1)!.context_complete, false); assert.equal((submissions.at(-1)!.output as { result: string }).result, 'inconclusive'); assert.equal((submissions.at(-1)!.failure as { code: string }).code, 'review_incomplete'); - assert.deepEqual((submissions.at(-1)!.failure as {completion:unknown}).completion,{context_complete:true,diff_complete:true,required_context_complete:true,output_complete:false,output_result:'inconclusive',quarantined:false,output_truncated:false,orchestration:{planned_units:1,completed_units:0,synthesis_complete:false,unfinished_reasons:['model_incomplete']}}); + assert.deepEqual((submissions.at(-1)!.failure as {completion:unknown}).completion,{context_complete:true,diff_complete:true,required_context_complete:true,output_complete:false,output_result:'inconclusive',quarantined:false,output_truncated:false,context_lookups:{requests:0,paths:0,cache_hits:0},orchestration:{planned_units:1,completed_units:0,synthesis_complete:false,unfinished_reasons:['model_incomplete']}}); containerOutput = { ...clean(), complete: false }; // A clean label cannot override incomplete analysis. await assert.rejects(reviewJob(), /review_output_invalid/); assert.equal((submissions.at(-1)!.output as { result: string }).result, 'inconclusive'); diff --git a/tests/inspection-check.mjs b/tests/inspection-check.mjs index 074ce20..8238ba2 100644 --- a/tests/inspection-check.mjs +++ b/tests/inspection-check.mjs @@ -28,12 +28,12 @@ function events(items,inputTokens=1) { ].map((value, sequence_number) => `data: ${JSON.stringify({ ...value, sequence_number })}\n\n`).join(''); } // Scripted Responses only; real app-server policy, shell execution, mounts and output validation. -for (const scenario of ['untrusted', 'findings', 'clean', 'incomplete','vision','vision-invalid','schema-invalid','json-invalid','large','continuation','synthesis-incomplete','evidence-tampered','global-budget','cross-group','provider-schema-rejected','reasons-duplicate','synthesis-no-reread','provided-group-evidence','large-compaction']) { +for (const scenario of ['untrusted', 'findings', 'clean', 'incomplete','vision','vision-invalid','schema-invalid','json-invalid','large','continuation','synthesis-incomplete','evidence-tampered','global-budget','cross-group','provider-schema-rejected','reasons-duplicate','synthesis-no-reread','provided-group-evidence','large-compaction','missing-required-context']) { const id = randomUUID(), proxy = `codexbot-inspection-proxy-${id}`, review = `codexbot-inspection-review-${id}`, volume = `codexbot-inspection-${id}`; const root = await mkdtemp(resolve('.local/inspection-')), directory = join(root, 'context'), head = 'a'.repeat(40); const config = globalThis.structuredClone(DEFAULT_CONFIG), request = { id, head, base: 'b'.repeat(40), config, command: { merge: false, effort: null, instructions: '' } }; const fixture = await writeInspectionFixture(directory, request, scenario !== 'clean'); - if(['cross-group','synthesis-incomplete','global-budget','synthesis-no-reread','provided-group-evidence'].includes(scenario)){ + if(['cross-group','synthesis-incomplete','global-budget','synthesis-no-reread','provided-group-evidence','missing-required-context'].includes(scenario)){ for(let i=1;i<=15;i++){const filename='src/padding-'+String(i).padStart(2,'0')+'.js';fixture.files.push({filename,status:'added',patch:'@@ -0,0 +1 @@\n+export const padding = true;'});await writeFile(join(directory,'source/head',filename),'export const padding = true;');} await mkdir(join(directory,'source/head/z-consumer'),{recursive:true}); const filename='z-consumer/handler.js',consumer="import { canAccess } from '../src/access.js';\nexport const serve = (role, record) => canAccess(role) ? record : null;\n"; @@ -58,7 +58,7 @@ for (const scenario of ['untrusted', 'findings', 'clean', 'incomplete','vision', const plan=await buildReviewPlan(directory,head,request.base,manifest.binaryFiles??[]); if(scenario==='evidence-tampered')await writeFile(join(directory,'source/head/src/access.js'),'tampered fixture'); const evidence = { diff: hash(JSON.stringify(fixture.files)), source: hash(fixture.source), base: hash(fixture.base), instructions: hash(JSON.stringify(fixture.instructions)), caller: hash(fixture.caller), hostile: hash(fixture.injection) }; - const result = { head_sha: head, complete: scenario !== 'incomplete', result: scenario === 'clean' ? 'clean' : 'findings', + const result = { head_sha: head, complete: !['incomplete','missing-required-context'].includes(scenario), result: scenario === 'clean' ? 'clean' : 'findings', summary: 'Scripted protocol fixture after verified file inspection; not a live AI review.', findings: scenario === 'clean' ? [] : [{ severity: 'high', confidence: 0.99, path: 'src/access.js', line: 3, side: 'RIGHT', title: 'Reject unauthorized roles', explanation: 'The nonempty owner string makes the authorization result truthy for every other role.', remediation: 'Compare role against owner explicitly.' }] }; @@ -100,9 +100,9 @@ for (const scenario of ['untrusted', 'findings', 'clean', 'incomplete','vision', console.log('fixture_inspection_verified'); const value=${JSON.stringify(scenario==='json-invalid'?'not-json':JSON.parse(encodedResult))};let finished=value; if(${JSON.stringify(scenario)}!=='json-invalid'){ - const incomplete=(${JSON.stringify(['continuation','large','large-compaction'])}.includes(${JSON.stringify(scenario)})&&unitId==='group-00'&&round===2)||(${JSON.stringify(scenario)}==='synthesis-incomplete'&&unitId==='synthesis')||${JSON.stringify(scenario)}==='incomplete'; + const incomplete=(${JSON.stringify(['continuation','large','large-compaction'])}.includes(${JSON.stringify(scenario)})&&unitId==='group-00'&&round===2)||(${JSON.stringify(scenario)}==='synthesis-incomplete'&&unitId==='synthesis')||['incomplete','missing-required-context'].includes(${JSON.stringify(scenario)}); if(incomplete&&value&&value.head_sha)value.complete=false;if(${JSON.stringify(scenario)}==='cross-group'&&unitId!=='synthesis'){value.findings=[];value.result='clean';} - finished=JSON.stringify({unit_id:unitId,review:value,unfinished_reasons:${JSON.stringify(scenario)}==='reasons-duplicate'?['unresolved_analysis','unresolved_analysis']:incomplete?['unreviewed_changes']:[]}); + finished=JSON.stringify({unit_id:unitId,review:value,unfinished_reasons:${JSON.stringify(scenario)}==='reasons-duplicate'?['unresolved_analysis','unresolved_analysis']:${JSON.stringify(scenario)}==='missing-required-context'?['missing_required_context']:incomplete?['unreviewed_changes']:[]}); } return new Response(events([{id:'msg_fixture_'+calls,type:'message',role:'assistant',status:'completed',content:[{type:'output_text',text:finished,annotations:[]}]}],${JSON.stringify(scenario)}==='large-compaction'&&unitId==='group-00'&&round===4?60000:1),{headers:{'content-type':'text/event-stream'}}); };const finite=globalThis.fetch;globalThis.fetch=async(url,init)=>{const value=await finite(url,init);if(calls%2!==0||!${JSON.stringify(['clean','incomplete'])}.includes(${JSON.stringify(scenario)}))return value;const bytes=new Uint8Array(await value.arrayBuffer());return new Response(new ReadableStream({start(controller){controller.enqueue(bytes);init.signal.addEventListener('abort',()=>controller.error(init.signal.reason),{once:true});}}),{status:value.status,headers:value.headers});};createInferenceServer(token,{minimumImages:${manifest.binaryFiles?.length??0},maxRequests:${scenario==='global-budget'?3:100}},detail=>process.stderr.write(executionEnvelope(detail)+'\\n'),value=>process.stderr.write(proxyRequestEnvelope(value)+'\\n')).listen('/run/proxy/inference.sock',()=>console.log('ready'));`; @@ -136,8 +136,8 @@ for (const scenario of ['untrusted', 'findings', 'clean', 'incomplete','vision', } else { assert.equal(run.status, 0, 'Actual Codex inspection failed; raw logs withheld'); assert.equal(inspected, true); - const execution=validateExecution(JSON.parse(run.stdout),plan);assert.equal(execution.synthesis,!['incomplete','synthesis-incomplete','global-budget'].includes(scenario));assert.equal(execution.coverage.length,plan.units.length);const output = validateReview(execution.review, head, fixture.files); assert.deepEqual(output, ['synthesis-incomplete','global-budget'].includes(scenario)?{...result,complete:false}:result);if(scenario==='cross-group'){assert.equal(execution.coverage.length,2);assert.equal(execution.turns,3);}if(scenario.startsWith('large')){assert.equal(plan.files.length,93);assert.equal(plan.binaries.length,4);assert.equal(execution.coverage.length,6);assert.equal(new Set(plan.binaries.flatMap(b=>b.snapshots.map(s=>s.sha))).size,4);assert.equal(execution.coverage[0].attempts,2);assert.equal(execution.turns,8);assert.equal(proxyFailure.request?.request_index,scenario==='large-compaction'?17:16);if(scenario==='large-compaction')assert.equal((logs.stdout??'').split('\n').filter(line=>line==='fixture_compaction_verified').length,1);}if(scenario==='continuation'){assert.equal(execution.coverage[0].attempts,2);assert.equal(execution.turns,2);}if(scenario==='global-budget'){assert.equal(execution.failure.code,'codex_turn_failed');assert.equal(proxyFailure.last.code,'proxy_request_limit');} - assert.equal(decision(output, config, true), scenario === 'clean' ? 'APPROVE' : ['incomplete','synthesis-incomplete','global-budget'].includes(scenario)?'COMMENT':'REQUEST_CHANGES'); + const execution=validateExecution(JSON.parse(run.stdout),plan);assert.equal(execution.synthesis,!['incomplete','synthesis-incomplete','global-budget','missing-required-context'].includes(scenario));assert.equal(execution.coverage.length,scenario==='missing-required-context'?1:plan.units.length);if(scenario==='missing-required-context'){assert.equal(execution.turns,1);assert.equal(execution.review.findings.length,1);assert(execution.reasons.includes('missing_required_context'));assert.equal(proxyFailure.request?.request_index,2);}const output = validateReview(execution.review, head, fixture.files); assert.deepEqual(output, ['synthesis-incomplete','global-budget'].includes(scenario)?{...result,complete:false}:result);if(scenario==='cross-group'){assert.equal(execution.coverage.length,2);assert.equal(execution.turns,3);}if(scenario.startsWith('large')){assert.equal(plan.files.length,93);assert.equal(plan.binaries.length,4);assert.equal(execution.coverage.length,6);assert.equal(new Set(plan.binaries.flatMap(b=>b.snapshots.map(s=>s.sha))).size,4);assert.equal(execution.coverage[0].attempts,2);assert.equal(execution.turns,8);assert.equal(proxyFailure.request?.request_index,scenario==='large-compaction'?17:16);if(scenario==='large-compaction')assert.equal((logs.stdout??'').split('\n').filter(line=>line==='fixture_compaction_verified').length,1);}if(scenario==='continuation'){assert.equal(execution.coverage[0].attempts,2);assert.equal(execution.turns,2);}if(scenario==='global-budget'){assert.equal(execution.failure.code,'codex_turn_failed');assert.equal(proxyFailure.last.code,'proxy_request_limit');} + assert.equal(decision(output, config, true), scenario === 'clean' ? 'APPROVE' : ['incomplete','synthesis-incomplete','global-budget','missing-required-context'].includes(scenario)?'COMMENT':'REQUEST_CHANGES'); } if(['clean','incomplete'].includes(scenario)){assert.equal(proxyFailure.request?.request_index,scenario==='clean'?2:6);assert.equal(proxyFailure.request.primary,null);assert.equal(proxyFailure.request.cleanup,null);assert.equal(proxyFailure.request.terminal_event,'completed');assert.equal(proxyFailure.last,null);} if(['synthesis-no-reread','provided-group-evidence'].includes(scenario)){const execution=validateExecution(JSON.parse(run.stdout),plan);assert.equal(execution.coverage.length,2);assert.equal(execution.synthesis,true);assert.equal(execution.turns,3);assert.equal(execution.review.complete,true);assert.equal(proxyFailure.request?.request_index,scenario==='synthesis-no-reread'?5:4);} diff --git a/tests/orchestration.test.ts b/tests/orchestration.test.ts index 9613c09..de472f1 100644 --- a/tests/orchestration.test.ts +++ b/tests/orchestration.test.ts @@ -106,13 +106,27 @@ await test('a model cannot waive its missing essential or visual evidence by cla const p = plan(2); let calls = 0; const result = await orchestrateReview(p, async () => '', async (id) => { calls++; return id === 'synthesis' ? response(id) : { ...response(id, false, [finding]), unfinished_reasons: [reason] }; }); - assert.equal(calls, 3); + assert.equal(calls, 1); + assert.equal(result.synthesis,false); + assert.deepEqual(result.review.findings,[finding]); assert.equal(result.review.complete, false); assert.equal(result.coverage[0]!.complete, false); assert(result.reasons.includes(reason as typeof result.reasons[number])); validateExecution(result, p); } }); +await test('unavailable essential evidence stops further groups and synthesis while preserving validated findings',async()=>{ + const p=plan(6),calls:string[]=[];const result=await orchestrateReview(p,async()=>'',async id=>{calls.push(id);return{...response(id,false,[finding]),unfinished_reasons:['missing_required_context']};}); + assert.deepEqual(calls,['group-00']);assert.equal(result.synthesis,false);assert.equal(result.review.complete,false);assert.deepEqual(result.review.findings,[finding]);assert(result.reasons.includes('unreviewed_changes'));assert.equal(decision(result.review,DEFAULT_CONFIG,true),'COMMENT');validateExecution(result,p); +}); +await test('modified-file evidence keeps the complete diff and verified base without retransmitting the full old source',async()=>{ + const directory=await mkdtemp(join(tmpdir(),'cb-lean-evidence-'));try{ + for(const side of['base','head']){await mkdir(join(directory,'source',side,'src'),{recursive:true});await writeFile(join(directory,'source',side,'src','auth.ts'),side==='base'?'old source':'new source');} + await writeFile(join(directory,'diff.json'),JSON.stringify([{filename:'src/auth.ts',status:'modified',patch:'@@ -1 +1 @@\n-old source\n+new source'}]));await writeFile(join(directory,'trusted-instructions.json'),'[]'); + const p=await buildReviewPlan(directory,HEAD,BASE,[]),data=JSON.parse(await unitEvidence(directory,p.units[0]!));assert.equal(data.evidence.length,2);assert.equal(data.evidence[0].text,undefined);assert.equal(data.evidence[1].text,'new source');assert(data.changes[0].patch.includes('-old source')); + await writeFile(join(directory,'source/base/src/auth.ts'),'tampered');await assert.rejects(unitEvidence(directory,p.units[0]!),/review_plan_unverifiable/); + }finally{await rm(directory,{recursive:true,force:true});} +}); await test('malformed, wrongly scoped, unsafe and fabricated unit outputs never satisfy coverage', async () => { for (const bad of [{ unexpected: true }, response('group-99'), { ...response('group-00'), review: { ...clean(), head_sha: BASE } }, { ...response('group-00'), review: { ...clean(), summary: 'Bearer sk-proj-' + 'x'.repeat(45) } }, { ...response('group-00'), all_groups_inspected: true }]) await assert.rejects(orchestrateReview(plan(1), async () => '', async () => bad));