Skip to content

Latest commit

 

History

History
78 lines (41 loc) · 18.8 KB

adaptive-context.md

File metadata and controls

78 lines (41 loc) · 18.8 KB

Bounded context exploration and review presentation

Bounded review orchestration

After collection passes, the trusted Actions host writes a private review-plan.json and immutable review-units/ inputs. Changed paths are ordered with changed relative JS/TS dependencies nearby, then grouped within 16 files / 2 MB / 1,500 changed text lines per input, at most 32 groups. A single larger changed file stays intact within the file/byte bounds. Inputs contain the actual textual patches and complete current/deleted changed-file text and hash receipts for every collected base/head file; modified old source remains read-only at its verified path instead of being retransmitted, or SHA-bound verified PNG metadata, plus applicable trusted instructions, the whole-PR path overview and an inventory of shared collected source. Every group keeps access to all original source, callers, dependencies, the file index and the same bounded context socket; grouping never removes cross-file context. Trusted instruction input is bounded to 1 MB; unrepresentable essential evidence fails before inference. Original Git/snapshot and binary verification remain required.

The orchestrator is immutable application code, not a model-provided plan. It checks hashes of host-prepared inputs and referenced text/image files before delivery. The actual pinned app-server processes groups sequentially in one ephemeral thread, with one required successful inspection command per turn and matching thread/turn identity. It gives every group an initial pass before spending at most two targeted continuations across the whole review, including synthesis. Per-file checkpoints are constrained to assigned paths and independently hash-verified evidence; unfinished paths cannot be waived. Continuations receive pending paths, the previous validated review and the immutable evidence receipt rather than another full evidence payload. Shared manifest/user context is sent once to the same thread. Optional continuations stop when the synthesis reserve (up to three minutes) is reached; this does not guarantee a model finishes within the deadline. Explicit missing mandatory or visual evidence cannot be cleared by a later model claim. For multiple groups, a separate final synthesis reconciles all unit results and cross-group behavior; it cannot waive unfinished units. A sole group instead receives an explicit final whole-PR/cross-file check in its own response, avoiding a redundant turn. Its verified evidence, inspection and completeness gates still apply. No model field such as “all groups inspected” controls scheduling or coverage.

One credential proxy and one engine timer cover all turns: the existing 100 inference-request ceiling and selected execution timeout (default 30 minutes) is not multiplied per group. The context socket allows 24 distinct uncached snapshot/path lookups and 100 total requests; the 768 GitHub-call, text/image-byte and model/effort limits remain in force. Protocol receipts and scope checks establish supplied evidence and completed scheduled responses, not proof of a model's semantic reasoning. More turns can change actual consumption within those ceilings; no real-provider cost or quality improvement is claimed from synthetic tests.

Valid incomplete results return to the trusted runner as data. It independently checks every expected unit ID/hash, bounded attempts and synthesis status against its original plan, drains context lookups, and rechecks host and model completeness. Incomplete work fails the Actions review job and may publish only COMMENT, preserving validated summary and findings. Invalid or quarantined output remains rejected. Later execution failures preserve earlier safe findings and the primary execution diagnosis; generic workflow failure does not overwrite it. Findings are deduplicated by original path/side/line and normalized title; more than 40 findings or other output loss is explicitly incomplete. The model cannot delete earlier valid findings during synthesis.

Diagnostics and incomplete COMMENT reports expose only bounded group counts, synthesis status and fixed unfinished-reason enums. The host overwrites that advisory status from its validated plan/results; model output cannot supply it or use it to authorize approval. They never expose prompts, source, arbitrary reasons or provider bodies. Scripted Docker acceptance exercises all scheduled groups for 93 changes / four distinct valid PNGs, read-only shared callers, continuation, incomplete synthesis, altered evidence and one global proxy allowance. This is not real-model quality acceptance. Any future rollout requires normal source review, CI and coordinated immutable Worker/caller pins; this change does not deploy or change existing pins.

The existing immutable collection and isolated Codex execution remain in place. Changed base/head files and trusted default-branch instructions are still required evidence. Direct JS/TS dependencies, simple Python relative imports and Rust sibling modules are resolved without executing project code. Python/Go/Java test naming heuristics supplement the existing nearby tests/manifests.

The collector writes /workspace/file-index.json: at most 12 nonrecursive directory walks and 5,000 regular-file paths. A partial supplementary index does not invalidate a small PR. It does not imply exhaustive semantic coverage. No full recursive tree or unrelated blob download is required.

A separate trusted host Unix socket at /run/proxy/context.sock accepts POST /file for required contents or /probe for optional existence, with JSON containing snapshot: base|head and a relative path. Successful retrieval materializes a verified file under the existing read-only reviewer mount. Paths, immutable commit/tree/blob identity, object modes, UTF-8, size and Git blob SHA are validated by the same collector. It cannot request another repository/ref, fetch URLs, follow symlinks or execute commands. There are at most 24 distinct uncached snapshot/path lookups and 100 total socket requests, including cache hits and invalid input, sharing the 768-call/16-MB-source/512-KB-file limits and SHA cache. Missing /file evidence makes the result incomplete and fails the review job. A /probe returns only {exists: true|false} from verified immutable tree metadata and never materializes source or satisfies content evidence. Verified absence is neutral; API/tree verification failures are errors and remain fail-closed, not false absence.

The GitHub read token remains on the trusted host through bounded exploration, rather than being revoked before inference. It is never mounted into the reviewer or credential proxy. On completion/failure, the host aborts/drains outstanding lookups, closes the context service, revokes the token and removes temporary source. Token cleanup failure fails the review job; it does not authorize approval. The separate publisher still uses the job conclusion and exact commit to decide whether a safe COMMENT is appropriate.

The concise permanent prompt asks Codex to understand changed behavior and trace important unchanged callers, data flows, tests, security and compatibility. Trusted project guidance cannot override safety/output rules; task-specific comment context only steers attention. Untrusted repository instructions cannot change credentials, authorize publication/merge or grant tools. Existing client-side approval requests stay denied and Docker stays network=none, nonroot, read-only, capability-free and credential-free.

Human-readable summaries distinguish clean completion, advisory findings, changes requested and incomplete evidence. They omit routine “merge not requested”/mechanical decision noise. Actual native APPROVE/COMMENT/REQUEST_CHANGES decisions, confidence policy, DLP, changed-line validation, publication fingerprints and merge restrictions remain independent of that wording.

Validation: immutable lookup/path/type/budget tests, small PR in a 1,200-file repository, native Codex/Docker reads of exact diff/base/head/trusted instructions/unchanged caller/hostile-text hashes, denied writes/network/credentials and full severity/confidence/completeness summary checks. Scripted Responses exercise real tools and sandbox behavior, not real AI recognition or prompt-quality improvement. Live old/revised-prompt comparison on identical clean/buggy/security/cross-file fixtures requires separate owner authorization; no paid or ChatGPT inference is part of CI.

Runner changes require coordinated Worker/caller pins to the actual merged source revision. Never guess a merge SHA or use a branch/tag. Close the production merge gate for the coordinated upgrade, then restore it only after verifying matching Worker/caller pins.

Prompt evaluation fixtures (opt-in live, no default inference)

tests/review-evaluation-fixtures.mjs defines clean authorization, introduced arithmetic defect, introduced authorization bypass and a changed cents conversion with an unchanged checkout caller. Fixtures include untrusted instruction text. Offline checks validate diffs/expected changed locations and distinguish clean/findings/incomplete outcomes; they do not score an LLM's semantic quality.

The existing guarded provider acceptance supports --case clean|buggy|security|multi-file, --label baseline|revised and optional --report-file in an owner-protected local directory. It records only completeness, finding count, expected-line match, label/model/effort and the resolved immutable local image ID. It never stores raw model text or credentials. Failure still exits nonzero. Build the baseline and revised images from their reviewed source revisions; hold Codex version/model/effort/fixture constant and compare all four cases. Use the credential-file, deployment authorization, explicit --allow-live-oauth or --allow-paid-api and 1–3 request budget controls already documented in acceptance.md. API output remains capped. Browser-managed sessions must instead use the authorized repository workflow; do not export them to a credential file.

No live A/B run has been authorized or executed in this engineering pass. Keep quality/entitlement conclusions pending until owner-authorized acceptance; local fixture scores and scripted Docker responses are not model-quality evidence.

Large PRs and binary coverage

The exact base-to-head aggregate diff covers all PR commits together. The collector does not promise per-intermediate-commit analysis. Up to 200 changed files, 6 MB of serialized diff, 16 MB of materialized text, 512 KB per text file, 768 GitHub calls and 120 seconds of initial collection are supported. Individual API calls time out after 15 seconds. Commit/tree/blob caches are local to this tenant and review; identical blobs are fetched once. Changed files and trusted AGENTS.md precede optional context. On-demand requests share the remaining byte/call budget. Precise failure codes and non-approving summaries explain exhausted budgets.

Missing patches are never assumed binary merely from extensions. Supported PNGs require exact immutable tree paths, regular non-executable blobs, verified Git blob SHA/size and PNG signature/chunk framing through IEND without trailing payload. Both base/head versions are verified as applicable to addition, modification, deletion and rename. Binary reads are bounded to 2 MB/file and 8 MB total unique content. The trusted collector never decompresses or renders images, and binary bytes are not installed as UTF-8 source. It materializes verified base/head PNGs separately under /workspace/media. The engine supplies official App Server localImage inputs with immutable path/side labels, deduplicated by blob SHA (at most 16 unique images). The selected model must advertise image input. The credential proxy requires the expected number of actual base64 image inputs on each inference request and denies remote image URLs, uploads, audio/video and generic file inputs. Missing or rejected images cannot produce an approval. Image text is untrusted data, never instructions. Unknown formats (including audio, executables, archives and extension spoofing), missing/truncated text and unverifiable essential files remain fail-closed.

The publisher independently verifies PNG evidence within 32 additional Git calls. A complete text-and-image analysis uses the ordinary confidence/verdict policy; unavailable visual evidence remains incomplete and cannot approve or merge. Image findings use line: null and appear in the overall review, since GitHub cannot anchor inline comments to image pixels. Text findings still require actual changed lines.

The pinned Codex 0.169.0 App Server supports localImage; its current adapter has no general local-file input. Official SIWC limitations allow model-supported images/files on Responses, but exclude audio/video input, transcription and Files API uploads. This implementation supports verified PNG images, not every binary format. It does not execute, unpack, transcribe or silently skip EXE/archive/audio/PDF files. Those changes require a supported explicit inspection path before approval. No private ChatGPT endpoints or paid fallback are used.

The Docker acceptance uses the actual pinned Codex binary to prove PNG conversion, transport, tool inspection and isolation against scripted Responses. It is not proof of real-model visual understanding; no new real-provider inference is part of this hotfix validation.

Final completion is checked again by the trusted host after the isolated engine returns and the context service is closed/drained. A missing file requested through /file remains essential missing evidence even if the model returns complete findings or has inspected every attached PNG. The host preserves validated findings in the non-approving submission, fails the review job and logs only completion booleans/result enum plus an approved context failure code. On-demand failures enter required coverage diagnostics rather than being labeled supplementary omissions. Paths, model text and credentials are not diagnostic fields. Neither a later successful lookup nor a complete model result clears a previous essential-evidence failure.

Optional discovery never uses a model-supplied required/optional flag. The existing /file endpoint stays mandatory and rejects extra fields such as required: false; the separate /probe operation supplies existence only, not readable or verified contents. A probe can report an object exists even when a later content read rejects its Git type, size or blob hash. A successful probe cannot waive initial changed-file/instruction/dependency requirements or clear a previous required failure. Both operations share 24 distinct uncached snapshot/path lookups and 100 total requests; cached verified contents remain reusable after the distinct-path allowance is consumed. The original GitHub API and byte limits are unchanged. Exhausted optional probes fail without invalidating otherwise complete evidence; a denied required fetch remains fatal. There is no extra inference pass or model/budget change. Host checks do not claim to prove exhaustive model reasoning about every possible dependency.

Runtime envelope and image retention

For G planned groups, successful baseline scheduling uses T = G + (G > 1 ? 1 : 0) turns. At most two targeted continuations are added globally. In native scripted inspection-then-report acceptance, each turn uses two provider requests. That observed minimum is not a prediction of real-model tool iterations, latency or tokens: all actual calls and retries still share the 100-request limit and selected execution timeout.

Shape (byte limits can create more groups) Groups Baseline turns Scripted requests Turns / scripted requests with two continuations
One bounded group 1 1 2 3 / 6
Medium example 3 4 8 6 / 12
93-file/four-image fixture 6 7 14 9 / 18
Maximum plan 32 33 66 35 / 70

The maximum plan leaves only 30 requests beyond that two-step/35-turn pattern and an arithmetic average of roughly 51 seconds per turn within the default 30 minutes, before startup overhead. There is no guaranteed large-PR completion or wall-clock/token cost estimate. Real models may need more inspection, compaction or reasoning; exhaustion produces a failed, non-approving partial review. Use representative acceptance; a longer configured timeout is not evidence of improved analysis. A controlled rollout should first observe representative advisory/no-merge reviews and split work that does not fit the envelope.

The native acceptance now creates four different PNG pixel patterns and verifies four distinct Git blob identities and four distinct encoded image inputs on every request across groups, continuation and final synthesis (16 requests in the large/one-continuation fixture). The first-turn-only attachment behavior is supported by the tested pinned app-server history; image loss was not reproduced. The proxy additionally binds fingerprints of the first trusted required visual inputs and rejects subsequent loss, repetition-as-replacement or substitution. Distinct Git images can have identical native encodings (for example after lossy conversion); the guard retains the minimum attachment count and the first trusted fingerprint set rather than inventing a one-to-one Git-to-encoding identity. Fingerprints and image bytes never enter public diagnostics. No images are blindly reattached per turn, which would inflate history. Native context compaction/re-encoding behavior at real large input sizes remains unmeasured; loss of required evidence fails closed rather than authorizing approval.

Context reuse and terminal failures

Read already supplied source directly before using the context socket. A probe and subsequent content request for the same immutable snapshot/path consume one distinct-path slot. Repeated verified content requests use the existing mount and hash-verified cache, never another repository, ref or credential. Total requests stay separately bounded, including invalid bodies; crossing that bound cancels context retrieval and makes the host result non-approving. New essential paths cannot bypass the distinct-path or API/byte/type/integrity limits.

Missing required files return a fixed context_path_missing reason; infrastructure, integrity, object-type and size failures are distinct fixed errors, never invented absence or raw exception text. Primary execution failures retain safe host completion flags and bounded lookup counts/code. The selected deadline is explicitly identified as execution-wide (default 30 minutes); the global 100-provider-request budget is unchanged.

A unit reporting unavailable required/visual evidence or truncated output ends further inference immediately, preserving validated partial summaries/findings as COMMENT with a failed job. It cannot be repaired by a later model completeness claim. Soft unfinished analysis still uses the existing shared continuation allowance. Complete multi-group reviews still require all groups and final synthesis.