Improve agent-memory evidence and benchmark integrity - #225
Open
Coding-Dev-Tools wants to merge 67 commits into
Open
Coding-Dev-Tools wants to merge 67 commits into
Coding-Dev-Tools wants to merge 67 commits into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Engraphis can currently expose an exact value after packing has omitted a condition from its source. This PR requires the complete meaningful source before advertising an exact binding, preserving literal bytes and original offsets in any language. Coverage withholds bound occurrences that cannot fit; legacy retains its selected text but clears incomplete binding metadata. Content corrections require explicit replacement bindings, and compact responses keep bindings only with admitted source context.
The PR also makes benchmark results reproducible and attributable: datasets, fixtures, oracles, reports and charts use verified byte snapshots; producer names are unique; scope, trust, case identity, retry receipts and observed failure usage remain consistent across fresh and resumed runs. Checkpoint locking prevents competing execution and permits explicit recovery after process death. Campaign pricing uses disjoint ordinary/cached/written input categories and a single final rounding step. Recall receipts retain the selected recipe, effective depth and packing mode, with strict validation of persisted mode labels.
Local queue and capacity locks now release automatically on process death while retaining interrupted receipts. Completed jobs remain cached, and interrupted jobs still require explicit reconciliation. Upstream diagnostics are verified while holding their released producer lock; persistent markers no longer cause an endless wait. Prerequisite paths stay within the repository, and pre-open marker disappearance or replacement fails closed. Real-process tests cover live exclusion, forced termination, preserved receipts and cached resume.
Benchmark analysis also rejects malformed scoring flags and binds repair digests to the identified manifest, including agreement with retained dataset-integrity metadata. Chart transport claims require actual JSON booleans. All 36 retained benchmark envelopes and eight external diagnostics remain compatible; the focused analysis and renderer suite passes 192 tests.
External analysis requires finite rates within [0,1], including when invalid rows agree with their aggregate. Capacity completion requires 24 unique cell identities and an exact matching hash index. Paid campaign and continuation runners use a shared persistent OS-held lock, preserving unfinished reservations, budget approval, single-child allocation and cached no-dispatch behavior after process death. The shared lock implementation is included in new frozen campaign source snapshots.
Campaign preparation derives public execution metadata locally and rejects all caller-supplied configuration overrides before binding. A bounded CLI-version grammar prevents warning banners or arbitrary process output from entering public manifests. Preparation and manifest loading share strict field, type, basename, digest and timeout validation; the three retained OAuth campaign manifests remain compatible. The CodeQL finding also exposed a name-based metadata classification; helpers now describe execution metadata accurately. No security query, gate or suppression was changed.
Provider usage is counted only after validating the complete TokenUsage schema, finite nonnegative counters, token relationships and consistent invocation metadata. Empty or malformed checksummed checkpoints cannot publish complete zero-cost accounting. Fresh failures preserve valid measurements and explicit missingness through terminal resume; legacy unknown rows stay distinct from proven zero invocations. Explicit zero-call rows cannot be scored complete. Reserved estimates remain distinct from observed usage. The combined campaign, continuation, API, ledger, audit, metadata, lock and CodeQL regression group passes 236 tests without provider calls.
Campaign history mutations now enforce the exact effective workspace, repository, session and scope. Logical child labels resolve under the declared parent; foreign or unknown physical child IDs fail closed. Core correction lineage uses mapped memory IDs, retaining fixture labels separately. Duplicate write IDs, missing targets, invalid intervals and backdated or already-closed corrections fail validation before the first memory write. Repeated invalidation preserves the earliest close. Memory creation and its validity closures use the existing engine transaction hooks, with target revalidation under the transaction and rollback on closure failure. The canonical ledger follows the same scope and temporal rules. All 400 retained ledger snapshots are unchanged; real-engine replay validates 1,040 stored records and all 80 correction chains. These are offline integrity checks, not new answer-quality scores or a promise of whole-batch rollback on arbitrary backend failures. Focused campaign and replay validation: 423 passed, 1 skipped in 22.95s.
Candidate code could forge a passing oracle result through stdout or shutdown hooks. A versioned expression interpreter now treats candidate source as data, exposes only primitive operations, and reserves stdout for one trusted result frame. Expected values remain in the host. Unsupported syntax/resource limits are unscored and cannot trigger corrections. Both Docker campaigns and the local corpus helper use the same contract; reader instructions disclose it and producer snapshots bind it. All 400 v1 and 400 generated v2 deterministic repairs pass, while stale fixtures remain incorrect. Historical campaign scores are unchanged; new execution requires a newly frozen manifest. Focused campaign validation passes 456 tests with one Docker-dependent skip.
Measured development fixtures:
These comparisons use identical inputs and budgets against
c3a86295. The three tight query-retention losses remain explicit. The earlier five unpunctuated retention fixtures also remain 0/5 versus 5/5 on98e8f5c2; omission is not counted as retention. Prior safety populations remain 5/5, 8/8 and 16/16; all 14 action and four source-integrity cases pass. A separate 35-token packing fixture retains three sources in 33 tokens with coverage versus one in 35 with legacy. The CI gate independently checks rendered token counts, full binding/span retention, suppression and roomy controls.Evidence is retained in
evidence-contracts-20260921-v11.json,offline-fixtures-v70.jsonanduser-journeys-20260921-v4.json. The offline registry binds 579 producer files; all 122 historical artifact checksums verify. All seven local user journeys pass with 116 unique producer identities. Historical results and run provenance remain unchanged.Validation at
655fb8ee: campaign accounting, continuation, audit, API, checkpoint recovery, compact REST/MCP binding, source identity, and evidence/documentation regressions pass, along with Ruff, Pyright 1.1.414, all eight offline evaluation gates, commercial boundary checks, and dashboard asset checks. Full local suite: 6,470 passed, 63 skipped. GitHub: 29/29 checks pass on this head.These are deterministic development and functional results, not new external QA, provider billing, deployment or untouched-holdout qualification. Production retrieval and packing defaults remain unchanged. The synchronized Railway fixes from #224 are included; dependency updates remain in #226, #227 and #228.