tools: retain chronological stability evidence without admission

CHANGELOG.md

@@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### 0.55.67 descriptive stability evidence (2026-11-03, proposed)
+
+- Add a non-admitting, hash-bound report over all chronological windows in
+  retained complete qualification rounds. Reuse original validation and
+  nearest-rank percentiles; preserve all samples and original failed checks.
+- Add eight numerical/CLI/negative regression contracts. Focused tooling:
+  58 to 66 PASS, zero skips or failures. Negative tests require expected
+  validation errors rather than accepting unrelated exceptions.
+- Close two independent-review findings after reproducing both: retain the
+  actual unsorted observations, not only summaries, and hash the same
+  once-read buffers used for parsing. Reconstruct all 16,000 real observations
+  and preserve all twelve original checks/four failures in a separate
+  corrected report. The CLI regression changes both inputs after reading.
+- Actual current-image Linux qualification `41615647439` fails four first
+  same-image checks. Functional 66, privacy/logging 200 and exact cleanup
+  pass independently. No later block, protocol change or unchanged retry.
+  Existing application source, dependencies, version and lockfiles unchanged;
+  no new application count or deployment admission claimed.
+
 ### 0.55.67 exact-checkout qualification image (2026-11-03, proposed)
 
 - Build and bind the actual qualification checkout rather than testing only

Session_starter.md

@@ -1,5 +1,21 @@
 ## SCIMServer - Condensed Session Memory
 
+**2026-11-03 calibration evidence:** branch
+`fix/qualification-calibration-20261103` is based on reviewed merge6f846cf8.
+Actual source-bound Linux run41615647439 fails4/12 same-image checks;
+functional66/privacy200/exact cleanup PASS, later blocks not run. New
+non-admitting descriptive collector retains all16,000 samples in64 windows,
+hash-binds inputs and reproduces all original checks exactly. Focused tests
+66/66 PASS after semantic RED, including review regressions for missing raw
+observations and a second-read hash race. Corrected output includes all
+unsorted samples and hashes exactly the once-read parsed buffers; all12
+original checks/four failures remain identical. No protocol, warmup, sample
+or limit changed.
+PR251 is draft again to avoid unchanged automatic retries while a prospective
+stability design is prepared. Operator unavailable is not hardware, gate-
+waiver or deployment approval. Latest failed evidence is protected separately
+in qualification-failure-20261103T1011035881166Z with verified roundtrip.
+
 **Last verified:** 2026-11-03
 
 ### Exact-checkout qualification implementation (2026-11-03)

docs/CONTEXT_INSTRUCTIONS.md

@@ -9,6 +9,20 @@
 
 ## Active Delivery Process
 
+**2026-11-03 measured calibration gap:** #263 merged into #251 at6f846cf8;
+ordinary exact-tip CI passes. Actual source-bound Linux run41615647439 fails
+four same-image comparisons, while functional66/privacy200/exact cleanup
+pass. No old/current comparison runs. #251 is draft again, not admitted.
+The new descriptive collector retains all64 chronological windows/16,000
+samples and hash-binds inputs without changing any original assessment,
+protocol or limit. Focused tooling66/66 PASS after RED. Review exposed and
+closed missing raw observations and a second-read provenance race: output
+now reconstructs every original sample in order, and hashes share exactly
+the parsed input buffers. Numerical, adversarial input-replacement and actual
+CLI tests ensure the report cannot claim convergence or deployment admission.
+Review/prospective stabilization design remains separate; operator unavailable
+does not authorize hardware, gate waiver or deployment.
+
 **2026-11-03 exact-checkout qualification:** isolated implementation binds the
 actual .67 checkout image rather than treating frozen .58 performance as new
 artifact admission. Historical protocol/results and all latency/resource/sample

docs/ENDPOINT_SNAPSHOT_REVISION.md

@@ -7,6 +7,72 @@ Performance admission and the independent release gates remain open.
 
 ## Current-checkout image binding (2026-11-03, not released)
 
+### Descriptive calibration prerequisite
+
+The new `--describe-stability` mode reuses the actual original assessor and
+nearest-rank estimator. It describes all 64 chronological 250-sample windows
+in a complete four-round block, retains both visits and every measured
+sample, and records the source image roles plus SHA-256 hashes of the exact
+protocol/round files. It cannot change a protocol, overwrite a report or
+admit a release. Its output explicitly says uncertainty and warmup convergence
+are not established. Window ratios are descriptive, not new acceptance
+tests on smaller sample sets.
+
+Focused RED reports the missing collector assertion before implementation;
+GREEN is **64/64**, comprising 58 existing contracts and six new cases.
+Independent review then found that summaries did not actually include the
+raw observations claimed above, and input hashes used a second pathname
+read. The corrected report includes each window's unsorted observations;
+percentiles use a separate sorted copy. Parsing and hashing now share the
+same once-read buffers. Both defects were reproduced before correction.
+Focused GREEN is now **66/66**, including chronological reconstruction and
+an actual CLI test that changes both input paths after their first read.
+The separately generated corrected current-image report reproduces every original failed comparison
+exactly and retains all 16,000 observations. In the baseline's second visit,
+populated-read p95 across its four windows is 4.574270, 30.201782,
+22.543207 and 16.514300 ms. The candidate's second visit is 2.465563,
+3.469971, 3.396606 and 2.432962 ms. This is not explained simply by
+discarding the first cold window. No extra benchmark was run.
+
+**Test-correctness RCA (Low):** during RED, a broad negative `assert.throws`
+could succeed because the missing collector itself threw TypeError. Require
+the expected receipt-validation assertion messages, alongside the positive
+collector and real CLI contracts. This closes a false-green assertion class
+at the earliest focused test stage.
+
+**Next boundary:** a reviewed prospective study must distinguish within-visit
+and across-visit instability before selecting any changed warmup or
+measurement policy. The operator was unavailable for the design question;
+that is not approval for paid resources, relaxed bounds or deployment.
+The pragmatic continuation is non-admitting analysis on existing evidence,
+under the requested instruction to diagnose/fix issues.
+
+**Self-improvement applied:** test numerical windows and exact input/output
+identity, not the mere presence of an analysis file or a thrown exception.
+**Design disposition: accepted.** A small descriptive collector and shared
+nearest-rank helper extend the existing assessor. No new framework,
+production dependency, resource lifecycle or admission strategy is added.
+
+**Hosted follow-up:** reviewed prerequisite #263 is merged into #251 at
+`6f846cf8`. Current-image Linux run
+[41615647439](https://github.com/pranems/SCIMServer/actions/runs/41615647439)
+successfully builds and binds actual checkout `bc60f7f3` and the immutable
+0.55.67 image starting `92cd5024`. Its first same-image block fails four
+of twelve checks: populated-read p95 ratio 0.571817, remove p95 1.123033,
+empty-read p95 0.440720 and empty-read p99 0.844341. Faster out-of-range
+controls also fail repeatability. All 17,628 checked requests, 16,000 retained
+timings, 66 functional checks, 200 private-log checks, capture and exact
+cleanup complete. The opposite control and both old/current blocks do not
+run. No unchanged retry or new performance admission is claimed.
+
+The fixed 100 warmups do not establish stable observed timings: first-visit
+populated-read p95 falls from 6.355253/6.801990 ms in the first 250 measured
+samples to 2.652742/2.667220 ms in the last 250. App-cgroup CPU changes across
+visits while SQL counts stay 25,591-25,593. This demonstrates a measurement
+stability gap, not its unique cause. The next design requires prospective
+calibration and review, not speculative production-code removal or
+post-hoc selection of a favorable warmup count.
+
 The existing qualification still selected a pinned 0.55.58 candidate even
 after source/dependency reconciliation proposed 0.55.67. A passing historical
 image comparison could not qualify that new artifact. The new prospective
@@ -27,6 +93,8 @@ resource limits, operation order and latency bounds are not modified.
 | Missing isolated root/web tooling / Low | The first chosen web build and documentation check failed on missing vite/jsdom. Restore each worktree-owned dependency set using npm ci only after its observed missing-dependency failure. The protected hook owns consolidated validation. | Prerequisite build/documentation checks; no shared dependency mutation or lock regeneration. |
 | Missing generated Prisma model types / Low | API dependencies were deliberately installed with scripts disabled, leaving the generated client absent. The API build exposed model-type errors; explicit Prisma generation succeeds with a codegen-only local URL and no database connection. The protected hook owns the subsequent full build/test verification. | First API build; normal code-generation prerequisite, not a production source defect. |
 | Stale mutable local base / Medium | The first local build reused Node 24.16.0 despite newer historical images. Require docker build --pull, retain the first image identity, and verify the refreshed runtime separately. | Actual built-image runtime readback; earliest possible gate is a fresh-base build plus runtime readback. |
+| Raw-observation retention claim / Medium | Window summaries retained counts and percentiles but omitted raw observations. Include the unsorted samples and prove all 16,000 can be reconstructed in original order from the report alone. Existing immutable reports remain unchanged; corrected output uses a new path. | Independent review; earliest possible focused test previously checked counts rather than reconstructability. The new outcome assertion closes that escape. |
+| Input-byte provenance race / Medium | Hashing reread mutable paths after parsing, allowing different bytes to identify the analysis. Read each input once and use that same buffer for parsing and hashing. An adversarial real CLI preload changes both files immediately after each read and verifies the hashes still identify the parsed bytes. | Independent review; earliest possible focused CLI test previously hashed only unchanged files. The deterministic replacement test closes that escape. |
 
 The initial source-binding test failed before implementation. The complete
 focused suite then passed 58/58, up from 51, with no skips. A separate real

docs/strategy/ENGINEERING_LESSONS_AND_PATTERNS.md

@@ -235,6 +235,23 @@ properties and runtime before adopting them. The narrow Linux qualification
 reuses the original lifecycle instead of building a new measurement framework.
 See [protocol and RCA](../ENDPOINT_SNAPSHOT_REVISION.md#prospective-linux-performance-qualification).
 
+A fixed warmup count is not evidence that observations have stabilized.
+Preserve chronological windows and distinguish within-visit drift from
+across-visit differences before attributing a failed identical-image control
+to production code. A descriptive report must preserve all original samples
+and assessments, bind its inputs, and explicitly avoid causal attribution,
+false-failure estimates or new admission criteria. Smaller windows are not
+independent replications and do not inherit pooled-sample limits. See the
+[current-image calibration prerequisite](../ENDPOINT_SNAPSHOT_REVISION.md#descriptive-calibration-prerequisite).
+
+Counts and percentiles are not retained observations. Reconstruct the complete
+chronological input from the report in a regression test, including unsorted
+values. Bind provenance to the exact buffers parsed, not a second read of
+the same pathname. A deterministic CLI test must replace or update the inputs
+after their first read and still obtain hashes of the analyzed bytes.
+Independent review caught both gaps in the descriptive collector; they are
+now closed in-place without changing any benchmark or acceptance result.
+
 ### Failure evidence needs its own completion budget and contract
 
 A failed measurement is not the same thing as an absent receipt. Preserve

scripts/performance/assess-membership.cjs

@@ -52,6 +52,10 @@ function rolesFor(protocol, block) {
   return { baseline: block.sameImage ? protocol.candidate : protocol.baseline, candidate: protocol.candidate };
 }
 
+function sortedPercentile(values, percentile) {
+  return values[Math.ceil(values.length * percentile / 100) - 1];
+}
+
 function assessBlock(rows, block, roles) {
   assert.ok(Object.hasOwn(schedules, block.order), 'Unknown round order');
   assert.equal(typeof block.sameImage, 'boolean');
@@ -78,9 +82,8 @@ function assessBlock(rows, block, roles) {
     const baseline = values('baseline');
     const candidate = values('candidate');
     return [50, 95, 99].map(percentile => {
-      const index = Math.ceil(2000 * percentile / 100) - 1;
-      const baselineMs = baseline[index];
-      const candidateMs = candidate[index];
+      const baselineMs = sortedPercentile(baseline, percentile);
+      const candidateMs = sortedPercentile(candidate, percentile);
       const ratio = candidateMs / baselineMs;
       return {
         operation, percentile, samplesPerRole: 2000, baselineMs, candidateMs, ratio,
@@ -91,7 +94,51 @@ function assessBlock(rows, block, roles) {
   return { block: block.name, sameImage: block.sameImage, passed: checks.every(check => check.passed), checks };
 }
 
-module.exports = { assessBlock, checkoutProtocol, rolesFor, validateProtocol };
+function describeStability(rows, block, roles) {
+  const qualificationAssessment = assessBlock(rows, block, roles);
+  const visits = { baseline: 0, candidate: 0 };
+  const rounds = rows.map((row, index) => ({
+    round: index + 1,
+    role: row.role,
+    visit: ++visits[row.role],
+    operations: Object.fromEntries(operations.map(operation => {
+      const windows = Array.from({ length: 4 }, (_, window) => {
+        const startCycle = window * 250;
+        const samples = row.samples[operation].slice(startCycle, startCycle + 250);
+        const values = [...samples].sort((a, b) => a - b);
+        return {
+          startCycle, endCycleExclusive: startCycle + 250, sampleCount: values.length,
+          samples,
+          p50: sortedPercentile(values, 50),
+          p95: sortedPercentile(values, 95),
+          p99: sortedPercentile(values, 99),
+        };
+      });
+      return [operation, {
+        windows,
+        lastToFirstP95Ratio: windows[3].p95 / windows[0].p95,
+      }];
+    })),
+  }));
+  return {
+    diagnosticOnly: true,
+    admissionEligible: false,
+    uncertaintyEstimated: false,
+    warmupConvergenceEstablished: false,
+    images: structuredClone(roles),
+    notes: [
+      'Chronological windows describe retained observations, not independent replications.',
+      'Cycle bounds are zero-based measured cycles after the protocol warmup.',
+      'Window ratios are not subject to the original pooled 2000-sample admission bounds.',
+      'No causal attribution, false-failure probability or acceptable warmup is inferred.',
+      'The original assessment and every sample are preserved without changing the protocol.',
+    ],
+    qualificationAssessment,
+    rounds,
+  };
+}
+
+module.exports = { assessBlock, checkoutProtocol, describeStability, rolesFor, validateProtocol };
 
 if (require.main === module) {
   try {
@@ -103,6 +150,23 @@ if (require.main === module) {
         JSON.parse(fs.readFileSync(identityPath, 'utf8')), sourceCommit, version,
       );
       fs.writeFileSync(output, `${JSON.stringify(current, null, 2)}\n`, { flag: 'wx' });
+    } else if (process.argv[2] === '--describe-stability') {
+      assert.equal(process.argv.length, 7, 'Explicit protocol, block, receipts and new output required');
+      const [protocolPath, blockName, rowsPath, output] = process.argv.slice(3);
+      const protocolBytes = fs.readFileSync(protocolPath);
+      const rowsBytes = fs.readFileSync(rowsPath);
+      const protocol = JSON.parse(protocolBytes.toString('utf8'));
+      const block = protocol.blocks.find(item => item.name === blockName);
+      assert.ok(block, 'Unknown preregistered block');
+      const report = describeStability(
+        JSON.parse(rowsBytes.toString('utf8')), block, rolesFor(protocol, block),
+      );
+      const { createHash } = require('node:crypto');
+      report.inputHashes = Object.fromEntries([
+        ['protocol', protocolBytes], ['rounds', rowsBytes],
+      ].map(([name, bytes]) => [name, createHash('sha256').update(bytes).digest('hex')]));
+      fs.writeFileSync(output, `${JSON.stringify(report, null, 2)}\n`, { flag: 'wx' });
+      console.log('Descriptive analysis complete; no stability or deployment admission.');
     } else {
       const [protocolPath, blockName, rowsPath, output] = process.argv.slice(2);
       assert.ok(protocolPath && blockName && rowsPath && output, 'Explicit protocol, block, receipts and output required');