Skip to content

perf: bound whole-visit CDF qualification feasibility - #268

Draft
pranems wants to merge 1 commit into
fix/qualification-method-precision-20261103from
fix/qualification-cluster-cdf-20261103
Draft

pranems wants to merge 1 commit into
fix/qualification-method-precision-20261103from
fix/qualification-cluster-cdf-20261103

Conversation

@pranems

@pranems pranems commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Summary

Implement and analytically verify an offline whole-visit cluster-CDF method,
preserving original pooled quantiles, and demonstrate the exact coverage/power
blocker. This is not an implemented admission correction.

Draft child of #266. Keep #266 and #251 draft. No readiness event, campaign,
new hardware, version substitution, target deployment or customer-production
action. The migration worktree and #267 are untouched.

At actual 0.55.67, original 10% ceiling and 0.5 CPU/1 GiB:

  • Conditional KL bounds do not assume independent visits, but need a prospective
    quantitative predictable-CDF drift restriction not established by the receipts.
  • Common/ergodic regime controls prove a count-independent uniform 5% false
    admission /80% neutral-power obstruction without that restriction.
  • Even an independent whole-visit counterexample subclass needs at least 276
    informative visits; the measured 45-minute projection fits only 80 total.
    This is a counterexample, not an actual-data independence assumption.
  • This conditional procedure needs 793 visits/role/block just for a finite upper
    p99 endpoint at zero drift and 0.05/144. That count is neither universal nor
    sufficient for 10% precision, and is not authorized.
  • Both actual diagnostic treatments reproduce all 24 original point ratios and
    input hashes exactly. Complete prior precision outputs are also unchanged.
  • STOP allocation until externally justified dependence, within-visit variation
    and tail-power model input can support coverage and power inside the unchanged
    budget. Original failures and raw observations remain immutable.

Linked Defect / Plan Reference

  • DELIVERY_PLAN.md defect ID: N/A; bounded continuation of the existing ci: qualify membership performance on isolated Linux runners #251 qualification hold, not a new product feature.
  • Audit doc reference: docs/ENDPOINT_SNAPSHOT_REVISION.md, "Whole-visit cluster-CDF construction and demonstrated feasibility limit (2026-11-03)".
  • ADR (if applicable): N/A; no runtime architecture or admission policy change.

TDD Process

  • RED test written first; observed it fail for the expected reason
  • GREEN: minimal implementation that turns the test green
  • Refactor (if any) keeps the test green
  • Permanent regression guard added (where applicable - see existing patterns in api/src/security/)

Semantic RED: missing whole-visit CDF implementation. Further discriminating RED
confirmed the sharper KL count (39828 versus 793 for the prior loose range-only
bound) and the missing method-independent finite-count power calculation.
Final focused run: 29/29 PASS.

Feature / Bug-Fix Commit Checklist (standing rule)

Every behavioral change must include all that apply. Tick or strike through with N/A + reason.

  • Unit Tests - service-level and controller-level coverage of the new behavior. N/A for API services/controllers; 16 new Node tooling contracts plus 13 preserved precision contracts pass.
  • E2E Tests - end-to-end spec under api/test/e2e/ exercising the feature through HTTP. N/A for unchanged API; actual native CLI end-to-end coverage is included.
  • Live Integration Tests - new section in scripts/live-test.ps1 runnable against local + Docker + Azure. N/A; no serving behavior or deployment changed.
  • Feature Documentation - dedicated doc in docs/ with architecture, RFC references, Mermaid diagrams, test coverage. Existing focused feature doc extended with full derivation, estimand, controls, counts, evidence limits and finite stop; no diagram added/edited.
  • INDEX.md Update - add the new doc reference to docs/INDEX.md. N/A; existing already-indexed feature doc, no new document.
  • CHANGELOG.md Update - [Unreleased] entry with full test counts and feature summary
  • Session & Context Updates - Session_starter.md and docs/CONTEXT_INSTRUCTIONS.md
  • Version Management - bump api/package.json (and any other version refs) when behavior changed. N/A; required actual 0.55.67 preserved, runtime inputs unchanged.
  • Response Contract Tests - API responses contain ONLY documented fields (key allowlist assertion at unit + E2E + live). N/A; API unchanged. Diagnostic contracts assert non-admission, no allocation and no success-shaped pass field.

Validation Output

Focused cluster/precision: 29/29 PASS, zero skips/failures.
Retained data:             24/24 point ratios identical; hashes identical.
Preserved precision:      both complete previous outputs identical.
Independent review:       no actionable findings; statistical blockers retained.
Protected Validate:       30/30 PASS, 1791.1s, exact 36297d4f.
Membership toolkit:       276 PASS, 8 explicit existing skips, 0 FAIL.
Web Vitest:               1661 PASS /125 files.
API unit:                 6254 PASS /216 suites.
API HTTP E2E:             2514 PASS /118 suites; 42 tests/2 suites skipped.
Exact-tip hosted CI:      41661345096 PASS at 36297d4f.
New campaign/live/browser/database admission: NOT RUN and NOT CLAIMED.

Commands and results:

  • node --test scripts\test\membership-cluster-cdf.test.cjs scripts\test\membership-precision.test.cjs - PASS, 29 tests.
  • node --check scripts\performance\membership-cluster-cdf.cjs and node --check scripts\test\membership-cluster-cdf.test.cjs - PASS.
  • git diff --check - PASS.
  • Protected PREPUSH_MODE=Validate publication below - PASS, all30 gates. A process-local non-serving codegen URL is used only for Prisma generation; it is not PostgreSQL validation evidence.

Exact protected publication command:

$env:PREPUSH_MODE = 'Validate'
$env:DATABASE_URL = 'postgresql://codegen:codegen@127.0.0.1:9/codegen?schema=public'
git push --set-upstream origin fix/qualification-cluster-cdf-20261103

Both exact actual-data commands below passed as offline analysis only:
admissionEligible:false, no allocation, no established actual coverage.
Their outputs were created exclusively; choose new filenames for reproduction.

node scripts\performance\membership-cluster-cdf.cjs `
  test-results\warmup-study-41632439081\membership-warmup-diagnostic-41632439081-1\warmup-control-100\protocol.json `
  test-results\warmup-study-41632439081\membership-warmup-diagnostic-41632439081-1\warmup-control-100\aa-baab\rounds.json `
  test-results\warmup-study-41632439081\membership-warmup-diagnostic-41632439081-1\warmup-control-100\campaign.json `
  test-results\cluster-cdf-final-control-100.json

node scripts\performance\membership-cluster-cdf.cjs `
  test-results\warmup-study-41632439081\membership-warmup-diagnostic-41632439081-1\warmup-treatment-1000\protocol.json `
  test-results\warmup-study-41632439081\membership-warmup-diagnostic-41632439081-1\warmup-treatment-1000\aa-baab\rounds.json `
  test-results\warmup-study-41632439081\membership-warmup-diagnostic-41632439081-1\warmup-treatment-1000\campaign.json `
  test-results\cluster-cdf-final-treatment-1000.json

The shared frozen benchmark, original protocol, assessor, qualification runner,
workflows, and every API/web source file are unchanged from the base.
Exact-tip ordinary hosted CI
passed validate (8m19s) and build-and-push (2m30s). This is not a new
performance campaign or release-admission result. Existing CI warnings remain;
no warning-free claim is made.

Standing Rules Acknowledgment

  • No em-dash (Unicode U+2014) anywhere in the diff. Verified via Select-String -Pattern ([char]0x2014).
  • No git commit --amend on history that has been pushed.
  • No new credentials, tokens, or secrets committed to source. Forbidden patterns regression spec at api/src/security/forbidden-source-patterns.spec.ts will catch known anti-patterns.
  • Migrations are additive-only (expand-contract for column changes). Migration linter at api/src/scripts/lint-migrations.ts enforces this. N/A; no migration.

Destructive Migration Override (skip if not applicable)

  • This PR contains a NEW destructive Prisma migration (DROP TABLE/COLUMN, ALTER COLUMN ... TYPE, RENAME, INSERT ... SELECT FROM)
  • I have set ALLOW_DESTRUCTIVE_MIGRATION=1 in the CI environment for this PR
  • Justification (required if box above is ticked):
    N/A. No schema or migration change.
    <!-- Why this destructive change is necessary, what the rollback path is, and which release window it ships in -->
    

Reviewer Hint

Diff areas worth special attention from the reviewer:

  • Conditional MGF/supermartingale/Jensen proof, predictable-CDF drift restriction
    and rank inversion with ties. No independent visits/windows assumed.
  • Difference between exact original pooled-quantile estimator, fixed marginal
    inference target, and inadmissible substitution of a history-conditioned target.
  • Count-independent power proof and independent-regime finite-count necessary
    condition; neither is an actual-host probability estimate.
  • All12 metrics and the inclusive original1.10 boundary, no pooling/discarding,
    no allocation or original-assessor wiring.
  • Design disposition accepted: one cohesive offline module and shared validation;
    no speculative framework. Gate improvement applied: pooled-median shock/ties
    and assumption-misspecification controls. No full-transcript RCA claim.

Derive conditional visit-CDF KL bounds without assuming independent visits. Preserve original pooled quantiles and retained ratios. Prove persistent-regime and finite-budget power obstructions; no admission correction or campaign allocation. Focused analytical and native CLI contracts: 29 passed. Independent review found no actionable defects and retained the actual-model blocker.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@pranems pranems left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent cluster-CDF review

Reviewed implementation tip: 36297d4fc9d28017301ee48b44d2ee7b4df10851.

A read-only, separate-context code-review agent examined the new implementation,
new tests, shared-helper extraction and feature-document derivation. It reported
no actionable correctness findings. This is not human approval or performance
admission.

The reviewer explicitly retained these substantive blockers:

  • The predictable-CDF drift restriction is not established for actual receipts.
    Actual-data coverage and 80% neutral power have not been demonstrated.
  • Without quantitative dependence restrictions, a common-regime counterexample
    prevents uniform 5% false admission and 80% neutral power at any finite count.
  • Even an independent whole-visit-regime counterexample subclass needs at least
    276 optimistically informative visits, versus 80 total projected in 45 minutes.
    This constructs a counterexample; it does not assume actual visits independent.
  • 793 role visits is only the conditional KL method's minimum for a finite upper
    p99 endpoint with zero predictable drift and one-sided error 0.05/144. It is
    neither a universal count nor sufficient for 10% ratio precision.
  • Diagnostics remain non-admitting; no valid admission correction is implemented.

The final commit includes only subsequent documentation status/provenance
clarifications after the independent review; mathematical code and tests are
the reviewed versions. No campaign, deployment or customer-production action ran.

Exact-tip publication receipt

  • Draft child PR: #268, targeting
    fix/qualification-method-precision-20261103 (#266). Do not make it,
    #266 or #251 ready or infer admission from the green tooling checks.
  • Protected PREPUSH_MODE=Validate publication: 30/30 PASS, 1791.1 seconds,
    exact 36297d4fc9d28017301ee48b44d2ee7b4df10851. The total includes the
    successful non-native Prisma-codegen configuration preflight.
  • New cluster/precision focused contracts: 29/29 PASS after semantic RED.
    Complete membership tooling: 276 PASS, 8 explicit existing retained/opt-in
    skips, zero failures.
  • API unit: 6254 PASS /216 suites. API HTTP E2E: 2514 PASS /118 suites,
    with 42 tests/2 suites skipped by the unchanged runner.
  • Web Vitest: 1661 PASS /125 files. Existing static, Mermaid-render,
    documentation, ownership, orchestration and supply-chain gates passed.
  • Ordinary exact-tip hosted CI:
    https://github.com/pranems/SCIMServer/actions/runs/41661345096
    validate PASS (8m19s), build-and-push PASS (2m30s), exact same head.
    Existing warnings are not presented as zero-warning evidence.
  • The codegen URL was process-local and non-serving. No new PostgreSQL,
    live HTTP, browser, performance-admission or deployment evidence is claimed.

Both final offline analyses preserve all24 original ratios and protocol/
round hashes. Both complete original precision outputs are deeply identical
after helper extraction. Every raw input was read-only; outputs use exclusive
creation. Final artifact identities:

Artifact SHA256
membership-cluster-cdf.cjs 234333ebc57b2f769f9026f89933e1004d9c0915817301c32a93b63ed2be60f8
membership-precision.cjs b86f26e47c37bca45df948913509936a5ed1df256273dd7d0e621bbd88da6709
membership-cluster-cdf.test.cjs da839ea5b5cc10ba391281153486aec7384a95e447218331148290bdd2ae9170
test-results\cluster-cdf-final-control-100.json 996208334fc4777f42f26d10678bb5faeb6fa4c59753733494ab3d31de160b5a
test-results\cluster-cdf-final-treatment-1000.json 4f61c8f0d56c2471d9c9ff4a7498b605f0231e1cb707b8817738643c05d154a2

Required stop and concrete missing input

This is a demonstrated infeasibility result and runnable conditional/offline
calculator, not a runnable corrected admission policy. The missing input is
externally justified quantitative restrictions on predictable-CDF drift and
within-visit variation/timing dependence, plus a neutral/tail-separation model
supporting coverage and power inside the unchanged45-minute work limit.
Neither independent visits alone, more retained windows, generic approval,
nor an unchanged hosted retry supplies that information. No new allocation
is authorized. Keep actual0.55.67, 10%, original resources and old failures.

Test/gate improvement applied: explicit pooled-median shock/ties and drift-
misspecification controls. Design disposition accepted: cohesive offline
module, shared validation with two actual callers, no speculative framework.
The optional graph overlay was unavailable because no knowledge graph exists.
No full-transcript RCA completeness, off-device recovery or retained image
binary is claimed. Parent migration tooling/cutover and customer production
were not touched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant