Skip to content

Quantify unresolved qualification precision without changing admission - #266

Draft
pranems wants to merge 3 commits into
fix/performance-qualification-20261022from
fix/qualification-method-precision-20261103
Draft

pranems wants to merge 3 commits into
fix/performance-qualification-20261022from
fix/qualification-method-precision-20261103

Conversation

@pranems

@pranems pranems commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Summary

Implement a bounded offline precision-feasibility check and freeze the specific
unresolved dependence/sample-sufficiency blocker. This does not implement a
qualification pass rule or allocate a performance campaign.

Keep actual 0.55.67, original 10% limits, all operation/percentile metrics,
0.5 CPU/1 GiB, original protocol and every historical failure unchanged.
The tool always emits UNRESOLVED and admissionEligible: false. Conditional
IID quantile bounds are sensitivity calculations, not coverage for the
sequential traced diagnostic.

Parent #251 stays draft. Do not mark this child ready to trigger the fixed
warmup workflow. No deployment, customer-production action, paid hardware,
source-side recovery or migration-worktree change.

Linked Defect / Plan Reference

TDD Process

  • RED test written first; observed it fail for the expected reason
  • GREEN: minimal implementation that turns the test green
  • Refactor (if any) keeps the test green
  • Permanent regression guard added (where applicable - see existing patterns in api/src/security/)

RED: missing conditional quantile-rank feasibility calculator. GREEN:
initial 81/81 original performance/stability/new precision contracts, zero skips or
failures. Final focused precision/stability: 21/21, including the inclusive
1.10/ties guard; new precision contracts alone: 13/13. Real CLI checks include marker/image/quota/cardinality rejection,
overwrite refusal, input byte bounds and descriptor-replacement provenance.

Feature / Bug-Fix Commit Checklist (standing rule)

Every behavioral change must include all that apply. Tick or strike through with N/A + reason.

  • Unit Tests - service-level and controller-level coverage of the new behavior. N/A to API services/controllers; 13 offline numerical/CLI precision contracts cover the actual changed tool.
  • E2E Tests - end-to-end spec under api/test/e2e/ exercising the feature through HTTP. N/A, no API behavior change; actual child-process CLI covered.
  • Live Integration Tests - new section in scripts/live-test.ps1 runnable against local + Docker + Azure. N/A, no live behavior change or admission allocation.
  • Feature Documentation - dedicated doc in docs/ with architecture, RFC references, Mermaid diagrams, test coverage. Existing feature doc extended with source citations, all 24 observed ratios, assumptions, alternatives, budget and finite-stop acceptance plan. No diagrams added or changed.
  • INDEX.md Update - add the new doc reference to docs/INDEX.md. N/A, existing indexed feature doc only.
  • CHANGELOG.md Update - [Unreleased] entry with full test counts and feature summary
  • Session & Context Updates - Session_starter.md and docs/CONTEXT_INSTRUCTIONS.md
  • Version Management - bump api/package.json (and any other version refs) when behavior changed. N/A, offline tooling only; requested actual 0.55.67 preserved.
  • Response Contract Tests - API responses contain ONLY documented fields (key allowlist assertion at unit + E2E + live). N/A, no API response change; diagnostic result explicitly non-admitting and no passed field.

Validation Output

node --test scripts\test\membership-precision.test.cjs scripts\test\membership-stability.test.cjs scripts\test\membership-performance.test.cjs
PASS: 81 tests, zero skips/failures.

node --test scripts\test\membership-precision.test.cjs scripts\test\membership-stability.test.cjs
PASS: 21 tests, zero skips/failures after inclusive-margin clarification.

node --check scripts\performance\membership-precision.cjs
PASS.
node --check scripts\test\membership-precision.test.cjs
PASS.
git diff --check
PASS.
pwsh -NoProfile -File scripts\sync-rfcs.ps1 -Quiet
PASS, existing documented waivers only.

Actual retained control and treatment precision CLI:
PASS execution, both UNRESOLVED/admissionEligible false.
No new performance or deployment evidence.

Protected Validate publication:
PASS all 30 gates at initial 69fd8e5c, 1574.7 seconds.
PASS all 30 gates at final 302e7d9080da03c92908acbfc9249811d5917121,
1694.7 seconds, using PREPUSH_MODE=Validate and the process-scoped
non-serving Prisma codegen URL.

Exact final publication command:
$env:PREPUSH_MODE = 'Validate'
$env:DATABASE_URL = 'postgresql://codegen:codegen@127.0.0.1:9/codegen?schema=public'
git push
PASS, exit0, all30 gates.

Final protected counts:
Membership tooling: 260 PASS / 8 explicit skips / 0 FAIL.
API unit: 6254 PASS / 216 suites.
API HTTP E2E: 2514 PASS / 118 suites, plus 42 skipped tests in 2 skipped
suites; existing pre-push exclusions remain unchanged.
Web Vitest: 1661 PASS / 125 files.
All static, source, documentation-render, ownership and orchestration
contract gates PASS under their existing ratchets.

No Prisma-backed matrix, live deployment, Playwright or new performance
campaign is claimed by this offline change.

Final exact-tip hosted CI: run41645985419 PASS at
302e7d9080da03c92908acbfc9249811d5917121.
validate PASS (8m37s); build-and-push PASS (2m50s).
Historical run41641357029 passed both jobs at69fd8e5c; it is not substituted
for final-tip CI.
First push was blocked before any suite because fresh-process DATABASE_URL
was absent. Retained separately. Process-local codegen-only placeholder
does not connect and is not database-backed validation evidence.

Independent separate-context read-only code review: no actionable findings.
Narrow follow-up independent mathematical review: inclusive-boundary/ties
clarification is correct, no actionable findings. The error-control target
concerns a true breach strictly above 1.10; equality remains allowed by the
original ceiling and still cannot admit a diagnostic receipt.
Residual blocker: neither a valid dependence model nor sufficient fixed-budget
precision has been established. The conditional calculation is explicitly
not qualification evidence.

Test/gate self-improvement applied: exhaustive small-n binomial enumeration,
792/793 p99 finite-bound boundary, correlated-visit counterexample, provenance
race, original-estimator compatibility and genuine 20% slowdown retention.
Design disposition accepted: one cohesive offline tool, no runner or serving
code responsibility and no speculative framework.

Both the broad code review and the narrow equality/ties follow-up are retained
in a COMMENTED review at exact published source302e7d90. This records
separate-context agent review, not independent human approval.
There are no review threads, and both this PR and parent251 remain draft.

Final ordinary CI:
https://github.com/pranems/SCIMServer/actions/runs/41645985419

The owned worktree is clean. Original protocol/assessor/client/runner and
all API/web files are byte-unchanged from991a9ce0. No new performance
campaign was allocated. This PR is ready for bounded consolidation review,
not ready-for-review event activation, parent release admission or deployment.

Standing Rules Acknowledgment

  • No em-dash (Unicode U+2014) anywhere in the diff. Verified via Select-String -Pattern ([char]0x2014).
  • No git commit --amend on history that has been pushed.
  • No new credentials, tokens, or secrets committed to source. Forbidden patterns regression spec at api/src/security/forbidden-source-patterns.spec.ts will catch known anti-patterns.
  • Migrations are additive-only (expand-contract for column changes). Migration linter at api/src/scripts/lint-migrations.ts enforces this. N/A, no migration change.

Destructive Migration Override (skip if not applicable)

  • This PR contains a NEW destructive Prisma migration (DROP TABLE/COLUMN, ALTER COLUMN ... TYPE, RENAME, INSERT ... SELECT FROM)
  • I have set ALLOW_DESTRUCTIVE_MIGRATION=1 in the CI environment for this PR
  • Justification (required if box above is ticked):
    <!-- Why this destructive change is necessary, what the rollback path is, and which release window it ships in -->
    

N/A, no migration change.

Reviewer Hint

Diff areas worth special attention from the reviewer:

  • Binomial lower/upper order-statistic tails and numerical boundary tests.
  • 144 one-sided components for illustrative simultaneous bounds versus an
    intersection-union global pass test; neither repairs invalid sampling units.
  • Conditional versus actual uncertainty, preserved pooled-quantile estimator,
    no window independence, no guessed sample increase or policy weakening.
  • Diagnostic markers, immutable image/source metadata, once-read input hashes,
    finite byte bounds and exclusive new output.
  • No qualification/native/warmup execution, no parent readiness/merge,
    unchanged runtime and migration worktree.

Prashant Rane (ALLYIS INC) and others added 3 commits November 3, 2026 06:37
Keep original admission and all failed evidence unchanged. Add bounded offline conditional quantile sensitivity and freeze the model/precision blocker before any further allocation.

Focused contracts: 81/81 PASS; semantic RED retained. Independent review: no actionable findings. No application, protocol, quota, version or deployment change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record the fresh-process Prisma codegen preflight and process-scoped non-serving placeholder resolution. No qualification allocation or database-backed evidence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Clarify the proposed false-admission target as a true breach above the original inclusive 1.10 ceiling. Add exact-boundary non-admission and 298/299 finite-tail controls; no admission or runtime code changes.

Narrow independent mathematical review: no actionable findings. Focused precision: 13/13 PASS; precision plus stability: 21/21 PASS. Retain initial 30-gate publication and its failed configuration preflight separately.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@pranems pranems left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separate-context independent review

Result: no actionable findings, with the release hold preserved.
This comment records read-only reviews performed in separate agent contexts;
it is not a claim of independent human approval.

The broad review examined binomial order-statistic tails, endpoints,
144-component simultaneous-bound accounting, original estimator preservation,
diagnostic isolation, bounded input/once-read provenance and negative controls.
It found no actionable defects in the offline calculator.

Final specification scrutiny then identified an equality/ties ambiguity.
The narrowly scoped follow-up review independently confirmed:

  • For a true ratio R strictly above 1.10, a valid upper bound U at or below
    1.10 implies a coverage failure. This implication does not hold at R=1.10,
    because U=R can occur with positive probability, including constant ties.
  • Correcting the proposed error target to a true breach strictly above 1.10
    preserves the original inclusive ceiling and does not establish a pass rule.
  • The new constant-candidate-1.1/baseline-1 test leaves point and conditional
    ratio endpoints at 1.1 while requiring UNRESOLVED/admissionEligible false.
  • 0.99^298 is above 0.05, while 0.99^299 is below it. The separate 792/793
    boundary for 0.05/144 is also correct. Neither is sufficient sample size
    for 10% quantile-ratio precision.

Residual limitations are substantive, not approval paperwork: no valid
dependence model, actual-data confidence coverage, campaign power or
fixed-budget replacement admission procedure has been established.
Conditional IID calculations remain sensitivity analyses, not evidence that
the sequential traced study qualifies 0.55.67.

No hosted performance workload, new quota, version substitution, historical
failure rewrite or deployment was performed by the review. The parent and
this child must stay draft; ordinary CI success does not clear performance
admission.

@pranems

pranems commented Nov 3, 2026

Copy link
Copy Markdown
Owner Author

Final bounded implementation receipt

Exact head: 302e7d9080da03c92908acbfc9249811d5917121.

Evidence Result
Initial semantic RED Missing conditional quantile feasibility calculator
New offline precision contracts 13/13 PASS
Precision plus original stability contracts 21/21 PASS
Initial combined original performance/stability/precision 81/81 PASS; final full tooling below includes the later boundary guard
Final protected PREPUSH_MODE=Validate; git push 30/30 gates PASS, 1694.7 seconds
Final membership tooling aggregate 260 PASS, 8 explicit existing retained-fixture/opt-in skips, 0 FAIL
API unit 6254 PASS / 216 suites
API HTTP E2E 2514 PASS / 118 suites; 42 tests / 2 suites skipped; existing runner exclusions unchanged
Web Vitest 1661 PASS / 125 files
Final hosted CI 41645985419, exact head above, validate and build-and-push SUCCESS
Independent agent reviews Broad offline implementation plus narrow equality/ties review, no actionable findings; COMMENTED receipt at exact head
Historical diagnostic preservation All 24 computed point ratios match retained audit exactly; both reports UNRESOLVED/non-admitting
Frozen/runtime preservation Original protocol, assessor, benchmark client, runner and every API/web file unchanged from991a9ce0
Ownership Physical owned worktree/dependencies; final worktree clean; migration worktree untouched
Allocation boundary No new performance campaign, readiness event, release merge, target deployment or customer-production action

Both #266 and parent #251 remain draft. Parent head remains
0b6490f7cbc1d728fe338579d764d2e29fabb3e2; this work has not changed master.
No independent human approval is claimed.

Substantive unresolved blocker

The retained sequential traced observations establish neither a valid
dependence model nor tail precision/power inside the fixed original budget.
Even the explicitly unverified IID sensitivity gives upper ratios 2.485493
for treatment remove p95 and 8.105010 for empty-read p99. These are not valid
confidence coverage statements for the actual traces. A larger fixed sample
count is not justified, and no new admission pass rule has been implemented.

Keep actual0.55.67, original inclusive1.10 ceiling, all12 metrics per block,
original0.5CPU/1GiB and50,000/100,000 quota/period. Original failures and raw
receipts stay unchanged. A defensible prospective dependence/precision
argument, frozen design, tests and review must precede further allocation.
This is not a request for generic authorization or a retrospective waiver.

Retained owned evidence

  • test-results/method-precision-red.log
  • test-results/method-precision-control-100.json
  • test-results/method-precision-treatment-1000.json
  • test-results/method-precision-final-focused.log
  • test-results/method-precision-protected-push.log (initial configuration failure)
  • test-results/method-precision-protected-push-codegen.log (initial 30-gate pass)
  • test-results/method-precision-final-protected-push.log
  • test-results/method-precision-final-ci.json
  • test-results/method-precision-final-ci-watch.log

The initial configuration failure was resolved using an explicit process-local
non-serving codegen URL, not a shared database or database-backed evidence.
One follow-up message to the synchronous reviewer was rejected by the tool;
the distinct narrow mathematical question was reviewed in a separate context
instead. No workload was allocated by that routing failure.

Test/gate self-improvement applied: exhaustive small-n binomial enumeration,
792/793 simultaneous and298/299 individual p99 finite-bound controls,
perfect within-visit dependence counterexample, provenance replacement,
same-point/wide-bound case and exact-inclusive-margin/ties regression.
Design disposition accepted: one cohesive offline tool, no serving or
resource-orchestrator growth. No full-transcript RCA completeness,
off-device recovery or retained image-binary claim.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant