Skip to content

ci: bind qualification to the actual checkout image - #263

Merged
pranems merged 2 commits into
fix/performance-qualification-20261022from
fix/qualification-image-binding-20261103
Nov 3, 2026
Merged

pranems merged 2 commits into
fix/performance-qualification-20261022from
fix/qualification-image-binding-20261103

Conversation

@pranems

@pranems pranems commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Summary

Qualify the exact checkout image and dependency set instead of the frozen
0.55.58 candidate. This is a small prerequisite stacked on #251, not a second
copy of that release candidate or permission to admit/deploy a failed build.

The workflow builds Linux/amd64 from its actual checkout, refreshes the mutable
base with --pull, binds immutable image identity/source/version, and supplies
a separate generated protocol. The historical protocol, failed evidence,
latency bounds, quotas, warmups, samples and operation order remain unchanged.
Build, measurement, cleanup and evidence upload remain inside the existing
75-minute job cap. No serving API/browser source, Prisma migration, dependency
manifest or lockfile changes.

Linked Defect / Plan Reference

  • DELIVERY_PLAN.md defect ID: N/A; exact-image qualification prerequisite for ci: qualify membership performance on isolated Linux runners #251.
  • Audit doc reference: docs/ENDPOINT_SNAPSHOT_REVISION.md, Current-checkout image binding and execution issues.
  • ADR (if applicable): N/A; extends the existing protocol/runner without a new architecture.

TDD Process

  • RED test written first; observed it fail for the expected reason
  • GREEN: minimal implementation that turns the test green
  • Refactor (if any) keeps the test green
  • Permanent regression guard added (where applicable - see existing patterns in api/src/security/)

Observed RED: absent checkoutProtocol helper; real CLI argument-count mismatch;
missing separate build/cleanup budget; missing fresh-base --pull. GREEN:
58 focused contracts, including actual CLI and extracted real PowerShell guard.

Feature / Bug-Fix Commit Checklist (standing rule)

Every behavioral change must include all that apply. Tick or strike through with N/A + reason.

  • Unit Tests - service-level and controller-level coverage of the new behavior. N/A for serving services/controllers; the changed tooling has permanent Node/actual PowerShell contracts.
  • E2E Tests - end-to-end spec under api/test/e2e/ exercising the feature through HTTP. N/A for the changed tooling; actual CLI and image/protocol integration checks cover its output.
  • Live Integration Tests - new section in scripts/live-test.ps1 runnable against local + Docker + Azure. N/A for serving behavior; no HTTP contract changed. Release/deployment live gates remain independent.
  • Feature Documentation - dedicated doc in docs/ with architecture, RFC references, Mermaid diagrams, test coverage. Updated the existing snapshot and infrastructure documents; no new protocol or diagram.
  • INDEX.md Update - add the new doc reference to docs/INDEX.md. N/A; no new document.
  • CHANGELOG.md Update - [Unreleased] entry with full test counts and feature summary
  • Session & Context Updates - Session_starter.md and docs/CONTEXT_INSTRUCTIONS.md
  • Version Management - bump api/package.json (and any other version refs) when behavior changed. N/A; remains part of the unreleased parent 0.55.67 candidate, with no second product version or lockfile rewrite.
  • Response Contract Tests - API responses contain ONLY documented fields (key allowlist assertion at unit + E2E + live). N/A; serving API source and public response contracts unchanged.

Validation Output

node --test scripts/test/membership-performance.test.cjs
PASS: 58 tests, zero skips or failures.

node --test scripts/test/membership-*.test.cjs
PASS: 227 tests; 8 explicit retained-fixture/opt-in skips; zero failures.
The files were expanded and passed as concrete paths on Windows.

node scripts/performance/assess-membership.cjs <original protocol> aa-baab <retained rounds> <new assessment>
Expected rejection, exit 1. Independent deep equality with original assessment
PASS: all 12 checks identical, including the same three failures.

docker build --pull --build-arg NPM_REGISTRY=https://packagefeedproxy.microsoft.io/npm/ --label org.opencontainers.image.revision=abdef7207428ec26ce54e76ec0e26a08921ea6ea --build-arg IMAGE_TAG=abdef7207428ec26ce54e76ec0e26a08921ea6ea -t scimserver-qualification:abdef720-currentbase-20261103 .
PASS: actual runtime 0.55.67 / Node24.22.2 / linux/x64; actual immutable
identity successfully generates the separate source-bound protocol.
This Windows Docker Desktop build is not Linux-host qualification.

PowerShell parse, Node syntax, workflow YAML/budget parsing: PASS.
Documentation freshness: PASS, 27/27.
Infrastructure C1/C2/C3, dependency ownership, git diff --check: PASS.
Manifest/lockfile Git differences: zero.

Protected push with PREPUSH_MODE=Validate:
$env:PREPUSH_MODE='Validate'
$env:DATABASE_URL='postgresql://localhost:5432/scim_codegen?schema=public'
git push --set-upstream origin fix/qualification-image-binding-20261103
PASS: all 30 gates, exit 0, 1195 seconds. Normal hooks enabled.
API unit: 6254 PASS, 216 suites.
API inmemory E2E: 2514 PASS, 42 explicit skips; 118 suites PASS, 2 skipped.
Web Vitest: 1661 PASS, 125 files; zero skipped or failed tests.
API lint: 0 errors, 507 existing warnings; API source is unchanged.
API build, web tsc/build, real-browser Mermaid rendering, source inclusion,
codegen, migration lint, ownership, supply chain and governance gates PASS.
Snapshot normalized Git bytes match HEAD; no baseline content accepted.

Hosted performance qualification is NOT RUN yet. The workflow is absent from
default master, so the supported path is reviewed integration into #251 and
its normal ready-for-review event, not an assumed branch dispatch. Neither
the historical failures nor final-image admission is declared cleared.

Standing Rules Acknowledgment

  • No em-dash (Unicode U+2014) anywhere in the diff. Verified via Select-String -Pattern ([char]0x2014).
  • No git commit --amend on history that has been pushed.
  • No new credentials, tokens, or secrets committed to source. Forbidden patterns regression spec at api/src/security/forbidden-source-patterns.spec.ts will catch known anti-patterns.
  • Migrations are additive-only (expand-contract for column changes). Migration linter at api/src/scripts/lint-migrations.ts enforces this. N/A; no migration changed.

Destructive Migration Override (skip if not applicable)

N/A; no database or migration change.

  • This PR contains a NEW destructive Prisma migration (DROP TABLE/COLUMN, ALTER COLUMN ... TYPE, RENAME, INSERT ... SELECT FROM)
  • I have set ALLOW_DESTRUCTIVE_MIGRATION=1 in the CI environment for this PR
  • Justification (required if box above is ticked):
    <!-- Why this destructive change is necessary, what the rollback path is, and which release window it ships in -->
    

Reviewer Hint

Diff areas worth special attention from the reviewer:

  • checkoutProtocol and validateProtocol: immutable source/image binding;
    published immutable GHCR baseline remains mandatory; no unbound local image.
  • Assert-CheckoutImage: reject source, package version or revision mismatch
    before pulling/allocating; published historical protocol remains usable.
  • The workflow's separate 10-minute build / 55-minute measurement step, actual
    45-minute cooperative work cap and four-minute always-upload reserve.
  • Regression tests exercise actual CLI, refusal to overwrite evidence, actual
    PowerShell guard and unchanged role selection, not just string presence.
  • Design disposition: accepted. Extend existing validation rather than add a
    new framework, speculative measurement policy or paid benchmark host.
  • Self-improvement applied: bind source, dependency-bearing artifact and actual
    runtime; preserve failed historical evidence and normal publication gates.

Prashant Rane (ALLYIS INC) and others added 2 commits November 3, 2026 03:20
Bind source, package version and immutable local image identity before qualification. Preserve frozen historical results, resource limits and latency bounds. Refresh mutable bases and reserve build/cleanup/upload time within the existing job cap. Focused contracts: 58 pass; complete local tooling: 227 pass, 8 explicit skips. No performance or deployment admission claimed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion

Record isolated dependency and Prisma generation prerequisites. Confirm the new assessor preserves the original control assessment exactly, including all three failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The source-bound lane has not run in its required hosted Linux environment, and Windows builds or extracted guard contracts cannot replace the exact-image campaign and retained cleanup/artifact evidence needed for qualification.

No open issues 🎉

What changed in this PR

Updates membership performance qualification to measure the actual checkout image and dependency set rather than the frozen candidate from #251. Historical protocols, failed evidence, and performance thresholds remain unchanged; hosted qualification is still pending.

Changes:

  • Builds with refreshed Docker bases and generates a separate protocol binding the immutable image ID, source commit, and package version.
  • Adds checkout/image preflight checks and adjusts build and measurement budgets within the existing 75-minute job cap.
  • Adds regression contracts and documents the implementation, evidence, and remaining qualification requirements.
File Description
Session_starter.md Records implementation status and pending qualification.
scripts/​test/​membership-performance.test.cjs Adds image-binding, CLI, preflight, and deadline contracts.
scripts/​performance/​run-membership-qualification.ps1 Accepts generated protocols and verifies checkout/image consistency.
scripts/​performance/​assess-membership.cjs Generates and validates source-bound candidate protocols.
docs/​ENDPOINT_SNAPSHOT_REVISION.md Documents binding behavior, execution issues, and validation evidence.
docs/​DEPLOYMENT_INFRASTRUCTURE_AND_FORM_FACTORS.md Updates qualification image provenance and time budgets.
docs/​CONTEXT_INSTRUCTIONS.md Summarizes current qualification status and constraints.
CHANGELOG.md Records the unreleased tooling changes and reported coverage.
.github/​workflows/​performance-qualification.yml Builds the checkout image, passes its protocol, and retains evidence.

Review effort: Balanced

@pranems
pranems merged commit 6f846cf into fix/performance-qualification-20261022 Nov 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants