Skip to content

fix(ops): prepare safe registry-driven tenant rollover - #267

Merged
pranems merged 11 commits into
masterfrom
ops/tenant10-migration-20261030
Nov 3, 2026
Merged

pranems merged 11 commits into
masterfrom
ops/tenant10-migration-20261030

Conversation

@pranems

@pranems pranems commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Summary

Prepare the existing tenant rollover entry points for the registered next tenant with safe provisioning, snapshot-bound database fidelity, explicit writer quiescence, owned cleanup and continuous offline regression coverage. This PR does not deploy an application, switch tenant roles, admit release 0.55.67 or change customer production.

Authoritative tooling disposition: exact tip 883fb6c64a55b05399142d45bbf3b8548c46cf8a passes all29 protected Validate gates in1133.6s. Full hosted41694778191, CodeQL41694778189 and freshness41694778305 pass. The actual default hosted PostgreSQL step emits the full fourteen-environment-value and worker-text assertion PASS at2026-11-03T18:33:10Z, followed by real identity, termination, extra-table rejection,2000-row fidelity and concurrent-write isolation through18:33:24Z. Independent exact-tip review5757623103 reports no open issues, confirming both latest findings resolved. Its state is COMMENTED, not formal APPROVED. Every inline thread is resolved with evidence. Tooling can be considered for ordinary reviewed merge; source/target provider bindings, writer quiescence, recovery readiness, release admission and actual target acceptance remain separate mandatory operational gates. No deployment, destructive restore, role cutover or customer-production action is claimed. Earlier paragraphs are historical evidence.

Latest status: e3fbf7c4 passes all29 protected gates in1133.1s, full hosted41689528595, CodeQL and freshness. Actual generated-YAML-to-PostgreSQL identity succeeds at2026-11-03T18:00:41Z; snapshot/signal/fidelity/isolation complete by18:00:47Z. Independent review5757233449 confirms High4458211788 resolved, but finds Medium4458623530: optional distinct-password/worker-text assertions are skipped by the default all path. An actual-default probe, with Docker intercepted before allocation, reproduces the intended RED. The two-line correction shares the assertions with all and returns early only for serialization; default-path probe and focused mode are now GREEN. Corrected protected publication, hosted default PostgreSQL and fresh independent review remain required. The PR stays draft. Subsequent pending statements describe earlier preparation, not current gate status.

Current correction: full hosted 41682922245, CodeQL and freshness passed at 7a938859. Fifth independent review confirms its parsed-login fix but reports High 4458211788: unescaped generated YAML can reinterpret otherwise valid connection values. All ten connection scalars now use JSON string serialization. The actual-generator assertion has behavioral RED/GREEN; pwsh -NoProfile -File scripts\test-tenant-copy-snapshot.ps1 -Case serialization passes the complete existing js-yaml parser for all 14 environment values, distinct raw/alternate synthetic passwords and actual worker text, without Docker. The real PostgreSQL fixture now consumes those actual YAML outputs and production URI/password helpers. Its hosted step follows existing API npm ci without adding dependencies or changing lockfiles. Twelve review cases run within eleven offline scripts. Corrected protected publication, hosted PostgreSQL/full CI and fresh independent review remain pending. The PR remains draft. Earlier paragraphs below are historical evidence, not current admission.

Independent review of the first hosted-green tip found six real integration defects. Published commit 05adbacd corrects them with actual-entry-point RED/GREEN coverage and passes all 29 protected Validate gates in 1,120 seconds. All six original threads have correction/test replies and are resolved. Corrected-tip hosted CI and refreshed review remain pending. The PR stays draft.

Update: hosted 41651296099 at 05adbacd passes, and refreshed review confirms the original six findings resolved. It identifies two new High issues and five previously missed Medium observations. Published commit 7f383242 addresses all seven: canonical database identities, portable Unix permissions, caller-wired subnet delegation, primary-first provisioning cleanup, idle input-held source snapshots with explicit signal/FIFO handling, accurate stubbed evidence labels and owned tmpfs fixtures. The 11-script offline gate (ten review cases), five compiled provisioning outcomes and all four real PostgreSQL termination/fidelity/isolation cases pass, including fixture ownership and exact-ID cleanup.

All 29 protected Validate gates pass in 1,144.8 seconds. Both second-review inline findings have evidence replies and are resolved; the five other observations have a consolidated evidence reply. Hosted 41660995714 passes at exact tip 7f383242, including validation, image build/push, container smoke and Trivy. CodeQL and freshness also pass. The third independent review confirms those two findings resolved but reports a new High connection-boundary issue: decoded whitespace can inject unquoted libpq parameters and override the validated host/database. This draft must not merge. A new real PostgreSQL identity regression is prepared locally; Docker is unresponsive, so behavioral RED/GREEN is not yet established and production code is unchanged.

Linked Defect / Plan Reference

  • DELIVERY_PLAN.md defect ID: N/A; authorized quarterly prov09-to-prov10 dev/canary rollover.
  • Audit doc reference: docs/TENANT_10_MIGRATION_EXECUTION_2026_10_30.md, including execution issues, measured evidence, open gates and recovery limitations.
  • ADR (if applicable): N/A; reuse existing estate registry, provisioning and carry entry points.

TDD Process

  • RED test written first; observed it fail for the expected reason
  • GREEN: minimal implementation that turns the test green
  • Refactor (if any) keeps the test green
  • Permanent regression guard added (where applicable - see existing patterns in api/src/security/)

Actual final hardening REDs:

  • Extracted production copy lifecycle returned fixture job cleanup failure instead of fixture primary transfer failure.
  • Real PostgreSQL fixture reported completed copy worker left 1 source transactions holding its exported snapshot.

Five lifecycle combinations now pass. The real database fixture verifies source transaction release after failed, successful and concurrent-write copies while retaining complete-manifest rejection, 2,000-row fidelity and exported-snapshot isolation. Six more review regression cases cover caller ACL failure policy, provisioning absence integration, pre-allocation private credential recovery, delegated job names, effective app/database identity and the bootstrap runtime boundary. Each reproduced the actual reviewed failure before its production correction.

Feature / Bug-Fix Commit Checklist (standing rule)

Every behavioral change must include all that apply. Tick or strike through with N/A + reason.

  • Unit Tests - eleven isolated offline PowerShell contract scripts exercise actual lifecycle and helper behavior; API service/controller changes are N/A.
  • E2E Tests - real PostgreSQL copy/preflight and compiled-template fixtures; new API HTTP behavior is N/A.
  • Live Integration Tests - actual target application acceptance and final frozen carry remain deployment gates, not claimed by this tooling PR.
  • Feature Documentation - migration execution document, deployment guides, infrastructure inventory and engineering lessons.
  • INDEX.md Update - migration document is indexed.
  • CHANGELOG.md Update - scoped Unreleased tooling entry; no fabricated release or aggregate test counts.
  • Session & Context Updates - current migration status and remaining release/deployment gates.
  • Version Management - N/A for tooling-only change; serving API/web and their versions and lockfiles are unchanged.
  • Response Contract Tests - N/A; no public API response changes.

Validation Output

Final published-tip commands and outcomes:

PREPUSH_MODE=Validate; DATABASE_URL=<non-secret localhost codegen URL>; git push
  PASS: 29/29 gates at883fb6c6 in1133.6 seconds.
pwsh -NoProfile -File scripts\test-tenant-copy-snapshot.ps1
  PASS in exact-tip hosted41694778191:
  default all-mode distinct-password/worker-text assertions, then actual
  PostgreSQL identity, termination, extra-table rejection,2000-row fidelity
  and concurrent-write isolation. No optional test mode was required.
gh pr checks 267 --repo pranems/SCIMServer
  PASS: validate, build-and-push, CodeQL analysis/aggregate and freshness.
Independent bot review5757623103 at883fb6c6:
  No open issues; COMMENTED, not formal APPROVED.
Target dev/canary deployment, final frozen carry and release0.55.67 admission:
  NOT RUN / NOT ESTABLISHED by this tooling PR.

Current focused correction: pwsh -NoProfile -File scripts\test-tenant-rollover-review.ps1 -Case jobyaml PASS after the expected actual-generator RED. pwsh -NoProfile -File scripts\test-tenant-copy-snapshot.ps1 -Case serialization PASS for complete YAML transport only, not a database test. Modified PowerShell syntax, workflow YAML parsing and git diff --check PASS. Full protected/hosted results for this correction are pending; historical results follow.

pwsh -NoProfile -File scripts\test-tenant-rollover.ps1
  PASS: 11 isolated offline contract scripts, including six independent-review regressions.
pwsh -NoProfile -File scripts\test-tenant-copy-snapshot.ps1
  PASS: worker TERM release, complete-manifest rejection, 2000-row fidelity,
        concurrent-write isolation and source transaction release after all exits.
        Unique run ownership, tmpfs PGDATA and exact container cleanup verified.
pwsh -NoProfile -File scripts\test-tenant-copy-preflight.ps1
  PASS: Docker-hosted shell contract with stubbed psql for alternate credential
        and enumeration failure decisions. This is NOT a real database fixture.
pwsh -NoProfile -File scripts\test-tenant-postgres-provisioning.ps1
  PASS: actual Bicep compilation and five provisioning controls, including
        immutable credential preservation and immediate connection recovery.
pwsh -NoProfile -File scripts\test-tenant-environment-template.ps1
  PASS: actual Consumption environment and Log Analytics template compilation.
PowerShell parser over 29 changed scripts: PASS.
Literal JSON documentation blocks: 1/1 PASS.
git diff --check: PASS before commit.
Six dependency manifest/lockfile hashes: unchanged after npm ci.
PREPUSH_MODE=Validate; DATABASE_URL=<non-secret localhost codegen URL>; git push
  PASS: 29/29 gates at f440c9bc in 1051.3 seconds.
  API unit: 6254 tests / 216 suites PASS.
  API inmemory HTTP: 2514 PASS / 42 explicit skips / 118 passed suites.
  Web vitest: 1661 tests / 125 files PASS.
Hosted run 41645490136 at f440c9bc: PASS, including validation, image build/push,
  container smoke and Trivy; CodeQL and freshness also PASS.
Independent review at f440c9bc: COMMENTED, six real findings, not approval.
Corrected 05adbacd protected publication: PASS, 29/29 gates in 1120 seconds.
Hosted run 41651296099 at05adbacd: PASS.
Refreshed review at05adbacd: original six resolved; two High and five Medium
  observations addressed in7f383242 with focused results above.
7f383242 protected publication: PASS, 29/29 gates in1144.8 seconds.
Hosted41660995714: PASS, exact tip7f383242. Validation, image build/push, container smoke and Trivy pass; CodeQL and freshness also pass.
Refreshed exact-tip independent bot review: changes recommended, new High finding4456642211. Empty author reply reviews are not independent review evidence.

Latest publication: f3ad6852 passes all29 protected gates in970.8 seconds. Hosted41674035838 executes the corrected SQL and reproduces the intended actual assertion: `RED: TARGET libpq parameters redirected the validated database/login identity.` This is the behavioral RED, unlike earlier harness failures.

Published correctionb8f9b847 quotes every libpq host/port/database/login/SSL-mode value and escapes quotes/backslashes in shared preflight/full-copy setup. All29 protected gates PASS in1118.6s. Hosted41678029555 confirms actual PostgreSQL step GREEN, including three literal identity combinations, TERM release, extra-table rejection,2000-row fidelity, concurrent-write isolation and fixture ownership/tmpfs/exact cleanup. The corrected third-review thread has actual RED/GREEN evidence and is resolved. Full hosted completion and refreshed independent bot review remain pending; resolving the thread does not replace them. The PR stays draft.

Fourth review atb8f9b847 confirms injection resolved but identifies Medium4457883278: login cases bypass URI username parsing. Actual-parser userinfo RED/GREEN covers one-time decoding with password semantics unchanged. Published correction7a938859 encodes each actual fixture username in its URI and forwards parsed usernames into the worker. All29 protected gates PASS in817.2s, including11 review cases within the11-script gate. Hosted41682922245 confirms actual PostgreSQL step GREEN with the complete parsed-login path. The corrected login thread has actual evidence and is resolved. Full hosted completion and refreshed independent bot review remain pending; the PR stays draft.

Obsolete unmergeable hosted41678029555 was cancelled after its PostgreSQL step passed but its web Vitest step did not complete. This is not a full hosted success. Its evidence is retained separately; new7a938859 must complete the normal exact-tip hosted matrix. No failure, test or required gate is waived.

RED-first follow-up: test-onlyfc4e7c32 adds actual parser-to-libpq database/login identity controls and an ordinary hosted PostgreSQL fixture step. Production remains unchanged. Its first protected publication was blocked after28 passing gates by15 web worker-startup timeouts, not failing assertions. All352 tests in the15 missing files subsequently pass with bounded diagnostic concurrency. Commitd70f983b records that RCA; normal full protected publication is running again. Local Docker remains unresponsive and has not been restarted.

The offline contract entry point is wired into both protected pre-push and hosted CI. The database and Bicep fixture results are independent integration claims, not implied by that offline gate.

The retained local PostgreSQL browser acceptance is 348/348 with no skips, failures or flaky results. It is not target Azure acceptance and does not admit a release. The failed current-image qualification and non-admitting warmup diagnostic remain unchanged. No target app is serving.

Standing Rules Acknowledgment

  • No em-dash (Unicode U+2014) anywhere in the diff. Verified via Select-String -Pattern ([char]0x2014).
  • No git commit --amend on history that has been pushed.
  • No new credentials, tokens, or secrets committed to source. Forbidden patterns regression spec at api/src/security/forbidden-source-patterns.spec.ts will catch known anti-patterns.
  • Migrations are additive-only (expand-contract for column changes). Migration linter at api/src/scripts/lint-migrations.ts enforces this. No Prisma migration is added.

Destructive Migration Override (skip if not applicable)

  • This PR contains a NEW destructive Prisma migration (DROP TABLE/COLUMN, ALTER COLUMN ... TYPE, RENAME, INSERT ... SELECT FROM)
  • I have set ALLOW_DESTRUCTIVE_MIGRATION=1 in the CI environment for this PR
  • Justification (required if box above is ticked):
    <!-- Why this destructive change is necessary, what the rollback path is, and which release window it ships in -->
    

N/A: no Prisma schema or migration change. Existing database carry is explicitly guarded by target identity, writer quiescence and complete snapshot comparison.

Reviewer Hint

Diff areas worth special attention from the reviewer:

  • Registry-derived identities and paired next-to-active/active-to-retiring role transition; no actual role transition in this PR.
  • Rejection of permanent customer-production targets, same source/target databases, ambiguous or empty Azure results and unproven no-app assumptions.
  • Private credential/parameter handling, administrator-password exclusion from native command arguments and template outputs, effective directory permission checks.
  • Actual exported snapshot shared by manifest and dump, extra-table rejection and source transaction cancellation via the owned client only.
  • Failure paths: preserve original work errors, report secondary cleanup failures, attempt independent cleanup and never reactivate target writers after failed work.
  • Eleven offline contract scripts are continuous pre-push/CI evidence; real database/template, serving application, release, final freeze and target browser claims remain separate.
  • Provisioning refuses existing/unverifiable servers before deployment, persists/verifies owner-restricted server-scoped credentials before allocation and a separate URI record before extensions, and propagates ACL/file errors even from permissive callers.
  • App/database identity uses actual direct or secret-backed DATABASE_URL, with host/port/database checks before suspension or restore. Bootstrap and deployment require PowerShell 7.2+ rather than the unsupported Windows PowerShell fallback.
  • Test/gate self-improvement applied with combined-failure and source-transaction-release assertions. Design disposition accepted: retain cohesive existing orchestrator/helper; a new abstraction without a second caller would be speculative.

Prashant Rane (ALLYIS INC) and others added 2 commits November 3, 2026 07:02
Preserve primary copy failures, release owned exported snapshots, verify complete snapshot fidelity and run offline rollover contracts in pre-push and CI. Keep tenant role cutover and serving deployment gated separately.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record the failed and corrected code-generation environment without claiming completed publication or target acceptance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical findings risk plaintext credential exposure, unintended password resets, and destructive restores with active writers, while invalid generated names block rollover.

6 open issues
What changed in this PR

Prepares SCIMServer’s existing deployment and database-copy tooling for registry-driven dev/canary tenant rollover. It does not activate the next tenant, deploy an application, admit a release, or change customer production.

Changes:

  • Adds next-tenant registry entries and paired activation/retirement support.
  • Hardens PostgreSQL provisioning, snapshot-based copy verification, writer suspension, credential handling, and cleanup.
  • Adds offline regression contracts to pre-push and CI, separate integration fixtures, and migration documentation.
File Description
setup.ps1 Downloads new provisioning helper dependencies.
Session_starter.md Records migration evidence and remaining gates.
scripts/​wif-e2e-proof.ps1 Reuses the shared JWT decoder.
scripts/​test-tenant-rollover.ps1 Runs ten isolated offline contracts.
scripts/​test-tenant-registry-resolution.ps1 Checks registry-derived tenant identities and scopes.
scripts/​test-tenant-proof-metadata.ps1 Tests token metadata validation.
scripts/​test-tenant-postgres-provisioning.ps1 Checks compiled outputs and private provisioning parameters.
scripts/​test-tenant-postgres-absence.ps1 Tests explicit server-absence decisions.
scripts/​test-tenant-environment-template.ps1 Checks compiled Consumption and logging configuration.
scripts/​test-tenant-directory-grant.ps1 Tests effective grants and repeatable permission setup.
scripts/​test-tenant-cutover.ps1 Tests paired role changes and rejected transitions.
scripts/​test-tenant-credential-storage.ps1 Tests private credential-file protection.
scripts/​test-tenant-copy-snapshot.ps1 Tests database fidelity, snapshot isolation, and transaction release.
scripts/​test-tenant-copy-safety.ps1 Tests writer suspension and owned-job cleanup.
scripts/​test-tenant-copy-preflight.ps1 Tests shell-level password fallback and enumeration failures.
scripts/​test-tenant-copy-no-apps.ps1 Tests fail-closed absent-app checks.
scripts/​test-tenant-copy-log-transport.ps1 Tests logging-command availability checks.
scripts/​test-tenant-copy-cleanup.ps1 Tests combined transfer and cleanup failures.
scripts/​tenant-copy-safety.ps1 Adds shared CLI, suspension, privacy, and cleanup helpers.
scripts/​setup-deploy-sp.ps1 Hardens identity checks and credential persistence.
scripts/​setup-auth-proof-apps.ps1 Supports the next tenant and validates actual token metadata.
scripts/​scim-estates.ps1 Supports paired tenant activation and retirement.
scripts/​scim-estates.json Registers the next tenant and staged estates.
scripts/​rotate-tenant-data.ps1 Adds guarded snapshot copying and failure-safe cleanup.
scripts/​replicate-estate.ps1 Prints the paired cutover command.
scripts/​pre-push-checks.ps1 Adds the offline rollover gate.
scripts/​postgres-provisioning.ps1 Adds private parameter transport and local URI construction.
scripts/​jwt-parts.ps1 Extracts diagnostic JWT decoding helpers.
scripts/​grant-deploy-sp-directory-access.ps1 Resolves, assigns, and verifies the selected directory role.
scripts/​deploy-azure.ps1 Uses masked input and shared PostgreSQL provisioning.
scripts/​az-tenant.ps1 Derives tenant mappings from the estate registry.
infra/​postgres.bicep Adds optional zone placement and removes secret-bearing output.
infra/​containerapp-env.bicep Explicitly declares the Consumption workload profile.
docs/​strategy/​ENGINEERING_LESSONS_AND_PATTERNS.md Records migration safety and evidence lessons.
docs/​SOVEREIGN_AND_GOV_CLOUD_DEPLOYMENT.md Clarifies cloud-specific validation requirements.
docs/​INDEX.md Links the migration execution plan.
docs/​DEPLOYMENT_INFRASTRUCTURE_AND_FORM_FACTORS.md Documents staged infrastructure and validation boundaries.
docs/​CONTEXT_INSTRUCTIONS.md Updates rollover status and release holds.
docs/​AZURE_DEPLOYMENT_AND_USAGE_GUIDE.md Explains environment and provisioning changes.
DEPLOYMENT.md Documents private PostgreSQL provisioning transport.
CHANGELOG.md Adds an unreleased tooling summary.
.github/​workflows/​build-test.yml Runs offline rollover contracts in hosted CI.

Review effort: Balanced

Comment thread scripts/postgres-provisioning.ps1 Resolved
Comment thread scripts/postgres-provisioning.ps1 Resolved
Comment thread scripts/rotate-tenant-data.ps1 Resolved
Comment thread scripts/rotate-tenant-data.ps1 Resolved
Comment thread scripts/deploy-azure.ps1 Resolved
Comment thread scripts/deploy-azure.ps1 Resolved
Fail closed on credential ACL errors, wire target absence into provisioning, retain verified pre-allocation credentials and immediate URI recovery, bind restored database to the suspended app, bound delegated job names and require PowerShell 7.2+. Six actual-entry-point RED/GREEN controls join continuous offline gates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A zero-padded port can bypass the same-database safeguard and permit a destructive restore onto the source, while credential persistence uses APIs unavailable in PowerShell 7.2.

2 open issues
5 previously missed issues

In code that hasn't changed since last review

Medium severity Existing VNet path does not ensure subnet delegation

infra/​containerapp-env.bicep:43

Declaring workload profiles makes the infrastructure subnet's Microsoft.App/environments delegation required, but the existing-VNet path in deploy-azure.ps1:523-553 accepts any existing subnet and creates missing ones without a delegation. A deployment that reuses a VNet without a delegated aca-infra subnet will therefore fail environment creation, despite the fresh-VNet Bicep path being valid. Validate/ensure the required delegation in the reuse path, or reject it before allocation, and cover that path alongside the compiled-template check.

Medium severity Cleanup failure masks provisioning error and skips directory cleanup

scripts/​postgres-provisioning.ps1:96

A parameter-file removal failure here replaces any original allocation or recovery error and prevents directory cleanup from being attempted. Preserve the provisioning error, attempt each cleanup independently, and report secondary failures while still failing when cleanup alone fails. Add a combined deployment/cleanup failure regression.

Medium severity Cancel owned client on worker termination

scripts/​rotate-tenant-data.ps1:297

The owned-client cancellation runs only on shell EXIT, not on termination signals. If the coordinator deletes a still-running job after its 45-minute deadline, container shutdown can bypass this trap and leave the exported source transaction executing pg_sleep(3600), retaining its snapshot and delaying vacuum. Handle worker termination explicitly, cancel the owned client before shutdown, and add a regression that terminates a running copy rather than testing only normal success/error exits.

Medium severity Stubbed psql does not verify real PostgreSQL preflight

scripts/​test-tenant-copy-preflight.ps1:25

This runner replaces psql with a stub and starts no PostgreSQL server, so neither password fallback nor enumeration failure is exercised against a real database. The PR's validation output labels this command as real PostgreSQL preflight evidence, which it does not establish. Either add an actual database-backed preflight fixture or label this result as a stubbed shell contract and keep real preflight evidence separate.

Medium severity Detached database can survive without ownership metadata

scripts/​test-tenant-copy-snapshot.ps1:46

This detached database uses PostgreSQL's default on-disk anonymous volume and has no run ownership labels. If the runner is interrupted before finally, --rm does not stop the container, so the database and writable storage can survive without discoverable ownership metadata. Give the fixture a unique task/run-labelled name, place PGDATA on tmpfs, and verify ownership and exact container-ID absence during cleanup.

6 resolved since last review

Review effort: Balanced

Comment thread scripts/rotate-tenant-data.ps1 Resolved
Comment thread scripts/setup-deploy-sp.ps1 Resolved
Close second-review boundaries with canonical ports/database names, portable credential permissions, caller-wired subnet delegation and primary-first provisioning cleanup. Replace server-side snapshot sleep with idle client input, explicit signal handling and owned FIFO cleanup. Verify owned tmpfs fixture lifecycle and classify stubbed preflight evidence accurately.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@pranems

pranems commented Nov 3, 2026

Copy link
Copy Markdown
Owner Author

All five previously missed second-review observations are addressed in7f383242: required subnet delegation is wired through the actual infrastructure caller and checked on reuse/create; provisioning preserves work errors while attempting both cleanup operations, with combined/cleanup-only controls; idle FIFO-held snapshot input replaces server pg_sleep with explicit signal handling and owned unlink, and the real PostgreSQL run passes TERM plus all three fidelity/isolation cases; preflight evidence is correctly labelled a Docker shell contract with stubbed psql, separate from real PostgreSQL/Azure receipts; and detached fixtures have unique ownership/name, tmpfs PGDATA, ownership-checked deletion and exact-ID absence checks. All29 protected gates PASS in1144.8s. Hosted41660995714 and refreshed review are pending; no waiver, merge or deployment. Tooling note: the first thread-disposition GraphQL query had a missing closing brace and changed no threads; corrected multiline query succeeds.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An active critical issue allows encoded whitespace in the database name to override the validated host, potentially directing a destructive restore to the source despite separation checks.

1 open issue
2 resolved since last review

Review effort: Balanced

Comment thread scripts/rotate-tenant-data.ps1 Resolved
Prashant Rane (ALLYIS INC) and others added 4 commits November 3, 2026 09:27
Add the real parser-to-libpq regression before changing production. Hosted integration is expected to expose the new review finding; local Docker is unresponsive and is not restarted. Keep offline contracts and real PostgreSQL evidence distinct. Design accepted: retain the shared worker boundary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retain the fifteen worker-startup failures separately from the intended libpq assertion RED. All352 tests in the missing15 files pass under bounded diagnostic concurrency; the unchanged full protected gate remains required.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the current_user SQL keyword without parentheses and distinguish setup errors from identity assertion RED. Bound hosted integration to three minutes; preserve the real harness-failure RCA. Production copy code remains unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hosted41674035838 reproduces actual target-identity assertion RED before this change. Quote every connection field and escape quotes/backslashes in shared preflight/full-copy setup. Full worker POSIX syntax and exact escaping controls pass; real PostgreSQL GREEN and full protected/hosted review are pending. Self-improvement applied through continuous parser-to-worker identity readback; design accepted: reuse the existing setup seam.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The login tests bypass production username parsing, masking a percent-decoding bug that prevents encoded special-character usernames from authenticating through the actual copy entry point.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread scripts/test-tenant-copy-snapshot.ps1 Resolved
Actual production-parser username assertion is RED before one-time decoding and GREEN afterwards, including encoded delimiters and literal percent escapes without changing password semantics. Encode fixture usernames in actual URIs and forward parsed identities, eliminating expected-value replacement. Record prior real libpq GREEN and the new review RCA. Full corrected-tip gates/database/review remain required. Design accepted: retain parser normalization and worker serialization as separate existing boundaries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unescaped database and login values can be altered or produce invalid job YAML, while the snapshot test bypasses serialization and therefore misses the defect.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread scripts/rotate-tenant-data.ps1 Resolved
JSON-string encode all connection scalars. Traverse the actual job generator, complete YAML parser, secret references and worker text in PostgreSQL regression fixtures. Record fifth-review RED/GREEN and remaining release/deployment gates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The default CI test path skips the distinct-password serialization assertion, allowing swapped secret values or references to pass undetected.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread scripts/test-tenant-copy-snapshot.ps1 Resolved
Reproduce default-mode assertion omission before intercepted Docker allocation, then execute the existing distinct-password and worker-text checks in all mode. Retain the focused early return and record actual hosted YAML-to-PostgreSQL evidence plus review RCA.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Destructive cross-tenant restore requires operator confirmation of Azure database bindings, writer quiescence, and recovery readiness because offline tests cannot establish provider guarantees, and hosted validation and target acceptance remain pending.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@pranems
pranems marked this pull request as ready for review November 3, 2026 18:42
@pranems
pranems merged commit 8409929 into master Nov 3, 2026
6 checks passed
@pranems

pranems commented Nov 3, 2026

Copy link
Copy Markdown
Owner Author

Postmerge acceptance: merged master84099293570af5ab06f7d8e2f93e7e6ae2c0236f passes authoritative BuildTestImage41696483989, CodeQL41696338523 and freshness41696488100. Default YAML/password/worker transport and real PostgreSQL identity/signal/manifest/fidelity/isolation pass through2026-11-03T18:44:18Z. Image smoke and HIGH/CRITICAL Trivy pass. The unique tooling test image test-tenant10-tooling-84099293570a was pushed and smoke-pulled as sha256:225ebe85030d76593dc1256f19e641f74ba3d886f53796443c6ca8dc68eed1e3. This unchanged-version tooling artifact is not an admitted0.55.67 release or a substitute. No performance campaign, deployment, final frozen carry, role cutover or customer-production action was performed. Actual provider bindings, writer quiescence/recovery readiness and target acceptance remain mandatory operational gates. Direct package REST inventory is unavailable with current read scopes; digest evidence comes from the already authorized CI producer and smoke-consumer logs, not a package-permission bypass. The local execution handoff retains these exact results and limitations for normal later documentation publication.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants