diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 9d9044607..b189e420f 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -371,6 +371,58 @@ cd terraform/environments/aws terraform apply -replace="module.compute_lambda[0].aws_lambda_function.main" -auto-approve ``` +### Release ordering: Redshift re-drive (#1832) after the attempt journal (#1824) + +A Redshift purchase dispatched before the attempt journal existed in an +environment has no journal record. If AWS changes that tuple's offering within +the 24h re-drive horizon, a Retry of such a row could buy a second reserved +node. So in every environment (dev, staging, prod, any other): + +- Deploy the release that contains #1832 (it lifts the Redshift re-drive and + Retry refusal) no earlier than 24h after the release that contains #1824 and + migration 000161 is live there. +- Never ship #1824 and #1832 in the same release, or #1832 within 24h of #1824. +- Start the 24h clock when migration 000161 is confirmed applied in that + environment, not when the deploy finishes. Migrations run at app start and a + failed run is only reported by the `migrations` check of `/health`, so a + finished deploy is not proof. Confirm with `SELECT version, dirty FROM + schema_migrations;` (version 161 or higher and `dirty` false). Confirming + later only lengthens the wait. The oldest `purchase_attempts` row is not + proof either: the table stays empty until the first Redshift purchase after + the deploy. + +Both changes are already on `main` (#1824 is `71b5e29a`, #1832 is `af8f5434`), +so an environment still running a build from before #1824 cannot take both from +one deploy of `main`. Do it in two deploys: + +1. Deploy `6fd9a2ef` (the parent of `af8f5434`: it has #1824 but not #1832). + Use the Rollback section below (`git checkout 6fd9a2ef`, then + `./scripts/tf-deploy.sh `, for example `aws prod`). + The CI deploy workflow deploys the ref it is dispatched on, and whether a + ref at that commit passes an environment's deploy trust conditions depends + on the environment, so check before relying on it. +2. Wait 24h from the moment 000161 is confirmed applied, then `git checkout main` + and deploy it the same way. + +An environment with no Redshift row in `purchase_executions` has no such rows +and can skip the wait. Check with: + +```sql +SELECT count(*) FROM purchase_executions e, + jsonb_array_elements(CASE WHEN jsonb_typeof(e.recommendations) = 'array' + THEN e.recommendations ELSE '[]'::jsonb END) r + WHERE r->>'service' IN ('redshift', 'data-warehouse'); +``` + +A count of 0 means the exemption applies; anything else means wait. + +Rolling back to a build without #1824 (see Rollback below) leaves Redshift rows +dispatched in the meantime without a journal record. After such a rollback the +24h wait restarts from the next deploy of #1824. + +Dev was handled by merge timing (#1824 went live there at 2026-11-06T23:38:42Z). +See `known_issues/39_redshift_dedupe_window.md`. + ### CI/CD Integration ```yaml diff --git a/known_issues/39_redshift_dedupe_window.md b/known_issues/39_redshift_dedupe_window.md index f078b7ffe..8fa3c95a6 100644 --- a/known_issues/39_redshift_dedupe_window.md +++ b/known_issues/39_redshift_dedupe_window.md @@ -21,6 +21,7 @@ - The window is measured from the other row's last save, so a later save of that row (for example a revocation request) restarts it. That only refuses longer; it never allows a double buy. - A `failed` row without the no-submission proof (reaped mid-buy, or an unknown outcome) may hold a real node. Inside the window the gate refuses it like a finished row, settled or not and whether or not Retry is refused (#1839); outside it, the gate counts it as an open lineage only while Retry can still re-drive it (below). - Now that the Redshift re-drive guard is lifted, a retry can be refused for up to 24h while another lineage's completed node sits inside its window (liveness only, no double buy). +- Deploy ordering: a Redshift row dispatched before the attempt journal (#1824, migration 000161) is live in an environment has no journal record, so #1832 (the lifted re-drive refusal) must be deployed there no earlier than 24h after #1824; otherwise a Retry inside the 24h horizon could buy a second node if AWS changed the offering. Recorded as a step in `docs/DEPLOYMENT.md` (#1835). - Plan rows that PR2b refused before submission (failed, `no_submission`, attempt 0) used to hold their ramp step until the account was detached. They are now retryable, so that hold clears with a Retry once the other node leaves its window. **Status:** Mitigated for platform-recorded buys. Tracked in #702 and LeanerCloud/cloud-commitments-go#515.