From 85df3e44145f9a94742bf63c97ae9bbd1fdd2789 Mon Sep 17 00:00:00 2001 From: Cristian Magherusan-Stanciu Date: Mon, 9 Nov 2026 14:10:21 +0100 Subject: [PATCH 1/2] docs(deploy): order the Redshift re-drive release after the attempt journal A Redshift row dispatched before the attempt journal (#1824, migration 000161) is live in an environment has no journal record, so #1832 must be deployed there at least 24h later. Add the two-step deploy (the parent of #1832, wait 24h, then main), the never-dispatched exemption and the rollback restart to DEPLOYMENT.md, and record the ordering in known_issues/39. Closes #1835 Co-Authored-By: Claude Sonnet 5.5 --- docs/DEPLOYMENT.md | 32 +++++++++++++++++++++++ known_issues/39_redshift_dedupe_window.md | 1 + 2 files changed, 33 insertions(+) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 9d9044607..93a5eb3fe 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -371,6 +371,38 @@ cd terraform/environments/aws terraform apply -replace="module.compute_lambda[0].aws_lambda_function.main" -auto-approve ``` +### Release ordering: Redshift re-drive (#1832) after the attempt journal (#1824) + +A Redshift purchase dispatched before the attempt journal existed in an +environment has no journal record. If AWS changes that tuple's offering within +the 24h re-drive horizon, a Retry of such a row could buy a second reserved +node. So in every environment (dev, staging, prod, any other): + +- Deploy the release that contains #1832 (it lifts the Redshift re-drive and + Retry refusal) no earlier than 24h after the release that contains #1824 and + migration 000161 is live there. +- Never ship #1824 and #1832 in the same release, or #1832 within 24h of #1824. +- Take the "live since" time from that environment's deploy history. The oldest + `purchase_attempts` row is not proof: the table stays empty until the first + Redshift purchase after the deploy. + +Both changes are already on `main` (#1824 is `71b5e29a`, #1832 is `af8f5434`), +so an environment still running a build from before #1824 cannot take both from +one deploy of `main`. Do it in two deploys: + +1. Deploy `6fd9a2ef` (the parent of `af8f5434`: it has #1824 but not #1832). +2. Wait 24h from that deploy, then deploy `main`. + +An environment that has never dispatched a Redshift purchase through this platform has no such rows +and can skip the wait. + +Rolling back to a build without #1824 (see Rollback below) leaves Redshift rows +dispatched in the meantime without a journal record. After such a rollback the +24h wait restarts from the next deploy of #1824. + +Dev was handled by merge timing (#1824 went live there at 2026-11-06T23:38:42Z). +See `known_issues/39_redshift_dedupe_window.md`. + ### CI/CD Integration ```yaml diff --git a/known_issues/39_redshift_dedupe_window.md b/known_issues/39_redshift_dedupe_window.md index f078b7ffe..8fa3c95a6 100644 --- a/known_issues/39_redshift_dedupe_window.md +++ b/known_issues/39_redshift_dedupe_window.md @@ -21,6 +21,7 @@ - The window is measured from the other row's last save, so a later save of that row (for example a revocation request) restarts it. That only refuses longer; it never allows a double buy. - A `failed` row without the no-submission proof (reaped mid-buy, or an unknown outcome) may hold a real node. Inside the window the gate refuses it like a finished row, settled or not and whether or not Retry is refused (#1839); outside it, the gate counts it as an open lineage only while Retry can still re-drive it (below). - Now that the Redshift re-drive guard is lifted, a retry can be refused for up to 24h while another lineage's completed node sits inside its window (liveness only, no double buy). +- Deploy ordering: a Redshift row dispatched before the attempt journal (#1824, migration 000161) is live in an environment has no journal record, so #1832 (the lifted re-drive refusal) must be deployed there no earlier than 24h after #1824; otherwise a Retry inside the 24h horizon could buy a second node if AWS changed the offering. Recorded as a step in `docs/DEPLOYMENT.md` (#1835). - Plan rows that PR2b refused before submission (failed, `no_submission`, attempt 0) used to hold their ramp step until the account was detached. They are now retryable, so that hold clears with a Retry once the other node leaves its window. **Status:** Mitigated for platform-recorded buys. Tracked in #702 and LeanerCloud/cloud-commitments-go#515. From 2da74161083056bc6721bfb5d593d8c73f796f80 Mon Sep 17 00:00:00 2001 From: Cristian Magherusan-Stanciu Date: Mon, 9 Nov 2026 17:17:17 +0100 Subject: [PATCH 2/2] docs(deploy): start the 24h clock at the confirmed migration and say how to deploy Address the cold review of the ordering step: count the 24h from when migration 000161 is confirmed applied (a finished deploy is not proof, migrations are non-fatal at startup), say how to deploy the parent commit and return to main, and give the never-dispatched exemption a check (guarded against non-array recommendations rows). Refs #1835 Co-Authored-By: Claude Sonnet 5.5 --- docs/DEPLOYMENT.md | 32 ++++++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 93a5eb3fe..b189e420f 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -382,19 +382,39 @@ node. So in every environment (dev, staging, prod, any other): Retry refusal) no earlier than 24h after the release that contains #1824 and migration 000161 is live there. - Never ship #1824 and #1832 in the same release, or #1832 within 24h of #1824. -- Take the "live since" time from that environment's deploy history. The oldest - `purchase_attempts` row is not proof: the table stays empty until the first - Redshift purchase after the deploy. +- Start the 24h clock when migration 000161 is confirmed applied in that + environment, not when the deploy finishes. Migrations run at app start and a + failed run is only reported by the `migrations` check of `/health`, so a + finished deploy is not proof. Confirm with `SELECT version, dirty FROM + schema_migrations;` (version 161 or higher and `dirty` false). Confirming + later only lengthens the wait. The oldest `purchase_attempts` row is not + proof either: the table stays empty until the first Redshift purchase after + the deploy. Both changes are already on `main` (#1824 is `71b5e29a`, #1832 is `af8f5434`), so an environment still running a build from before #1824 cannot take both from one deploy of `main`. Do it in two deploys: 1. Deploy `6fd9a2ef` (the parent of `af8f5434`: it has #1824 but not #1832). -2. Wait 24h from that deploy, then deploy `main`. + Use the Rollback section below (`git checkout 6fd9a2ef`, then + `./scripts/tf-deploy.sh `, for example `aws prod`). + The CI deploy workflow deploys the ref it is dispatched on, and whether a + ref at that commit passes an environment's deploy trust conditions depends + on the environment, so check before relying on it. +2. Wait 24h from the moment 000161 is confirmed applied, then `git checkout main` + and deploy it the same way. -An environment that has never dispatched a Redshift purchase through this platform has no such rows -and can skip the wait. +An environment with no Redshift row in `purchase_executions` has no such rows +and can skip the wait. Check with: + +```sql +SELECT count(*) FROM purchase_executions e, + jsonb_array_elements(CASE WHEN jsonb_typeof(e.recommendations) = 'array' + THEN e.recommendations ELSE '[]'::jsonb END) r + WHERE r->>'service' IN ('redshift', 'data-warehouse'); +``` + +A count of 0 means the exemption applies; anything else means wait. Rolling back to a build without #1824 (see Rollback below) leaves Redshift rows dispatched in the meantime without a journal record. After such a rollback the