Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,58 @@ cd terraform/environments/aws
terraform apply -replace="module.compute_lambda[0].aws_lambda_function.main" -auto-approve
```

### Release ordering: Redshift re-drive (#1832) after the attempt journal (#1824)

A Redshift purchase dispatched before the attempt journal existed in an
environment has no journal record. If AWS changes that tuple's offering within
the 24h re-drive horizon, a Retry of such a row could buy a second reserved
node. So in every environment (dev, staging, prod, any other):

- Deploy the release that contains #1832 (it lifts the Redshift re-drive and
Retry refusal) no earlier than 24h after the release that contains #1824 and
migration 000161 is live there.
- Never ship #1824 and #1832 in the same release, or #1832 within 24h of #1824.
- Start the 24h clock when migration 000161 is confirmed applied in that
environment, not when the deploy finishes. Migrations run at app start and a
failed run is only reported by the `migrations` check of `/health`, so a
finished deploy is not proof. Confirm with `SELECT version, dirty FROM
schema_migrations;` (version 161 or higher and `dirty` false). Confirming
later only lengthens the wait. The oldest `purchase_attempts` row is not
proof either: the table stays empty until the first Redshift purchase after
the deploy.

Both changes are already on `main` (#1824 is `71b5e29a`, #1832 is `af8f5434`),
so an environment still running a build from before #1824 cannot take both from
one deploy of `main`. Do it in two deploys:

1. Deploy `6fd9a2ef` (the parent of `af8f5434`: it has #1824 but not #1832).
Use the Rollback section below (`git checkout 6fd9a2ef`, then
`./scripts/tf-deploy.sh <cloud> <environment>`, for example `aws prod`).
The CI deploy workflow deploys the ref it is dispatched on, and whether a
ref at that commit passes an environment's deploy trust conditions depends
on the environment, so check before relying on it.
2. Wait 24h from the moment 000161 is confirmed applied, then `git checkout main`
and deploy it the same way.

An environment with no Redshift row in `purchase_executions` has no such rows
and can skip the wait. Check with:

```sql
SELECT count(*) FROM purchase_executions e,
jsonb_array_elements(CASE WHEN jsonb_typeof(e.recommendations) = 'array'
THEN e.recommendations ELSE '[]'::jsonb END) r
WHERE r->>'service' IN ('redshift', 'data-warehouse');
```

A count of 0 means the exemption applies; anything else means wait.

Rolling back to a build without #1824 (see Rollback below) leaves Redshift rows
dispatched in the meantime without a journal record. After such a rollback the
24h wait restarts from the next deploy of #1824.

Dev was handled by merge timing (#1824 went live there at 2026-11-06T23:38:42Z).
See `known_issues/39_redshift_dedupe_window.md`.

### CI/CD Integration

```yaml
Expand Down
1 change: 1 addition & 0 deletions known_issues/39_redshift_dedupe_window.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
- The window is measured from the other row's last save, so a later save of that row (for example a revocation request) restarts it. That only refuses longer; it never allows a double buy.
- A `failed` row without the no-submission proof (reaped mid-buy, or an unknown outcome) may hold a real node. Inside the window the gate refuses it like a finished row, settled or not and whether or not Retry is refused (#1839); outside it, the gate counts it as an open lineage only while Retry can still re-drive it (below).
- Now that the Redshift re-drive guard is lifted, a retry can be refused for up to 24h while another lineage's completed node sits inside its window (liveness only, no double buy).
- Deploy ordering: a Redshift row dispatched before the attempt journal (#1824, migration 000161) is live in an environment has no journal record, so #1832 (the lifted re-drive refusal) must be deployed there no earlier than 24h after #1824; otherwise a Retry inside the 24h horizon could buy a second node if AWS changed the offering. Recorded as a step in `docs/DEPLOYMENT.md` (#1835).
- Plan rows that PR2b refused before submission (failed, `no_submission`, attempt 0) used to hold their ramp step until the account was detached. They are now retryable, so that hold clears with a Retry once the other node leaves its window.

**Status:** Mitigated for platform-recorded buys. Tracked in #702 and LeanerCloud/cloud-commitments-go#515.
Expand Down
Loading