From ebf75936b1b6a1c8eda2ec38d8ec39cd07ffc799 Mon Sep 17 00:00:00 2001 From: DJ Mountney Date: Thu, 12 Nov 2026 11:03:05 -0700 Subject: [PATCH] docs: list the 2026-11-10 index builds and migrations with measured times [ENG-2184] Measured on a 2026-10-14-002 Compose install with 3.1M instances, upgraded to release/2026-11-10-003: the scheduler finished startup 18 s after `docker compose up -d`, and rolling back to 2026-10-14 built and dropped nothing. Co-Authored-By: Claude Opus 5.5 --- docs/upgrading.md | 35 +++++++++++++++++++++++++++++++++++ on-prem/CHANGELOG.md | 4 ++++ 2 files changed, 39 insertions(+) diff --git a/docs/upgrading.md b/docs/upgrading.md index f56f659..0d3be8d 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -99,6 +99,39 @@ Check running container versions: docker compose ps --format "table {{.Name}}\t{{.Image}}" ``` +## Upgrading from 2026-10-14 to 2026-11-10 + +`2026-11-10` builds six MongoDB indexes and runs five migrations when the scheduler starts. None of the indexes is on `instances`, `runs` or `testRecord`, and the migrations read only keys and secrets, so there is nothing to build ahead of the upgrade. On a test install with 3.1 million `instances` (41 GB), 310,000 runs and 124,000 orchestration sessions, the scheduler logged `Startup tasks complete` 18 seconds after `docker compose up -d`. + +Run the check in [Index Builds Before an Upgrade](#index-builds-before-an-upgrade) before `docker compose up -d`. On the test install it exited `0` with an estimate of 25 seconds, and the builds took 1 second. + +Of this release's indexes, only the one on `orchestrationSession` grows with your history, and only if you use orchestration. At the check's default rate the collection would need about 18 GB to pass the 10-minute limit. If the check exits `2` and names `orchestrationSession`, build it first as in [Building the Indexes First](#building-the-indexes-first). + +### What the scheduler builds and runs + +| Index | Collection | +|---|---| +| `orchestrationSession` `{ runId: 1 }` | Grows with orchestrated runs. 0.79 GB took about 1 second in testing. | +| `oauthRefreshToken` `{ rotationReplayExpiresAt: 1 }` | Small: refresh tokens of MCP clients. | +| `orgStorageDaily` `{ date: 1, orgId: 1, bytes: 1 }` | Empty on-prem. | +| `scimPendingUsers` `{ orgId: 1, email: 1 }` (unique) | Written only by SCIM provisioning. | +| `customerRequests`, two indexes | Empty on-prem. | + +| Migration | What it does | Time in testing | +|---|---|---| +| `20261028-drop-superseded-instances-indexes` | Drops two `instances` indexes that newer ones replace, once the replacements exist. `2026-10-14` already dropped them, so it finds nothing. | 3 ms | +| `20261103-redact-audit-log-secrets` | Masks credentials in stored audit log records. On-prem doesn't store audit records in MongoDB, so it usually finds none. | 553 ms for 10,000 records | +| `20261109-backfill-api-key-hashes` | Stores a hash and the last 4 characters of each organization API key. | 202 ms for 3,000 keys | +| `20261110-backfill-record-key-hashes` | Stores a hash and the last 4 characters of each record key. | 25 ms for 400 keys | +| `20261110-backfill-scim-secret-hashes` | Stores a hash and the last 4 characters of the SCIM secret. | 2 ms | + +The two key migrations leave a key without a hash when another document holds the same value, and log a warning with the documents' ids: + +- `left N organization API key(s) in M group(s) without a hash`: API keys that share a value already fail to authenticate. Delete the extra copies. +- `left N record key(s) without a hash`: a record key shared by two organizations. Have all but one of them rotate their record key; deleting a copy breaks that organization's CI. + +Still on `2026-07-26`? The scheduler also builds the `2026-10-14` indexes on `instances`, `runs` and `testRecord`: estimate and build them first as in [Upgrading from 2026-07-26 to 2026-10-14](#upgrading-from-2026-07-26-to-2026-10-14). `2026-11-10` builds the replacement `instances` indexes before it drops the old ones, so reading a run's spec files keeps an index throughout. + ## Upgrading from 2026-07-26 to 2026-10-14 Upgrade to `2026-10-14-002` or later. On `2026-10-14-001`, users who signed in before the upgrade can't sign in again: SAML SSO returns them to the sign-in page with `account_not_linked`, and the root admin's password sign-in answers `INVALID_EMAIL_OR_PASSWORD`. `2026-10-14-002` signs them in with no change to the database. @@ -242,6 +275,8 @@ git checkout docker compose up -d ``` +Rolling back from `2026-11-10` to `2026-10-14` builds and drops nothing: the `2026-10-14` scheduler keeps the new indexes, and the stored hashes are fields it doesn't read. + Rolling back from `2026-10-14` to `2026-07-26` rebuilds indexes. When the `2026-07-26` scheduler starts, it drops the two `instances` indexes `2026-10-14` builds and rebuilds its own, which takes as long as the `instances` builds in the [estimate](#estimate-the-upgrade-time). Reading a run's spec files has no index until the first one finishes, so roll back in a maintenance window too. ## Troubleshooting diff --git a/on-prem/CHANGELOG.md b/on-prem/CHANGELOG.md index 596e2b5..0dcae17 100644 --- a/on-prem/CHANGELOG.md +++ b/on-prem/CHANGELOG.md @@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Upgrade Notes +- **The scheduler builds six MongoDB indexes and runs five migrations when it starts, which takes seconds.** None of the indexes is on `instances`, `runs` or `testRecord`. On a test install with 3.1 million `instances`, startup finished 18 seconds after `docker compose up -d`. Run the index check in the upgrade guide before `docker compose up -d`. See [Upgrading from 2026-10-14 to 2026-11-10](../docs/upgrading.md#upgrading-from-2026-10-14-to-2026-11-10). +- **Organization API keys, record keys and the SCIM secret get a stored hash.** A key whose value another document also holds is left without one, and the scheduler logs a warning with the documents' ids. The upgrade guide says how to resolve them. + ## [2026-10-14-002] - 2026-10-27 Image-only update on `2026-10-14-001`: no compose file or environment variable changes.