diff --git a/.changeset/sealed-log-opt-in.md b/.changeset/sealed-log-opt-in.md deleted file mode 100644 index 1489d802ae..0000000000 --- a/.changeset/sealed-log-opt-in.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -'@workflow/world': patch -'@workflow/world-vercel': patch -'@workflow/world-local': patch -'@workflow/world-postgres': patch -'@workflow/core': patch ---- - -New runs are no longer created with the sealed-log event identity (specVersion 7) by default; set `WORKFLOW_SEALED_LOG=1` to opt in. Every runtime still reads sealed logs, and a run's version is fixed at creation, so runs already created at specVersion 7 keep working. diff --git a/.changeset/world-testing-minted-floor.md b/.changeset/world-testing-minted-floor.md new file mode 100644 index 0000000000..77d235870c --- /dev/null +++ b/.changeset/world-testing-minted-floor.md @@ -0,0 +1,5 @@ +--- +'@workflow/world-testing': patch +--- + +The event-id conformance test now floors a run's stamped `specVersion` at `mintedSpecVersion()` rather than `SPEC_VERSION_CURRENT`, so a World is not failed for stamping the version it was told to stamp while a spec bump is staged. diff --git a/docs/content/docs/v5/configuration/runtime-tuning.mdx b/docs/content/docs/v5/configuration/runtime-tuning.mdx index b7d6acf2e2..fb7cfcb0ad 100644 --- a/docs/content/docs/v5/configuration/runtime-tuning.mdx +++ b/docs/content/docs/v5/configuration/runtime-tuning.mdx @@ -124,12 +124,12 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL ### `WORKFLOW_SEALED_LOG` -- Default: disabled -- Set `1` to create new runs at the sealed-log spec version, in which the World's backend assigns each event its position *before* the write commits rather than letting concurrent writers race for one. Concurrent writes then never contend for a position, which is what makes a wide fan-out cheap. +- Default: enabled +- New runs are created at the sealed-log spec version, in which the World's backend assigns each event its position *before* the write commits rather than letting concurrent writers race for one. Concurrent writes then never contend for a position, which is what makes a wide fan-out cheap. - The price of assigning positions in advance is that a writer which claims one and then dies leaves a position no writer will ever fill. The backend closes such a position by writing a `noop` event into it once it can prove the position was abandoned, so a reader still sees the dense log it needs. Replay steps over a `noop` without delivering it to the workflow or advancing the deterministic clock. Its timestamp belongs to whichever reader sealed it, not to the run. -- Left off, a deployment stays on the previous scheme, where each position is allocated by the write that occupies it. That is the default while two things remain outstanding: an abandoned claim currently strands its run between a step outcome and the resume that should follow it, recovered only by the queue's own redelivery some minutes later, and the Python runtime cannot yet read a sealed log at all. -- Existing runs are unaffected either way. A run's spec version is stamped once, at creation, and read from the run for the rest of its life, so flipping this changes only what *new* runs get, and a run in flight keeps the scheme it started on. Every build reads sealed logs regardless of the setting, so runs created while it was on stay readable after it goes off. -- A run created at the sealed-log version can only be replayed by a reader that knows to skip `noop` events. That includes every runtime on this release train, but a runtime that pins its own accepted spec range separately, such as the Python runtime, has to catch up before it can read these runs. Leave this off in any environment that serves one. +- Set `0` to put a deployment back on the previous scheme, where each position is allocated by the write that occupies it. Use this as the kill switch if position assignment turns out to be at fault for event-log problems. +- Existing runs are unaffected either way. A run's spec version is stamped once, at creation, and read from the run for the rest of its life, so flipping this changes only what *new* runs get, and a run in flight keeps the scheme it started on. Every build reads sealed logs regardless of the setting. +- A run created at the sealed-log version can only be replayed by a reader that knows to skip `noop` events. That includes every runtime on this release train, but a runtime that pins its own accepted spec range separately, such as the Python runtime, has to catch up before it can read these runs. Switch this off in an environment where it has not. - Only the Vercel World seals. The Local and Postgres Worlds allocate each position at the commit that occupies it, so they cannot leave a hole and never write a `noop`; the setting still moves the version they stamp, so the fleet stays on one spec. ### `WORKFLOW_PRECONDITION_MAX_INPROCESS_RESTARTS` diff --git a/docs/content/docs/v5/how-it-works/event-sourcing.mdx b/docs/content/docs/v5/how-it-works/event-sourcing.mdx index e9e9d2f4c3..cb6091010d 100644 --- a/docs/content/docs/v5/how-it-works/event-sourcing.mdx +++ b/docs/content/docs/v5/how-it-works/event-sourcing.mdx @@ -269,7 +269,7 @@ The observability UI grays out events it can identify this way and shows the rea ## Sealed positions (noop events) -Runs at `specVersion` 7 and above use a *sealed log*. The backend gives each write its position from a per-run sequencer **before** the write commits, so concurrent writers hold distinct positions and never race for a slot. New runs opt in to this behavior with [`WORKFLOW_SEALED_LOG=1`](/docs/configuration/runtime-tuning#workflow_sealed_log); left unset, they stay on the previous scheme. Every runtime reads a sealed log regardless, and a run's version is fixed at creation, so changing the setting never affects an in-flight run. However, a writer can claim a position and then stop because of a crashed process or canceled transaction. This leaves a hole that no writer will fill, and a hole looks like an event the reader failed to load. +Runs at `specVersion` 7 and above use a *sealed log*. The backend gives each write its position from a per-run sequencer **before** the write commits, so concurrent writers hold distinct positions and never race for a slot. New runs use this behavior by default. [`WORKFLOW_SEALED_LOG=0`](/docs/configuration/runtime-tuning#workflow_sealed_log) returns a deployment to the previous scheme. Every runtime reads a sealed log regardless, and a run's version is fixed at creation, so changing the setting never affects an in-flight run. However, a writer can claim a position and then stop because of a crashed process or canceled transaction. This leaves a hole that no writer will fill, and a hole looks like an event the reader failed to load. The backend restores the dense log at read time by **sealing** these positions. Once a hole is provably abandoned, bounded by the commit time of later positions, the backend writes a `noop` event into it. Positions are assigned in order, so a committed later position proves how long the hole has been open. A `noop` occupies its position, and length-based completeness checks, cursors, and pagination all count it. It has no other effect: diff --git a/docs/content/worlds/v5/building-a-world.mdx b/docs/content/worlds/v5/building-a-world.mdx index 3b853c6c18..c30b1237ee 100644 --- a/docs/content/worlds/v5/building-a-world.mdx +++ b/docs/content/worlds/v5/building-a-world.mdx @@ -193,7 +193,7 @@ Spec version 7 supports one alternative to allocate-at-commit for Worlds whose s The runtime skips `noop` events during replay. It never delivers them to a consumer or uses them to advance the deterministic clock, so a sealed log replays identically to one whose writers filled the holes. `noop` isn't user-creatable and is never sent to `events.create()`. Only your read path may write one. Worlds that allocate at the commit, through a synchronous counter or unique-constraint append, maintain perfect density and don't need sealing. `world-local` and `world-postgres` never seal, and a World that allocates at the commit is spec 7 compliant without additional work. -The version a World stamps comes from `mintedSpecVersion()`: the slot-identity version by default, or 7 when [`WORKFLOW_SEALED_LOG=1`](/docs/configuration/runtime-tuning#workflow_sealed_log) opts in. Declare `mintedSpecVersion()` instead of a literal so your World moves with the fleet. A runtime other than the one that created a spec 7 run may read it, so readers must understand `noop` before anything stamps 7 in that environment. +The version a World stamps comes from `mintedSpecVersion()`: 7 by default or the slot-identity version when [`WORKFLOW_SEALED_LOG=0`](/docs/configuration/runtime-tuning#workflow_sealed_log) disables it. Declare `mintedSpecVersion()` instead of a literal so your World moves with the fleet. A runtime other than the one that created a spec 7 run may read it, so readers must understand `noop` before anything stamps 7 in that environment. `events.create()` params carry `eventCount`: how many events the writer held in the log it replayed from, which is the position it expects to land on minus one. Attempt `eventCount + 1`. When that position is taken, **do not reject the write**. Advance to the next free position, commit there, and return the events occupying the positions you skipped over on the success response, in `events` with a matching `cursor` and `hasMore`. The writer merges them into its own log and replays once, rather than paying a second round trip to discover it was behind. A caller with a stale count is the normal case for a fan-out, and rejecting it would serialize writes the runtime issues in parallel. diff --git a/packages/world/src/spec-version.test.ts b/packages/world/src/spec-version.test.ts index 75d7abeb92..e03e735b01 100644 --- a/packages/world/src/spec-version.test.ts +++ b/packages/world/src/spec-version.test.ts @@ -21,32 +21,25 @@ describe('spec version constants', () => { }); describe('mintedSpecVersion', () => { - it('stamps the slot-identity version by default', () => { - // Stamping trails reading. Until every reader in the fleet accepts - // spec 7 — the Python runtime still caps at 6 — and pre-assigned - // positions stop stranding runs, a new run gets the version the whole - // fleet can already serve. - expect(mintedSpecVersion({})).toBe(SPEC_VERSION_SUPPORTS_SLOT_IDENTITY); - expect(mintedSpecVersion({})).not.toBe(SPEC_VERSION_SUPPORTS_SEALED_LOG); + it('stamps the sealed-log version by default', () => { + expect(mintedSpecVersion({})).toBe(SPEC_VERSION_CURRENT); + expect(mintedSpecVersion({})).toBe(SPEC_VERSION_SUPPORTS_SEALED_LOG); }); - it('stamps the sealed-log version when opted in', () => { - for (const on of ['1', 'true']) { - expect(mintedSpecVersion({ [SEALED_LOG_ENV_VAR]: on })).toBe( - SPEC_VERSION_CURRENT - ); - expect(mintedSpecVersion({ [SEALED_LOG_ENV_VAR]: on })).toBe( - SPEC_VERSION_SUPPORTS_SEALED_LOG + it('falls back to slot identity when switched off', () => { + for (const off of ['0', 'false']) { + expect(mintedSpecVersion({ [SEALED_LOG_ENV_VAR]: off })).toBe( + SPEC_VERSION_SUPPORTS_SLOT_IDENTITY ); } }); - it('stays off for an unset, empty, or malformed value', () => { - // Opting in is deliberate: a typo must not silently move a deployment - // onto a scheme its readers may not accept. - for (const raw of ['', '0', 'false', 'yes-please']) { + it('stays on by default for an unset or malformed value', () => { + // A flag is an escape hatch, not a hard requirement: a typo must not + // silently move a deployment onto the older identity scheme. + for (const raw of ['', '1', 'true', 'yes-please']) { expect(mintedSpecVersion({ [SEALED_LOG_ENV_VAR]: raw })).toBe( - SPEC_VERSION_SUPPORTS_SLOT_IDENTITY + SPEC_VERSION_CURRENT ); } }); diff --git a/packages/world/src/spec-version.ts b/packages/world/src/spec-version.ts index e9dc90fd46..41b1d157d9 100644 --- a/packages/world/src/spec-version.ts +++ b/packages/world/src/spec-version.ts @@ -102,7 +102,7 @@ export const SPEC_VERSION_CURRENT = SPEC_VERSION_SUPPORTS_SEALED_LOG as SpecVersion; /** - * Environment variable that opts new runs IN to the sealed log. + * Environment variable that opts new runs OUT of the sealed log. * * Read per `createWorld()` call rather than at module load, so a test or a * single process can create worlds in both modes. @@ -110,43 +110,43 @@ export const SPEC_VERSION_CURRENT = export const SEALED_LOG_ENV_VAR = 'WORKFLOW_SEALED_LOG'; /** - * The spec version a World should stamp on the runs it creates: the - * slot-identity version unless {@link SEALED_LOG_ENV_VAR} opts in to the - * sealed log that supersedes it. - * - * Reading and stamping are separate stages of a spec bump, and this is the - * first of them: every build already reads a sealed log and skips `noop` (see - * {@link SPEC_VERSION_MAX_SUPPORTED}), while stamping stays behind the flag - * until the version is safe to mint everywhere. Two things have to be true - * before that default flips, and neither is yet: - * - * - **Every reader in the fleet has to accept spec 7.** A runtime that pins - * its own accepted range separately does not move with this constant. The - * Python runtime validates `specVersion <= 6` and rejects a spec-7 - * `run_started` outright, so stamping 7 by default makes every run it serves - * unrunnable. - * - **Pre-assigned positions have to be free of the stall they currently - * cause.** Assigning a position before the write commits is what lets a - * claim be abandoned, and abandoned claims are observably stranding runs: - * spec-7 runs stall between a step outcome and the resume that should follow - * it, and only the queue's own redelivery (order of ten minutes later) moves - * them on. Measured against spec-6 runs on the same backend in the same - * window, spec 7 stalls roughly 30x as often. - * - * The fallback is a real fallback, not a formality. Stamping the lower version - * has to leave a World the runtime still admits, which is why + * The spec version a World should stamp on the runs it creates: the sealed log + * unless {@link SEALED_LOG_ENV_VAR} switches it off, in which case the + * slot-identity version it supersedes. + * + * Same shape, and the same reasoning, as the flag slot identity itself shipped + * behind before going unconditional: default on, with one env var to put a + * deployment back on the previous scheme without a release. + * + * What default-on rests on is the density requirement in + * `Storage['events']`: a reader's log must be a PREFIX of the run's log, so + * that the number of events it holds tells it whether it has the whole thing. + * A sealed log satisfies that by repair rather than by construction — a + * position is handed out before its write commits, so a read can land while + * one is still empty — and it is only equivalent if a read that cannot see + * past such a position waits for it to be filled or sealed instead of + * reporting a log that ends there. It has to be the READ that waits, because + * a shorter prefix is a legal log state and nothing downstream can tell the + * two apart. The first rollout of this default shipped without that: the + * backend's in-request poll budget was shorter than the age a position must + * reach before it can be sealed, so the read always gave up and truncated, + * and a replay took a step whose completion sat above the gap to be still + * running — then sat on it for a full inline-ownership lease. + * + * The fallback is a real fallback, not a formality. Turning this off has to + * leave a World the runtime still admits, which is why * `assertWorldSupportsRuntimeProtocol` floors at the slot-identity version - * rather than at {@link SPEC_VERSION_CURRENT} because a default that made the - * runtime reject its own World would be no default at all. + * rather than at {@link SPEC_VERSION_CURRENT} because a kill switch that made + * the runtime reject its own World would be no kill switch at all. * * Every World reads runs up to {@link SPEC_VERSION_MAX_SUPPORTED} whatever - * this returns, so leaving it off here does not make runs another process - * created unreadable — including the spec-7 runs created while it was on. + * this returns, so switching it off here does not make runs another process + * created unreadable. */ export function mintedSpecVersion( env: Record = process.env ): SpecVersion { - return envFlag(SEALED_LOG_ENV_VAR, false, env) + return envFlag(SEALED_LOG_ENV_VAR, true, env) ? SPEC_VERSION_CURRENT : SPEC_VERSION_SUPPORTS_SLOT_IDENTITY; }