Skip to content

fix: Metadata record optimization - #33

Merged
isTravis merged 1 commit into
mainfrom
tr/record-optim
Aug 19, 2026
Merged

fix: Metadata record optimization#33
isTravis merged 1 commit into
mainfrom
tr/record-optim

Conversation

@isTravis

Copy link
Copy Markdown
Member

Fixes two problems with PATCH /api/collections/:owner/:slug/metadata on large collections.

1. The endpoint reported failure on writes that succeeded

Building a metadata patch version takes longer than 100s at a few million records, so Cloudflare
returned 524. Nothing aborts the handler on client disconnect, so the write then committed anyway.
This has already happened on dev: a 524 to the caller, and a complete patch version created a couple
of minutes later. A caller who retried raced the version that did get created and failed on the
(collection_id, semver) unique constraint after tens of minutes of work.

  • ?async=true returns 202 {job_id}; poll GET …/metadata/jobs/:jobId for running / completed /
    failed. Modelled on the existing async negotiate commit.
  • New metadata_jobs table. Stranded jobs are failed out by tool:cleanupSessions, already on cron.
  • One job per collection at a time — a concurrent write gets 409 with the in-flight job_id instead
    of racing for the same semver.
  • Sync remains the default, so the CLI and existing scripts are unaffected.
  • The UI now uses the async path and polls, so editing a readme in settings works at any size.

Async is opted into by query param only, deliberately not a body flag as negotiate allows: here the
body is the metadata, so an async key would be merged in and persisted.

Retrying the same edit after a silent success is already safe — latest has moved on, the metadata
matches, and the existing unchanged-check returns {unchanged: true} before doing any work.

2. A metadata edit copied the entire record set

A patch version has, by definition, the same records as its base, but we wrote it a full private copy:
one version_records row per record. On a collection of a few million records that is roughly 2 GB with
indexes, spent to correct a readme, and again on every subsequent edit.

  • New versions.records_from_version_id. A metadata patch points at the version that owns the rows
    instead of copying them: one row written instead of millions.
  • Always one hop — a patch inherits its base's pointer, so it never chains.
  • ON DELETE RESTRICT, so deleting a version whose rows others share fails loudly rather than silently
    emptying them.
  • ~35 call sites now resolve through recordsVersionId(). Record bodies were never duplicated;
    record_objects is unchanged.

recordsFromVersionId is a required field on recordsVersionId(), so any version fetched with a
projection missing the column is a compile error rather than a silently empty record set. That caught 6
real cases during the sweep.

Two read paths needed judgement rather than mechanical resolution: the record provenance list now
OR-joins on the pointer (otherwise patch versions vanish from a record's provenance), while the
access-control and file-reference joins deliberately stay ownership-only — they are collection-scoped,
and the owning version is always in the same collection. Both are commented in place.

The pointer is internal storage detail and is stripped from API responses.

Migrations

Both additive — no table rewrite, no backfill.

  • 0013metadata_jobs
  • 0014versions.records_from_version_id (nullable + self-FK)

Existing versions keep NULL and go on owning their rows.

Testing

  • tsc --noEmit, oxlint, oxfmt --check clean; 97 unit tests pass.
  • All 15 migrations apply cleanly to an empty database.
  • pnpm tool:verifyRecordSharing — 13 assertions against real Postgres, including that the
    unresolved query returns 0 (the failure mode is confirmed, not assumed), RESTRICT blocking
    deletion of a shared base, and chained edits staying one hop. Kept out of pnpm test, which is pure
    unit tests that run anywhere.

@isTravis
isTravis merged commit 5f5d53f into main Aug 19, 2026
1 check passed
@isTravis
isTravis deleted the tr/record-optim branch August 19, 2026 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant