Add keyset pagination and time-range filters to the run export api

[giulianoconte]

Overview

Add keyset pagination and time-range filters to the run export api.

The bulk GET /api/exports/runs is currently hundreds of thousands of runs. Pulling it in one shot is unreliable - the on-the-fly stream can get reset partway through (observed: Cloudflare closing the HTTP/2 stream before the dump completed), in addition to taking a while. This adds an opt-in way to pull it in resumable, bounded chunks. When calling the endpoint with no parameters results in the same behavior as before (a full export).

Changes:

• backend/app/routers/exports.py
  • Add limit, start, end, cursor params
  • Implement time-range and cursor query logic
  • Update rate limit logic from 2/h to 120/h - full export costs 60, one page costs 1
• backend/app/services/runs_db_mongo.py
  • Add new (submitted_at, _id) index
  • _ensure_run_validator validates that new run submissions have submitted_at
• backend/app/metrics.py
  • Add new run_export_pages counter
• backend/tests/test_export_helpers.py, backend/pytest.ini, backend/requirements-dev.txt
  • Add pytest (single dev dependency) + pure-function unit tests for the export pagination helpers

Behavior

GET /api/exports/runs gains four optional query params:

Param	Meaning
limit	Max runs in this page (1 to 50000). Enables keyset pagination.
start	Inclusive lower bound on submitted_at (ISO-8601).
end	Exclusive upper bound on submitted_at (ISO-8601).
cursor	Opaque keyset token from a prior page's X-Next-Cursor header.

• Ordering: (submitted_at, _id) ascending, always.
• Pagination: with limit, a full page returns an opaque X-Next-Cursor response header. Pass it back as cursor= to get the next page; its absence means you've reached the end. Ascending order means runs submitted during a paginated scan will sort after the cursor, so a forward pager never misses or double-counts. In other words, You can page through the whole corpus while clients submit run concurrently; the runs will be picked up before you've reached the end.
• Time-range: start/end give a half-open [start, end) window on submitted_at, for incremental "everything since my last sync" pulls. Combine with limit to also bound each page.
• No params behavior is unchanged: full export (same body, same semantics).

# bootstrap the whole corpus in 5000-run pages
GET /api/exports/runs?limit=5000
GET /api/exports/runs?limit=5000&cursor=<X-Next-Cursor from previous>
...until no X-Next-Cursor

# incremental: just the new runs since the last sync
GET /api/exports/runs?start=2026-06-20T00:00:00Z&limit=5000

Implementation

• Index (runs_db_mongo.py): adds ascending (submitted_at, _id). The existing {submitted_at: -1} index is descending and lacks the _id tiebreaker, so it can't serve the ordered keyset scan. With the new index the scan is index-ordered with no blocking in-memory sort (IXSCAN -> FETCH -> LIMIT, no SORT stage). The export filters to official characters ($in OFFICIAL_CHARACTERS); since character isn't in the index, it stays a post-fetch filter (the FETCH evaluates it), so the query isn't covered. That's fine today because official-character runs dominate the corpus, so few fetched docs are rejected. The submit path does store modded-character runs (they're excluded only at read/snapshot time), so if modded runs ever became a large fraction of stored runs, a (character, submitted_at, _id) index would make character an indexed predicate (SORT_MERGE over the official values, fully covered, no post-fetch rejection). I kept the simpler index since I assume the fraction to be negligible currently.
• Two effective rate limits: full dumps stay at 2/hour, paged reads get 120/hour. The split is expressed with slowapi's per-request cost: the route is 120/hour where a bounded (limit) page costs 1 and a the full export costs 60 - so the old 2/hour for full export is reserved.
• Null/legacy submitted_at (older runs that predate upload-time tracking and haven't been stamped) sort as a leading block ordered by _id; the cursor handles that block explicitly, so a full bootstrap (no start/end) still returns them. A start/end window filters on submitted_at and therefore excludes them - intended for incremental sync, documented on the endpoint. Null volume doesn't degrade paging (the non-sparse index keys nulls too) - see Verification.
• Metric split so paging doesn't silently change an existing dashboard: spire_codex_run_exports_total still counts unbounded dumps; new spire_codex_run_export_pages_total counts bounded pages.
• Schema validator enforcing the ordering invariant (runs_db_mongo.py): the export's "a forward pager never misses a new run" guarantee holds only because every new run sorts at the end - i.e. is stamped with a real submitted_at. A run inserted without submitted_at sorts into the leading null block. So _ensure_run_validator attaches a $jsonSchema validator requiring submitted_at to be a BSON date. The validationLevel: "moderate" means the validator enforces new runs bot not pre-existing runs. Note that a bulk re-import of untimestamped legacy runs would now be rejected. You should be able to get around it by adding submitted_at in the import or setting validationAction: "warn" for that window.
• Cursor format is base64url of submitted_at|_id.

Verification

Tested manually against a local instance (Docker, in-memory slowapi), seeded with runs. To reproduce:

# in-memory rate limit note: a 429 returns a (non-gzip) JSON error body; reset
# the limiter any time with: docker compose -f docker-compose.local.yml restart backend

# 1. start the stack (Mongo + backend on :8000)
docker compose -f docker-compose.local.yml up -d

# 2. seed raw run blobs (POST /api/runs dedups on a canonical run_hash)
for f in seed-runs/*.json; do
  curl -s -X POST "http://localhost:8000/api/runs?username=seed_local" \
    -H "Content-Type: application/json" --data-binary "@$f" >/dev/null
done

# 3. full (unbounded) export - the unchanged path - count lines
curl -s "http://localhost:8000/api/exports/runs" | gunzip | wc -l

# 4. page through, following X-Next-Cursor until it's gone, concatenating pages
cursor=""; : > /tmp/paged.jsonl
while :; do
  url="http://localhost:8000/api/exports/runs?limit=50"
  [ -n "$cursor" ] && url="$url&cursor=$cursor"
  curl -s -D /tmp/hdr "$url" | gunzip >> /tmp/paged.jsonl
  cursor=$(grep -i '^x-next-cursor:' /tmp/hdr | tr -d '\r' | awk '{print $2}')
  [ -z "$cursor" ] && break
done
wc -l /tmp/paged.jsonl          # has equal line set as full export

# 5. half-open [start, end) window (incremental "since my last sync")
curl -s "http://localhost:8000/api/exports/runs?start=2026-06-01T00:00:00Z&end=2026-06-20T00:00:00Z" \
  | gunzip | wc -l

# 6. inspect the query plan (expect IXSCAN -> FETCH -> LIMIT, no SORT stage)
docker exec spire-codex-local-mongo mongosh spire_codex --quiet --eval '
  db.runs.find({character:{$in:["IRONCLAD","SILENT","DEFECT","NECROBINDER","REGENT"]}})
    .sort({submitted_at:1,_id:1}).limit(50).explain().queryPlanner.winningPlan'

# 7. (optional) null-volume test: inject 8k null-submitted_at docs straight into
#    Mongo (bypasses the submit path on purpose), then plan the null-block
#    continuation query - expect SORT_MERGE over two IXSCANs, not a COLLSCAN/SORT
docker exec spire-codex-local-mongo mongosh spire_codex --quiet --eval '
  const bulk = [];
  for (let i = 0; i < 8000; i++)
    bulk.push({_id:"null"+i, character:"IRONCLAD", submitted_at:null});
  db.runs.insertMany(bulk);
  db.runs.find({$and:[
      {character:{$in:["IRONCLAD","SILENT","DEFECT","NECROBINDER","REGENT"]}},
      {$or:[{submitted_at:null,_id:{$gt:"null0"}},{submitted_at:{$ne:null}}]}]})
    .sort({submitted_at:1,_id:1}).limit(50).explain("executionStats").executionStats'

Results:

• Paginated export == full export: the concatenated pages (step 4) are the identical line set as the unbounded export (step 3) - no gaps, overlaps, or double-counts. (Line count can differ from run count: unreadable/missing run files are skipped, and a multiplayer run emits one line per player - but both paths skip the same ones, so the sets match.)
• Half-open ranges partition exactly: splitting the corpus at a midpoint timestamp into [min, mid) and [mid, max] yields disjoint windows whose counts sum to the whole (e.g. 146 + 147 = 293), and [mid, mid) is empty - the run sitting exactly on the boundary lands in exactly one window, never both or neither.
• Query plan: IXSCAN -> FETCH -> LIMIT, no SORT stage (step 6).
• Null volume doesn't degrade paging: with the 8k injected null docs (step 7), the null-block continuation query plans as SORT_MERGE over two IXSCANs, examining docsExamined ~= 2x limit (sub-ms) - bounded by page size, not null count; no collection scan, no in-memory sort.
• Param validation: bad cursor/start -> 400, limit outside [1, 50000] -> 422, future-dated window -> empty 200. Validation runs before rate-limit consumption, so malformed requests don't burn budget.

Notes

• Difference in pagination practices. Every other paginated endpoint here uses offset pagination (page + limit) and returns its metadata in the JSON body (total/page/per_page/has_next); this is the only endpoint using a keyset cursor and a data-bearing X-Next-Cursor response header. I diverged because: (1) the response is a gzipped JSONL stream, which has no JSON envelope to carry next/has_next without either wrapping (and breaking) the stream or violating the "no params = unchanged body" guarantee; and (2) offset pagination means a deep skip/OFFSET walks and discards that many index entries per page (O(offset)) and double-counts or skips under concurrent inserts, whereas keyset is O(page-size) and insert-stable, which is what a full-corpus export needs. Let me know if you want to change the approach.
• The new index makes the current {submitted_at: -1} index redundant. (submitted_at, _id) ascending is a superset: it serves the keyset export and everything that uses {submitted_at: -1} today - the unfiltered newest-first runs list (sort=date), the admin "last submission" find_one, and the 24h-count range - because MongoDB serves a sort and its reverse from one index and range filters ignore direction. I kept the previous index in this PR so it's additive-only. You could have just one (submitted_at: 1, _id: 1) or (submitted_at: -1, _id: -1) index to serve current use cases and this PR. The {username|character|user_id: 1, submitted_at: -1} compounds have different prefixes and are unaffected.
• Direction (ascending vs descending) doesn't matter for pagination. I chose ascending. The only sort a uniform-direction index can't serve is a mixed-direction one such as submitted_at: 1, _id: -1, which afaict nothing in the codebase needs currently.
• Confirm submitted_at is stored as a BSON Date everywhere. This repo flags ETL type drift elsewhere (e.g. win/was_abandoned as 0/1 vs bool). If any legacy ETL'd docs hold submitted_at as a string, BSON type-bracketing would make the ordered scan and $gt/$ne comparisons silently skip them. I couldn't check against prod. I think you can check with this:

  db.runs.aggregate([{$group: {_id: {$type: "$submitted_at"}, n: {$sum: 1}}}])
  

  (Expected: only date and null. A string bucket means we'd want a normalization pass first.)
• The design assumes new runs are never null submitted_at The live submit path always fills submitted_at, so new runs get added at the end. If a future bug or query ever inserted runs without submitted_at, they'd be silently missed for in-progress paginated scans - nulls sort first, so they land behind any pager that has advanced past the null block (and the start/end window excludes nulls entirely). They would only be picked up if you rescanned the null block from the beginning. This To mitigate this:
  • (Implemented) Enforce the invariant at write time: The $jsonSchema validator (see "How") makes a buggy null/missing insert a loud failure, applied automatically on deploy. If you'd rather ease it in, set validationAction: "warn" to log violations without rejecting.
  • Monitor the null count: Alert if count_documents({submitted_at: null}) trends up; under the invariant it should be flat or shrinking. A second signal even with the validator on.
  • Rescan the null block on the client side: Periodically rescan to pick up new rows in the null block.
• cost keys on limit presence, so a windowed-but-unbounded pull (start/end, no limit) still costs 60. Defensible (it could return the whole corpus), and the guidance is simply "always pass limit for the cheap path."
• CORS: X-Next-Cursor isn't in expose_headers, so a browser fetch can't read it. The current consumer is a C# HttpClient (unaffected); if your frontend ever consumes the export, add expose_headers=["X-Next-Cursor"].
• Tests (backend/tests/test_export_helpers.py). The repo had no test suite, so this adds pytest as a single dev dependency (requirements-dev.txt + a minimal pytest.ini); run with pytest from backend/. The added tests are pure-function unit tests - no database, no app, no CI needed (the router imports _get_collection lazily, so the helpers import clean): cursor encode/decode round-trip (including the null-submitted_at block) and malformed-cursor rejection (a real base64 error and a valid-base64-but-no-separator token, both -> 400), _build_match keyset/range clause construction (no-params, half-open window, null-block continuation, past-nulls continuation), ISO parsing + its 400, and the rate-limit cost. For more test coverage, we need a Mongo-backed test fixture, which this PR does not introduce. This would allow endpoint behavior tests, e.g. paginate == full export and range partitioning.