Enrich in Search batches instead of one request per entity

[batbrainy]

Linked issue

Closes #45 (and #44, whose durable-backlog work this builds on).

Problem

#44 made the enrichment backlog durable: quota exhaustion defers work instead of
discarding it. It did not make the backlog drain. One core request per entity gives 40
completions an hour against roughly 2,280 cold actor and repository arrivals an hour, and
GitHub authentication is out of scope. Durability without throughput means the backlog and
its oldest pending age grow forever.

Scope

Replaces the one-request-per-entity normal path with derivation-first staged batch
enrichment, and makes catch-up a measured, published property rather than a claim.

In scope

• Derive every field computable from the stored event with no network call.
• Normal path: unauthenticated GitHub Search with repeated exact user: / repo:
  qualifiers, up to SEARCH_BATCH_SIZE (≤ 10) entities per request, on a separate
  per-minute Search ledger.
• Payload-URL detail fetching demoted to a bounded fallback for missing, renamed,
  mismatched, and contract-invalid batch items.
• Append-only observations and batch envelopes; latest projection on the entity rows.
• /status reports stages, throughput, batch quality, both quotas, and a tri-state
  catch-up verdict.

Deliberately not in scope

• Any credential or token.
• Full user-profile hydration (actor name, company, location, bio, followers). The
  useful-data contract is explicit about what "complete" means.
• Pausing polling to make room for enrichment.
• Any form of sampling, dropping, or terminal outcome caused by budget.

Technical decisions

Search batches, not OR-joined queries. Repeated exact qualifiers separated by spaces.
Joining them with OR answers HTTP 422 and is not the supported batching form.

Two ledgers, because they are two rate-limit resources. Search reports
x-ratelimit-resource: search with a limit of 10 per minute; core is 60 per hour. The
core ledger refuses to reconcile a foreign resource, so a single ledger would have imported
a 60-second reset as the hourly window boundary. The core formula becomes 12 poll + 40
detail fallback + 8 reserve = 60. Search pressure
can neither consume nor block the polling allocation.

Stable-ID validation, never name matching. A returned item is applied only when its
immutable GitHub id equals the claimed row's. facebook/react redirecting to react/react
is exactly why: a name-matched apply would have written one repository's data onto another.

Status and stage are separate columns. enrichment_status remains the business
outcome; enrichment_stage carries pipeline position through seven resting stages.
Stage ownership is disjoint — the batch lane owns retry_scheduled, the detail lane owns
detail_pending — so the two claims can never contend for a row.

Explicit leases. lease_token and leased_until replace the overloaded next_retry_at
lease, so a retry instant and a live claim are no longer the same field. Every projection
is guarded by the token, so a reclaimed row cannot be double-applied by a resurrected
worker.

One looping cycle job, not a job per request. Pacing is 6 seconds and the recurring
tick is 60; a job-per-request design could not exceed ~1,200 entities an hour whatever the
budget allowed. The cycle waits out pacing on the dedicated single-thread enrichment
worker, never inside a gate hold or a ledger lock.

Rejected alternatives. Authentication (out of scope by decision). Keeping per-entity
enrichment and accepting unbounded backlog growth (#44's own finding). Suspending polling
to fund enrichment — it trades event capture, which is unrecoverable, for enrichment,
which is not.

Testing performed

docker compose run --rm --build test — 1,959 examples, 0 failures; stress suite 10
examples, 0 failures; RuboCop 259 files, no offenses; Brakeman 0 warnings, 0 errors;
docker compose config --quiet; git diff --check.

Coverage added for every case issue #45 lists, including: event-native completion with no
network call, duplicate-demand coalescing, both batch lanes, stable-ID validation, partial
/ empty / malformed / renamed / unrequested / incomplete_results responses, nullable
contract fields, fallback admission and its bounded core consumption, reserve enforcement
on both ledgers, lane fairness and borrowing, quota deferral across Search minute windows
and core hourly windows in one scenario, retry / stale lease / restart / concurrent
worker behavior, append-only observations surviving a refresh, the absence of any
quota-based terminal state, /status per-stage and catch-up metrics, and configuration
validation.

Docker verification

Live, unauthenticated, no credentials present, from isolated Compose projects with the
development stack stopped. Full transcript:
docs/evidence/2026-08-02-live-staged-batch-enrichment.md.

• One poll persisted 92 events and 176 cold entities with zero enrichment requests;
  owner_login derived locally, 184 event observations committed with the events.
• One Search request settled 9 of 10 actors; the next settled 10 of 10
  repositories. Both batch rows recorded x-ratelimit-resource: search, limit 10,
  while the core ledger's enrichment counter stayed at 0.
• A full production cycle settled 74 entities in 42.5 seconds across 8 requests — fill
  ratio 1.00 — stopping at the configured reserve rather than running the limit to zero.
• facebook/react reproduced the rename: Search answered 422, the members were admitted
  to the fallback, and the stored payload URL resolved it through GitHub's redirect with
  the stable id validated.
• With the Search ledger blocked at its reserve, five pending rows were fingerprinted, the
  database container restarted, and the fingerprint recomputed identical — ledger
  state, batch envelopes, and observations all intact.
• Sustained run at shipped defaults, sampling /status every 60 seconds:
  26 minutes, 53 samples, 872 arrivals. The Search lane
  requested all 872 across 92 requests at a fill ratio of 0.981–0.986, applying 858
  directly and sending 14 to the fallback, and drained each five-minute arrival burst in
  roughly two minutes; 870 of 872 exited the pipeline (99.8%). Across the
  24 samples past the minimum sample window /status reported keeping_up 8 times and
  not_keeping_up 16 times, and the reason is in the numbers: the 1.6% of items Search
  does not return wait on the detail lane, which that run had capped at 4 requests an
  hour. The service did not claim to keep up when it was not — which is the behaviour
  the issue asks for. That measurement is why CORE_DETAIL_FALLBACK_ALLOWANCE now
  defaults to 40 (the whole core remainder after polling and the reserve); the evidence
  file is explicit that the new value is arithmetic from a measured miss rate rather than
  a second measured outcome.

Documentation updates

IMPLEMENTATION_PLAN.md Appendix G plus in-place amendments to §5, §7, §10, §11, §12,
and §16 (including the new forbidden claim: no guaranteed catch-up, only a dated measured
comparison). New ADR 0013; amendment lines on ADRs 0004, 0007, 0008, 0010. README,
.env.example, CLAUDE.md, docs/DESIGN_BRIEF.md, docs/SUBMISSION_CHECKLIST.md, and
the fixtures README follow the same numbers throughout.

Known limitations

• Catch-up is a measurement, not a guarantee. /status compares measured completion
  and arrival rates over a trailing window and reports keeping_up, not_keeping_up, or
  insufficient_sample. It publishes no drain estimate. A past rate does not bound future
  arrivals.
• Event capture remains incomplete by construction. The public feed is a sliding
  window with no cursor; a long enough outage loses events permanently. Polling therefore
  keeps first claim on the core budget.
• Search is an index, not a directory. An entity absent from it is resolved by the
  bounded fallback, not by a larger batch.
• One IP, one feed, one sample. The evidence file covers the window it measured.

Three defects found only by running it live

Recorded because each is a case where the code and its tests agreed with each other and
disagreed with the world:

1. An unparsable payload URL was treated as retryable. github-actions[bot] supplies
   a URL containing brackets; the SSRF policy correctly refuses it, but the retry ladder
   would have spent three of the four hourly core detail requests re-refusing the same
   stored string. Detail fallback now terminates on permanent classifications, per §10.

2. Search answers 422, not an empty result, when every requested identifier is
   unsearchable. A batch of ten silently omits its unsearchable members, which is why the
   exploratory probe never saw this. Read as a generic client error, a renamed repository
   retried forever on the one lane that cannot resolve it.

3. queues: polling,control in config/queue.yml declared one queue literally named
   "polling,control". SolidQueue::QueueSelector wraps its input in Array(), so that
   worker claimed nothing for the life of the process — the always-on container never
   polled or reconciled. This predates this issue; the suite missed it because the spec
   split the configured string itself and asserted intent rather than runtime behavior.

4. Independent review of the branch raised seven further findings, all addressed in
   453fea6: secondary rate limits now bind both ledgers (they are IP-scoped);
   MAX_HTTP_RETRIES was inert on Search because the backoff landed inside the pacing
   window; the borrow flag stated which slot was taken rather than what the ledger reads
   it to mean; only one exact English 422 message reached the fallback; a deferred batch
   restored rows to a status/stage pair the model does not list as legal; and Compose
   forwarded none of the new configuration.

🤖 Generated with Claude Code