fix(orb): validate the sender-supplied ingest timestamps before they reach orb_signals - #10156
Conversation
…reach orb_signals handleOrbIngest is the open, unauthenticated fleet-calibration collector, and every field is whitelist- or range-validated before storage — except decision_timestamp and outcome_timestamp, which got a bare typeof === 'string' check: no length cap, no format check (outcome_timestamp is also written verbatim into sent_at). decision_timestamp is the day bucket for the public fleet-accuracy trend and the retention rollup key, so a non-instant string (e.g. 'unknown') sorts above every ISO date in the lexicographic window bound, is then dropped in JS as unparseable — silently excluding the signal from the published trend — and lands a permanent junk day in orb_signal_rollups' composite PK. Add MAX_TIMESTAMP_CHARS (64) and a pure normalizeIngestTimestamp that returns the value only when it is a length-capped, Date.parse-able instant, routing all three binds through it — mirroring the REUSE_DAY_PATTERN + clampReuseCount pair one field family over. A malformed timestamp is stored NULL rather than dropping the event, so COALESCE(decision_timestamp, received_at) falls back to the server clock and the signal still counts. Validation only: no migration, no schema change, and every other field's behaviour is unchanged. Closes JSONbored#10028
|
Tip ✅ LoopOver review result - approve/merge recommendedReview updated: 2026-07-31 10:11:13 UTC
Review summary Nits — 4 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10156 +/- ##
===========================================
- Coverage 92.05% 80.16% -11.90%
===========================================
Files 931 283 -648
Lines 114046 58843 -55203
Branches 27543 6999 -20544
===========================================
- Hits 104988 47171 -57817
- Misses 7759 11381 +3622
+ Partials 1299 291 -1008
Flags with carried forward coverage won't be shown. Click here to find out more.
|
What & why
handleOrbIngestis the open, unauthenticated fleet-calibration collector, and its header states the posture: every field of an incoming event is whitelist- or range-validated before storage. Two fields were the exception:A bare
typeof === "string"— no length cap, no format check. Any string up to 1 MiB × 500 events per request is persisted three times per row. Anddecision_timestampis not inert: it is the day bucket for the public fleet-accuracy trend and the retention rollup key (substr(COALESCE(decision_timestamp, received_at), 1, 10)). So a malformed value reachable today from a self-host build that emits one:WHERE COALESCE(...) >= ?1is a lexicographic string compare, and"unknown"sorts above every ISO date.substr(...,1,10)is unparseable →!Number.isFinite(dayMs)→continue. The signal is excluded from the published accuracy series with no warning.dayinorb_signal_rollups' composite primary key, a row no window query reads again and no prune removes.The fix
Add
MAX_TIMESTAMP_CHARS = 64(alongside the existingMAX_HASH_CHARS/MAX_VERDICT_CHARSblock) and a purenormalizeIngestTimestampthat returns the value only when it is a length-capped,Date.parse-able instant, otherwisenull— routing all three binds through it. This mirrors theREUSE_DAY_PATTERN+clampReuseCountpair already in this file, one field family over.A malformed timestamp is stored NULL rather than dropping the event, so
COALESCE(decision_timestamp, received_at)falls back to the server-sidereceived_atand the signal still counts toward the trend — the behaviour difference that matters (today the signal is silently lost). Validation only: no migration, no schema change, and every other field's whitelist/clamp behaviour, the per-event skip rules, and theacceptedcount are unchanged.Tests (
test/integration/orb-ingest.test.ts)normalizeIngestTimestamp("2026-07-30T12:00:00.000Z")→ unchanged;"unknown", a 65-char string,12345,undefined→null(all three branches, both arms).decision_timestamp: "unknown"event is still accepted ({ accepted: 1 }) with the stored columnNULL; an over-length (65+) timestamp likewise storedNULLand accepted.decision_timestamp,outcome_timestamp,sent_at) — the currently-correct behaviour pinned.COALESCE(decision_timestamp, received_at)on the stored row parses to a finite instant — the signal survives into the public trend.main.Validation
src/orb/ingest.tsis 100% line and branch.npm run typecheckclean for these files;npm run engine-parity:drift-checkpasses (host-only file, not a twin);npm run dead-exports:checkclean; the orb-ingest suite (55 tests) green.git diff --checkclean; no migration / schema / generated-artifact change.Closes #10028