perf(rpc): fix POST /links/create 3–6s latency during Redis instability - #677
perf(rpc): fix POST /links/create 3–6s latency during Redis instability#677izadoesdev wants to merge 4 commits into
Conversation
…link create When beginCachedLinkMutation fails due to Redis being unavailable, the subsequent backfillLinkCache call hits the same dead Redis and blocks for another 1.5s (the LINK_CACHE_OPERATION_DEADLINE_MS timeout) before failing. This adds 1.5s of wasted latency to every link create request during Redis instability. Changes: - Track cacheUnavailable flag when begin-mutation errors on Redis - Skip backfillLinkCache entirely when Redis is known to be down (both in the success path and in the reconciliation path) - Make backfillLinkCache fire-and-forget (unawaited) in the cache-bypass path since the PG row is already committed and backfill errors are already caught internally - Make writeMutationAudit fire-and-forget in runAuditedMutation since appendAuditEvent already has outbox fallback for durability Net effect: eliminates ~1.5s of dead Redis waiting per create request during cache outages, plus removes the sequential audit INSERT from the response path for all tracked mutations. Co-authored-by: iza <izadoesdev@users.noreply.github.com>
Co-authored-by: iza <izadoesdev@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
The latest updates on your projects. Learn more about Unkey Deploy
|
2 participants
Description
Fixes POST
/links/createtaking 3–6.5 seconds in production during Redis instability periods. Root cause analysis showed two compounding code-level bottlenecks that amplify Redis timeout penalties:Bottleneck 1 — Redundant Redis backfill after known failure (~1.5s saved)
When
beginCachedLinkMutationfails due to Redis being unavailable, the handler falls back to a PG-only create (correct), but then callsbackfillLinkCachewhich hits the same dead Redis and blocks for another 1.5s (LINK_CACHE_OPERATION_DEADLINE_MS) before failing. This call always fails when Redis is down — it's pure waste.Bottleneck 2 — Awaited audit INSERT blocks the response (~50–100ms saved)
runAuditedMutationawaitswriteMutationAuditafter every successful tracked mutation. SinceappendAuditEventalready has an outbox fallback for durability, blocking the HTTP response on this INSERT is unnecessary latency.Full request timeline during Redis instability (before fix):
resolveApiKey/getMemberRoleviacacheablebeginCachedLinkMutationvia link cache RedisbackfillLinkCachevia link cache RediswriteMutationAuditPG INSERTAfter fix, steps 4 and 5 are eliminated from the response path: step 4 is skipped entirely when Redis is known down, step 5 is fire-and-forget.
Remaining latency from Redis instability (~3.5s) is infrastructure, not code:
cacheable2s timeout onresolveApiKey/getMemberRoleis an infra-level Redis health issuebeginCachedLinkMutationis the designed failover windowChanges:
cacheUnavailableflag in the link create handler when begin-mutation errorsbackfillLinkCachewhen Redis is known to be down (success path + reconciliation path)backfillLinkCachefire-and-forget (unawaited) in the non-error cache-bypass pathwriteMutationAuditfire-and-forget inrunAuditedMutation(audit has outbox fallback)Slice
rpc(links router + audit middleware)Checklist
stagingand does not include another unmerged PR unless it is named above.staging; after it closes, this branch will not be reused for another change.Summary by cubic
Fixes POST
/links/createtaking 3–6.5s during Redis instability by cutting redundant Redis waits and adding fail-fast behavior so one timeout no longer compounds into sequential 1.5–2s penalties per request.Changes
backfillLinkCachepath entirely, removing ~1.5s of dead-Redis waiting per create in cache-bypass and reconciliation flows.writeMutationAudit(fire-and-forget) sinceappendAuditEventhas an outbox fallback; healthy-Redis behavior is unchanged.Written for commit 9bfee91. Summary will update on new commits.