Summary
When brpoplpush in ProcessingList#fetch raises an exception (Redis connection drop, timeout, protocol error), the while true loop in Server#start! retries immediately with no backoff and no error handling, causing 100% CPU usage.
Steps to reproduce
- Start a server with one or more Redis-backed queues
- Kill or pause Redis (
redis-cli DEBUG SLEEP 30 or network drop)
- Observe CPU pinned at 100% per queue server fiber
Assumed root cause
server.rb:59-64:
def start!
return false if @task
@task = true
@parent.async(transient: true, annotation: self.class.name) do |task|
@task = task
while true
self.dequeue(task) # no rescue around this call
end
ensure
@task = nil
end
enddequeue calls @processing_list.fetch, which runs brpoplpush with timeout 0. On success this blocks I guess the fiber cooperatively. On failure, the exception propagates out of dequeue — the ensure block in dequeue is a no-op because _id is nil — and the while true loop immediately retries. No sleep, no backoff, no logging.
With multiple queues, each queue has its own Server instance and its own while true fiber. A single Redis hiccup causes all of them to spin simultaneously.
Expected behavior
Failed dequeue calls should be caught and retried with backoff (e.g., exponential backoff capped at 30s), with the error logged.
Our Workaround
We are currently monkey-patching Server#start! via prepend to add rescue => error with exponential backoff around self.dequeue(task).
Summary
When
brpoplpushinProcessingList#fetchraises an exception (Redis connection drop, timeout, protocol error), thewhile trueloop inServer#start!retries immediately with no backoff and no error handling, causing 100% CPU usage.Steps to reproduce
redis-cli DEBUG SLEEP 30or network drop)Assumed root cause
server.rb:59-64:dequeue calls @processing_list.fetch, which runs brpoplpush with timeout 0. On success this blocks I guess the fiber cooperatively. On failure, the exception propagates out of dequeue — the ensure block in dequeue is a no-op because _id is nil — and the while true loop immediately retries. No sleep, no backoff, no logging.
With multiple queues, each queue has its own Server instance and its own while true fiber. A single Redis hiccup causes all of them to spin simultaneously.
Expected behavior
Failed dequeue calls should be caught and retried with backoff (e.g., exponential backoff capped at 30s), with the error logged.
Our Workaround
We are currently monkey-patching Server#start! via prepend to add rescue => error with exponential backoff around self.dequeue(task).