diff --git a/serverless/load-balancing/overview.mdx b/serverless/load-balancing/overview.mdx index c75f76a2b..a44c35c32 100644 --- a/serverless/load-balancing/overview.mdx +++ b/serverless/load-balancing/overview.mdx @@ -166,13 +166,26 @@ if health_check_with_retry("https://ENDPOINT_ID.api.runpod.ai", "RUNPOD_API_KEY" Use at least 3 retries with 5-10 second delays. +## When to use queue-based endpoints + +Use queue-based endpoints when you need: + +- Job-based or long-running workloads that run for extended periods. Load balancing is not suited for long-running tasks and would time out or drop the request. +- Guaranteed execution where every request is queued and processed, even during traffic spikes. No requests are dropped when workers are busy. +- Batch or offline workloads processed asynchronously where latency is not critical, such as nightly dataset processing, pre-computing embeddings, or running evaluations. +- Automatic retries on failure without any client-side logic. +- Configurable concurrency that supports both low and high concurrency workloads depending on your needs. + +Choose queue-based when your workload can tolerate higher latency in exchange for reliability and guaranteed delivery. If your use case is a short, real-time request/response transaction, use a load balancing endpoint instead. ## When to use load balancing endpoints Use load balancing endpoints when you need: -- Direct access to your model's HTTP server. -- Internal batching systems (like vLLM). -- Non-JSON payloads. -- Multiple endpoints within a single worker. -- Lower latency for real-time applications. \ No newline at end of file +- Direct access to your model's HTTP server without queueing overhead. +- Internal batching systems like vLLM that manage their own request batching internally. +- Non-JSON payloads such as binary data or multipart uploads. +- Multiple endpoints within a single worker using custom URL paths and any HTTP framework. +- Low-latency real-time applications where responses are immediate and ephemeral, such as serving a file download or returning page stats. + +Choose load balancing when your workload is latency-sensitive and responses are immediate. If your use case involves long-running jobs or requires guaranteed delivery, use a queue-based endpoint instead.