From d0b1a095cd333ea440748ee1f6271dbb710b0a65 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Wed, 26 Aug 2026 12:16:00 -0400 Subject: [PATCH 1/4] Update overview.mdx --- serverless/load-balancing/overview.mdx | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/serverless/load-balancing/overview.mdx b/serverless/load-balancing/overview.mdx index c75f76a2b..3a5858f38 100644 --- a/serverless/load-balancing/overview.mdx +++ b/serverless/load-balancing/overview.mdx @@ -166,13 +166,28 @@ if health_check_with_retry("https://ENDPOINT_ID.api.runpod.ai", "RUNPOD_API_KEY" Use at least 3 retries with 5-10 second delays. +## When to use queue-based endpoints + +Use queue-based endpoints when you need: + +- **Job-based or long-running workloads** — queue-based is the right choice for jobs that run for extended periods. Load balancing is not suited for long-running tasks and would time out or drop the request. +- **Guaranteed execution** — every request is queued and processed, even during traffic spikes. No requests are dropped when workers are busy. +- **Batch or offline workloads** — processing large volumes of requests asynchronously where latency is not critical, such as nightly dataset processing, pre-computing embeddings, or running evaluations. +- **Automatic retries** — failed requests are automatically retried without any client-side logic. +- **Low-concurrency workloads** — tasks that run one at a time per worker and don't need multiple simultaneous connections. + +Choose queue-based when your workload can tolerate higher latency in exchange for reliability and guaranteed delivery. If your use case is a short, real-time request/response transaction, use a load balancing endpoint instead. + +## When to use load balancing endpoints ## When to use load balancing endpoints Use load balancing endpoints when you need: -- Direct access to your model's HTTP server. -- Internal batching systems (like vLLM). -- Non-JSON payloads. -- Multiple endpoints within a single worker. -- Lower latency for real-time applications. \ No newline at end of file +- **Direct server access** — route requests straight to your model's HTTP server without queueing overhead. +- **Internal batching systems** — frameworks like vLLM manage their own request batching internally and work best with direct access. +- **Non-JSON payloads** — send binary data, multipart uploads, or any payload format your HTTP server supports. +- **Multiple endpoints within a single worker** — expose custom URL paths and API contracts using any HTTP framework. +- **Low-latency real-time applications** — short, quick request/response transactions where the output is ephemeral, such as serving a file download or returning page stats. + +Choose load balancing when your workload is latency-sensitive and responses are immediate. If your use case involves long-running jobs or requires guaranteed delivery, use a queue-based endpoint instead. From 5af303e605a6a1f86a4d2c8231e853091472f56b Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Wed, 26 Aug 2026 12:50:47 -0400 Subject: [PATCH 2/4] Update overview.mdx --- serverless/load-balancing/overview.mdx | 2 -- 1 file changed, 2 deletions(-) diff --git a/serverless/load-balancing/overview.mdx b/serverless/load-balancing/overview.mdx index 3a5858f38..259c1ca94 100644 --- a/serverless/load-balancing/overview.mdx +++ b/serverless/load-balancing/overview.mdx @@ -180,8 +180,6 @@ Choose queue-based when your workload can tolerate higher latency in exchange fo ## When to use load balancing endpoints -## When to use load balancing endpoints - Use load balancing endpoints when you need: - **Direct server access** — route requests straight to your model's HTTP server without queueing overhead. From 3929fda8d7ee8dc649a12c8cd108524e7546b15b Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Wed, 26 Aug 2026 16:23:06 -0400 Subject: [PATCH 3/4] Update overview.mdx --- serverless/load-balancing/overview.mdx | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/serverless/load-balancing/overview.mdx b/serverless/load-balancing/overview.mdx index 259c1ca94..d7d360f28 100644 --- a/serverless/load-balancing/overview.mdx +++ b/serverless/load-balancing/overview.mdx @@ -168,13 +168,19 @@ Use at least 3 retries with 5-10 second delays. ## When to use queue-based endpoints +## When to use queue-based endpoints + +## When to use queue-based endpoints + Use queue-based endpoints when you need: -- **Job-based or long-running workloads** — queue-based is the right choice for jobs that run for extended periods. Load balancing is not suited for long-running tasks and would time out or drop the request. -- **Guaranteed execution** — every request is queued and processed, even during traffic spikes. No requests are dropped when workers are busy. -- **Batch or offline workloads** — processing large volumes of requests asynchronously where latency is not critical, such as nightly dataset processing, pre-computing embeddings, or running evaluations. -- **Automatic retries** — failed requests are automatically retried without any client-side logic. -- **Low-concurrency workloads** — tasks that run one at a time per worker and don't need multiple simultaneous connections. +- Job-based or long-running workloads that run for extended periods. Load balancing is not suited for long-running tasks and would time out or drop the request. +- Guaranteed execution where every request is queued and processed, even during traffic spikes. No requests are dropped when workers are busy. +- Batch or offline workloads processed asynchronously where latency is not critical, such as nightly dataset processing, pre-computing embeddings, or running evaluations. +- Automatic retries on failure without any client-side logic. +- Configurable concurrency that supports both low and high concurrency workloads depending on your needs. + +Choose queue-based when your workload can tolerate higher latency in exchange for reliability and guaranteed delivery. If your use case is a short, real-time request/response transaction, use a load balancing endpoint instead. Choose queue-based when your workload can tolerate higher latency in exchange for reliability and guaranteed delivery. If your use case is a short, real-time request/response transaction, use a load balancing endpoint instead. @@ -182,10 +188,10 @@ Choose queue-based when your workload can tolerate higher latency in exchange fo Use load balancing endpoints when you need: -- **Direct server access** — route requests straight to your model's HTTP server without queueing overhead. -- **Internal batching systems** — frameworks like vLLM manage their own request batching internally and work best with direct access. -- **Non-JSON payloads** — send binary data, multipart uploads, or any payload format your HTTP server supports. -- **Multiple endpoints within a single worker** — expose custom URL paths and API contracts using any HTTP framework. -- **Low-latency real-time applications** — short, quick request/response transactions where the output is ephemeral, such as serving a file download or returning page stats. +- Direct access to your model's HTTP server without queueing overhead. +- Internal batching systems like vLLM that manage their own request batching internally. +- Non-JSON payloads such as binary data or multipart uploads. +- Multiple endpoints within a single worker using custom URL paths and any HTTP framework. +- Low-latency real-time applications where responses are immediate and ephemeral, such as serving a file download or returning page stats. Choose load balancing when your workload is latency-sensitive and responses are immediate. If your use case involves long-running jobs or requires guaranteed delivery, use a queue-based endpoint instead. From 49621cd09b630139c369bd52b10e2e6badd597ae Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Wed, 26 Aug 2026 16:29:26 -0400 Subject: [PATCH 4/4] Update overview.mdx --- serverless/load-balancing/overview.mdx | 6 ------ 1 file changed, 6 deletions(-) diff --git a/serverless/load-balancing/overview.mdx b/serverless/load-balancing/overview.mdx index d7d360f28..a44c35c32 100644 --- a/serverless/load-balancing/overview.mdx +++ b/serverless/load-balancing/overview.mdx @@ -168,10 +168,6 @@ Use at least 3 retries with 5-10 second delays. ## When to use queue-based endpoints -## When to use queue-based endpoints - -## When to use queue-based endpoints - Use queue-based endpoints when you need: - Job-based or long-running workloads that run for extended periods. Load balancing is not suited for long-running tasks and would time out or drop the request. @@ -182,8 +178,6 @@ Use queue-based endpoints when you need: Choose queue-based when your workload can tolerate higher latency in exchange for reliability and guaranteed delivery. If your use case is a short, real-time request/response transaction, use a load balancing endpoint instead. -Choose queue-based when your workload can tolerate higher latency in exchange for reliability and guaranteed delivery. If your use case is a short, real-time request/response transaction, use a load balancing endpoint instead. - ## When to use load balancing endpoints Use load balancing endpoints when you need: