queue based and load balancing changes

serverless/load-balancing/overview.mdx

@@ -1,14 +1,14 @@
 ---
 title: "Overview"
 sidebarTitle: "Overview"
 description: "Deploy custom direct-access REST APIs with load balancing Serverless endpoints. Review configuration and operations guidance for Runpod Serverless."
 ---
 
 import { RequestsTooltip, QueueBasedEndpointsTooltip } from "/snippets/tooltips.jsx";
 
 Load balancing endpoints route incoming traffic directly to available workers, bypassing the queueing system. Unlike <QueueBasedEndpointsTooltip /> that process requests sequentially, load balancing distributes requests across your worker pool for lower latency.
 
 You can create custom REST endpoints accessible via a unique URL:
 
 ```
 https://ENDPOINT_ID.api.runpod.ai/YOUR_CUSTOM_PATH
@@ -28,15 +28,15 @@
 
 ### Queue-based endpoints
 
 With queue-based endpoints, <RequestsTooltip /> are placed in a queue and processed in order. They use the standard handler pattern (`def handler(job)`) and are accessed through fixed endpoints like `/run` and `/runsync`.
 
 These endpoints are better for tasks that can be processed asynchronously and guarantee request processing, similar to how TCP guarantees packet delivery in networking.
 
 ### Load balancing endpoints (new)
 
 Load balancing endpoints send requests directly to workers without queuing. You can use any HTTP framework such as FastAPI or Flask, and define custom URL paths and API contracts to suit your specific needs.
 
 These endpoints are ideal for real-time applications and streaming, but provide no queuing mechanism for request backlog, similar to UDP's behavior in networking.
 
 
 ## Endpoint type comparison table
@@ -44,15 +44,15 @@
  Aspect | Load balancing | Queue-based |
 |--------|----------------|-------------|
 | **Request flow** | Direct to worker HTTP server | Through queueing system |
 | **Implementation** | Custom HTTP server (FastAPI, Flask, etc.) | Handler function |
 | **API flexibility** | Custom URL paths, any HTTP capability | Fixed `/run` and `/runsync` endpoints |
 | **Backpressure** | Drops requests when overloaded | Queue buffering |
 | **Latency** | Lower (single-hop) | Higher (queue + worker) |
 | **Error handling** | No built-in retry | Automatic retries |
 
 ## Worker comparison
 
 **Queue-based worker** (traditional):
 
 ```python
 import runpod
@@ -64,7 +64,7 @@
 runpod.serverless.start({"handler": handler})
 ```
 
 **Load balancing worker** (custom HTTP server):
 
 ```python
 from fastapi import FastAPI
@@ -116,19 +116,19 @@
 
 | Variable | Default | Description |
 |----------|---------|-------------|
 | `PORT` | `80` | Main application server port |
 | `PORT_HEALTH` | Same as `PORT` | Health check endpoint port |
 | `HEALTH_CHECK_PATH` | `/ping` | Path the load balancer polls to check worker health |
 
 If using a custom port, add it to your endpoint's environment variables and expose it in container configuration (under **Expose HTTP Ports (Max 10)**).
 
 ## Timeouts and limits
 
 | Limit | Value |
 |-------|-------|
 | **Request timeout** | 2 min (no worker available) |
 | **Processing timeout** | 5.5 min (per request) |
 | **Payload limit** | 30 MB (request and response) |
 
 For payloads larger than 30 MB, use [network volumes](/storage/network-volumes) or implement chunking.
 
@@ -166,13 +166,26 @@
 
 Use at least 3 retries with 5-10 second delays.
 
+## When to use queue-based endpoints
+
+Use queue-based endpoints when you need:
+
+- Job-based or long-running workloads that run for extended periods. Load balancing is not suited for long-running tasks and would time out or drop the request.
+- Guaranteed execution where every request is queued and processed, even during traffic spikes. No requests are dropped when workers are busy.
+- Batch or offline workloads processed asynchronously where latency is not critical, such as nightly dataset processing, pre-computing embeddings, or running evaluations.
+- Automatic retries on failure without any client-side logic.
+- Configurable concurrency that supports both low and high concurrency workloads depending on your needs.
+
+Choose queue-based when your workload can tolerate higher latency in exchange for reliability and guaranteed delivery. If your use case is a short, real-time request/response transaction, use a load balancing endpoint instead.
 
 ## When to use load balancing endpoints
 
 Use load balancing endpoints when you need:
 
-- Direct access to your model's HTTP server.
-- Internal batching systems (like vLLM).
-- Non-JSON payloads.
-- Multiple endpoints within a single worker.
-- Lower latency for real-time applications.
+- Direct access to your model's HTTP server without queueing overhead.
+- Internal batching systems like vLLM that manage their own request batching internally.
+- Non-JSON payloads such as binary data or multipart uploads.
+- Multiple endpoints within a single worker using custom URL paths and any HTTP framework.
+- Low-latency real-time applications where responses are immediate and ephemeral, such as serving a file download or returning page stats.
+
+Choose load balancing when your workload is latency-sensitive and responses are immediate. If your use case involves long-running jobs or requires guaranteed delivery, use a queue-based endpoint instead.