From 82323086574956ee655935a61bfa03a7e553133a Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 25 Aug 2026 10:42:31 -0400 Subject: [PATCH] docs: clarify Flash Endpoint gpu= single-type vs list placement semantics --- flash/configuration/gpu-types.mdx | 43 ++++++++++++++++++++---------- flash/configuration/parameters.mdx | 14 +++++++--- flash/create-endpoints.mdx | 4 ++- flash/troubleshooting.mdx | 8 +++--- 4 files changed, 48 insertions(+), 21 deletions(-) diff --git a/flash/configuration/gpu-types.mdx b/flash/configuration/gpu-types.mdx index 91889b89c..74578cec2 100644 --- a/flash/configuration/gpu-types.mdx +++ b/flash/configuration/gpu-types.mdx @@ -64,6 +64,8 @@ async def dev_infer(data: dict) -> dict: The `GpuType` enum provides access to specific GPU models. Use these when you need exact hardware characteristics. +A single `GpuType` requests that exact card — this is the only form guaranteed to pin workers to a specific GPU model. If you pass a list of `GpuType` values, placement is advisory at the pool/VRAM-tier level instead. See [GPU selection behavior](/flash/configuration/gpu-types#gpu-selection-behavior) for details. + ### Available GPU types | GpuType | GPU Model | VRAM | Architecture | @@ -95,24 +97,26 @@ The `GpuType` enum provides access to specific GPU models. Use these when you ne ```python from runpod_flash import Endpoint, GpuType -# Single specific GPU +# Single specific GPU (pins the exact card) @Endpoint(name="inference", gpu=GpuType.NVIDIA_A100_80GB_PCIe) async def infer(data: dict) -> dict: ... -# Multiple specific GPUs (fallback strategy) +# Multiple specific GPUs (requests any card in this set) @Endpoint( name="flexible", gpu=[ - GpuType.NVIDIA_A100_80GB_PCIe, # Try A100 PCIe first - GpuType.NVIDIA_A100_SXM4_80GB, # Fall back to A100 SXM4 - GpuType.NVIDIA_A40 # Final fallback to A40 + GpuType.NVIDIA_A100_80GB_PCIe, + GpuType.NVIDIA_A100_SXM4_80GB, + GpuType.NVIDIA_A40 ] ) async def flexible_infer(data: dict) -> dict: ... ``` +When you pass a list, the Flash SDK converts each `GpuType` to its GPU pool (`GpuGroup`) when it creates the endpoint, excluding any pool members you didn't list. Placement within the selected pools is advisory: workers may be scheduled on a different card with equivalent VRAM rather than one of the exact models you listed, and list order is not preserved. Pass a single `GpuType` when your workload requires an exact GPU model. + ## Advanced fallback strategies Combine `GpuGroup` and `GpuType` for robust availability: @@ -123,9 +127,9 @@ from runpod_flash import Endpoint, GpuGroup, GpuType @Endpoint( name="hybrid-selection", gpu=[ - GpuType.NVIDIA_A100_80GB_PCIe, # Specific GPU first - GpuGroup.AMPERE_48, # Pool fallback - GpuGroup.ANY # Ultimate fallback + GpuType.NVIDIA_A100_80GB_PCIe, # Specific GPU model + GpuGroup.AMPERE_48, # Any card in the 48GB Ampere pool + GpuGroup.ANY # Any GPU: expands to all pools for maximum availability ] ) async def infer(data: dict) -> dict: @@ -135,18 +139,25 @@ async def infer(data: dict) -> dict: ## GPU selection behavior **Single GPU type:** -Flash waits for this specific GPU to become available. Jobs stay in queue until capacity is available. +Passing a single `GpuType` requests that exact GPU model. This is the only form guaranteed to pin workers to a specific card. Flash waits for that model to become available, and jobs stay in queue until capacity is free. ```python -gpu=GpuGroup.AMPERE_80 # Only A100 80GB +gpu=GpuType.NVIDIA_GEFORCE_RTX_4090 # Only the RTX 4090 ``` -**Multiple GPU types (fallback):** -Flash attempts to provision in the order specified. +**Single GPU pool:** +Passing a single `GpuGroup` requests any card in that pool. + +```python +gpu=GpuGroup.AMPERE_80 # Any A100 80GB card +``` + +**A list of GPU types or pools:** +A list requests any one of the listed options. It is not an ordered preference: the SDK does not preserve list order, and placement within the selected pools is advisory — workers may be scheduled on a different card with equivalent VRAM rather than one of the exact models you listed. Be aware of this when benchmarking or when your workload depends on a specific GPU model. ```python gpu=[GpuGroup.AMPERE_80, GpuGroup.AMPERE_48, GpuGroup.ADA_24] -# Tries: A100 → A40/A6000 → RTX 4090 +# Any card from the 80GB, 48GB, or 24GB pools ``` **GpuGroup.ANY:** @@ -156,8 +167,12 @@ Flash selects the first available GPU based on current capacity. gpu=GpuGroup.ANY # Fastest provisioning, unpredictable GPU type ``` + +There is currently no API field that reports which GPU model a worker was actually placed on. The endpoint configuration reflects what you requested, not where workers ran, so GPU substitution is invisible in the API. To verify placement, query the GPU from inside the worker (for example, with `nvidia-smi`). + + -**For production**: Use specific GPU types for predictable cost and performance. +**For production**: Pass a single `GpuType` when you need a specific card for predictable cost and performance. **For development**: Use `GpuGroup.ANY` for fastest iteration. diff --git a/flash/configuration/parameters.mdx b/flash/configuration/parameters.mdx index b1f551b23..eb4752bba 100644 --- a/flash/configuration/parameters.mdx +++ b/flash/configuration/parameters.mdx @@ -71,7 +71,11 @@ result = await ep.post("/inference", {"data": "..."}) **Type**: `GpuGroup`, `GpuType`, or `list[GpuGroup | GpuType]` **Default**: `GpuGroup.ANY` (if neither `gpu` nor `cpu` is specified) -Specifies GPU hardware for the endpoint. Accepts a single GPU type/group or a list for fallback strategies. +Specifies GPU hardware for the endpoint. Accepts a single GPU pool/type or a list of pools and types: + +- A single `GpuType` requests that exact GPU model. This is the only form guaranteed to pin workers to a specific card. +- A single `GpuGroup` requests any card in that pool. +- A list requests any of the listed options. The SDK converts each `GpuType` in a list to its GPU pool, so list placement is advisory at the pool/VRAM-tier level: workers may run on a different card with equivalent VRAM rather than one of the exact models you listed, and list order is not preserved. ```python from runpod_flash import Endpoint, GpuType, GpuGroup @@ -84,12 +88,16 @@ async def infer(data): ... @Endpoint(name="rtx-worker", gpu=GpuType.NVIDIA_GEFORCE_RTX_4090) async def process(data): ... -# Multiple types for fallback +# Multiple types: requests any card in the set @Endpoint(name="flexible", gpu=[GpuType.NVIDIA_A100_80GB_PCIe, GpuType.NVIDIA_RTX_A6000, GpuType.NVIDIA_GEFORCE_RTX_4090]) async def flexible_infer(data): ... ``` -See [GPU types](/flash/configuration/gpu-types) for all available options. + +There is currently no API field that reports which GPU model a worker was actually placed on — the endpoint configuration reflects what you requested, not where workers ran. To verify placement, query the GPU from inside the worker (for example, with `nvidia-smi`). + + +See [GPU types](/flash/configuration/gpu-types) for all available options, including [GPU selection behavior](/flash/configuration/gpu-types#gpu-selection-behavior). ### cpu diff --git a/flash/create-endpoints.mdx b/flash/create-endpoints.mdx index b2ce31eae..74e814ae5 100644 --- a/flash/create-endpoints.mdx +++ b/flash/create-endpoints.mdx @@ -140,11 +140,13 @@ async def infer(data: dict) -> dict: ... @Endpoint(name="rtx-worker", gpu=GpuType.NVIDIA_GEFORCE_RTX_4090) async def render(data: dict) -> dict: ... -# Use multiple GPU types for better availability +# Use multiple GPU types for better availability (requests any card in the set) @Endpoint(name="flexible", gpu=[GpuType.NVIDIA_GEFORCE_RTX_4090, GpuType.NVIDIA_RTX_A5000]) async def process(data: dict) -> dict: ... ``` +A single `GpuType` requests that exact card; exact-card placement is only guaranteed with a single `GpuType`. A list requests any card in the set and is treated at the pool/VRAM-tier level, so a worker may run on a different card with equivalent VRAM rather than one of the models you listed. See [GPU selection behavior](/flash/configuration/gpu-types#gpu-selection-behavior) for details. + If neither `gpu=` nor `cpu=` is specified, GPU defaults to `GpuGroup.ANY`. ### CPU endpoints diff --git a/flash/troubleshooting.mdx b/flash/troubleshooting.mdx index cb6da5a08..a5c86efc9 100644 --- a/flash/troubleshooting.mdx +++ b/flash/troubleshooting.mdx @@ -558,13 +558,15 @@ Circuit breaker is open. Retry in [N] seconds @Endpoint( name="flexible", gpu=[ - GpuType.NVIDIA_A100_80GB_PCIe, # First choice - GpuType.NVIDIA_RTX_A6000, # Fallback - GpuType.NVIDIA_GEFORCE_RTX_4090 # Second fallback + GpuType.NVIDIA_A100_80GB_PCIe, + GpuType.NVIDIA_RTX_A6000, + GpuType.NVIDIA_GEFORCE_RTX_4090 ] ) ``` + A list requests any card in the set — the SDK does not preserve list order. See [GPU selection behavior](/flash/configuration/gpu-types#gpu-selection-behavior) for details. + 2. **Use GpuGroup.ANY**: For development, accept any available GPU: ```python gpu=GpuGroup.ANY