Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions flash/configuration/parameters.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---

Check warning on line 1 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L1

Try to keep the LIX score (45.31) below 35.

Check warning on line 1 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L1

Try to keep the Coleman–Liau Index grade (12.55) below 9.

Check warning on line 1 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L1

Try to keep the Flesch–Kincaid grade level (9.26) below 8.

Check warning on line 1 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L1

Try to keep the SMOG grade (10.42) below 10.

Check warning on line 1 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L1

Try to keep the Flesch reading ease score (42.76) above 70.

Check warning on line 1 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L1

Try to keep the Automated Readability Index (8.23) below 8.
title: "Endpoint parameters"
sidebarTitle: "Parameters"
description: "Complete reference for all Endpoint class parameters. Review setup, configuration, deployment, and usage guidance for Runpod Flash."
Expand All @@ -10,34 +10,34 @@

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `name` | `str` | Endpoint name (required unless `id=` is used) | - |

Check warning on line 13 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L13

In general, use active voice instead of passive voice ('is used').
| `id` | `str` | Connect to existing endpoint by ID | `None` |
| `gpu` | `GpuGroup`, `GpuType`, or list | GPU type(s) for the endpoint | `GpuGroup.ANY` |

Check warning on line 15 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L15

Don't use plurals in parentheses such as in 'type(s)'.

Check warning on line 15 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L15

Use parentheses judiciously.
| `cpu` | `str` or `CpuInstanceType` | CPU instance type (mutually exclusive with `gpu`) | `None` |
| `workers` | `int` or `(min, max)` | Worker scaling configuration | `(0, 1)` |
| `idle_timeout` | `int` | Seconds before scaling down idle workers | `60` |
| `dependencies` | `list[str]` | Python packages to install | `None` |
| `system_dependencies` | `list[str]` | System packages to install (apt) | `None` |

Check warning on line 20 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L20

Use parentheses judiciously.
| `accelerate_downloads` | `bool` | Enable download acceleration | `True` |
| `volume` | `NetworkVolume` or list | Network volume(s) for persistent storage | `None` |

Check warning on line 22 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L22

Don't use plurals in parentheses such as in 'volume(s)'.

Check warning on line 22 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L22

Use parentheses judiciously.
| `datacenter` | `DataCenter`, list, or `None` | Datacenter(s) for deployment | `None` (all DCs) |

Check warning on line 23 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L23

Don't use plurals in parentheses such as in 'Datacenter(s)'.

Check warning on line 23 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L23

Use parentheses judiciously.

Check warning on line 23 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L23

Use parentheses judiciously.
| `env` | `dict[str, str]` | Environment variables | `None` |
| `gpu_count` | `int` | GPUs per worker | `1` |
| `execution_timeout_ms` | `int` | Max execution time in milliseconds | `0` (no limit) |

Check warning on line 26 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L26

Use parentheses judiciously.
| `flashboot` | `bool` | Enable Flashboot fast startup | `True` |
| `image` | `str` | Custom Docker image to deploy | `None` |
| `scaler_type` | `ServerlessScalerType` | Scaling strategy | auto |
| `scaler_value` | `int` | Scaling threshold | `4` |
| `template` | `PodTemplate` | Pod template overrides | `None` |
| `min_cuda_version` | `str` or `CudaVersion` | Minimum CUDA version for GPU host selection | `"12.8"` (GPU) / `None` (CPU) |

Check warning on line 32 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L32

Spell out 'CUDA', if it's unfamiliar to the audience.
| `python_version` | `str` | Python version for the worker image | Local Python |

## Parameter details

### name

Check warning on line 37 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L37

'name' should use sentence-style capitalization.

**Type**: `str`
**Required**: Yes (unless `id=` is specified)

Check warning on line 40 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L40

'Yes' should be in lowercase.

Check warning on line 40 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L40

In general, use active voice instead of passive voice ('is specified').

The endpoint name visible in the [Runpod console](https://console.runpod.io/serverless). Use descriptive names to easily identify endpoints.

Expand All @@ -50,12 +50,12 @@
Use naming conventions like `image-generation-prod` or `batch-processor-dev` to organize your endpoints.
</Tip>

### id

Check warning on line 53 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L53

'id' should use sentence-style capitalization.

**Type**: `str`
**Default**: `None`

Connect to an existing deployed endpoint by its ID. When `id` is specified, `name` is not required.

Check warning on line 58 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L58

In general, use active voice instead of passive voice ('is specified').

Check warning on line 58 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L58

Use 'isn't' instead of 'is not'.

```python
# Connect to existing endpoint
Expand All @@ -66,10 +66,10 @@
result = await ep.post("/inference", {"data": "..."})
```

### gpu

Check warning on line 69 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L69

'gpu' should use sentence-style capitalization.

**Type**: `GpuGroup`, `GpuType`, or `list[GpuGroup | GpuType]`
**Default**: `GpuGroup.ANY` (if neither `gpu` nor `cpu` is specified)

Check warning on line 72 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L72

In general, use active voice instead of passive voice ('is specified').

Specifies GPU hardware for the endpoint. Accepts a single GPU type/group or a list for fallback strategies.

Expand All @@ -91,7 +91,7 @@

See [GPU types](/flash/configuration/gpu-types) for all available options.

### cpu

Check warning on line 94 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L94

'cpu' should use sentence-style capitalization.

**Type**: `str` or `CpuInstanceType`
**Default**: `None`
Expand All @@ -112,12 +112,12 @@

See [CPU types](/flash/configuration/cpu-types) for all available options.

### workers

Check warning on line 115 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L115

'workers' should use sentence-style capitalization.

**Type**: `int` or `tuple[int, int]`
**Default**: `(0, 1)`

Controls worker scaling. Accepts either a single integer (max workers with min=0) or a tuple of (min, max).

Check warning on line 120 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L120

Use parentheses judiciously.

Check warning on line 120 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L120

Use parentheses judiciously.

```python
# Just max: scales from 0 to 5
Expand All @@ -135,13 +135,13 @@
- `workers=(1, N)`: Avoid cold starts by keeping at least one worker warm
- `workers=(N, N)`: Fixed worker count for consistent performance

### idle_timeout

Check warning on line 138 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L138

'idle_timeout' should use sentence-style capitalization.

**Type**: `int`
**Default**: `60`
**Valid range**: 1-3600 seconds

Number of seconds workers will stay active (running) after completing a request, waiting for additional requests before scaling down (to minimum workers).

Check warning on line 144 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L144

Avoid using 'will'.

Check warning on line 144 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L144

Use parentheses judiciously.

Check warning on line 144 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L144

Use parentheses judiciously.

```python
# Quick scale-down for cost savings
Expand All @@ -156,7 +156,7 @@
- `60-120 seconds`: Balanced, variable traffic patterns
- `120-300 seconds`: Latency-optimized, consistent traffic

### dependencies

Check warning on line 159 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L159

'dependencies' should use sentence-style capitalization.

**Type**: `list[str]`
**Default**: `None`
Expand All @@ -176,7 +176,7 @@
Packages must be imported **inside** the function body, not at the top of your file.
</Warning>

### system_dependencies

Check warning on line 179 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L179

'system_dependencies' should use sentence-style capitalization.

**Type**: `list[str]`
**Default**: `None`
Expand All @@ -193,12 +193,12 @@
async def process_video(data): ...
```

### accelerate_downloads

Check warning on line 196 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L196

'accelerate_downloads' should use sentence-style capitalization.

**Type**: `bool`
**Default**: `True`

Enables faster downloads for dependencies, models, and large files. Disable if you encounter compatibility issues.

Check warning on line 201 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L201

Use 'turn off' or 'off' instead of 'Disable'.

```python
@Endpoint(
Expand All @@ -209,12 +209,12 @@
async def process(data): ...
```

### volume

Check warning on line 212 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L212

'volume' should use sentence-style capitalization.

**Type**: `NetworkVolume` or `list[NetworkVolume]`
**Default**: `None`

Attaches network volume(s) for persistent storage. Volumes are mounted at `/runpod-volume/`. Flash uses the volume `name` to find an existing volume or create a new one. Each volume is tied to a specific datacenter.

Check warning on line 217 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L217

Don't use plurals in parentheses such as in 'volume(s)'.

Check warning on line 217 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L217

Use parentheses judiciously.

Check warning on line 217 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L217

In general, use active voice instead of passive voice ('are mounted').

Check warning on line 217 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L217

In general, use active voice instead of passive voice ('is tied').

```python
from runpod_flash import Endpoint, GpuGroup, DataCenter, NetworkVolume
Expand All @@ -234,7 +234,7 @@
...
```

For multi-datacenter deployments, pass a list of volumes (one per datacenter):

Check warning on line 237 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L237

Use parentheses judiciously.

```python
from runpod_flash import Endpoint, GpuGroup, DataCenter, NetworkVolume
Expand Down Expand Up @@ -265,12 +265,12 @@

See [Storage](/flash/configuration/storage) for setup instructions.

### datacenter

Check warning on line 268 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L268

'datacenter' should use sentence-style capitalization.

**Type**: `DataCenter`, `list[DataCenter]`, `str`, `list[str]`, or `None`
**Default**: `None` (all available datacenters)

Check warning on line 271 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L271

Use parentheses judiciously.

Specifies the datacenter(s) for worker deployment. When set to `None`, the endpoint is available in all datacenters.

Check warning on line 273 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L273

Don't use plurals in parentheses such as in 'datacenter(s)'.

Check warning on line 273 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L273

Use parentheses judiciously.

```python
from runpod_flash import Endpoint, GpuGroup, DataCenter
Expand Down Expand Up @@ -308,12 +308,12 @@

| Value | Location |
|-------|----------|
| `DataCenter.US_CA_2` | US - California |

Check warning on line 311 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L311

Try to avoid using first-person plural like 'US'.
| `DataCenter.US_GA_2` | US - Georgia |

Check warning on line 312 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L312

Try to avoid using first-person plural like 'US'.
| `DataCenter.US_IL_1` | US - Illinois |

Check warning on line 313 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L313

Try to avoid using first-person plural like 'US'.
| `DataCenter.US_KS_2` | US - Kansas |

Check warning on line 314 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L314

Try to avoid using first-person plural like 'US'.
| `DataCenter.US_MD_1` | US - Maryland |

Check warning on line 315 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L315

Try to avoid using first-person plural like 'US'.
| `DataCenter.US_MO_1` | US - Missouri |

Check warning on line 316 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L316

Try to avoid using first-person plural like 'US'.
| `DataCenter.US_MO_2` | US - Missouri |
| `DataCenter.US_NC_1` | US - North Carolina |
| `DataCenter.US_NC_2` | US - North Carolina |
Expand All @@ -328,7 +328,7 @@
CPU endpoints are restricted to `CPU_DATACENTERS`, which currently only includes `EU_RO_1`.
</Note>

### env

Check warning on line 331 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L331

'env' should use sentence-style capitalization.

**Type**: `dict[str, str]`
**Default**: `None`
Expand Down Expand Up @@ -374,7 +374,7 @@
Environment variables are excluded from configuration hashing. Changing environment values won't trigger endpoint recreation, making it easy to rotate API keys.
</Note>

### gpu_count

Check warning on line 377 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L377

'gpu_count' should use sentence-style capitalization.

**Type**: `int`
**Default**: `1`
Expand All @@ -391,12 +391,12 @@
async def train(data): ...
```

### execution_timeout_ms

Check warning on line 394 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L394

'execution_timeout_ms' should use sentence-style capitalization.

**Type**: `int`
**Default**: `0` (no limit)

Check warning on line 397 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L397

Use parentheses judiciously.

Maximum execution time for a single job in milliseconds. Jobs exceeding this timeout are terminated.

Check warning on line 399 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L399

In general, use active voice instead of passive voice ('are terminated').

```python
# 5 minute timeout
Expand All @@ -420,7 +420,7 @@
The Flash SDK's `runsync()` method uses your `execution_timeout_ms` value as the client-side HTTP timeout. If set to a positive value, the SDK waits that duration for the job to complete. If unset or set to `0`, the SDK defaults to a 60-second timeout. For long-running inference jobs, set `execution_timeout_ms` to prevent premature timeouts.
</Note>

### flashboot

Check warning on line 423 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L423

'flashboot' should use sentence-style capitalization.

**Type**: `bool`
**Default**: `True`
Expand All @@ -438,7 +438,7 @@

Set to `False` for debugging or compatibility reasons.

### image

Check warning on line 441 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L441

'image' should use sentence-style capitalization.

**Type**: `str`
**Default**: `None`
Expand All @@ -461,14 +461,14 @@

See [Custom Docker images](/flash/custom-docker-images) for complete documentation.

### scaler_type

Check warning on line 464 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L464

'scaler_type' should use sentence-style capitalization.

**Type**: `ServerlessScalerType`
**Default**: Auto-selected based on endpoint type

Check warning on line 467 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L467

'Auto' should be in lowercase.

Scaling algorithm strategy. Defaults are automatically set:
- Queue-based: `QUEUE_DELAY` (scales based on queue depth)

Check warning on line 470 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L470

Use parentheses judiciously.
- Load-balanced: `REQUEST_COUNT` (scales based on active requests)

Check warning on line 471 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L471

Use parentheses judiciously.

```python
from runpod_flash import Endpoint, ServerlessScalerType
Expand All @@ -481,7 +481,7 @@
async def process(data): ...
```

### scaler_value

Check warning on line 484 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L484

'scaler_value' should use sentence-style capitalization.

**Type**: `int`
**Default**: `4`
Expand All @@ -498,7 +498,7 @@
async def process(data): ...
```

### template

Check warning on line 501 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L501

'template' should use sentence-style capitalization.

**Type**: `PodTemplate`
**Default**: `None`
Expand All @@ -519,37 +519,41 @@
async def process(data): ...
```

## PodTemplate

Check warning on line 522 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L522

'PodTemplate' should use sentence-style capitalization.

`PodTemplate` provides advanced pod configuration options:

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `containerDiskInGb` | `int` | Container disk size in GB | 64 |
| `containerRegistryAuthId` | `str` | ID of saved registry credentials for pulling private container images | `None` |
| `env` | `list[dict]` | Environment variables as list of `{"key": "...", "value": "..."}` | `None` |

```python
from runpod_flash import PodTemplate

template = PodTemplate(
containerDiskInGb=100,
containerRegistryAuthId="REGISTRY_CREDENTIAL_ID",
env=[
{"key": "PYTHONPATH", "value": "/workspace"},
{"key": "CUDA_VISIBLE_DEVICES", "value": "0"}
]
)
```

To find the registry credential ID for `containerRegistryAuthId`, list your saved credentials with `runpodctl registry list` or check **Container Registry Authentication** in the [Runpod console settings](https://console.runpod.io/user/settings). See [Private images](/flash/custom-docker-images#private-images) for a complete example.

<Tip>
For simple environment variables, use the `env` parameter on `Endpoint` instead of `PodTemplate.env`.
</Tip>

### min_cuda_version

Check warning on line 551 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L551

'min_cuda_version' should use sentence-style capitalization.

**Type**: `str` or `CudaVersion`
**Default**: `"12.8"` for GPU endpoints, `None` for CPU endpoints

Specifies the minimum CUDA driver version required on the host machine. GPU endpoints default to `"12.8"` to ensure workers run on hosts with recent CUDA drivers.

Check warning on line 556 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L556

Spell out 'CUDA', if it's unfamiliar to the audience.

Check warning on line 556 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L556

Spell out 'CUDA', if it's unfamiliar to the audience.

```python
from runpod_flash import Endpoint, GpuType, CudaVersion
Expand Down Expand Up @@ -581,18 +585,18 @@
Valid CUDA versions: `CudaVersion.V11_1`, `V11_4`, `V11_7`, `V11_8`, `V12_0`, `V12_1`, `V12_2`, `V12_3`, `V12_4`, `V12_6`, `V12_8` (or equivalent strings like `"12.4"`). Invalid values raise a `ValueError`.
</Note>

### python_version

Check warning on line 588 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L588

'python_version' should use sentence-style capitalization.

**Type**: `str`
**Default**: Local Python version

Check warning on line 591 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L591

'Local' should be in lowercase.

Sets the Python version for the worker image. Supported values: `"3.10"`, `"3.11"`, `"3.12"`, and `"3.13"`.

When you don't specify a Python version, Flash matches your local interpreter (the Python version you run Flash from). The resolution order is:

Check warning on line 595 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L595

Use parentheses judiciously.

1. `--python-version` CLI flag (highest priority)

Check warning on line 597 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L597

Use 'command-line tool' instead of 'CLI'.

Check warning on line 597 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L597

Use parentheses judiciously.
2. `python_version` declared on resource configs
3. Your local Python version (`sys.version_info`)

Check warning on line 599 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L599

Use parentheses judiciously.

```python
from runpod_flash import Endpoint, GpuGroup
Expand Down Expand Up @@ -624,11 +628,11 @@
Python 3.10 reaches end-of-life on 2026-10-31. Consider migrating to Python 3.11 or later.
</Note>

If your local Python version is not supported (for example, 3.9 or 3.14), the build fails with an actionable error message listing the supported versions.

Check warning on line 631 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L631

Use 'isn't' instead of 'is not'.

Check warning on line 631 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L631

Use parentheses judiciously.

The `--python-version` CLI flag on `flash build` and `flash deploy` overrides both per-resource declarations and local interpreter detection.

Check warning on line 633 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L633

Use 'command-line tool' instead of 'CLI'.

## EndpointJob

Check warning on line 635 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L635

'EndpointJob' should use sentence-style capitalization.

When using `Endpoint(id=...)` or `Endpoint(image=...)`, the `.run()` method returns an `EndpointJob` object for async operations:

Expand Down Expand Up @@ -661,25 +665,25 @@
### Changes that recreate workers

These changes restart all workers:
- GPU configuration (`gpu`, `gpu_count`)

Check warning on line 668 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L668

Use parentheses judiciously.
- CPU instance type (`cpu`)

Check warning on line 669 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L669

Use parentheses judiciously.
- Docker image (`image`)

Check warning on line 670 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L670

Use parentheses judiciously.
- Storage (`volume`)

Check warning on line 671 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L671

Use parentheses judiciously.
- Datacenter (`datacenter`)

Check warning on line 672 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L672

Use parentheses judiciously.
- Flashboot setting (`flashboot`)

Check warning on line 673 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L673

Use parentheses judiciously.
- CUDA version requirement (`min_cuda_version`)

Check warning on line 674 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L674

Use parentheses judiciously.

Check warning on line 674 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L674

Spell out 'CUDA', if it's unfamiliar to the audience.
- Python version (`python_version`)

Check warning on line 675 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L675

Use parentheses judiciously.

Workers are temporarily unavailable during recreation (typically 30-90 seconds).

Check warning on line 677 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L677

Use parentheses judiciously.

### Changes that update settings only

These changes apply immediately with no downtime:
- Worker scaling (`workers`)

Check warning on line 682 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L682

Use parentheses judiciously.
- Timeouts (`idle_timeout`, `execution_timeout_ms`)

Check warning on line 683 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L683

Use parentheses judiciously.
- Scaler settings (`scaler_type`, `scaler_value`)

Check warning on line 684 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L684

Use parentheses judiciously.
- Environment variables (`env`)

Check warning on line 685 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L685

Use parentheses judiciously.
- Endpoint name (`name`)

Check warning on line 686 in flash/configuration/parameters.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/configuration/parameters.mdx#L686

Use parentheses judiciously.

```python
# First deployment
Expand Down
32 changes: 31 additions & 1 deletion flash/custom-docker-images.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---

Check warning on line 1 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L1

Try to keep the Flesch reading ease score (46.69) above 70.

Check warning on line 1 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L1

Try to keep the SMOG grade (10.46) below 10.

Check warning on line 1 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L1

Try to keep the Flesch–Kincaid grade level (8.82) below 8.

Check warning on line 1 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L1

Try to keep the Coleman–Liau Index grade (12.50) below 9.

Check warning on line 1 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L1

Try to keep the LIX score (39.39) below 35.

Check warning on line 1 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L1

Try to keep the Automated Readability Index (8.25) below 8.
title: "Use custom containers with Flash"
sidebarTitle: "Custom containers"
description: "Deploy pre-built Docker images with Flash using Endpoint. Review setup, configuration, deployment, and usage guidance for Runpod Flash."
Expand All @@ -12,8 +12,8 @@

Use custom Docker images when you need:

- **Pre-built inference servers**: vLLM, TensorRT-LLM, or other specialized serving frameworks.

Check warning on line 15 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L15

Spell out 'LLM', if it's unfamiliar to the audience.
- **System-level dependencies**: Custom CUDA versions, cuDNN, or system libraries not installable via `pip`.

Check warning on line 16 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L16

Spell out 'CUDA', if it's unfamiliar to the audience.
- **Baked-in models**: Large models pre-downloaded in the image to avoid runtime downloads.
- **Existing Serverless workers**: You already have a working Runpod Serverless Docker image.

Expand All @@ -23,7 +23,7 @@

## Available Docker images

### Official Runpod workers

Check warning on line 26 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L26

'Official Runpod workers' should use sentence-style capitalization.

Runpod provides pre-built worker images for common frameworks:

Expand Down Expand Up @@ -108,7 +108,37 @@
</Step>
</Steps>

## Private images

To deploy an image from a private registry, first save your registry credentials in the Runpod console, then reference the credential ID in your endpoint's `PodTemplate` using the `containerRegistryAuthId` field:

```python
from runpod_flash import Endpoint, GpuType, PodTemplate

vllm = Endpoint(
name="private-vllm",
image="ghcr.io/your-org/private-worker:latest",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
template=PodTemplate(
containerDiskInGb=64,
containerRegistryAuthId="REGISTRY_CREDENTIAL_ID"
)
)
```

<Warning>
Without `containerRegistryAuthId`, Runpod cannot pull a private image, and workers fail to start.
</Warning>

To find the credential ID, list your saved registry credentials with the Runpod CLI:

Check warning on line 133 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L133

Use 'command-line tool' instead of 'CLI'.

```bash
runpodctl registry list
```

To create a new credential, use [`runpodctl registry create`](/runpodctl/reference/runpodctl-registry), or navigate to [Settings](https://console.runpod.io/user/settings) in the Runpod console and scroll down to **Container Registry Authentication**. For GitHub Container Registry, use a personal access token scoped to `read:packages`.

## Complete example: vLLM inference

Check warning on line 141 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L141

'Complete example: vLLM inference' should use sentence-style capitalization.

This example deploys vLLM and makes inference requests:

Expand Down Expand Up @@ -176,7 +206,7 @@
)
```

### CPU endpoints

Check warning on line 209 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L209

'CPU endpoints' should use sentence-style capitalization.

For CPU workloads, use the `cpu` parameter:

Expand Down Expand Up @@ -209,7 +239,7 @@
print(job.error) # Error message if failed
```

### HTTP requests

Check warning on line 242 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L242

'HTTP requests' should use sentence-style capitalization.

Use `.get()`, `.post()`, `.put()`, `.delete()` for direct HTTP calls:

Expand All @@ -228,7 +258,7 @@
)
```

## EndpointJob reference

Check warning on line 261 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L261

'EndpointJob reference' should use sentence-style capitalization.

The `.run()` method returns an `EndpointJob` for async operations:

Expand All @@ -250,7 +280,7 @@
## Limitations

- **Input format**: Queue-based calls require `{"input": {...}}` format.
- **Code execution**: Cannot execute arbitrary Python code remotely. Your Docker image must include all logic.

Check warning on line 283 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L283

Use 'can't' instead of 'Cannot'.
- **@Endpoint decorator**: The decorator pattern doesn't work with `image=`. Use the instance pattern instead.
- **Handler required**: Your Docker image must implement a Runpod Serverless [handler function](/serverless/workers/handler-functions).

Expand All @@ -258,7 +288,7 @@

### Endpoint fails to initialize

**Problem**: Workers fail to start or crash immediately.

Check warning on line 291 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L291

'Workers' should be in lowercase.

**Solutions**:
- Verify your Docker image is compatible with [Runpod Serverless](/serverless/overview).
Expand All @@ -268,7 +298,7 @@

### Out of memory errors

**Problem**: Workers crash with CUDA OOM or RAM errors.

Check warning on line 301 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L301

'Workers' should be in lowercase.

Check warning on line 301 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L301

Spell out 'CUDA', if it's unfamiliar to the audience.

Check warning on line 301 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L301

Spell out 'OOM', if it's unfamiliar to the audience.

**Solutions**:
- Use a larger GPU: `gpu=GpuType.NVIDIA_A100_80GB_PCIe`
Expand All @@ -278,11 +308,11 @@

### Authentication errors

**Problem**: Cannot download gated models or private images.

Check warning on line 311 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L311

Use 'can't' instead of 'Cannot'.

Check warning on line 311 in flash/custom-docker-images.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

flash/custom-docker-images.mdx#L311

'Cannot' should be in lowercase.

**Solutions**:
- Add `HF_TOKEN` to `env` for Hugging Face gated models.
- Configure Docker registry authentication in [Runpod console](https://console.runpod.io/user/settings) for private images.
- For private images, save registry credentials in the [Runpod console](https://console.runpod.io/user/settings) and pass the credential ID to your endpoint with `PodTemplate(containerRegistryAuthId=...)`. See [Private images](#private-images).

## Next steps

Expand Down
Loading