Skip to content

feat: upgrade vllm from 0.27.1 to 0.28.0 for CUDA and ROCm backends #261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
axsapronov wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: upgrade vllm from 0.27.1 to 0.28.0 for CUDA and ROCm backends #261

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ The following table lists the supported accelerated backends and their correspon

| CUDA Version <br/> (Variant) | vLLM | SGLang | VoxBox |
|------------------------------|------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------|
| 13.0 | **`0.27.1`**, `0.25.1`, `0.24.0`, `0.22.1`, <br/>`0.21.0`, `0.20.2`, <br/>`0.19.1`, `0.18.1` | `0.5.18`, `0.5.15.post1`, <br/>`0.5.14`, `0.5.12.post1` | |
| 12.9 | **`0.27.1`**, `0.25.1`, `0.24.0`, `0.22.1`, <br/>`0.21.0`, `0.20.2`, <br/>`0.19.1`, `0.18.1`, <br/>`0.17.1`, `0.16.0`, <br/>`0.15.1`, `0.14.1`, <br/>`0.13.0`, `0.12.0`, <br/>`0.11.2` | `0.5.18`, `0.5.15.post1`, `0.5.14`, `0.5.12.post1`, <br/>`0.5.9`, `0.5.8.post1`, <br/>`0.5.7`, `0.5.6.post2` | |
| 13.0 | **`0.28.0`**, `0.27.1`, `0.25.1`, `0.24.0`, `0.22.1`, <br/>`0.21.0`, `0.20.2`, <br/>`0.19.1`, `0.18.1` | `0.5.18`, `0.5.15.post1`, <br/>`0.5.14`, `0.5.12.post1` | |
| 12.9 | **`0.28.0`**, `0.27.1`, `0.25.1`, `0.24.0`, `0.22.1`, <br/>`0.21.0`, `0.20.2`, <br/>`0.19.1`, `0.18.1`, <br/>`0.17.1`, `0.16.0`, <br/>`0.15.1`, `0.14.1`, <br/>`0.13.0`, `0.12.0`, <br/>`0.11.2` | `0.5.18`, `0.5.15.post1`, `0.5.14`, `0.5.12.post1`, <br/>`0.5.9`, `0.5.8.post1`, <br/>`0.5.7`, `0.5.6.post2` | |
| 12.8 | `0.17.1`, `0.16.0`, <br/>`0.15.1`, `0.14.1`, <br/>`0.13.0`, `0.12.0`, <br/>`0.11.2`, `0.10.2` | `0.5.9`, `0.5.8.post1`, <br/>`0.5.7`, `0.5.6.post2`, <br/>`0.5.5.post3` | `0.0.21` |
| 12.6 | `0.15.1`, `0.14.1`, <br/>`0.13.0`, `0.12.0`, <br/>`0.11.2`, `0.10.2` | | `0.0.21` |

Expand Down Expand Up @@ -114,7 +114,7 @@ The following table lists the supported accelerated backends and their correspon

| ROCm Version <br/> (Variant) | vLLM | SGLang |
|------------------------------|-------------------------------------------------------------------------------------|-----------------------------------------------------------|
| 7.2 | **`0.27.1`**, `0.25.1`, `0.24.0`, `0.22.1`, <br/>`0.21.0`, `0.20.2`, <br/>`0.19.1` | `0.5.18`, `0.5.15.post1`, <br/>`0.5.14`, `0.5.12.post1` |
| 7.2 | **`0.28.0`**, `0.27.1`, `0.25.1`, `0.24.0`, `0.22.1`, <br/>`0.21.0`, `0.20.2`, <br/>`0.19.1` | `0.5.18`, `0.5.15.post1`, <br/>`0.5.14`, `0.5.12.post1` |
| 7.1 | `0.17.1` | |
| 7.0 | `0.18.1`, <br/>`0.16.0`, `0.15.1`, <br/>`0.14.1`, `0.13.0`, <br/>`0.12.0`, `0.11.2` | `0.5.9`, `0.5.8.post1`, <br/>`0.5.7`, `0.5.6.post2` |
| 6.4 | `0.16.0`, <br/>`0.15.1`, `0.14.1`, <br/>`0.13.0`, `0.12.0`, <br/>`0.11.2`, `0.10.2` | `0.5.8.post1`, `0.5.7`, <br/>`0.5.6.post2`, `0.5.5.post3` |
Expand Down
6 changes: 3 additions & 3 deletions pack/cuda/Dockerfile.vllm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@ ARG PYTHON_VERSION=3.12
ARG CMAKE_MAX_JOBS
ARG CUDA_VERSION=13.0.1
ARG CUDA_ARCHS
ARG VLLM_BASE_IMAGE=vllm/vllm-openai:v0.27.1-ubuntu2404
ARG VLLM_VERSION=0.27.1
ARG VLLM_BASE_IMAGE=vllm/vllm-openai:v0.28.0-ubuntu2404
ARG VLLM_VERSION=0.28.0
ARG VLLM_TORCH_VERSION=2.13.0
ARG VLLM_TORCH_CUDA_VERSION=${CUDA_VERSION}
## Built from source, replacing the base's own lmcache: that one is a PyPI wheel compiled
## against lmcache's build-time torch pin, which need not match the torch this base ships, and
## on a mismatch lmcache silently degrades to its torch baseline (vllm-project/vllm#53424).
## Keep in sync with the ROCm and SGLang pins —— the MP wire protocol has no version handshake.
ARG VLLM_LMCACHE_VERSION=0.5.4
ARG VLLM_OMNI_COMMIT=d77a35a3
ARG VLLM_OMNI_COMMIT=0d582c2da0

FROM ${VLLM_BASE_IMAGE} AS vllm-build
SHELL ["/bin/bash", "-eo", "pipefail", "-c"]
Expand Down
35 changes: 15 additions & 20 deletions pack/cuda/patches/vllm/001_wrong_dp_ray.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,33 +1,28 @@
diff --git a/vllm/utils/network_utils.py b/vllm/utils/network_utils.py
--- a/vllm/utils/network_utils.py
+++ b/vllm/utils/network_utils.py
@@ -146,6 +146,8 @@
def get_open_zmq_inproc_path() -> str:
return f"inproc://{uuid4()}"

@@ -171,6 +171,9 @@
return range(dp_master_port, dp_master_port + 10)

+_next_port: int | None = None
+
+
def get_open_port() -> int:
"""
Get an open port for the vLLM process to listen on.
@@ -156,13 +158,17 @@
@@ -180,10 +183,13 @@
Right now we reserve 10 ports for the data parallel master
process. Currently it uses 2 ports.
"""
+ global _next_port
if "VLLM_DP_MASTER_PORT" in os.environ:
dp_master_port = envs.VLLM_DP_MASTER_PORT
reserved_port_range = range(dp_master_port, dp_master_port + 10)
while True:
- candidate_port = _get_open_port()
+ candidate_port = _get_open_port(_next_port)
+ _next_port = candidate_port + 1
if candidate_port not in reserved_port_range:
return candidate_port
- return _get_open_port()
+ port = _get_open_port(_next_port)
reserved_port_range = _get_reserved_port_range()
- port = _get_open_port()
+ port = _get_open_port(start_port=_next_port)
+ _next_port = port + 1
+ return port


def get_open_ports_list(count: int = 5) -> list[int]:
if port in reserved_port_range:
port = _get_open_port(start_port=reserved_port_range.stop, max_attempts=1000)
+ _next_port = port + 1
Comment on lines +21 to +25

@gemini-code-assist gemini-code-assist Bot Aug 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If port is 65535, then _next_port will be set to 65536. Since port numbers must be in the range [0, 65535], passing 65536 as start_port to _get_open_port on the subsequent call will raise an OverflowError or ValueError during socket binding. To prevent this, we should reset _next_port to None (or wrap it) if it exceeds 65535.

    port = _get_open_port(start_port=_next_port)
    _next_port = (port + 1) if port < 65535 else None
    if port in reserved_port_range:
        port = _get_open_port(start_port=reserved_port_range.stop, max_attempts=1000)
        _next_port = (port + 1) if port < 65535 else None

return port


31 changes: 0 additions & 31 deletions pack/cuda/patches/vllm/002_shm_broadcast_port_race.patch
View file
Open in desktop

This file was deleted.

12 changes: 6 additions & 6 deletions pack/matrix.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,8 @@ rules:
- "linux/amd64"
args:
- "ROCM_VERSION=7.2.3"
- "VLLM_VERSION=0.27.1"
- "VLLM_BASE_IMAGE=vllm/vllm-openai-rocm:v0.27.1"
- "VLLM_VERSION=0.28.0"
- "VLLM_BASE_IMAGE=vllm/vllm-openai-rocm:v0.28.0"
## AMD ROCm 7.2.1 - SGLang 0.5.18 (rocm720-mi30x)
##
- backend: "rocm"
Expand All @@ -106,8 +106,8 @@ rules:
- "vllm"
args:
- "CUDA_VERSION=13.0.1"
- "VLLM_VERSION=0.27.1"
- "VLLM_BASE_IMAGE=vllm/vllm-openai:v0.27.1-ubuntu2404"
- "VLLM_VERSION=0.28.0"
- "VLLM_BASE_IMAGE=vllm/vllm-openai:v0.28.0-ubuntu2404"
## NVIDIA CUDA 13.0.1 - SGLang 0.5.18
##
- backend: "cuda"
Expand All @@ -124,8 +124,8 @@ rules:
- "vllm"
args:
- "CUDA_VERSION=12.9.1"
- "VLLM_VERSION=0.27.1"
- "VLLM_BASE_IMAGE=vllm/vllm-openai:v0.27.1-cu129-ubuntu2404"
- "VLLM_VERSION=0.28.0"
- "VLLM_BASE_IMAGE=vllm/vllm-openai:v0.28.0-cu129-ubuntu2404"
## NVIDIA CUDA 12.9.1 - SGLang 0.5.18
##
- backend: "cuda"
Expand Down
8 changes: 4 additions & 4 deletions pack/rocm/Dockerfile.vllm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@ ARG PYTHON_VERSION=3.12
ARG CMAKE_MAX_JOBS
ARG ROCM_VERSION=7.2.3
ARG ROCM_ARCHS
ARG VLLM_BASE_IMAGE=vllm/vllm-openai-rocm:v0.27.1
ARG VLLM_VERSION=0.27.1
ARG VLLM_TORCH_VERSION=2.13.0
ARG VLLM_BASE_IMAGE=vllm/vllm-openai-rocm:v0.28.0
ARG VLLM_VERSION=0.28.0
ARG VLLM_TORCH_VERSION=2.12.0
ARG VLLM_TORCH_ROCM_VERSION=${ROCM_VERSION}
## The ROCm base ships no lmcache of its own, so it is always built from source here.
## Keep in sync with the CUDA and SGLang pins —— see pack/cuda/Dockerfile.vllm.
ARG VLLM_LMCACHE_VERSION=0.5.4
ARG VLLM_MOONCAKE_VERSION=0.3.11.post1
ARG VLLM_OMNI_COMMIT=d77a35a3
ARG VLLM_OMNI_COMMIT=0d582c2da0

FROM ${VLLM_BASE_IMAGE} AS vllm-build
SHELL ["/bin/bash", "-eo", "pipefail", "-c"]
Expand Down
35 changes: 15 additions & 20 deletions pack/rocm/patches/vllm/001_wrong_dp_ray.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,33 +1,28 @@
diff --git a/vllm/utils/network_utils.py b/vllm/utils/network_utils.py
--- a/vllm/utils/network_utils.py
+++ b/vllm/utils/network_utils.py
@@ -146,6 +146,8 @@
def get_open_zmq_inproc_path() -> str:
return f"inproc://{uuid4()}"

@@ -171,6 +171,9 @@
return range(dp_master_port, dp_master_port + 10)

+_next_port: int | None = None
+
+
def get_open_port() -> int:
"""
Get an open port for the vLLM process to listen on.
@@ -156,13 +158,17 @@
@@ -180,10 +183,13 @@
Right now we reserve 10 ports for the data parallel master
process. Currently it uses 2 ports.
"""
+ global _next_port
if "VLLM_DP_MASTER_PORT" in os.environ:
dp_master_port = envs.VLLM_DP_MASTER_PORT
reserved_port_range = range(dp_master_port, dp_master_port + 10)
while True:
- candidate_port = _get_open_port()
+ candidate_port = _get_open_port(_next_port)
+ _next_port = candidate_port + 1
if candidate_port not in reserved_port_range:
return candidate_port
- return _get_open_port()
+ port = _get_open_port(_next_port)
reserved_port_range = _get_reserved_port_range()
- port = _get_open_port()
+ port = _get_open_port(start_port=_next_port)
+ _next_port = port + 1
+ return port


def get_open_ports_list(count: int = 5) -> list[int]:
if port in reserved_port_range:
port = _get_open_port(start_port=reserved_port_range.stop, max_attempts=1000)
+ _next_port = port + 1
Comment on lines +21 to +25

@gemini-code-assist gemini-code-assist Bot Aug 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If port is 65535, then _next_port will be set to 65536. Since port numbers must be in the range [0, 65535], passing 65536 as start_port to _get_open_port on the subsequent call will raise an OverflowError or ValueError during socket binding. To prevent this, we should reset _next_port to None (or wrap it) if it exceeds 65535.

    port = _get_open_port(start_port=_next_port)
    _next_port = (port + 1) if port < 65535 else None
    if port in reserved_port_range:
        port = _get_open_port(start_port=reserved_port_range.stop, max_attempts=1000)
        _next_port = (port + 1) if port < 65535 else None

return port


31 changes: 0 additions & 31 deletions pack/rocm/patches/vllm/002_shm_broadcast_port_race.patch
View file
Open in desktop

This file was deleted.