Skip to content

fix: retry transient vLLM adapter reload failures - #778

Open
yaodong-shen wants to merge 1 commit into
OpenPipe:mainfrom
yaodong-shen:fix/dedicated-lora-reload-retry
Open

fix: retry transient vLLM adapter reload failures#778
yaodong-shen wants to merge 1 commit into
OpenPipe:mainfrom
yaodong-shen:fix/dedicated-lora-reload-retry

Conversation

@yaodong-shen

Copy link
Copy Markdown

Summary

  • retry LoRA reloads after connection establishment failures and transient 502/503/504 responses
  • check the supervised vLLM child before every attempt so a real process exit stops retries immediately
  • report the request URL, process state, runtime log path, checkpoint step/path, and final error when retries are exhausted

Fixes #678

Retry safety

The request is retried at most four times with exponential backoff. Client errors and read/write failures are not retried because the server may already have applied the non-idempotent adapter load.

Validation

  • uv run --no-sync pytest -q tests/unit/test_vllm_runtime.py — 5 passed
  • existing runtime launcher tests — 14 passed; one macOS /tmp versus /private/tmp assertion failed identically on a clean origin/main@3bdbf229 worktree
  • prek ruff, ruff-format, ty, and uv-lock-check hooks — passed
  • Python 3.12 compile and git diff --check — passed

The complete unit suite requires optional backend and GPU dependencies that are not installable in this macOS environment; repository CI installs those extras in its CUDA runner.

Retry connection failures and transient gateway responses while reloading dedicated LoRA adapters.

Stop immediately when the supervised child exits and report runtime diagnostics after exhaustion.

Fixes OpenPipe#678
@yaodong-shen

Copy link
Copy Markdown
Author

The Prek, Package Install, and TrainerRank GPU workflows are waiting for maintainer approval and created no jobs for this external-fork PR. The focused retry tests and the repository Prek checks pass locally. Could a maintainer approve the workflow runs when convenient?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dedicated LocalBackend vLLM server can become unreachable during LoRA adapter reload

1 participant