fix(rlix-drivers): forward resize-timeout env knobs to MilesCoordinator - #36
Open
JunzheJoe wants to merge 1 commit into
Open
fix(rlix-drivers): forward resize-timeout env knobs to MilesCoordinator#36JunzheJoe wants to merge 1 commit into
JunzheJoe wants to merge 1 commit into
Conversation
RLIX_RESIZE_LOCK_TIMEOUT_S and MILES_RESIZE_RPC_TIMEOUT_S are read at import time inside the MilesCoordinator actor process (rlops/rlix#35). Ray actors do not inherit the driver shell environment on multi-node clusters, so a driver-side export silently never reached the actor. Add both keys to the existing explicit allowlists in run_miles_dual.py and run_miles_rlix.py, mirroring MILES_MAX_RESIDUAL_GPU_MEM_GB. The scheduler-side RLIX_RESIZE_RPC_TIMEOUT_S travels via rlix.init()'s own env forwarding (fixed rlix-side in rlops/rlix#35).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add
RLIX_RESIZE_LOCK_TIMEOUT_SandMILES_RESIZE_RPC_TIMEOUT_Sto the explicit env-forwarding allowlists in both RLix drivers (run_miles_dual.py,run_miles_rlix.py).Why
Kynnet's review of rlops/rlix#35 caught that the new resize-timeout knobs are read inside the MilesCoordinator actor process, but weren't on any explicit
runtime_envforwarding path — and as the drivers' own comment notes, Ray does not propagate the parent driver's env by default. On a single-node smoke the raylet inherits the driver shell so the override arrives implicitly; on multi-node it silently falls back to the compiled defaults (180s). Same class of gap as theMILES_MIN_FREE_GPU_MEM_GBforwarding issue found on rlops/rlix#21.Defaults are unaffected — this only makes the knobs actually tunable in the standard deploy path, mirroring how
MILES_MAX_RESIDUAL_GPU_MEM_GBis forwarded.Pairing
The scheduler-side
RLIX_RESIZE_RPC_TIMEOUT_Sis handled rlix-side (rlops/rlix#35 commit 3c27d4d:rlix.init()merges it from the driver env into the env_vars it already threads to the orchestrator/scheduler actors). Either PR is safe to land alone.Validation
ast.parseclean on both drivers