Skip to content

feat(compute)!: configure provisioning and bootstrap timeouts - #5321

Open
shiju-nv wants to merge 3 commits into
NVIDIA:mainfrom
shiju-nv:feat/4364-bootstrap-timeouts/shiju-nv
Open

shiju-nv wants to merge 3 commits into
NVIDIA:mainfrom
shiju-nv:feat/4364-bootstrap-timeouts/shiju-nv

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Allow operators to configure startup timeouts for slow isolation backends such as CoCo/peer-pods. Keep the five-minute defaults, persisted admission deadlines and normal reconnect limits.

Related Issue

Closes #4364.

Builds on Mike Hepburn's (@eformat) #4575, preserving his attribution and original DCO. Configuration validation, current-main integration, routed connections and regression adaptations are Shiju's work.

Changes

  • Add bounded gateway repair, Kubernetes bootstrap and supervisor attachment settings, accepting 1000 through 86400000 milliseconds.
  • Pass the attachment setting to dedicated supervisors. Shared supervisors take the setting from their own process configuration.
  • Apply the configured budget to discovery, attachment and the first routed connection; retain the normal 30-second reconnect window.
  • Preserve durable admission timing, readiness and late-stop protection; document configuration and restart requirements.
  • Report expiry without an assumed duration and keep lifecycle guidance consistent with the configured windows.

This change is based directly on main and does not depend on #5243's tracing change.

Rust compatibility

External exhaustive literals for Config, GatewayFileSection, KubernetesComputeConfig, BackendServices, SandboxInstanceConfig and SharedHostConfig need the added fields. PreparedBackend::discover_policy implementations and affected public policy-discovery callers need the timeout argument. Existing TOML and omitted-setting defaults remain supported. This requires a qualified minor release with Rust consumer updates; no patch backport.

Release Notes

Testing

Thirteen focused configuration/timer and Kubernetes stop/start cases passed for the unchanged timeout behavior. Both isolated backend timer mutations failed at their intended assertions, while the candidate passed. Existing Docker create/exec/delete and stop/start/exec cases passed in bridge mode with a 900000 ms gateway repair window. No CoCo/peer-pods deployment has been tested.

The diagnostic correction strengthens the existing 900000 ms timeout/cleanup regression. Final scoped all-target server Clippy compiled that case and passed with warnings denied; hosted Rust CI passed on e43b7474e825c0fa99d27b75fe413e64471fe9c9. The retried advisory compatibility review completed without reported findings. The Rust source-compatibility changes described above remain subject to minor-release qualification. Formatting, whitespace, SPDX, changed Markdown and docs/navigation checks passed. Docs validation reported zero errors and three warnings.

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (existing Docker lifecycle coverage exercised)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

shiju-nv and others added 2 commits November 6, 2026 03:39
Continue Mike Hepburn's configuration contribution from NVIDIA#4575
(de949b7), preserving its attribution and
original DCO provenance.

Adapt it to current session admission, registered and shared backends, bound
configuration input, and use the configured budget for the initial routed
connection. Add focused timeout and configuration regressions. These
adaptations are Shiju's work and are not represented as endorsed by the
original contributor.

BREAKING CHANGE: External Rust configuration struct literals must supply the
new timeout fields. PreparedBackend discovery implementors and callers of
public supervisor/runtime policy-discovery methods must accept or pass the
timeout argument. Target the next qualified minor release, not a patch release
or backport. Existing TOML defaults and normal reconnect limits are retained.

Co-authored-by: Mike Hepburn <eformat@gmail.com>
Signed-off-by: Mike Hepburn <eformat@gmail.com>
Signed-off-by: Shiju <shiju@nvidia.com>
Signed-off-by: Shiju <shiju@nvidia.com>

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change the new operator-facing timeout settings to use seconds before merging. Existing gateway configuration uses seconds for comparable windows (image_preparation_timeout_seconds, grpc_rate_limit_window_seconds, and supervisor_pool.recovery_timeout_secs). These startup/repair windows are measured in minutes, and this PR does not establish a need for sub-second precision.

Suggested TOML names:

  • provisioning_repair_window_seconds
  • runtime_bootstrap_grace_seconds
  • sandbox_attach_request_timeout_seconds

Use matching seconds-based environment variables and CLI flags, including OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS and --attach-request-timeout-seconds. Express the existing five-minute defaults as 300 and the bounds as 1..=86400 seconds. Convert to Duration or milliseconds at configuration boundaries as needed.

Please update the configuration validation, tests, examples, published docs, and affected skills together. This request concerns the public configuration interface; I still prefer this PR's explicit runtime configuration and timeout plumbing over the alternative implementation in #5316.

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv

shiju-nv commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

The initial ms interface followed accepted issue #4364, which explicitly proposed all three _MS variables with 300000 defaults. #4575 used those units, and I preserved them when adapting the implementation. They also matched the gateway’s internal deadline arithmetic.

I agree seconds fit these startup windows and the existing gateway settings better. I’ve updated the PR.

@shiju-nv
shiju-nv requested a review from TaylorMutch November 7, 2026 08:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hard-coded 5-minute provisioning/bootstrap deadlines make sandboxes unusable on slow isolation backends (CoCo/peer-pods)

2 participants