Add sandbox startup failure observability and improve error messages (REMOTE-2210)#112
Draft
seemeroland wants to merge 1 commit into
Draft
Add sandbox startup failure observability and improve error messages (REMOTE-2210)#112seemeroland wants to merge 1 commit into
seemeroland wants to merge 1 commit into
Conversation
- Add oz_worker_task_sandbox_startup_failures_total metric that tracks task failures occurring within the first 5 minutes of task assignment. This counter is labeled by failure reason and is primed at startup so dashboards can query it immediately. A spike indicates systemic sandbox provisioning or image-pull issues rather than long-running task errors. - Add RecordSandboxStartupFailure(reason) helper in the metrics package so worker code can increment the new counter when a startup-window failure is detected. - In executeTask, check whether a task failure occurred within the sandboxStartupWindow (5 min). When it did, emit a Warn-level log entry with the task ID, elapsed duration, failure phase, and reason, plus a task.sandbox_startup_failure trace event with the same attributes. - Improve userFacingTaskError to produce actionable, reason-specific messages for backendFailureError values instead of always falling through to the raw 'Failed to execute task: ...' default. The new messages cover: active_deadline, container_create/start, sidecar_prep, image_pull, unschedulable, and container_oom. - Update README metric catalog and sample alerts with the new counter and recommended PromQL alert queries for sandbox startup failures. Co-Authored-By: Oz <oz-agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses REMOTE-2210: Cloud agent sandbox fails to start before startup deadline.
Changes
1. New metric:
oz_worker_task_sandbox_startup_failures_total{reason}Tracks task failures that occur within the first 5 minutes of task assignment, labeled by failure reason. This counter is a subset of
oz_worker_task_failures_totaldesigned to isolate systemic sandbox provisioning issues (slow image pulls, cluster resource pressure, admission failures) from long-running task errors. The metric is primed at startup so dashboards and alerts can query it immediately.Recommended PromQL alert:
2. Startup window failure detection in
executeTaskWhen a task fails within the
sandboxStartupWindow(5 min), the worker now:Warn-level log with task ID, elapsed duration, failure phase, and reasontask.sandbox_startup_failuretrace event with those same attributesoz_worker_task_sandbox_startup_failures_totalcounter3. Improved user-facing error messages
userFacingTaskErrornow produces actionable, reason-specific messages for knownbackendFailureErrortypes instead of always falling back to the raw"Failed to execute task: ..."string. Covered reasons:active_deadlinecontainer_create/container_startsidecar_prepimage_pullunschedulablecontainer_oom4. README documentation
Added the new metric to the catalog and sample dashboards/alerts section.
Conversation: https://staging.warp.dev/conversation/3785204b-d602-4fb7-8a4f-e8a0d934d721
Run: https://oz.staging.warp.dev/runs/019f71e6-7b6c-7eed-95ef-49206074f37f
This PR was generated with Oz.