Skip to content

Add sandbox startup failure observability and improve error messages (REMOTE-2210)#112

Draft
seemeroland wants to merge 1 commit into
mainfrom
oz-agent/REMOTE-2210
Draft

Add sandbox startup failure observability and improve error messages (REMOTE-2210)#112
seemeroland wants to merge 1 commit into
mainfrom
oz-agent/REMOTE-2210

Conversation

@seemeroland

Copy link
Copy Markdown
Contributor

Summary

Addresses REMOTE-2210: Cloud agent sandbox fails to start before startup deadline.

Changes

1. New metric: oz_worker_task_sandbox_startup_failures_total{reason}

Tracks task failures that occur within the first 5 minutes of task assignment, labeled by failure reason. This counter is a subset of oz_worker_task_failures_total designed to isolate systemic sandbox provisioning issues (slow image pulls, cluster resource pressure, admission failures) from long-running task errors. The metric is primed at startup so dashboards and alerts can query it immediately.

Recommended PromQL alert:

sum by (reason) (rate(oz_worker_task_sandbox_startup_failures_total[5m])) > 0

2. Startup window failure detection in executeTask

When a task fails within the sandboxStartupWindow (5 min), the worker now:

  • Emits a Warn-level log with task ID, elapsed duration, failure phase, and reason
  • Records a task.sandbox_startup_failure trace event with those same attributes
  • Increments the new oz_worker_task_sandbox_startup_failures_total counter

3. Improved user-facing error messages

userFacingTaskError now produces actionable, reason-specific messages for known backendFailureError types instead of always falling back to the raw "Failed to execute task: ..." string. Covered reasons:

Reason New message
active_deadline "The agent sandbox did not complete before the job deadline was exceeded. This may indicate slow container image pulls, cluster resource pressure, or sandbox startup delays — please try again."
container_create / container_start "The agent sandbox failed to start. This may indicate a container image issue, insufficient cluster resources, or a configuration problem — please try again."
sidecar_prep "The agent sandbox failed to prepare its required dependencies..."
image_pull "The agent sandbox could not pull its container image..."
unschedulable "The agent sandbox could not be scheduled due to insufficient cluster resources..."
container_oom "The agent sandbox ran out of memory and was terminated..."

4. README documentation

Added the new metric to the catalog and sample dashboards/alerts section.


Conversation: https://staging.warp.dev/conversation/3785204b-d602-4fb7-8a4f-e8a0d934d721
Run: https://oz.staging.warp.dev/runs/019f71e6-7b6c-7eed-95ef-49206074f37f

This PR was generated with Oz.

- Add oz_worker_task_sandbox_startup_failures_total metric that tracks
  task failures occurring within the first 5 minutes of task assignment.
  This counter is labeled by failure reason and is primed at startup so
  dashboards can query it immediately. A spike indicates systemic sandbox
  provisioning or image-pull issues rather than long-running task errors.

- Add RecordSandboxStartupFailure(reason) helper in the metrics package
  so worker code can increment the new counter when a startup-window
  failure is detected.

- In executeTask, check whether a task failure occurred within the
  sandboxStartupWindow (5 min). When it did, emit a Warn-level log entry
  with the task ID, elapsed duration, failure phase, and reason, plus a
  task.sandbox_startup_failure trace event with the same attributes.

- Improve userFacingTaskError to produce actionable, reason-specific
  messages for backendFailureError values instead of always falling
  through to the raw 'Failed to execute task: ...' default. The new
  messages cover: active_deadline, container_create/start, sidecar_prep,
  image_pull, unschedulable, and container_oom.

- Update README metric catalog and sample alerts with the new counter and
  recommended PromQL alert queries for sandbox startup failures.

Co-Authored-By: Oz <oz-agent@warp.dev>
@seemeroland seemeroland added the from-feedback-bot Linear issue delegated from Feedback Bot label Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

from-feedback-bot Linear issue delegated from Feedback Bot

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant