Reduce OOM-kill risk for oz agent process in direct backend (REMOTE-2166)

[seemeroland]

Problem

Cloud agent runs were going silent for ~1 hour. The hypothesis (REMOTE-2166): the Linux OOM killer was targeting the oz agent process (the coordinator) instead of its memory-hungry child processes (e.g. cargo build with CARGO_BUILD_JOBS=1 still consuming significant memory). Once the agent process was killed, the run went dark until the VM was re-spun and a follow-up message resumed it.

Changes

internal/worker/direct.go

• Changed cmd.Run() to cmd.Start() + OOM protection + cmd.Wait().
• After starting the oz agent process, write oom_score_adj = -300 to /proc/<pid>/oom_score_adj. This tells the Linux OOM killer to prefer killing the agent's children (the heavy compilers/tools) over the agent coordinator itself. Requires CAP_SYS_RESOURCE for negative values; the call is best-effort — a warning is logged if the capability is unavailable and execution continues normally.
• Added classifyAgentExitError: detects SIGKILL exits on the oz agent process, logs a prominent error with a dmesg investigation hint, and emits an agent.sigkill_detected span event with a possible_oom_kill hint for tracing.

main.go

• At startup, protect the long-lived worker process itself with oom_score_adj = -500. The worker manages task lifecycle and server reconnects; losing it is more disruptive than losing a single oz agent run.
• Same best-effort approach: logs a warning if CAP_SYS_RESOURCE is missing.

internal/metrics/metrics.go

• Added TaskFailureReasonAgentOOM = "agent_oom" to the bounded failure reason constants and taskFailureReasons slice, so SIGKILL exits surface in the oz_worker_task_failures_total metric with a distinct reason label.

internal/worker/errors.go

• Added a user-facing message for agent_oom failures guiding operators to check VM memory (dmesg) and reduce build parallelism (CARGO_BUILD_JOBS=1).

Notes

• Capability requirement: Negative oom_score_adj values require CAP_SYS_RESOURCE. Warp's own factory VMs should have this; self-hosted workers running as non-root without the capability will see a warning log but are otherwise unaffected.
• This change addresses the direct backend. The Docker/Kubernetes backends are unaffected (Docker already tracks OOMKilled via the container state; Kubernetes handles OOM at the cgroup/pod level).
• The oz CLI itself would ideally raise the OOM score of its own spawned tool children (e.g. cargo build) — that would give even stronger protection, but requires changes to the oz CLI binary rather than this repo.

Conversation: https://staging.warp.dev/conversation/f2fe5d7a-21e4-425b-a07d-e6476974386b
Run: https://oz.staging.warp.dev/runs/019f6729-eba5-70ff-908a-7e9a34214faa

This PR was generated with Oz.