Skip to content

feat(idl): typed gpu_fault on core.ExecutionError and workflow.ClusterEvent - #7878

Merged
samhita-alla merged 1 commit into
mainfrom
gpu-forward/idl-gpu-fault
Aug 26, 2026
Merged

feat(idl): typed gpu_fault on core.ExecutionError and workflow.ClusterEvent#7878
samhita-alla merged 1 commit into
mainfrom
gpu-forward/idl-gpu-fault

Conversation

@samhita-alla

@samhita-alla samhita-alla commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Why

Today a GPU fault that hits a running task surfaces, at best, as a free-text Kubernetes Event message copied into the attempt's cluster_events. Failure classification and the console then have to regex the text to know that a fault happened, which device, and whether it was the workload's fault or the hardware's.

This adds the data shape: GpuFault in flyteidl2.core, carried both on the cluster event that reported it and on the error the task failed with.

What

  • GpuFault { kind (XID | SXID), code, name, severity (USER | WARN | CRITICAL), gpu_uuid, optional gpu_index, pci_bus_id, node, pid, process } with field-level comments on semantics (host device index, not a pod ordinal; SXid carries only the PCI id; gpu_index optional because 0 is valid).
  • It lives in flyteidl2/core/execution.proto, next to ExecutionError, because both ExecutionError and workflow.ClusterEvent have to carry it and core cannot import workflow.
  • core.ExecutionError.gpu_fault = 8, set when a failure was classified as caused by or coincident with a GPU fault, so consumers do not parse message.
  • workflow.ClusterEvent.gpu_fault = 3, unset for every other kind of cluster event.
  • workflow.ErrorInfo.code = 3, the short failure code from the plugin's ExecutionError (OOMKilled, Interrupted, GpuFallenOffBus), empty when the producer did not set one, and workflow.ErrorInfo.gpu_fault = 4 mirroring the one on ExecutionError. The executor builds ErrorInfo and had nowhere to put the code, so it was dropped before the user saw it.
  • Purely additive; existing consumers are unaffected.
  • Regenerated Go, Python, TypeScript and Rust.

Source of the data

A node-level agent (any component that watches the kernel log, for example a node-problem-detector style DaemonSet) tails the node kernel log for NVRM: Xid (PCI:…): <code>, pid=… lines, attributes them to the pod(s) holding the GPU, and emits a Warning Event on the task pod with a stable message format. The executor's existing pod-event watcher forwards those Events into cluster_events. The executor side that fills these typed fields from the message is stacked on top of this PR.

Notes

  • No behaviour change in any service; this PR is IDL and generated code only.
  • The gen/rust diff is large because the embedded FILE_DESCRIPTOR_SET byte array is one byte per line, so any descriptor change rewrites it. The generated Rust types themselves change only where the protos did.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

@samhita-alla

Copy link
Copy Markdown
Contributor Author

/regen

@samhita-alla
samhita-alla force-pushed the gpu-forward/idl-gpu-fault branch from 7e2e9a9 to 669ff2e Compare August 19, 2026 11:45
@samhita-alla

Copy link
Copy Markdown
Contributor Author

/regen

@samhita-alla
samhita-alla force-pushed the gpu-forward/idl-gpu-fault branch from 669ff2e to 2525156 Compare August 19, 2026 12:08
@samhita-alla
samhita-alla marked this pull request as ready for review August 19, 2026 16:01
@samhita-alla
samhita-alla force-pushed the gpu-forward/idl-gpu-fault branch from 2525156 to 37b04aa Compare August 19, 2026 16:34
@samhita-alla samhita-alla changed the title feat(idl): typed gpu_fault on workflow.ClusterEvent feat(idl): typed gpu_fault on core.ExecutionError and workflow.ClusterEvent Aug 19, 2026
@samhita-alla
samhita-alla force-pushed the gpu-forward/idl-gpu-fault branch from 37b04aa to 447635b Compare August 19, 2026 16:47
@samhita-alla
samhita-alla marked this pull request as draft August 19, 2026 16:55
@samhita-alla
samhita-alla force-pushed the gpu-forward/idl-gpu-fault branch from 447635b to b967437 Compare August 20, 2026 05:32
@samhita-alla

Copy link
Copy Markdown
Contributor Author

Rebased onto main after #7882 landed: ClusterEvent field numbers 3–6 are now taken by the Kubernetes event metadata, so gpu_fault moved to field 7. No other changes.

@samhita-alla
samhita-alla marked this pull request as ready for review August 20, 2026 09:51
@samhita-alla
samhita-alla force-pushed the gpu-forward/idl-gpu-fault branch 4 times, most recently from 7d00299 to 78f852f Compare August 21, 2026 17:03
@samhita-alla
samhita-alla requested a review from pingsutw August 24, 2026 10:18
@samhita-alla
samhita-alla force-pushed the gpu-forward/idl-gpu-fault branch from 78f852f to 7b56cee Compare August 24, 2026 10:18
…rEvent

Adds GpuFault to flyteidl2.core and an optional gpu_fault field on
ClusterEvent so a GPU fault (NVIDIA Xid or NVSwitch SXid) attributed to an
action attempt is carried as data next to the human-readable message: kind,
code, name, severity (user / warn / critical), the faulting device (GPU UUID,
host index, PCI bus id, node) and the blamed process when the driver named
one. The message is unchanged; the field is unset for every other cluster
event, so existing consumers are unaffected.

GpuFault lives in flyteidl2/core/execution.proto next to ExecutionError
because both the error and the cluster event have to carry it and core cannot
import workflow. core.ExecutionError gains gpu_fault = 8, set when a failure
was classified as caused by or coincident with a GPU fault, and
workflow.ErrorInfo gains code = 3 (the short failure code from the plugin's
ExecutionError, for example OOMKilled or GpuFallenOffBus, which the executor
used to drop) and gpu_fault = 4 mirroring the one on ExecutionError, so both
survive to the user without anyone parsing the message text.

A node-level agent that watches the kernel log emits these as
Kubernetes Events on the task pod from the node kernel log; the executor's
event watcher already forwards pod events into the attempt's cluster_events,
and this field lets the console and failure classification consume them
without parsing text.

Go, Python, TypeScript and Rust regenerated with the pinned buf 1.58.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
@samhita-alla
samhita-alla force-pushed the gpu-forward/idl-gpu-fault branch from 7b56cee to 83a1de7 Compare August 26, 2026 06:43
@samhita-alla
samhita-alla merged commit 940b399 into main Aug 26, 2026
24 checks passed
@samhita-alla
samhita-alla deleted the gpu-forward/idl-gpu-fault branch August 26, 2026 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants