feat(idl): typed gpu_fault on core.ExecutionError and workflow.ClusterEvent - #7878
Merged
Conversation
Contributor
Author
|
/regen |
samhita-alla
force-pushed
the
gpu-forward/idl-gpu-fault
branch
from
August 19, 2026 11:45
7e2e9a9 to
669ff2e
Compare
Contributor
Author
|
/regen |
samhita-alla
force-pushed
the
gpu-forward/idl-gpu-fault
branch
from
August 19, 2026 12:08
669ff2e to
2525156
Compare
samhita-alla
marked this pull request as ready for review
August 19, 2026 16:01
samhita-alla
force-pushed
the
gpu-forward/idl-gpu-fault
branch
from
August 19, 2026 16:34
2525156 to
37b04aa
Compare
samhita-alla
force-pushed
the
gpu-forward/idl-gpu-fault
branch
from
August 19, 2026 16:47
37b04aa to
447635b
Compare
samhita-alla
marked this pull request as draft
August 19, 2026 16:55
samhita-alla
force-pushed
the
gpu-forward/idl-gpu-fault
branch
from
August 20, 2026 05:32
447635b to
b967437
Compare
Contributor
Author
|
Rebased onto main after #7882 landed: ClusterEvent field numbers 3–6 are now taken by the Kubernetes event metadata, so gpu_fault moved to field 7. No other changes. |
samhita-alla
marked this pull request as ready for review
August 20, 2026 09:51
samhita-alla
force-pushed
the
gpu-forward/idl-gpu-fault
branch
4 times, most recently
from
August 21, 2026 17:03
7d00299 to
78f852f
Compare
samhita-alla
force-pushed
the
gpu-forward/idl-gpu-fault
branch
from
August 24, 2026 10:18
78f852f to
7b56cee
Compare
pingsutw
approved these changes
Aug 26, 2026
…rEvent Adds GpuFault to flyteidl2.core and an optional gpu_fault field on ClusterEvent so a GPU fault (NVIDIA Xid or NVSwitch SXid) attributed to an action attempt is carried as data next to the human-readable message: kind, code, name, severity (user / warn / critical), the faulting device (GPU UUID, host index, PCI bus id, node) and the blamed process when the driver named one. The message is unchanged; the field is unset for every other cluster event, so existing consumers are unaffected. GpuFault lives in flyteidl2/core/execution.proto next to ExecutionError because both the error and the cluster event have to carry it and core cannot import workflow. core.ExecutionError gains gpu_fault = 8, set when a failure was classified as caused by or coincident with a GPU fault, and workflow.ErrorInfo gains code = 3 (the short failure code from the plugin's ExecutionError, for example OOMKilled or GpuFallenOffBus, which the executor used to drop) and gpu_fault = 4 mirroring the one on ExecutionError, so both survive to the user without anyone parsing the message text. A node-level agent that watches the kernel log emits these as Kubernetes Events on the task pod from the node kernel log; the executor's event watcher already forwards pod events into the attempt's cluster_events, and this field lets the console and failure classification consume them without parsing text. Go, Python, TypeScript and Rust regenerated with the pinned buf 1.58.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
samhita-alla
force-pushed
the
gpu-forward/idl-gpu-fault
branch
from
August 26, 2026 06:43
7b56cee to
83a1de7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Today a GPU fault that hits a running task surfaces, at best, as a free-text Kubernetes Event message copied into the attempt's
cluster_events. Failure classification and the console then have to regex the text to know that a fault happened, which device, and whether it was the workload's fault or the hardware's.This adds the data shape:
GpuFaultinflyteidl2.core, carried both on the cluster event that reported it and on the error the task failed with.What
GpuFault { kind (XID | SXID), code, name, severity (USER | WARN | CRITICAL), gpu_uuid, optional gpu_index, pci_bus_id, node, pid, process }with field-level comments on semantics (host device index, not a pod ordinal; SXid carries only the PCI id; gpu_index optional because 0 is valid).flyteidl2/core/execution.proto, next toExecutionError, because bothExecutionErrorandworkflow.ClusterEventhave to carry it and core cannot import workflow.core.ExecutionError.gpu_fault = 8, set when a failure was classified as caused by or coincident with a GPU fault, so consumers do not parsemessage.workflow.ClusterEvent.gpu_fault = 3, unset for every other kind of cluster event.workflow.ErrorInfo.code = 3, the short failure code from the plugin'sExecutionError(OOMKilled,Interrupted,GpuFallenOffBus), empty when the producer did not set one, andworkflow.ErrorInfo.gpu_fault = 4mirroring the one onExecutionError. The executor buildsErrorInfoand had nowhere to put the code, so it was dropped before the user saw it.Source of the data
A node-level agent (any component that watches the kernel log, for example a node-problem-detector style DaemonSet) tails the node kernel log for
NVRM: Xid (PCI:…): <code>, pid=…lines, attributes them to the pod(s) holding the GPU, and emits a Warning Event on the task pod with a stable message format. The executor's existing pod-event watcher forwards those Events intocluster_events. The executor side that fills these typed fields from the message is stacked on top of this PR.Notes
gen/rustdiff is large because the embeddedFILE_DESCRIPTOR_SETbyte array is one byte per line, so any descriptor change rewrites it. The generated Rust types themselves change only where the protos did.Co-Authored-By: Claude Fable 5 noreply@anthropic.com
main