Skip to content

Trident 26.06 node plugin registers CSINode without topologyKeys #1172

Description

@iohenkies

Summary

Trident 26.06 node plugin registers CSINode without topologyKeys, breaking topology-aware provisioning.

After upgrading from 25.10.0 to 26.06.0, the Trident node plugin registers itself in the CSINode object with topologyKeys: null on every node. Any StorageClass using volumeBindingMode: WaitForFirstConsumer together with allowedTopologies can no longer provision volumes. Existing volumes and mounts are unaffected.

Restarting the node DaemonSet after the controller is healthy fixes most nodes but never all of them, which points at a race between the CSI gRPC endpoint becoming available and the node completing its registration with the controller.

Environment

  • Trident 26.06.0, installed with the trident-operator Helm chart 100.2606.0
  • Kubernetes 1.33.7
  • Container runtime containerd 2.2.1
  • Kubernetes ochestrator Rancher v2.14.3
  • OS Ubuntu 24.04.4 LTS
  • Backends: ontap-nas and ontap-san, each with supportedTopologies for two zones
  • Nodes correctly labelled topology.kubernetes.io/region and topology.kubernetes.io/zone
  • Previously working on Trident 25.10.0 with identical configuration

Symptom

New PVCs stay Pending:

Warning  ProvisioningFailed  csi.trident.netapp.io_trident-controller-...
error generating accessibility requirements: no topology key found for node <node>

The scheduler rejects every candidate node:

0/120 nodes are available: 1 node(s) didn't find available persistent volumes to bind, ...

The CSINode object shows the driver registered, but without topology:

csi.vsphere.vmware.com     topologyKeys=["topology.csi.vmware.com/k8s-region","topology.csi.vmware.com/k8s-zone"]
csi.trident.netapp.io      topologyKeys=null

The node itself has the labels:

topology.kubernetes.io/region=region-a
topology.kubernetes.io/zone=zone-a

Another CSI driver (vSphere) on the same node registers its topology keys correctly, so this is specific to Trident.

Reproduction

  1. Label nodes with topology.kubernetes.io/region and topology.kubernetes.io/zone.
  2. Create a StorageClass with volumeBindingMode: WaitForFirstConsumer and allowedTopologies matching those labels.
  3. Install Trident 25.10.0 via the operator. Confirm every node has topology keys:
    kubectl get csinode -o json | jq '.items[].spec.drivers[] | select(.name=="csi.trident.netapp.io") | .topologyKeys'
  4. Upgrade to 26.06.0.
  5. Re-check. All nodes now report null.
  6. Create a PVC on that StorageClass with a consumer pod. It stays Pending with the error above.

Impact

  • Topology-aware provisioning is broken after every upgrade, on every node at once, because controller and node pods restart together and the nodes are ready before the controller.
  • Restarting the DaemonSet is not a reliable workaround: a fraction of nodes lose the race even sequentially against an idle controller, so it takes repeated check-and-restart cycles with no guarantee of convergence. On a 120-node cluster that is not practical.
  • There is no configuration option to work around it. Neither the Helm chart values nor the TridentOrchestrator CRD schema in 25.10.0 or 26.06.0 expose anything topology-related, and the node plugin container args are identical between the two versions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions