fix(preflight): remove hardcoded NIC/GPU count assumptions in health checks

[paklui]

Summary

Follow-up to #375, addressing the two other issues flagged during that review
(hardcoded NIC/GPU count assumptions in the cluster health checks).

• get_nic_ethtool_stats_dict() (cvs/lib/linux_utils.py) derived its
  per-node command batch size from the first node's backend RDMA NIC count
  and indexed every other node with it. A cluster where one node has fewer
  NICs than its peers (e.g. a downed link) hit IndexError. Batch size is
  now driven by the max count across nodes; nodes short an interface at a
  given index run a harmless true no-op for that round and are skipped
  when parsing results.

• verify_gpu_pcie_bus_width() (cvs/lib/verify_lib.py) defaulted
  expected_cards=8, and check_cluster_health.py always called it with no
  override, so the check unconditionally failed on any platform with a
  different GPU count per node (e.g. a 4-GPU Helios-R tray always reported
  "Number of cards not matching expected no 8"). The default now auto-detects
  the expected count from the first node's actual GPU count; check_cluster_health
  also gains an --expected_gpu_count flag for explicit override when strict
  enforcement of a known count is wanted.

Testing

• New unit tests: test_asymmetric_nic_counts_across_nodes (linux_utils),
  test_auto_detects_expected_cards_when_omitted and
  test_flags_node_disagreeing_with_auto_detected_count (verify_lib).
• make fmt-check and make lint clean (pylint 10.00/10).
• Live-validated against a 2-node Helios-R cluster (4 GPUs/node, 9 backend
  RDMA NICs/node):
  • verify_gpu_pcie_bus_width(phdl) with no expected_cards now
    auto-detects 4 instead of failing against the hardcoded 8.
  • get_nic_ethtool_stats_dict(phdl) runs cleanly across both nodes with
    correct per-node interface sets; no regression on the normal (symmetric)
    path. Both nodes happened to have all links up during this test, so the
    exact asymmetric-count path itself is covered by the unit test rather
    than a live repro (didn't want to force a link down on shared hardware
    to manufacture one).

Test plan

• make fmt-check
• make lint
• python -m unittest cvs.lib.unittests.test_linux_utils cvs.lib.unittests.test_verify_lib
• Live smoke test against a 2-node cluster (see above)

🤖 Generated with Claude Code