CVS RCCL: Capture amd-smi ECC counters before and after test on each node and warn if any counter delta > 0

[sarachoi-amd]

… capture and delta reporting

Motivation

Capture amd-smi ECC counters on each node before and after the test, and warn when any counter increases. This makes it easier to spot silent ECC activity during validation and flag nodes that may need further investigation.

Technical Details

RCCL ECC delta checking is opt-in via cvs_params.verify_ecc_delta ("True"; default "False") in rccl_perf and rccl_regression. It captures amd-smi ECC counters (amd-smi metric -g all --json) before and after each collective and logs one INFO table per node (CE/UE/DE before/after/delta for all seven blocks by default). Use verify_ecc_blocks (e.g. ["UMC", "XGMI_WAFL"]) to limit which blocks are checked; omit or leave empty for all blocks. Requires passwordless sudo (same gate as dmesg). Counter increases log WARNING but do not fail the test. Grep the pytest log for ECC_BLOCKS.

Ticket: AIMVT-307 (DCCS-6489)

Test Plan

Run RCCL on a live cluster with verify_ecc_delta enabled and disabled, varying verify_ecc_blocks (omitted, empty, lowercase, valid subset, mixed valid/invalid, and invalid syntax), and confirm ECC capture, logging, warnings, and pass/fail behavior match expectations.

Test Result

Ran RCCL with varied cvs_params ECC settings on a live cluster. With verify_ecc_delta: "False" (or omitted), no ECC_BLOCKS output. With verify_ecc_delta: "True", confirmed full before/after ECC tables when verify_ecc_blocks is omitted or [], and block-filtered output for lowercase names and mixed valid/invalid lists (e.g. ["UMC","test","SDMA"] reports only UMC and SDMA). Invalid block names emit WARNING and fall back to all blocks when none remain valid. Invalid verify_ecc_blocks syntax is handled without breaking the run. ECC warnings do not fail the test when the collective itself fails.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
• Ran make fmt-check, make lint, and make test (or make ut) locally and all checks pass.
• Added/updated unit tests in cvs/lib/unittests/test_rocm_plib.py for new ECC delta helpers.
• Updated RCCL docs and config (docs/reference/configuration-files/rccl.rst, cvs/tests/rccl/README.md, rccl_config.json comments).
• Verified on a live cluster with verify_ecc_delta enabled/disabled and varied verify_ecc_blocks settings.
• Confirmed default behavior is unchanged (verify_ecc_delta defaults to "False").
• No cluster credentials, SSH keys, or customer-specific values committed.