Skip to content

AmdSmi gpu power mismatch check - #298

Open
jaspals3123 wants to merge 1 commit into
developmentfrom
jaspal_powerchecks
Open

jaspals3123 wants to merge 1 commit into
developmentfrom
jaspal_powerchecks

Conversation

@jaspals3123

Copy link
Copy Markdown
Collaborator

{ "name": "GPU power-cap validation", "plugins": { "AmdSmiPlugin": { "collection_args": {}, "analysis_args": { "power": { "power_cap_mismatch_allowed": false } } } } }

node-scraper --plugin-configs=/home/jaspals/node-scraper/config.json

@amd-ansonyim amd-ansonyim left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes for end user clarity

expected_power_cap = next(iter(power_caps.values()))
for gpu, power_cap in power_caps.items():
if power_cap != expected_power_cap:
self._log_event(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will log every time we have a mismatched value. If a node has i.e 8 GPUs and all 8 have different power cap values we'll get this event logged 8 times. Would be more clear if we keep track of all mismatched values and log one event at the end highlighting all mismatches.

category=EventCategory.PLATFORM,
description=f"Power cap inconsistency for gpu {gpu}",
priority=EventPriority.ERROR,
data={"power_caps": list(power_caps.values())},

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be better to pass in power_caps itself as a dict to the data, otherwise end user will just see a list of power cap values and not the corresponding GPU.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants