From 25ec689cac82e15afb9608bf57c98f7b601fec3e Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 24 Sep 2026 00:07:48 +0000 Subject: [PATCH] docs: Update plugin documentation [automated] --- docs/PLUGIN_DOC.md | 142 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 127 insertions(+), 15 deletions(-) diff --git a/docs/PLUGIN_DOC.md b/docs/PLUGIN_DOC.md index a0020ca5..935cdfd9 100644 --- a/docs/PLUGIN_DOC.md +++ b/docs/PLUGIN_DOC.md @@ -5,32 +5,33 @@ | Plugin | Collection | Analyzer Args | Collection Args | DataModel | Collector | Analyzer | | --- | --- | --- | --- | --- | --- | --- | | GenericCollectionPlugin | Runs each command from collection_args.commands on the target (in-band host or BMC over OOB SSH).
Commands are user-configured; there are no fixed CMD_\* class fields. | **Analyzer Args:**
- `checks`: list[nodescraper.plugins.generic_collection.analyzer_args.CommandCheck] — Per-command validation rules keyed by collected command name. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `commands`: list[nodescraper.plugins.generic_collection.collector_args.CommandSpec] — Named commands to run. Each entry must include 'name' and 'command'. Prefer small textual stdout; see class docstring...
- `sudo`: bool — Default sudo setting for commands that do not specify sudo.
- `timeout`: int — Default per-command timeout in seconds.
- `include_stdout`: bool — Default: include each command's stdout in collected results for analysis. When false, stdout is omitted from stored r... | [GenericCollectionDataModel](#GenericCollectionDataModel-Model) | [GenericCollectionCollector](#Collector-Class-GenericCollectionCollector) | [GenericAnalyzer](#Data-Analyzer-Class-GenericAnalyzer) | +| AmdSmiFlavorPlugin | bad-pages
firmware --json
list --json
metric -g all
partition --json
process --json
ras --cper --folder={folder}
ras --afid --cper-file {cper_file}
static -g all --json
static -g {gpu_id} --json
topology
version --json
xgmi -l
xgmi -m | **Analyzer Args:**
- `check_static_data`: bool — If True, run static data checks (e.g. driver version, partition mode).
- `expected_gpu_processes`: Optional[int] — Expected number of GPU processes.
- `expected_max_power`: Optional[int] — Expected maximum power value (e.g. watts).
- `expected_power_management`: Optional[str] — Expected amd-smi metric power_management value per GPU (e.g. DISABLED for active/full power, ENABLED for power-manage...
- `expected_driver_version`: Optional[str] — Expected AMD driver version string.
- `expected_memory_partition_mode`: Optional[str] — Expected memory partition mode (e.g. sp3, dp).
- `expected_compute_partition_mode`: Optional[str] — Expected compute partition mode.
- `expected_firmware_versions`: Optional[dict[str, str]] — Expected firmware versions keyed by amd-smi fw_id (e.g. PLDM_BUNDLE).
- `l0_to_recovery_count_error_threshold`: Optional[int] — L0-to-recovery count above which an error is raised.
- `l0_to_recovery_count_warning_threshold`: Optional[int] — L0-to-recovery count above which a warning is raised.
- `vendorid_ep`: Optional[str] — Expected endpoint vendor ID (e.g. for PCIe).
- `vendorid_ep_vf`: Optional[str] — Expected endpoint VF vendor ID.
- `devid_ep`: Optional[str] — Expected endpoint device ID.
- `devid_ep_vf`: Optional[str] — Expected endpoint VF device ID.
- `sku_name`: Optional[str] — Expected SKU name string for GPU.
- `expected_xgmi_speed`: Optional[list[float]] — Expected xGMI speed value(s) (e.g. link rate).
- `analysis_range_start`: Optional[datetime.datetime] — Start of time range for time-windowed analysis.
- `analysis_range_end`: Optional[datetime.datetime] — End of time range for time-windowed analysis. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `analysis_firmware_ids`: Optional[list[str]] — amd-smi fw_id values to record in analysis_ref.firmware_versions
- `cper_file_path`: Optional[str] — Path to CPER folder or file for RAS AFID collection (ras --afid --cper-file). | [AmdSmiFlavorDataModel](#AmdSmiFlavorDataModel-Model) | [AmdSmiFlavorCollector](#Collector-Class-AmdSmiFlavorCollector) | [AmdSmiAnalyzer](#Data-Analyzer-Class-AmdSmiAnalyzer) | | AmdSmiPlugin | bad-pages
firmware --json
list --json
metric -g all
partition --json
process --json
ras --cper --folder={folder}
ras --afid --cper-file {cper_file}
static -g all --json
static -g {gpu_id} --json
topology
version --json
xgmi -l
xgmi -m | **Analyzer Args:**
- `check_static_data`: bool — If True, run static data checks (e.g. driver version, partition mode).
- `expected_gpu_processes`: Optional[int] — Expected number of GPU processes.
- `expected_max_power`: Optional[int] — Expected maximum power value (e.g. watts).
- `expected_power_management`: Optional[str] — Expected amd-smi metric power_management value per GPU (e.g. DISABLED for active/full power, ENABLED for power-manage...
- `expected_driver_version`: Optional[str] — Expected AMD driver version string.
- `expected_memory_partition_mode`: Optional[str] — Expected memory partition mode (e.g. sp3, dp).
- `expected_compute_partition_mode`: Optional[str] — Expected compute partition mode.
- `expected_firmware_versions`: Optional[dict[str, str]] — Expected firmware versions keyed by amd-smi fw_id (e.g. PLDM_BUNDLE).
- `l0_to_recovery_count_error_threshold`: Optional[int] — L0-to-recovery count above which an error is raised.
- `l0_to_recovery_count_warning_threshold`: Optional[int] — L0-to-recovery count above which a warning is raised.
- `vendorid_ep`: Optional[str] — Expected endpoint vendor ID (e.g. for PCIe).
- `vendorid_ep_vf`: Optional[str] — Expected endpoint VF vendor ID.
- `devid_ep`: Optional[str] — Expected endpoint device ID.
- `devid_ep_vf`: Optional[str] — Expected endpoint VF device ID.
- `sku_name`: Optional[str] — Expected SKU name string for GPU.
- `expected_xgmi_speed`: Optional[list[float]] — Expected xGMI speed value(s) (e.g. link rate).
- `analysis_range_start`: Optional[datetime.datetime] — Start of time range for time-windowed analysis.
- `analysis_range_end`: Optional[datetime.datetime] — End of time range for time-windowed analysis. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `analysis_firmware_ids`: Optional[list[str]] — amd-smi fw_id values to record in analysis_ref.firmware_versions
- `cper_file_path`: Optional[str] — Path to CPER folder or file for RAS AFID collection (ras --afid --cper-file). | [AmdSmiDataModel](#AmdSmiDataModel-Model) | [AmdSmiCollector](#Collector-Class-AmdSmiCollector) | [AmdSmiAnalyzer](#Data-Analyzer-Class-AmdSmiAnalyzer) | -| BiosPlugin | sh -c 'cat /sys/devices/virtual/dmi/id/bios_version'
wmic bios get SMBIOSBIOSVersion /Value | **Analyzer Args:**
- `exp_bios_version`: list[str] — Expected BIOS version(s) to match against collected value (str or list).
- `regex_match`: bool — If True, match exp_bios_version as regex; otherwise exact match. | - | [BiosDataModel](#BiosDataModel-Model) | [BiosCollector](#Collector-Class-BiosCollector) | [BiosAnalyzer](#Data-Analyzer-Class-BiosAnalyzer) | +| BiosPlugin | sh -c 'cat /sys/devices/virtual/dmi/id/bios_version'
smbiosDump | grep -A5 'BIOS Info (Type 0)' | grep 'Version:' | head -1
wmic bios get SMBIOSBIOSVersion /Value | **Analyzer Args:**
- `exp_bios_version`: list[str] — Expected BIOS version(s) to match against collected value (str or list).
- `regex_match`: bool — If True, match exp_bios_version as regex; otherwise exact match. | - | [BiosDataModel](#BiosDataModel-Model) | [BiosCollector](#Collector-Class-BiosCollector) | [BiosAnalyzer](#Data-Analyzer-Class-BiosAnalyzer) | | CmdlinePlugin | cat /proc/cmdline | **Analyzer Args:**
- `required_cmdline`: Union[str, List] — Command-line parameters that must be present (e.g. 'pci=bfsort').
- `banned_cmdline`: Union[str, List] — Command-line parameters that must not be present.
- `os_overrides`: Dict[str, nodescraper.plugins.inband.cmdline.cmdlineconfig.OverrideConfig] — Per-OS overrides for required_cmdline and banned_cmdline (keyed by OS identifier).
- `platform_overrides`: Dict[str, nodescraper.plugins.inband.cmdline.cmdlineconfig.OverrideConfig] — Per-platform overrides for required_cmdline and banned_cmdline (keyed by platform). | - | [CmdlineDataModel](#CmdlineDataModel-Model) | [CmdlineCollector](#Collector-Class-CmdlineCollector) | [CmdlineAnalyzer](#Data-Analyzer-Class-CmdlineAnalyzer) | -| DeviceEnumerationPlugin | powershell -Command "(Get-WmiObject -Class Win32_Processor | Measure-Object).Count"
lspci -d {vendorid_ep}: | grep -iE 'VGA|Display|3D|Processing accelerators|Co-processor|Accelerator' | grep -vi 'Virtual Function' | wc -l
powershell -Command "(wmic path win32_VideoController get name | findstr AMD | Measure-Object).Count"
lscpu
lshw
lspci -d {vendorid_ep}: | grep -i 'Virtual Function' | wc -l
powershell -Command "(Get-VMHostPartitionableGpu | Measure-Object).Count" | **Analyzer Args:**
- `cpu_count`: Optional[list[int]] — Expected CPU count(s); pass as int or list of ints. Analysis passes if actual is in list.
- `gpu_count`: Optional[list[int]] — Expected GPU count(s); pass as int or list of ints. Analysis passes if actual is in list.
- `vf_count`: Optional[list[int]] — Expected virtual function count(s); pass as int or list of ints. Analysis passes if actual is in list. | - | [DeviceEnumerationDataModel](#DeviceEnumerationDataModel-Model) | [DeviceEnumerationCollector](#Collector-Class-DeviceEnumerationCollector) | [DeviceEnumerationAnalyzer](#Data-Analyzer-Class-DeviceEnumerationAnalyzer) | -| DimmPlugin | sh -c 'dmidecode -t 17 | tr -s " " | grep -v "Volatile\|None\|Module" | grep Size' 2>/dev/null
dmidecode
wmic memorychip get Capacity | - | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `skip_sudo`: bool — If True, do not use sudo when running dmidecode or wmic for memory info. | [DimmDataModel](#DimmDataModel-Model) | [DimmCollector](#Collector-Class-DimmCollector) | - | +| DeviceEnumerationPlugin | esxcli hardware cpu global get | awk '/CPU Packages:/ {print $NF}'
powershell -Command "(Get-WmiObject -Class Win32_Processor | Measure-Object).Count"
lspci -d {vendorid_ep}: | grep -iE 'VGA|Display|3D|Processing accelerators|Co-processor|Accelerator' | grep -vi 'Virtual Function' | wc -l
powershell -Command "(wmic path win32_VideoController get name | findstr AMD | Measure-Object).Count"
lscpu
lshw
esxcli hardware pci list | grep -iE '^ \*Device ID: 0x0\*{device_id}([^0-9a-f]|$)' | wc -l
lspci -d {vendorid_ep}: | grep -i 'Virtual Function' | wc -l
powershell -Command "(Get-VMHostPartitionableGpu | Measure-Object).Count" | **Analyzer Args:**
- `cpu_count`: Optional[list[int]] — Expected CPU count(s); pass as int or list of ints. Analysis passes if actual is in list.
- `gpu_count`: Optional[list[int]] — Expected GPU count(s); pass as int or list of ints. Analysis passes if actual is in list.
- `vf_count`: Optional[list[int]] — Expected virtual function count(s); pass as int or list of ints. Analysis passes if actual is in list. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `devid_ep`: Optional[int] — Expected GPU PF PCI device ID (int, e.g. 0x75a3) for ESXi device counting.
- `devid_ep_vf`: Optional[int] — Expected GPU VF PCI device ID (int) for ESXi VF counting. | [DeviceEnumerationDataModel](#DeviceEnumerationDataModel-Model) | [DeviceEnumerationCollector](#Collector-Class-DeviceEnumerationCollector) | [DeviceEnumerationAnalyzer](#Data-Analyzer-Class-DeviceEnumerationAnalyzer) | +| DimmPlugin | sh -c 'dmidecode -t 17 | tr -s " " | grep -v "Volatile\|None\|Module" | grep Size' 2>/dev/null
dmidecode
smbiosDump | grep -A15 'Memory Device (Type 17)' | grep 'Size:'
wmic memorychip get Capacity | - | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `skip_sudo`: bool — If True, do not use sudo when running dmidecode or wmic for memory info. | [DimmDataModel](#DimmDataModel-Model) | [DimmCollector](#Collector-Class-DimmCollector) | - | | DkmsPlugin | dkms status
dkms --version | **Analyzer Args:**
- `dkms_status`: Union[str, list] — Expected dkms status string(s) to match (e.g. 'amd/1.0.0'). At least one of dkms_status or dkms_version required.
- `dkms_version`: Union[str, list] — Expected dkms version string(s) to match. At least one of dkms_status or dkms_version required.
- `regex_match`: bool — If True, match dkms_status and dkms_version as regex; otherwise exact match. | - | [DkmsDataModel](#DkmsDataModel-Model) | [DkmsCollector](#Collector-Class-DkmsCollector) | [DkmsAnalyzer](#Data-Analyzer-Class-DkmsAnalyzer) | -| DmesgPlugin | dmesg --time-format iso -x
ls -1 /var/log/dmesg\* 2>/dev/null | grep -E '^/var/log/dmesg(\.[0-9]+(\.gz)?)?$' || true | **Built-in Regexes:**
- Out of memory error: `(?:oom_kill_process.*)|(?:Out of memory.*)`
- I/O Page Fault: `IO_PAGE_FAULT`
- Kernel Panic: `\bkernel panic\b.*`
- SQ Interrupt: `sq_intr`
- SRAM ECC: `sram_ecc.*`
- Failed to load driver. IP hardware init error.: `\[amdgpu\]\] \*ERROR\* hw_init of IP block.*`
- Failed to load driver. IP software init error.: `\[amdgpu\]\] \*ERROR\* sw_init of IP block.*`
- Real Time throttling activated: `sched: RT throttling activated.*`
- RCU preempt detected stalls: `rcu_preempt detected stalls.*`
- RCU preempt self-detected stall: `rcu_preempt self-detected stall.*`
- QCM fence timeout: `qcm fence wait loop timeout.*`
- General protection fault: `(?:[\w-]+(?:\[[0-9.]+\])?\s+)?general protectio...`
- Segmentation fault: `(?:segfault.*in .*\[)|(?:[Ss]egmentation [Ff]au...`
- Failed to disallow cf state: `amdgpu: Failed to disallow cf state.*`
- Failed to terminate tmr: `\*ERROR\* Failed to terminate tmr.*`
- Suspend of IP block failed: `\*ERROR\* suspend of IP block <\w+> failed.*`
- amdgpu Page Fault: `(amdgpu \w{4}:\w{2}:\w{2}\.\w:\s+amdgpu:\s+\[\S...`
- Page Fault: `page fault for address.*`
- Fatal error during GPU init: `(?:amdgpu)(.*Fatal error during GPU init)|(Fata...`
- PCIe AER Error Status: `(pcieport [\w:.]+: AER: aer_status:[^\n]*(?:\n[...`
- PCIe AER Correctable Error Status: `(.*aer_cor_status: 0x[0-9a-fA-F]+, aer_cor_mask...`
- PCIe AER Uncorrectable Error Status: `(.*aer_uncor_status: 0x[0-9a-fA-F]+, aer_uncor_...`
- PCIe AER Uncorrectable Error Severity with TLP Header: `(.*aer_uncor_severity: 0x[0-9a-fA-F]+.*)(\n.*TL...`
- Failed to read journal file: `Failed to read journal file.*`
- Journal file corrupted or uncleanly shut down: `journal corrupted or uncleanly shut down.*`
- ACPI BIOS Error: `ACPI BIOS Error`
- ACPI Error: `ACPI Error`
- Filesystem corrupted!: `EXT4-fs error \(device .*\):`
- Error in buffered IO, check filesystem integrity: `(Buffer I\/O error on dev)(?:ice)? (\w+)`
- PCIe card no longer present: `pcieport (\w+:\w+:\w+\.\w+):\s+(\w+):\s+(Slot\(...`
- PCIe Link Down: `pcieport (\w+:\w+:\w+\.\w+):\s+(\w+):\s+(Slot\(...`
- Mismatched clock configuration between PCIe device and host: `pcieport (\w+:\w+:\w+\.\w+):\s+(\w+):\s+(curren...`
- RAS Correctable Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- RAS Uncorrectable Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- RAS Deferred Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- RAS Corrected PCIe Error: `((?:\[Hardware Error\]:\s+)?event severity: cor...`
- GPU Reset: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- GPU reset failed: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- ACA Error: `(Accelerator Check Architecture[^\n]*)(?:\n[^\n...`
- ACA Error: `(Accelerator Check Architecture[^\n]*)(?:\n[^\n...`
- MCE Corrected Error: `\[Hardware Error\]:[^\n]*MC\d+_STATUS\[[^\]]*\|...`
- MCE Uncorrected Error: `\[Hardware Error\]:[^\n]*MC\d+_STATUS\[[^\]]*\|...`
- Mode 2 Reset Failed: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)? (...`
- RAS Corrected Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- SGX Error: `x86/cpu: SGX disabled by BIOS`
- MMP Error: `Failed to load MMP firmware qat_4xxx_mmp.bin`
- GPU Throttled: `amdgpu \w{4}:\w{2}:\w{2}.\w: amdgpu: WARN: GPU ...`
- RAS Poison Consumed: `amdgpu[ 0-9a-fA-F:.]+:(?:\s*amdgpu:)?\s+(?:{\d+...`
- RAS Poison created: `amdgpu[ 0-9a-fA-F:.]+:(?:\s*amdgpu:)?\s+(?:{\d+...`
- Bad page threshold exceeded: `(amdgpu: Saved bad pages (\d+) reaches threshol...`
- RAS Hardware Error: `Hardware error from APEI Generic Hardware Error...`
- Error Address: `Error Address.*(?:\s.*)`
- RAS EDR Event: `EDR: EDR event received`
- DPC Event: `DPC: .*`
- LNet: ko2iblnd has no matching interfaces: `(?:\[[^\]]+\]\s*)?LNetError:.*ko2iblnd:\s*No ma...`
- LNet: Error starting up LNI: `(?:\[[^\]]+\]\s*)?LNetError:\s*.*Error\s*-?\d+\...`
- Lustre: network initialisation failed: `LustreError:.*ptlrpc_init_portals\(\).*network ...`
**Analyzer Args:**
- `analysis_range_start`: Optional[datetime.datetime] — Start of time range for analysis (ISO format). Only events on or after this time are analyzed.
- `analysis_range_end`: Optional[datetime.datetime] — End of time range for analysis (ISO format). Only events before this time are analyzed.
- `check_unknown_dmesg_errors`: Optional[bool] — If True, treat unknown/unmatched dmesg error lines as failures.
- `exclude_category`: Optional[set[str]] — Set of error categories to exclude from analysis.
- `interval_to_collapse_event`: int — Seconds within which repeated events are collapsed into one (for rate limiting).
- `num_timestamps`: int — Number of timestamps to include per event in output.
- `error_regex`: Union[list[nodescraper.base.regexanalyzer.ErrorRegex], list[dict], NoneType] — Custom error regex patterns; each item can be ErrorRegex or dict with category/pattern.
- `priority_override_rules`: Optional[list[dict]] — Rules to override the priority of matched ErrorRegex objects. Each rule is a dict where all keys except 'new_priority...
- `mce_threshold`: Optional[int] — When set, raise ERROR if correctable MCE/RAS error count for any component (CPU, GPU BDF/block, etc.) reaches or exce...
- `ignore_match_rules`: Optional[list[dict[str, object]]] — Rules that skip regex matches during analysis. Each rule may use line_regex, match_regex, message, and/or mce_banks.... | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `collect_rotated_logs`: bool — If True, also collect rotated dmesg log files from /var/log/dmesg\*.
- `skip_sudo`: bool — If True, do not use sudo when running dmesg or listing log files.
- `log_dmesg_data`: bool — If True, log the collected dmesg output in artifacts. | [DmesgData](#DmesgData-Model) | [DmesgCollector](#Collector-Class-DmesgCollector) | [DmesgAnalyzer](#Data-Analyzer-Class-DmesgAnalyzer) | +| DmesgPlugin | dmesg --time-format iso -x
cat /var/log/vmkernel.log
ls -1 /var/log/dmesg\* 2>/dev/null | grep -E '^/var/log/dmesg(\.[0-9]+(\.gz)?)?$' || true
ls -1 /var/log/vmkernel.\* 2>/dev/null | grep -E '^/var/log/vmkernel\.[0-9]+(\.gz)?$' || true | **Built-in Regexes:**
- Out of memory error: `(?:oom_kill_process.*)|(?:Out of memory.*)`
- I/O Page Fault: `IO_PAGE_FAULT`
- Kernel Panic: `\bkernel panic\b.*`
- SQ Interrupt: `sq_intr`
- SRAM ECC: `sram_ecc.*`
- Failed to load driver. IP hardware init error.: `\[amdgpu\]\] \*ERROR\* hw_init of IP block.*`
- Failed to load driver. IP software init error.: `\[amdgpu\]\] \*ERROR\* sw_init of IP block.*`
- Real Time throttling activated: `sched: RT throttling activated.*`
- RCU preempt detected stalls: `rcu_preempt detected stalls.*`
- RCU preempt self-detected stall: `rcu_preempt self-detected stall.*`
- QCM fence timeout: `qcm fence wait loop timeout.*`
- General protection fault: `(?:[\w-]+(?:\[[0-9.]+\])?\s+)?general protectio...`
- Segmentation fault: `(?:segfault.*in .*\[)|(?:[Ss]egmentation [Ff]au...`
- Failed to disallow cf state: `amdgpu: Failed to disallow cf state.*`
- Failed to terminate tmr: `\*ERROR\* Failed to terminate tmr.*`
- Suspend of IP block failed: `\*ERROR\* suspend of IP block <\w+> failed.*`
- amdgpu Page Fault: `(amdgpu \w{4}:\w{2}:\w{2}\.\w:\s+amdgpu:\s+\[\S...`
- Page Fault: `page fault for address.*`
- Fatal error during GPU init: `(?:amdgpu)(.*Fatal error during GPU init)|(Fata...`
- PCIe AER Error Status: `(pcieport [\w:.]+: AER: aer_status:[^\n]*(?:\n[...`
- PCIe AER Correctable Error Status: `(.*aer_cor_status: 0x[0-9a-fA-F]+, aer_cor_mask...`
- PCIe AER Uncorrectable Error Status: `(.*aer_uncor_status: 0x[0-9a-fA-F]+, aer_uncor_...`
- PCIe AER Uncorrectable Error Severity with TLP Header: `(.*aer_uncor_severity: 0x[0-9a-fA-F]+.*)(\n.*TL...`
- Failed to read journal file: `Failed to read journal file.*`
- Journal file corrupted or uncleanly shut down: `journal corrupted or uncleanly shut down.*`
- ACPI BIOS Error: `ACPI BIOS Error`
- ACPI Error: `ACPI Error`
- Filesystem corrupted!: `EXT4-fs error \(device .*\):`
- Error in buffered IO, check filesystem integrity: `(Buffer I\/O error on dev)(?:ice)? (\w+)`
- PCIe card no longer present: `pcieport (\w+:\w+:\w+\.\w+):\s+(\w+):\s+(Slot\(...`
- PCIe Link Down: `pcieport (\w+:\w+:\w+\.\w+):\s+(\w+):\s+(Slot\(...`
- Mismatched clock configuration between PCIe device and host: `pcieport (\w+:\w+:\w+\.\w+):\s+(\w+):\s+(curren...`
- RAS Correctable Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- RAS Uncorrectable Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- RAS Deferred Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- RAS Uncorrectable Error: `(\d+ new uncorrectable hardware errors detected...`
- RAS Correctable Error: `(\d+ new correctable hardware errors detected i...`
- RAS ECC Fatal Error: `(GPU detected ECC Fatal Error\.)`
- GPU Reset: `(Issuing Whole GPU reset\.)`
- RAS Corrected PCIe Error: `((?:\[Hardware Error\]:\s+)?event severity: cor...`
- GPU Reset: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- GPU reset failed: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- ACA Error: `(Accelerator Check Architecture[^\n]*)(?:\n[^\n...`
- ACA Error: `(Accelerator Check Architecture[^\n]*)(?:\n[^\n...`
- MCE Corrected Error: `\[Hardware Error\]:[^\n]*MC\d+_STATUS\[[^\]]*\|...`
- MCE Uncorrected Error: `\[Hardware Error\]:[^\n]*MC\d+_STATUS\[[^\]]*\|...`
- Mode 2 Reset Failed: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)? (...`
- RAS Corrected Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....`
- SGX Error: `x86/cpu: SGX disabled by BIOS`
- MMP Error: `Failed to load MMP firmware qat_4xxx_mmp.bin`
- GPU Throttled: `amdgpu \w{4}:\w{2}:\w{2}.\w: amdgpu: WARN: GPU ...`
- RAS Poison Consumed: `amdgpu[ 0-9a-fA-F:.]+:(?:\s*amdgpu:)?\s+(?:{\d+...`
- RAS Poison created: `amdgpu[ 0-9a-fA-F:.]+:(?:\s*amdgpu:)?\s+(?:{\d+...`
- Bad page threshold exceeded: `(amdgpu: Saved bad pages (\d+) reaches threshol...`
- RAS Hardware Error: `Hardware error from APEI Generic Hardware Error...`
- Error Address: `Error Address.*(?:\s.*)`
- RAS EDR Event: `EDR: EDR event received`
- DPC Event: `DPC: .*`
- LNet: ko2iblnd has no matching interfaces: `(?:\[[^\]]+\]\s*)?LNetError:.*ko2iblnd:\s*No ma...`
- LNet: Error starting up LNI: `(?:\[[^\]]+\]\s*)?LNetError:\s*.*Error\s*-?\d+\...`
- Lustre: network initialisation failed: `LustreError:.*ptlrpc_init_portals\(\).*network ...`
**Analyzer Args:**
- `analysis_range_start`: Optional[datetime.datetime] — Start of time range for analysis (ISO format). Only events on or after this time are analyzed.
- `analysis_range_end`: Optional[datetime.datetime] — End of time range for analysis (ISO format). Only events before this time are analyzed.
- `check_unknown_dmesg_errors`: Optional[bool] — If True, treat unknown/unmatched dmesg error lines as failures.
- `exclude_category`: Optional[set[str]] — Set of error categories to exclude from analysis.
- `interval_to_collapse_event`: int — Seconds within which repeated events are collapsed into one (for rate limiting).
- `num_timestamps`: int — Number of timestamps to include per event in output.
- `error_regex`: Union[list[nodescraper.base.regexanalyzer.ErrorRegex], list[dict], NoneType] — Custom error regex patterns; each item can be ErrorRegex or dict with category/pattern.
- `priority_override_rules`: Optional[list[dict]] — Rules to override the priority of matched ErrorRegex objects. Each rule is a dict where all keys except 'new_priority...
- `mce_threshold`: Optional[int] — When set, raise ERROR if correctable MCE/RAS error count for any component (CPU, GPU BDF/block, etc.) reaches or exce...
- `ignore_match_rules`: Optional[list[dict[str, object]]] — Rules that skip regex matches during analysis. Each rule may use line_regex, match_regex, message, and/or mce_banks.... | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `collect_rotated_logs`: bool — If True, also collect rotated dmesg log files from /var/log/dmesg\*.
- `skip_sudo`: bool — If True, do not use sudo when running dmesg or listing log files.
- `log_dmesg_data`: bool — If True, log the collected dmesg output in artifacts. | [DmesgData](#DmesgData-Model) | [DmesgCollector](#Collector-Class-DmesgCollector) | [DmesgAnalyzer](#Data-Analyzer-Class-DmesgAnalyzer) | | FabricsPlugin | lspci | grep -i cassini
lsmod | grep cxi
cxi_stat
ibstat
ibv_devinfo
ls -l /sys/class/infiniband/\*/device/net
fi_info -p cxi
mst start
mst status -v
ip link show
ofed_info -s | - | - | [FabricsDataModel](#FabricsDataModel-Model) | [FabricsCollector](#Collector-Class-FabricsCollector) | - | | JournalPlugin | journalctl --no-pager --system --output=short-iso
journalctl --no-pager --system --output=json | **Analyzer Args:**
- `analysis_range_start`: Optional[datetime.datetime] — Start of time range for analysis (ISO format). Only events on or after this time are analyzed.
- `analysis_range_end`: Optional[datetime.datetime] — End of time range for analysis (ISO format). Only events before this time are analyzed.
- `check_priority`: Optional[int] — Check against journal log priority (0=emergency..7=debug). If an entry has priority <= check_priority, an ERROR event...
- `group`: bool — If True, group entries that have the same priority and message. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `boot`: Optional[int] — Optional boot ID to limit journal collection to a specific boot. | [JournalData](#JournalData-Model) | [JournalCollector](#Collector-Class-JournalCollector) | [JournalAnalyzer](#Data-Analyzer-Class-JournalAnalyzer) | | KernelPlugin | sh -c 'uname -a'
sh -c 'cat /proc/sys/kernel/numa_balancing'
wmic os get Version /Value | **Analyzer Args:**
- `exp_kernel`: Union[str, list] — Expected kernel version string(s) to match (e.g. from uname -a).
- `exp_numa`: Optional[int] — Expected value for kernel.numa_balancing (e.g. 0 or 1).
- `regex_match`: bool — If True, match exp_kernel as regex; otherwise exact match. | - | [KernelDataModel](#KernelDataModel-Model) | [KernelCollector](#Collector-Class-KernelCollector) | [KernelAnalyzer](#Data-Analyzer-Class-KernelAnalyzer) | | KernelModulePlugin | cat /proc/modules
modinfo amdgpu
wmic os get Version /Value | **Analyzer Args:**
- `kernel_modules`: dict[str, dict] — Expected kernel module name -> {version, etc.}. Analyzer checks collected modules match.
- `regex_filter`: list[str] — List of regex patterns to filter which collected modules are checked (default: amd). | - | [KernelModuleDataModel](#KernelModuleDataModel-Model) | [KernelModuleCollector](#Collector-Class-KernelModuleCollector) | [KernelModuleAnalyzer](#Data-Analyzer-Class-KernelModuleAnalyzer) | | MemoryPlugin | free -b
lsmem
numactl -H
wmic OS get FreePhysicalMemory /Value; wmic ComputerSystem get TotalPhysicalMemory /Value | **Analyzer Args:**
- `ratio`: float — Required free-memory ratio (0-1). Analysis fails if free/total < ratio.
- `memory_threshold`: str — Minimum free memory required (e.g. '30Gi', '1T'). Used when ratio is not sufficient. | - | [MemoryDataModel](#MemoryDataModel-Model) | [MemoryCollector](#Collector-Class-MemoryCollector) | [MemoryAnalyzer](#Data-Analyzer-Class-MemoryAnalyzer) | -| NetworkPlugin | ip addr show
curl
ethtool -i {interface}
ethtool -S {interface}
ethtool {interface}
lldpcli show neighbor
lldpctl
ip neighbor show
ping
ip route show
ip rule show
wget | - | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `url`: Optional[str] — Optional URL to probe for network connectivity (used with netprobe).
- `netprobe`: Optional[Literal['ping', 'wget', 'curl']] — Tool to use for network connectivity probe: ping, wget, or curl.
- `exclusion_regex`: Optional[list[str]] — Regex patterns matched (search) against netdev/interface names; any match is skipped (ethtool not run against it). | [NetworkDataModel](#NetworkDataModel-Model) | [NetworkCollector](#Collector-Class-NetworkCollector) | [NetworkAnalyzer](#Data-Analyzer-Class-NetworkAnalyzer) | -| NicPlugin | bcmcli_config query AN_PROTOCOL -d {device_id}
bcmcli_debug dscdump --lane 0 -d {device_id}
bcmcli_fwmanager show certificate -d {device_id}
bcmcli_fwmanager show fwpackage -d {device_id}
bcmcli_show version
bcmcli_show device_list
bcmcli_show health -d {device_id}
bcmcli_link show -d {device_id}
bcmcli_livepatch show -d {device_id}
bcmcli_debug loopback -d {device_id}
bcmcli_config msixmv query --pf all -d {device_id}
bcmcli_config show_confs -d {device_id}
bcmcli_nvm show -d {device_id}
bcmcli_debug pcie counters -d {device_id}
bcmcli_config query pcie_relaxed_ordering -d {device_id}
bcmcli_show pci -d {device_id}
bcmcli_config query performance_profile -d {device_id}
bcmcli_qos show qos -d {device_id}
bcmcli_qos show hw-maps -d {device_id}
bcmcli_qos show rx-port-ratelimit -d {device_id}
bcmcli_qos show tx-ep-ratelimit --port 0 -d {device_id}
bcmcli_qos show ingress --cosq -d {device_id}
bcmcli_qos show egress --cosq -d {device_id}
bcmcli_show status -d {device_id}
bcmcli_show temperature -d {device_id}
bcmcli_show phy -d {device_id}
bcmcli_show resource_counts -d {device_id}
bcmcli_hwdbg dump_tx_counters -d {device_id}
bcmcli_hwdbg dump_rx_counters -d {device_id}
bcmcli_debug verify -d {device_id}
bcmcli_debug serdes_tx --get --modtype NRZ --lane 0 -d {device_id}
bcmcli_tunnel show rss -d {device_id}
bcmcli_tunnel show vxlan --type ipv4 -d {device_id}
bcmcli_tunnel show vxlan --type ipv6 -d {device_id}
bcmcli_timesync show dll -d {device_id}
bcmcli_timesync show duty-cycle -d {device_id}
bcmcli_timesync show ts-pin -d {device_id}
bcmcli_dump snap_dump -d {device_id}
bcmcli_config query support_rdma -d {device_id}
bcmcli_timesync show ptp -d {device_id}
bcmcli_timesync show synce -d {device_id}
bcmcli_timesync show tsio -d {device_id}
niccli --dev {device_num} cable --module_info --show
niccli --dev {device_num} counters --pcie
niccli devid
niccli --listdev
niccli --list
niccli --list_devices
niccli --dev {device_num} fw --livepatch --show
niccli --dev {device_num} linkdiag --dscdump --lane 0
niccli --dev {device_num} linkdiag --loopback --show
niccli --dev {device_num} linkdiag --txfir --show --modulation_type NRZ --lane 0
niccli --dev {device_num} link --counters --show
niccli --dev {device_num} link --status
niccli --dev {device_num} msix --max_vectors --show --pf 0
niccli --dev {device_num} nvm --listoptions
niccli --dev {device_num} nvm --list
niccli --dev {device_num} nvm --verify
niccli --dev {device_num} nvm --view
niccli -dev {device_num} nvm -getoption pcie_relaxed_ordering
niccli --dev {device_num} nvm --getoption pcie_relaxed_ordering
niccli -dev {device_num} nvm -getoption performance_profile
niccli --dev {device_num} nvm --getoption performance_profile
niccli -dev {device_num} nvm -getoption support_rdma -scope 0
niccli -dev {device_num} getqos
niccli --dev {device_num} nvm --getoption support_rdma --scope 0
niccli --dev {device_num} qos --ets --show
niccli --dev {device_num} show --all
niccli --dev {device_num} show --health
niccli --dev {device_num} show --certificate
niccli --dev {device_num} show --pkg_ver
niccli --dev {device_num} qos --egress --cosq --show
niccli --dev {device_num} qos --ingress --cosq --show
niccli --dev {device_num} qos --rx_rate_limit --show
niccli --dev {device_num} qos --listmap --pri2cos
niccli --dev {device_num} timesync --ptp --show
niccli --dev {device_num} timesync --tsio --show
niccli --dev {device_num} tunnel --cfg --rss --show
niccli --dev {device_num} tunnel --cfg --vxlan --type ipv4 --show
niccli --dev {device_num} resmgmt --all --profile --show
niccli --dev {device_num} qos --dscp2prio
niccli --dev {device_num} qos --tx_ep_rate_limit --port 0 --show
niccli --dev {device_num} show --device_info
niccli --dev {device_num} show --device_pci_ids
niccli --dev {device_num} show
niccli --dev {device_num} timesync --synce --show
niccli verify
niccli --version
nicctl show card
nicctl --version
nicctl show card flash partition --json
nicctl show card interrupts --json
nicctl show card logs --non-persistent
nicctl show card logs --boot-fault
nicctl show card logs --persistent
nicctl show card profile --json
nicctl show card time --json
nicctl show card statistics packet-buffer summary --json
nicctl show lif statistics --json
nicctl show lif internal queue-to-ud-pinning
nicctl show pipeline internal anomalies
nicctl show pipeline internal rsq-ring
nicctl show pipeline internal statistics memory
nicctl show port fsm
nicctl show port transceiver --json
nicctl show port statistics --json
nicctl show port internal mac
nicctl show qos headroom --json
nicctl show rdma queue --json
nicctl show rdma queue-pair --detail --json
nicctl show version firmware
nicctl show dcqcn
nicctl show environment
nicctl show lif
nicctl show pcie ats
nicctl show port
nicctl show qos
nicctl show rdma statistics
nicctl show version host-software
nicctl show dcqcn --card {card_id} --json
nicctl show card hardware-config --card {card_id} | **Analyzer Args:**
- `expected_values`: Optional[Dict[str, Dict[str, Any]]] — Per-command expected checks keyed by canonical key (see command_to_canonical_key).
- `performance_profile_expected`: str — Expected Broadcom performance_profile value (case-insensitive). Default RoCE.
- `support_rdma_disabled_values`: List[str] — Values that indicate RDMA is not supported (case-insensitive).
- `pcie_relaxed_ordering_expected`: str — Expected Broadcom pcie_relaxed_ordering value (e.g. 'Relaxed ordering = enabled'); checked case-insensitively. Defaul...
- `expected_qos_prio_map`: Optional[Dict[Any, Any]] — Expected priority-to-TC map (e.g. {0: 0, 1: 1}; keys may be int or str in config). Checked per device when set.
- `expected_qos_pfc_enabled`: Optional[int] — Expected PFC enabled value (0/1 or bitmask). Checked per device when set.
- `expected_qos_tsa_map`: Optional[Dict[Any, Any]] — Expected TSA map for ETS (e.g. {0: 'ets', 1: 'strict'}; keys may be int or str in config). Checked per device when set.
- `expected_qos_tc_bandwidth`: Optional[List[int]] — Expected TC bandwidth percentages. Checked per device when set.
- `require_qos_consistent_across_adapters`: bool — When True and no expected_qos_\* are set, require all adapters to have the same prio_map, pfc_enabled, and tsa_map.
- `nicctl_log_error_regex`: Optional[List[Dict[str, Any]]] — Optional list of error patterns for nicctl show card logs. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `commands`: Optional[List[str]] — Optional list of niccli/nicctl/bcmcli commands to run. When None, default command set is used.
- `use_sudo_niccli`: bool — If True, run niccli commands with sudo when required.
- `use_sudo_nicctl`: bool — If True, run nicctl commands with sudo when required.
- `use_sudo_bcmcli`: bool — If True, run bcmcli commands with sudo when required.
- `broadcom_cli_override`: Optional[str] — Force 'niccli' or 'bcmcli' instead of auto-detecting which Broadcom CLI is present. | [NicDataModel](#NicDataModel-Model) | [NicCollector](#Collector-Class-NicCollector) | [NicAnalyzer](#Data-Analyzer-Class-NicAnalyzer) | +| NetworkPlugin | ip addr show
curl
ethtool -i {interface}
ethtool -S {interface}
ethtool {interface}
lldpcli show neighbor
lldpctl
ip neighbor show
ping
ip route show
ip rule show
wget | **Analyzer Args:**
- `expected_nic_firmware`: Optional[str] — Exact firmware version expected from each collected NIC. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `url`: Optional[str] — Optional URL to probe for network connectivity (used with netprobe).
- `netprobe`: Optional[Literal['ping', 'wget', 'curl']] — Tool to use for network connectivity probe: ping, wget, or curl.
- `exclusion_regex`: Optional[list[str]] — Regex patterns matched (search) against netdev/interface names; any match is skipped (ethtool not run against it). | [NetworkDataModel](#NetworkDataModel-Model) | [NetworkCollector](#Collector-Class-NetworkCollector) | [NetworkAnalyzer](#Data-Analyzer-Class-NetworkAnalyzer) | +| NicPlugin | bcmcli_config query AN_PROTOCOL -d {device_id}
bcmcli_debug dscdump --lane 0 -d {device_id}
bcmcli_fwmanager show certificate -d {device_id}
bcmcli_fwmanager show fwpackage -d {device_id}
bcmcli_show version
bcmcli_show device_list
bcmcli_show health -d {device_id}
bcmcli_link show -d {device_id}
bcmcli_livepatch show -d {device_id}
bcmcli_debug loopback -d {device_id}
bcmcli_config msixmv query --pf all -d {device_id}
bcmcli_config show_confs -d {device_id}
bcmcli_nvm show -d {device_id}
bcmcli_debug pcie counters -d {device_id}
bcmcli_config query pcie_relaxed_ordering -d {device_id}
bcmcli_show pci -d {device_id}
bcmcli_config query performance_profile -d {device_id}
bcmcli_qos show qos -d {device_id}
bcmcli_qos show hw-maps -d {device_id}
bcmcli_qos show rx-port-ratelimit -d {device_id}
bcmcli_qos show tx-ep-ratelimit --port 0 -d {device_id}
bcmcli_qos show ingress --cosq -d {device_id}
bcmcli_qos show egress --cosq -d {device_id}
bcmcli_show status -d {device_id}
bcmcli_show temperature -d {device_id}
bcmcli_show phy -d {device_id}
bcmcli_show resource_counts -d {device_id}
bcmcli_hwdbg dump_tx_counters -d {device_id}
bcmcli_hwdbg dump_rx_counters -d {device_id}
bcmcli_debug verify -d {device_id}
bcmcli_debug serdes_tx --get --modtype NRZ --lane 0 -d {device_id}
bcmcli_tunnel show rss -d {device_id}
bcmcli_tunnel show vxlan --type ipv4 -d {device_id}
bcmcli_tunnel show vxlan --type ipv6 -d {device_id}
bcmcli_timesync show dll -d {device_id}
bcmcli_timesync show duty-cycle -d {device_id}
bcmcli_timesync show ts-pin -d {device_id}
bcmcli_dump snap_dump -d {device_id}
bcmcli_config query support_rdma -d {device_id}
bcmcli_timesync show ptp -d {device_id}
bcmcli_timesync show synce -d {device_id}
bcmcli_timesync show tsio -d {device_id}
niccli --dev {device_num} cable --module_info --show
niccli --dev {device_num} counters --pcie
niccli devid
niccli --listdev
niccli --list
niccli --list_devices
niccli --dev {device_num} fw --livepatch --show
niccli --dev {device_num} linkdiag --dscdump --lane 0
niccli --dev {device_num} linkdiag --loopback --show
niccli --dev {device_num} linkdiag --txfir --show --modulation_type NRZ --lane 0
niccli --dev {device_num} link --counters --show
niccli --dev {device_num} link --status
niccli --dev {device_num} msix --max_vectors --show --pf 0
niccli --dev {device_num} nvm --listoptions
niccli --dev {device_num} nvm --list
niccli --dev {device_num} nvm --verify
niccli --dev {device_num} nvm --view
niccli -dev {device_num} nvm -getoption pcie_relaxed_ordering
niccli --dev {device_num} nvm --getoption pcie_relaxed_ordering
niccli -dev {device_num} nvm -getoption performance_profile
niccli --dev {device_num} nvm --getoption performance_profile
niccli -dev {device_num} nvm -getoption support_rdma -scope 0
niccli -dev {device_num} getqos
niccli --dev {device_num} nvm --getoption support_rdma --scope 0
niccli --dev {device_num} qos --ets --show
niccli --dev {device_num} show --all
niccli --dev {device_num} show --health
niccli --dev {device_num} show --certificate
niccli --dev {device_num} show --pkg_ver
niccli --dev {device_num} qos --egress --cosq --show
niccli --dev {device_num} qos --ingress --cosq --show
niccli --dev {device_num} qos --rx_rate_limit --show
niccli --dev {device_num} qos --listmap --pri2cos
niccli --dev {device_num} timesync --ptp --show
niccli --dev {device_num} timesync --tsio --show
niccli --dev {device_num} tunnel --cfg --rss --show
niccli --dev {device_num} tunnel --cfg --vxlan --type ipv4 --show
niccli --dev {device_num} resmgmt --all --profile --show
niccli --dev {device_num} qos --dscp2prio
niccli --dev {device_num} qos --tx_ep_rate_limit --port 0 --show
niccli --dev {device_num} show --device_info
niccli --dev {device_num} show --device_pci_ids
niccli --dev {device_num} show
niccli --dev {device_num} timesync --synce --show
niccli verify
niccli --version
nicctl show card
nicctl --version
nicctl show card flash partition --json
nicctl show card interrupts --json
nicctl show card logs --non-persistent
nicctl show card logs --boot-fault
nicctl show card logs --persistent
nicctl show card profile --json
nicctl show card time --json
nicctl show card statistics packet-buffer summary --json
nicctl show lif statistics --json
nicctl show lif internal queue-to-ud-pinning
nicctl show pipeline internal anomalies
nicctl show pipeline internal rsq-ring
nicctl show pipeline internal statistics memory
nicctl show port fsm
nicctl show port transceiver --json
nicctl show port statistics --json
nicctl show port internal mac
nicctl show qos headroom --json
nicctl show rdma queue --json
nicctl show rdma queue-pair --detail --json
nicctl show version firmware
nicctl show dcqcn
nicctl show environment
nicctl show lif
nicctl show pcie ats
nicctl show port
nicctl show qos
nicctl show rdma statistics
nicctl show version host-software
nicctl show dcqcn --card {card_id} --json
nicctl show card hardware-config --card {card_id} | **Analyzer Args:**
- `expected_nic_firmware`: Optional[str] — Exact firmware version expected from each collected NIC.
- `expected_values`: Optional[Dict[str, Dict[str, Any]]] — Per-command expected checks keyed by canonical key (see command_to_canonical_key).
- `performance_profile_expected`: str — Expected Broadcom performance_profile value (case-insensitive). Default RoCE.
- `support_rdma_disabled_values`: List[str] — Values that indicate RDMA is not supported (case-insensitive).
- `pcie_relaxed_ordering_expected`: str — Expected Broadcom pcie_relaxed_ordering value (e.g. 'Relaxed ordering = enabled'); checked case-insensitively. Defaul...
- `expected_qos_prio_map`: Optional[Dict[Any, Any]] — Expected priority-to-TC map (e.g. {0: 0, 1: 1}; keys may be int or str in config). Checked per device when set.
- `expected_qos_pfc_enabled`: Optional[int] — Expected PFC enabled value (0/1 or bitmask). Checked per device when set.
- `expected_qos_tsa_map`: Optional[Dict[Any, Any]] — Expected TSA map for ETS (e.g. {0: 'ets', 1: 'strict'}; keys may be int or str in config). Checked per device when set.
- `expected_qos_tc_bandwidth`: Optional[List[int]] — Expected TC bandwidth percentages. Checked per device when set.
- `require_qos_consistent_across_adapters`: bool — When True and no expected_qos_\* are set, require all adapters to have the same prio_map, pfc_enabled, and tsa_map.
- `nicctl_log_error_regex`: Optional[List[Dict[str, Any]]] — Optional list of error patterns for nicctl show card logs. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `commands`: Optional[List[str]] — Optional list of niccli/nicctl/bcmcli commands to run. When None, default command set is used.
- `use_sudo_niccli`: bool — If True, run niccli commands with sudo when required.
- `use_sudo_nicctl`: bool — If True, run nicctl commands with sudo when required.
- `use_sudo_bcmcli`: bool — If True, run bcmcli commands with sudo when required.
- `broadcom_cli_override`: Optional[str] — Force 'niccli' or 'bcmcli' instead of auto-detecting which Broadcom CLI is present. | [NicDataModel](#NicDataModel-Model) | [NicCollector](#Collector-Class-NicCollector) | [NicAnalyzer](#Data-Analyzer-Class-NicAnalyzer) | | NvmePlugin | nvme smart-log {dev}
nvme error-log {dev} --log-entries=256
nvme id-ctrl {dev}
nvme id-ns {dev}{ns}
nvme fw-log {dev}
nvme self-test-log {dev}
nvme get-log {dev} --log-id=6 --log-len=512
nvme telemetry-log {dev} --output-file={dev}_{f_name}
nvme list -o json | - | - | [NvmeDataModel](#NvmeDataModel-Model) | [NvmeCollector](#Collector-Class-NvmeCollector) | - | -| OsPlugin | sh -c '( lsb_release -ds || (cat /etc/\*release | grep PRETTY_NAME) || uname -om ) 2>/dev/null | head -n1'
cat /etc/\*release | grep VERSION_ID
wmic os get Version /value
wmic os get Caption /Value | **Analyzer Args:**
- `exp_os`: Union[str, list] — Expected OS name/version string(s) to match (e.g. from lsb_release or /etc/os-release).
- `exact_match`: bool — If True, require exact match for exp_os; otherwise substring match. | - | [OsDataModel](#OsDataModel-Model) | [OsCollector](#Collector-Class-OsCollector) | [OsAnalyzer](#Data-Analyzer-Class-OsAnalyzer) | +| OsPlugin | sh -c '( lsb_release -ds || (cat /etc/\*release | grep PRETTY_NAME) || uname -om ) 2>/dev/null | head -n1'
vmware -v
cat /etc/\*release | grep VERSION_ID
esxcli system version get
wmic os get Version /value
wmic os get Caption /Value | **Analyzer Args:**
- `exp_os`: Union[str, list] — Expected OS name/version string(s) to match (e.g. from lsb_release or /etc/os-release).
- `exact_match`: bool — If True, require exact match for exp_os; otherwise substring match. | - | [OsDataModel](#OsDataModel-Model) | [OsCollector](#Collector-Class-OsCollector) | [OsAnalyzer](#Data-Analyzer-Class-OsAnalyzer) | | PackagePlugin | dnf list --installed
dpkg-query -W
pacman -Q
cat /etc/\*release
wmic product get name,version | **Analyzer Args:**
- `exp_package_ver`: Dict[str, Optional[str]] — Map package name -> expected version (None = any version). Checked against installed packages.
- `regex_match`: bool — If True, match package versions with regex; otherwise exact or prefix match.
- `rocm_regex`: Optional[str] — Optional regex to identify ROCm package version (used when enable_rocm_regex is True).
- `enable_rocm_regex`: bool — If True, use rocm_regex (or default pattern) to extract ROCm version for checks. | - | [PackageDataModel](#PackageDataModel-Model) | [PackageCollector](#Collector-Class-PackageCollector) | [PackageAnalyzer](#Data-Analyzer-Class-PackageAnalyzer) | -| PciePlugin | lspci -d {vendor_id}: -nn
lspci -x
lspci -xxxx
lspci -PP
lspci -PP -d {vendor_id}:{dev_id}
lspci -PP -D -d {vendor_id}:{dev_id}
lspci -PP -D
lspci -vvv
lspci -vvvt | **Analyzer Args:**
- `exp_speed`: int — Expected PCIe link speed (generation 1–5).
- `exp_width`: int — Expected PCIe link width in lanes (1–16).
- `exp_sriov_count`: int — Expected SR-IOV virtual function count.
- `exp_gpu_count_override`: Optional[int] — Override expected GPU count for validation.
- `exp_max_payload_size`: Union[Dict[int, int], int, NoneType] — Expected max payload size: int for all devices, or dict keyed by device ID.
- `exp_max_rd_req_size`: Union[Dict[int, int], int, NoneType] — Expected max read request size: int for all devices, or dict keyed by device ID.
- `exp_ten_bit_tag_req_en`: Union[Dict[int, int], int, NoneType] — Expected 10-bit tag request enable: int for all devices, or dict keyed by device ID. | - | [PcieDataModel](#PcieDataModel-Model) | [PcieCollector](#Collector-Class-PcieCollector) | [PcieAnalyzer](#Data-Analyzer-Class-PcieAnalyzer) | +| PciePlugin | lspci -d {vendor_id}: -nn
lspci -x
lspci -xxxx
lspci -PP
lspci -PP -d {vendor_id}:{dev_id}
lspci -PP -D -d {vendor_id}:{dev_id}
lspci -PP -D
lspci -vvv
lspci -vvvt | **Analyzer Args:**
- `exp_speed`: int — Expected PCIe link speed (generation 1–5).
- `exp_width`: int — Expected PCIe link width in lanes (1–16).
- `exp_sriov_count`: int — Expected SR-IOV virtual function count.
- `exp_gpu_count_override`: Optional[int] — Override expected GPU count for validation.
- `exp_max_payload_size`: Union[Dict[int, int], int, NoneType] — Expected max payload size: int for all devices, or dict keyed by device ID.
- `exp_max_rd_req_size`: Union[Dict[int, int], int, NoneType] — Expected max read request size: int for all devices, or dict keyed by device ID.
- `exp_ten_bit_tag_req_en`: Union[Dict[int, int], int, NoneType] — Expected 10-bit tag request enable: int for all devices, or dict keyed by device ID. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `devid_ep`: Optional[int] — Expected GPU PF PCI device ID (int, e.g. 0x75a3) for ESXi BDF resolution.
- `devid_ep_vf`: Optional[int] — Expected GPU VF PCI device ID (int) for ESXi VF BDF resolution. | [PcieDataModel](#PcieDataModel-Model) | [PcieCollector](#Collector-Class-PcieCollector) | [PcieAnalyzer](#Data-Analyzer-Class-PcieAnalyzer) | | ProcessPlugin | for p in {pids}; do printf "%s:" "$p"; cat /proc/$p/comm 2>/dev/null || true; printf "\n"; done
printf "__SAMPLER__:%s\n" "$$"; for f in /proc/[0-9]\*/stat; do [ -r "$f" ] || continue; pid="${f#/proc/}"; pid="${pid%/stat}"; [ "$pid" = "$$" ] && continue; printf "%s|" "$pid"; cat "$f" 2>/dev/null || true; printf "\n"; done
cat /proc/stat | **Analyzer Args:**
- `max_cpu_usage`: float — Maximum allowed aggregate CPU execution percentage; idle and I/O-wait time are excluded. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `top_n_process`: int — Number of top processes by CPU usage to collect.
- `sample_interval_seconds`: float — Wall-clock interval between procfs CPU samples. | [ProcessDataModel](#ProcessDataModel-Model) | [ProcessCollector](#Collector-Class-ProcessCollector) | [ProcessAnalyzer](#Data-Analyzer-Class-ProcessAnalyzer) | -| RdmaPlugin | rdma link -j
rdma dev
rdma link
rdma statistic -j | **Analyzer Args:**
- `exclusion_regex`: Optional[list[str]] — Regex patterns matched against an interface netdev; matching interfaces are skipped. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output. | [RdmaDataModel](#RdmaDataModel-Model) | [RdmaCollector](#Collector-Class-RdmaCollector) | [RdmaAnalyzer](#Data-Analyzer-Class-RdmaAnalyzer) | +| RdmaPlugin | rdma link -j
rdma dev
rdma link
rdma statistic -j | **Analyzer Args:**
- `expected_nic_firmware`: Optional[str] — Exact firmware version expected from each collected RDMA NIC.
- `exclusion_regex`: Optional[list[str]] — Regex patterns matched against an interface netdev; matching interfaces are skipped. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output. | [RdmaDataModel](#RdmaDataModel-Model) | [RdmaCollector](#Collector-Class-RdmaCollector) | [RdmaAnalyzer](#Data-Analyzer-Class-RdmaAnalyzer) | | RegexSearchPlugin | No collector step: reads local text from the CLI --data path (file or directory).
Directory scans load each file's contents into RegexSearchData for analysis. | Runs RegexSearchAnalyzer: user-defined patterns via analysis_args.error_regex (same shape as Dmesg).
Emits regex match events with optional per-file source in the description when scanning directories.
**Analyzer Args:**
- `error_regex`: Optional[list[dict[str, Any]]] — Regex patterns to search for; each dict may include regex (str), message, event_category, event_priority (same as Dme...
- `interval_to_collapse_event`: int — Seconds within which repeated events are collapsed into one.
- `num_timestamps`: int — Number of timestamps to include per event in output. | - | [RegexSearchData](#RegexSearchData-Model) | - | [RegexSearchAnalyzer](#Data-Analyzer-Class-RegexSearchAnalyzer) | | RocmPlugin | {rocm_path}/opencl/bin/\*/clinfo
env | grep -Ei 'rocm|hsa|hip|mpi|openmp|ucx|miopen'
ls /sys/class/kfd/kfd/proc/
grep -i -E 'rocm' /etc/ld.so.conf.d/\*
{rocm_path}/bin/rocminfo
ls -v -d {rocm_path}\*
ls -v -d {rocm_path}-[3-7]\* | tail -1
ldconfig -p | grep -i -E 'rocm'
grep . -H -r -i {rocm_path}/.info/\* | **Analyzer Args:**
- `exp_rocm`: Union[str, list] — Expected ROCm version string(s) to match (e.g. from rocminfo).
- `exp_rocm_latest`: str — Expected 'latest' ROCm path or version string for versioned installs.
- `exp_rocm_sub_versions`: dict[str, Union[str, list]] — Map sub-version name (e.g. version_rocm) to expected string or list of allowed strings. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `rocm_path`: str — Base path to ROCm installation (e.g. /opt/rocm). Used for rocminfo, clinfo, and version discovery. | [RocmDataModel](#RocmDataModel-Model) | [RocmCollector](#Collector-Class-RocmCollector) | [RocmAnalyzer](#Data-Analyzer-Class-RocmAnalyzer) | -| StoragePlugin | sh -c 'df -lH -B1 | grep -v 'boot''
wmic LogicalDisk Where DriveType="3" Get DeviceId,Size,FreeSpace | **Analyzer Args:**
- `min_required_free_space_abs`: Optional[str] — Minimum required free space per mount (e.g. '10G', '1T').
- `min_required_free_space_prct`: Optional[int] — Minimum required free space as percentage of total (0–100).
- `ignore_devices`: Optional[list[str]] — Mount points or devices to exclude from free-space checks.
- `check_devices`: Optional[list[str]] — If non-empty, only these mount points or devices are checked.
- `regex_match`: bool — If True, match device/mount names with regex; otherwise exact match. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `skip_sudo`: bool — If True, do not use sudo when running df and related storage commands. | [StorageDataModel](#StorageDataModel-Model) | [StorageCollector](#Collector-Class-StorageCollector) | [StorageAnalyzer](#Data-Analyzer-Class-StorageAnalyzer) | +| StoragePlugin | sh -c 'df -lH -B1 | grep -v 'boot''
esxcli storage filesystem list
wmic LogicalDisk Where DriveType="3" Get DeviceId,Size,FreeSpace | **Analyzer Args:**
- `min_required_free_space_abs`: Optional[str] — Minimum required free space per mount (e.g. '10G', '1T').
- `min_required_free_space_prct`: Optional[int] — Minimum required free space as percentage of total (0–100).
- `ignore_devices`: Optional[list[str]] — Mount points or devices to exclude from free-space checks.
- `check_devices`: Optional[list[str]] — If non-empty, only these mount points or devices are checked.
- `regex_match`: bool — If True, match device/mount names with regex; otherwise exact match. | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `skip_sudo`: bool — If True, do not use sudo when running df and related storage commands. | [StorageDataModel](#StorageDataModel-Model) | [StorageCollector](#Collector-Class-StorageCollector) | [StorageAnalyzer](#Data-Analyzer-Class-StorageAnalyzer) | | ScaleOutAristaPlugin | show interfaces counters bins | json | no-more
show interfaces counters queue | no-more
show interfaces counters queue drop-precedence | no-more
show qos interfaces ecn counters queue | json | no-more
show interfaces counters errors | json | no-more
show interfaces phy | no-more
show interfaces phy detail | no-more
show interfaces counters ip | json | no-more
show ip interface | no-more
show lldp | no-more
show lldp neighbors | json | no-more
show interfaces counters | json | no-more
show interfaces flow-control | json | no-more
show interfaces counters queue detail | no-more
show priority-flow-control counters | json | no-more
show priority-flow-control status | no-more
show interfaces status | json | no-more
show qos interfaces | no-more
show qos interfaces ecn | no-more
show qos interfaces trust | no-more
show qos maps | no-more
show qos profile | no-more
show qos profile summary | no-more
show interfaces counters rates | json | no-more
show running-config | no-more
show startup-config | no-more
show system environment cooling | json | no-more
show platform trident mmu queue status | no-more
show version | json | no-more | **Analyzer Args:**
- `analysis_ports`: Optional[List[str]] — Restrict per-port analysis to the given ports. Ports are S/P/[SP] where subport is optional (e.g. ['1/1', '1/31', '1/...
- `expected_port_bandwidth`: int — Expected interface bandwidth (bps) from show interfaces status (AristaPortStatus.bandwidth). Ports with a different b... | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output. Re-runs successful... | [ScaleOutAristaDataModel](#ScaleOutAristaDataModel-Model) | [ScaleOutAristaCollector](#Collector-Class-ScaleOutAristaCollector) | [ScaleOutAristaAnalyzer](#Data-Analyzer-Class-ScaleOutAristaAnalyzer) | | ScaleOutDellPlugin | show alarm | no-more
show buffer pool | no-more
show buffer profile | no-more
show clock | no-more
show interface counters {port} | no-more
show event details | no-more
show interface fec status | no-more
show interface counters | no-more
show interface counters rate | no-more
show interface Eth | no-more
show interface phy counters | no-more
show interface status | no-more
show interface transceiver | no-more
show interface transceiver dom | no-more
show interface transceiver summary | no-more
show ip arp | no-more
show ip interfaces | no-more
show ip route | no-more
show lldp neighbor | no-more
show lldp table | no-more
show qos interface Ethall priority-flow-control statistics | no-more
show priority-flow-control watchdog | no-more
show qos interface Ethall queue all priority-flow-control watchdog-statistics | no-more
show platform environment | no-more
show platform firmware detail | no-more
show platform syseeprom | no-more
show qos interface Eth all | no-more
show qos interface Eth all queue all | no-more
show qos map dot1p-tc | no-more
show qos map dscp-tc | no-more
show qos map pfc-priority-pg | no-more
show qos map pfc-priority-queue | no-more
show qos map tc-dot1p | no-more
show qos map tc-dscp | no-more
show qos map tc-pg | no-more
show qos map tc-queue | no-more
show qos scheduler-policy | no-more
show qos wred-policy | no-more
show queue counters | no-more
show queue persistent-watermark multicast | no-more
show queue persistent-watermark unicast | no-more
show queue watermark multicast | no-more
show queue watermark unicast | no-more
show running-configuration | no-more
show version | no-more | **Analyzer Args:**
- `analysis_ports`: Optional[List[str]] — Restrict per-port analysis to the given ports. Accepts optional Eth prefix (e.g. ['1/1', '1/31', '1/1/1'] or ['Eth1/1...
- `expected_port_speed`: int — Expected interface speed (Mbps) from show interface status (DellInterfaceStatus.speed). Ports with a different speed... | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `collection_ports`: Optional[List[str]] — Restrict detail counter collection to these ports. Accepts the same tokens as analysis_ports (e.g. ['1/1', '1/1/2'] o... | [ScaleOutDellDataModel](#ScaleOutDellDataModel-Model) | [ScaleOutDellCollector](#Collector-Class-ScaleOutDellCollector) | [ScaleOutDellAnalyzer](#Data-Analyzer-Class-ScaleOutDellAnalyzer) | -| SysSettingsPlugin | cat /sys/{}
ls -1 /sys/{}
ls -l /sys/{} | **Analyzer Args:**
- `checks`: Optional[list[nodescraper.plugins.inband.sys_settings.analyzer_args.SysfsCheck]] — List of sysfs checks (path, expected values or pattern, display name). | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `paths`: list[str] — Sysfs paths to read (cat). Paths with '\*' are cat'd via bash (e.g. class/drm/card\*/device/vbios_version); if cat fails, ls -l is used.
- `directory_paths`: list[str] — Sysfs paths to list (ls -1); used for checks that match entry names by regex. | [SysSettingsDataModel](#SysSettingsDataModel-Model) | [SysSettingsCollector](#Collector-Class-SysSettingsCollector) | [SysSettingsAnalyzer](#Data-Analyzer-Class-SysSettingsAnalyzer) | +| SysSettingsPlugin | cat /sys/{}
ls -1 /sys/{}
ls -l /sys/{} | **Analyzer Args:**
- `checks`: Optional[list[nodescraper.plugins.inband.sys_settings.analyzer_args.SysfsCheck]] — List of sysfs checks (path, expected values or pattern, display name). | **Collection Args:**
- `html_view`: bool — When true, include logged command artifacts in command_artifacts.html using human-readable output.
- `paths`: list[str] — Sysfs paths to read (cat). Paths with '\*' are cat'd via bash (e.g. class/drm/card\*/device/vbios_version); if cat fail...
- `directory_paths`: list[str] — Sysfs paths to list (ls -1); used for checks that match entry names by regex. | [SysSettingsDataModel](#SysSettingsDataModel-Model) | [SysSettingsCollector](#Collector-Class-SysSettingsCollector) | [SysSettingsAnalyzer](#Data-Analyzer-Class-SysSettingsAnalyzer) | | SysctlPlugin | sysctl -n | **Analyzer Args:**
- `exp_vm_swappiness`: Optional[int] — Expected vm.swappiness value.
- `exp_vm_numa_balancing`: Optional[int] — Expected vm.numa_balancing value.
- `exp_vm_oom_kill_allocating_task`: Optional[int] — Expected vm.oom_kill_allocating_task value.
- `exp_vm_compaction_proactiveness`: Optional[int] — Expected vm.compaction_proactiveness value.
- `exp_vm_compact_unevictable_allowed`: Optional[int] — Expected vm.compact_unevictable_allowed value.
- `exp_vm_extfrag_threshold`: Optional[int] — Expected vm.extfrag_threshold value.
- `exp_vm_zone_reclaim_mode`: Optional[int] — Expected vm.zone_reclaim_mode value.
- `exp_vm_dirty_background_ratio`: Optional[int] — Expected vm.dirty_background_ratio value.
- `exp_vm_dirty_ratio`: Optional[int] — Expected vm.dirty_ratio value.
- `exp_vm_dirty_writeback_centisecs`: Optional[int] — Expected vm.dirty_writeback_centisecs value.
- `exp_kernel_numa_balancing`: Optional[int] — Expected kernel.numa_balancing value. | - | [SysctlDataModel](#SysctlDataModel-Model) | [SysctlCollector](#Collector-Class-SysctlCollector) | [SysctlAnalyzer](#Data-Analyzer-Class-SysctlAnalyzer) | | SyslogPlugin | ls -1 /var/log/syslog\* 2>/dev/null | grep -E '^/var/log/syslog(\.[0-9]+(\.gz)?)?$' || true
ls -1 /var/log/messages\* 2>/dev/null | grep -E '^/var/log/messages(\.[0-9]+(\.gz)?)?$' || true | - | - | [SyslogData](#SyslogData-Model) | [SyslogCollector](#Collector-Class-SyslogCollector) | - | | UptimePlugin | uptime | - | - | [UptimeDataModel](#UptimeDataModel-Model) | [UptimeCollector](#Collector-Class-UptimeCollector) | - | @@ -71,6 +72,44 @@ GenericCollectionDataModel - Runs each command from collection_args.commands on the target (in-band host or BMC over OOB SSH). - Commands are user-configured; there are no fixed CMD_\* class fields. +## Collector Class AmdSmiFlavorCollector + +### Description + +amd-smi collector with host/guest driver-flavor support (incl. ESXi mxGPU). + +**Bases**: ['AmdSmiCollector'] + +**Link to code**: [amdsmi_flavor_collector.py](https://github.com/amd/node-scraper/blob/HEAD/nodescraper/plugins/inband/amdsmi/amdsmi_flavor_collector.py) + +### Class Variables + +- **SUPPORTED_OS_FAMILY**: `{, }` +- **_HOST_DRIVER_MODULES**: `('gim', 'amdgpuv')` +- **_GUEST_DRIVER_MODULES**: `('amdgpu',)` +- **_PARTITION_COLUMNS**: `('gpu_id', 'memory', 'accelerator_type', 'accelerator_profile_index', 'partition_id')` + +### Provides Data + +AmdSmiFlavorDataModel + +### Commands + +- bad-pages +- firmware --json +- list --json +- metric -g all +- partition --json +- process --json +- ras --cper --folder={folder} +- ras --afid --cper-file {cper_file} +- static -g all --json +- static -g {gpu_id} --json +- topology +- version --json +- xgmi -l +- xgmi -m + ## Collector Class AmdSmiCollector ### Description @@ -133,8 +172,10 @@ Collect BIOS details ### Class Variables +- **SUPPORTED_OS_FAMILY**: `{, , }` - **CMD_WINDOWS**: `wmic bios get SMBIOSBIOSVersion /Value` - **CMD**: `sh -c 'cat /sys/devices/virtual/dmi/id/bios_version'` +- **CMD_ESXI**: `smbiosDump | grep -A5 'BIOS Info (Type 0)' | grep 'Version:' | head -1` ### Provides Data @@ -143,6 +184,7 @@ BiosDataModel ### Commands - sh -c 'cat /sys/devices/virtual/dmi/id/bios_version' +- smbiosDump | grep -A5 'BIOS Info (Type 0)' | grep 'Version:' | head -1 - wmic bios get SMBIOSBIOSVersion /Value ## Collector Class CmdlineCollector @@ -180,6 +222,7 @@ Collect CPU and GPU count ### Class Variables +- **SUPPORTED_OS_FAMILY**: `{, , }` - **CMD_GPU_COUNT_LINUX**: `lspci -d {vendorid_ep}: | grep -iE 'VGA|Display|3D|Processing accelerators|Co-processor|Accelerator' | grep -vi 'Virtual Function' | wc -l` - **CMD_VF_COUNT_LINUX**: `lspci -d {vendorid_ep}: | grep -i 'Virtual Function' | wc -l` - **CMD_LSCPU_LINUX**: `lscpu` @@ -187,6 +230,8 @@ Collect CPU and GPU count - **CMD_CPU_COUNT_WINDOWS**: `powershell -Command "(Get-WmiObject -Class Win32_Processor | Measure-Object).Count"` - **CMD_GPU_COUNT_WINDOWS**: `powershell -Command "(wmic path win32_VideoController get name | findstr AMD | Measure-Object).Count"` - **CMD_VF_COUNT_WINDOWS**: `powershell -Command "(Get-VMHostPartitionableGpu | Measure-Object).Count"` +- **CMD_CPU_COUNT_ESXI**: `esxcli hardware cpu global get | awk '/CPU Packages:/ {print $NF}'` +- **CMD_PCI_COUNT_ESXI**: `esxcli hardware pci list | grep -iE '^ *Device ID: 0x0*{device_id}([^0-9a-f]|$)' | wc -l` ### Provides Data @@ -194,11 +239,13 @@ DeviceEnumerationDataModel ### Commands +- esxcli hardware cpu global get | awk '/CPU Packages:/ {print $NF}' - powershell -Command "(Get-WmiObject -Class Win32_Processor | Measure-Object).Count" - lspci -d {vendorid_ep}: | grep -iE 'VGA|Display|3D|Processing accelerators|Co-processor|Accelerator' | grep -vi 'Virtual Function' | wc -l - powershell -Command "(wmic path win32_VideoController get name | findstr AMD | Measure-Object).Count" - lscpu - lshw +- esxcli hardware pci list | grep -iE '^ \*Device ID: 0x0\*{device_id}([^0-9a-f]|$)' | wc -l - lspci -d {vendorid_ep}: | grep -i 'Virtual Function' | wc -l - powershell -Command "(Get-VMHostPartitionableGpu | Measure-Object).Count" @@ -214,8 +261,10 @@ Collect data on installed DIMMs ### Class Variables +- **SUPPORTED_OS_FAMILY**: `{, , }` - **CMD_WINDOWS**: `wmic memorychip get Capacity` - **CMD**: `sh -c 'dmidecode -t 17 | tr -s " " | grep -v "Volatile\|None\|Module" | grep Size' 2>/dev/null` +- **CMD_ESXI**: `smbiosDump | grep -A15 'Memory Device (Type 17)' | grep 'Size:'` - **CMD_DMIDECODE_FULL**: `dmidecode` ### Provides Data @@ -226,6 +275,7 @@ DimmDataModel - sh -c 'dmidecode -t 17 | tr -s " " | grep -v "Volatile\|None\|Module" | grep Size' 2>/dev/null - dmidecode +- smbiosDump | grep -A15 'Memory Device (Type 17)' | grep 'Size:' - wmic memorychip get Capacity ## Collector Class DkmsCollector @@ -265,9 +315,11 @@ Read dmesg log ### Class Variables -- **SUPPORTED_OS_FAMILY**: `{}` +- **SUPPORTED_OS_FAMILY**: `{, }` - **CMD**: `dmesg --time-format iso -x` +- **CMD_ESXI**: `cat /var/log/vmkernel.log` - **CMD_LOGS**: `ls -1 /var/log/dmesg* 2>/dev/null | grep -E '^/var/log/dmesg(\.[0-9]+(\.gz)?)?$' || true` +- **CMD_LOGS_ESXI**: `ls -1 /var/log/vmkernel.* 2>/dev/null | grep -E '^/var/log/vmkernel\.[0-9]+(\.gz)?$' || true` ### Provides Data @@ -276,7 +328,9 @@ DmesgData ### Commands - dmesg --time-format iso -x +- cat /var/log/vmkernel.log - ls -1 /var/log/dmesg\* 2>/dev/null | grep -E '^/var/log/dmesg(\.[0-9]+(\.gz)?)?$' || true +- ls -1 /var/log/vmkernel.\* 2>/dev/null | grep -E '^/var/log/vmkernel\.[0-9]+(\.gz)?$' || true ## Collector Class FabricsCollector @@ -357,6 +411,7 @@ Read kernel version ### Class Variables +- **SUPPORTED_OS_FAMILY**: `{, , }` - **CMD_WINDOWS**: `wmic os get Version /Value` - **CMD**: `sh -c 'uname -a'` - **CMD_NUMA_BALANCING**: `sh -c 'cat /proc/sys/kernel/numa_balancing'` @@ -871,9 +926,12 @@ Collect OS details ### Class Variables +- **SUPPORTED_OS_FAMILY**: `{, , }` - **CMD_VERSION_WINDOWS**: `wmic os get Version /value` - **CMD_VERSION**: `cat /etc/*release | grep VERSION_ID` +- **CMD_VERSION_ESXI**: `esxcli system version get` - **CMD_WINDOWS**: `wmic os get Caption /Value` +- **CMD_ESXI**: `vmware -v` - **PRETTY_STR**: `PRETTY_NAME` - **CMD**: `sh -c '( lsb_release -ds || (cat /etc/*release | grep PRETTY_NAME) || uname -om ) 2>/dev/null | head -n1'` @@ -884,7 +942,9 @@ OsDataModel ### Commands - sh -c '( lsb_release -ds || (cat /etc/\*release | grep PRETTY_NAME) || uname -om ) 2>/dev/null | head -n1' +- vmware -v - cat /etc/\*release | grep VERSION_ID +- esxcli system version get - wmic os get Version /value - wmic os get Caption /Value @@ -951,7 +1011,8 @@ class for collection of PCIe data only supports Linux OS type. ### Class Variables -- **SUPPORTED_OS_FAMILY**: `{}` +- **SUPPORTED_OS_FAMILY**: `{, }` +- **_BDF_LINE**: `re.compile('^[0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}\\.[0-9a-f]+', re.IGNORECASE)` - **CMD_LSPCI_VERBOSE**: `lspci -vvv` - **CMD_LSPCI_VERBOSE_TREE**: `lspci -vvvt` - **CMD_LSPCI_PATH**: `lspci -PP` @@ -1085,8 +1146,10 @@ Collect disk usage details ### Class Variables +- **SUPPORTED_OS_FAMILY**: `{, , }` - **CMD_WINDOWS**: `wmic LogicalDisk Where DriveType="3" Get DeviceId,Size,FreeSpace` - **CMD**: `sh -c 'df -lH -B1 | grep -v 'boot''` +- **CMD_ESXI**: `esxcli storage filesystem list` ### Provides Data @@ -1095,6 +1158,7 @@ StorageDataModel ### Commands - sh -c 'df -lH -B1 | grep -v 'boot'' +- esxcli storage filesystem list - wmic LogicalDisk Where DriveType="3" Get DeviceId,Size,FreeSpace ## Collector Class ScaleOutAristaCollector @@ -1570,6 +1634,27 @@ Results for each command configured in collection_args. - **results**: `list[nodescraper.plugins.generic_collection.generic_collection_data.CommandCollectionResult]` +## AmdSmiFlavorDataModel Model + +### Description + +AmdSmiDataModel with host/guest driver-flavor model variants allowed. + +**Link to code**: [amdsmidata_flavor.py](https://github.com/amd/node-scraper/blob/HEAD/nodescraper/plugins/inband/amdsmi/amdsmidata_flavor.py) + +**Bases**: ['AmdSmiDataModel'] + +### Model annotations and fields + +- **version**: `Optional[Union[AmdSmiVersion, HostDriverAmdSmiVersion]]` +- **gpu_list**: `Optional[list[Union[AmdSmiListItem, HostDriverAmdSmiListItem]]]` +- **topology**: `Optional[list[Union[Topo, HostDriverTopo]]]` +- **bad_pages**: `Optional[list[Union[BadPages, HostDriverBadPages]]]` +- **static**: `Optional[list[Union[AmdSmiStatic, GuestAmdSmiStatic, HostDriverAmdSmiStatic]]]` +- **metric**: `Optional[list[Union[AmdSmiMetric, GuestAmdSmiMetric, HostDriverAmdSmiMetric]]]` +- **xgmi_metric**: `Optional[list[Union[XgmiMetrics, HostDriverXgmiMetrics]]]` +- **xgmi_link**: `Optional[list[Union[XgmiLinks, HostDriverXgmiLinks]]]` + ## AmdSmiDataModel Model ### Description @@ -1797,6 +1882,7 @@ Collected output of niccli (Broadcom) and nicctl (Pensando) commands. - **broadcom_nic_support_rdma**: `Dict[int, str]` - **broadcom_nic_performance_profile**: `Dict[int, str]` - **broadcom_nic_pcie_relaxed_ordering**: `Dict[int, str]` +- **broadcom_nic_firmware**: `Dict[int, str]` - **pensando_nic_cards**: `List[nodescraper.plugins.inband.nic.nic_data.PensandoNicCard]` - **pensando_nic_dcqcn**: `List[nodescraper.plugins.inband.nic.nic_data.PensandoNicDcqcn]` - **pensando_nic_environment**: `List[nodescraper.plugins.inband.nic.nic_data.PensandoNicEnvironment]` @@ -2190,7 +2276,7 @@ Check dkms matches expected status and version ### Description -Check dmesg for errors +Check dmesg (Linux) or vmkernel.log (ESXi) for errors **Bases**: ['RegexAnalyzer'] @@ -2198,6 +2284,7 @@ Check dmesg for errors ### Class Variables +- **ESXI_TIMESTAMP_PATTERN**: `re.compile('(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d+Z)')` - **ERROR_REGEX**: `[ regex=re.compile('(?:oom_kill_process.*)|(?:Out of memory.*)') message='Out of memory error' event_category= event_priority=, regex=re.compile('IO_PAGE_FAULT') message='I/O Page Fault' event_category= event_priority=, @@ -2234,6 +2321,10 @@ Check dmesg for errors regex=re.compile('(?:\\d{4}-\\d+-\\d+T\\d+:\\d+:\\d+,\\d+[+-]\\d+:\\d+)?(.* correctable hardware errors detected in total in \\w+ block.*)') message='RAS Correctable Error' event_category= event_priority=, regex=re.compile('(?:\\d{4}-\\d+-\\d+T\\d+:\\d+:\\d+,\\d+[+-]\\d+:\\d+)?(.* uncorrectable hardware errors detected in \\w+ block.*)') message='RAS Uncorrectable Error' event_category= event_priority=, regex=re.compile('(?:\\d{4}-\\d+-\\d+T\\d+:\\d+:\\d+,\\d+[+-]\\d+:\\d+)?(.* deferred hardware errors detected in \\w+ block.*)') message='RAS Deferred Error' event_category= event_priority=, + regex=re.compile('(\\d+ new uncorrectable hardware errors detected in \\w+ Block.*)') message='RAS Uncorrectable Error' event_category= event_priority=, + regex=re.compile('(\\d+ new correctable hardware errors detected in \\w+ Block.*)') message='RAS Correctable Error' event_category= event_priority=, + regex=re.compile('(GPU detected ECC Fatal Error\\.)') message='RAS ECC Fatal Error' event_category= event_priority=, + regex=re.compile('(Issuing Whole GPU reset\\.)') message='GPU Reset' event_category= event_priority=, regex=re.compile('((?:\\[Hardware Error\\]:\\s+)?event severity: corrected.*)\\n.*(\\[Hardware Error\\]:\\s+Error \\d+, type: corrected.*)\\n.*(\\[Hardware Error\\]:\\s+section_type: PCIe error.*)') message='RAS Corrected PCIe Error' event_category= event_priority=, regex=re.compile('(?:\\d{4}-\\d+-\\d+T\\d+:\\d+:\\d+,\\d+[+-]\\d+:\\d+)?(.*GPU reset begin.*)') message='GPU Reset' event_category= event_priority=, regex=re.compile('(?:\\d{4}-\\d+-\\d+T\\d+:\\d+:\\d+,\\d+[+-]\\d+:\\d+)?(.*GPU reset(?:\\(\\d+\\))? failed.*)') message='GPU reset failed' event_category= event_priority=, @@ -2257,10 +2348,11 @@ Check dmesg for errors regex=re.compile('(?:\\[[^\\]]+\\]\\s*)?LNetError:\\s*.*Error\\s*-?\\d+\\s+starting up LNI\\s+\\w+', re.IGNORECASE) message='LNet: Error starting up LNI' event_category= event_priority=, regex=re.compile('LustreError:.*ptlrpc_init_portals\\(\\).*network initiali[sz]ation failed', re.IGNORECASE) message='Lustre: network initialisation failed' event_category= event_priority= ]` +- **_FILTER_TIMESTAMP_PATTERN**: `re.compile('(\\d{4}-\\d+-\\d+T\\d+:\\d+:\\d+),(\\d+[+-]\\d+:\\d+)|(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d+Z)')` ### Regex Patterns -*58 items defined* +*62 items defined* - **Built-in Regexes:** - - Out of memory error: `(?:oom_kill_process.*)|(?:Out of memory.*)` @@ -2298,6 +2390,10 @@ Check dmesg for errors - - RAS Correctable Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....` - - RAS Uncorrectable Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....` - - RAS Deferred Error: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....` +- - RAS Uncorrectable Error: `(\d+ new uncorrectable hardware errors detected...` +- - RAS Correctable Error: `(\d+ new correctable hardware errors detected i...` +- - RAS ECC Fatal Error: `(GPU detected ECC Fatal Error\.)` +- - GPU Reset: `(Issuing Whole GPU reset\.)` - - RAS Corrected PCIe Error: `((?:\[Hardware Error\]:\s+)?event severity: cor...` - - GPU Reset: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....` - - GPU reset failed: `(?:\d{4}-\d+-\d+T\d+:\d+:\d+,\d+[+-]\d+:\d+)?(....` @@ -2768,6 +2864,20 @@ Arguments for journal analyzer - **ratio**: `float` — Required free-memory ratio (0-1). Analysis fails if free/total < ratio. - **memory_threshold**: `str` — Minimum free memory required (e.g. '30Gi', '1T'). Used when ratio is not sufficient. +## Analyzer Args Class NetworkAnalyzerArgs + +### Description + +Arguments for network and ethtool analysis. + +**Bases**: ['AnalyzerArgs'] + +**Link to code**: [analyzer_args.py](https://github.com/amd/node-scraper/blob/HEAD/nodescraper/plugins/inband/network/analyzer_args.py) + +### Annotations / fields + +- **expected_nic_firmware**: `Optional[str]` — Exact firmware version expected from each collected NIC. + ## Analyzer Args Class NicAnalyzerArgs ### Description @@ -2780,6 +2890,7 @@ Analyzer args for niccli/nicctl data, with expected_values keyed by canonical co ### Annotations / fields +- **expected_nic_firmware**: `Optional[str]` — Exact firmware version expected from each collected NIC. - **expected_values**: `Optional[Dict[str, Dict[str, Any]]]` — Per-command expected checks keyed by canonical key (see command_to_canonical_key). - **performance_profile_expected**: `str` — Expected Broadcom performance_profile value (case-insensitive). Default RoCE. - **support_rdma_disabled_values**: `List[str]` — Values that indicate RDMA is not supported (case-insensitive). @@ -2857,6 +2968,7 @@ Arguments for the RDMA analyzer. ### Annotations / fields +- **expected_nic_firmware**: `Optional[str]` — Exact firmware version expected from each collected RDMA NIC. - **exclusion_regex**: `Optional[list[str]]` — Regex patterns matched against an interface netdev; matching interfaces are skipped. ## Analyzer Args Class RegexSearchAnalyzerArgs