Skip to content

feat(ascend): implement DCMI V2 API support for device detection and usage - #20

Merged
gitlawr merged 1 commit into
gpustack:mainfrom
yxf0314:issue/6148-950pr
Aug 27, 2026
Merged

feat(ascend): implement DCMI V2 API support for device detection and usage#20
gitlawr merged 1 commit into
gpustack:mainfrom
yxf0314:issue/6148-950pr

Conversation

@yxf0314

@yxf0314 yxf0314 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

@yxf0314
yxf0314 requested review from gitlawr and thxCode and a lite review from Copilot August 27, 2026 07:29

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for the Ascend A5 generation (Ascend950) and the DCMI V2 API. It adds new CDI utility functions to handle directory-based device nodes and glob-based mounts, implements V2 API wrappers in the pydcmi package, and updates the Ascend detector to support V2-based device discovery, usage tracking, and topology reporting. Additionally, comprehensive tests have been added to verify these changes. A critical issue was identified in _get_device_memory_status_v2 where a premature return statement inside a loop prevents checking subsequent memory types (like DDR) if the first type is healthy or has its enable flag set to false.

Comment on lines +777 to +782
if dev_ecc_info.enable_flag and (
dev_ecc_info.single_bit_error_cnt > 0
or dev_ecc_info.double_bit_error_cnt > 0
):
return DeviceMemoryStatusEnum.UNHEALTHY
return DeviceMemoryStatusEnum.HEALTHY

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In _get_device_memory_status_v2, returning DeviceMemoryStatusEnum.HEALTHY immediately when enable_flag is false (or when there are no errors) prematurely terminates the loop. This prevents subsequent memory types (such as DDR on DDR-only devices where HBM is not enabled but the query succeeds with enable_flag = False) from being checked.

The loop should only return early if an unhealthy status is detected. If the loop completes without finding any unhealthy memory, it should fall through to return HEALTHY at the end of the function.

Suggested change
if dev_ecc_info.enable_flag and (
dev_ecc_info.single_bit_error_cnt > 0
or dev_ecc_info.double_bit_error_cnt > 0
):
return DeviceMemoryStatusEnum.UNHEALTHY
return DeviceMemoryStatusEnum.HEALTHY
if dev_ecc_info.enable_flag and (
dev_ecc_info.single_bit_error_cnt > 0
or dev_ecc_info.double_bit_error_cnt > 0
):
return DeviceMemoryStatusEnum.UNHEALTHY

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds Ascend DCMI V2 support to improve device detection/usage on newer Ascend generations (notably A5/950), and updates CDI generation to account for A5’s UB device/mount requirements.

Changes:

  • Add DCMI V2 initialization + V2 wrapper functions in the pydcmi binding and route Ascend detection/usage/topology logic based on the initialized API version.
  • Extend Ascend SoC naming/variant handling for the A5 (Ascend950*) generation, including fallback UUID behavior when die IDs aren’t readable.
  • Update CDI generation to enumerate UB device directories and mount additional A5 UB user-space libraries; add comprehensive tests for the new behaviors.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/gpustack_runtime/detector/test_ascend.py Adds tests covering A5 SoC naming, CDI UB enumeration/mounting, and DCMI V2-only driver behavior (info/usage/topology).
gpustack_runtime/detector/pydcmi/init.py Introduces V2 init + V2 wrappers and tracks which API version initialized the library.
gpustack_runtime/detector/ascend.py Implements V2 detection/usage paths, V2-aware topology behavior, and A5 (Ascend950*) naming support.
gpustack_runtime/deployer/cdi/ascend.py Enumerates UB device directories and conditionally mounts A5 UB libraries based on detected arch family/variant.
gpustack_runtime/deployer/cdi/utils.py Adds helpers to enumerate device nodes from a directory and to expand mount globs into CDI mounts.
Suppressed comments (1)

gpustack_runtime/detector/pydcmi/init.py:877

  • dcmiv2_init() sets the global _apiVersion to 2, but dcmi_shutdown() does not reset it. After a shutdown + re-init cycle (or simply after shutdown), dcmi_api_version() can keep reporting 2 even though the library is no longer initialized, which can send callers down the V2-only code path against a V1-only driver.

Reset _apiVersion in dcmi_shutdown() so the next initialization establishes the correct version again.

def dcmi_shutdown():
    global _libInitialized, _libInitializedException

    with libLoadLock:
        if not _libInitialized:

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@yxf0314

yxf0314 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author
root@179-67-12-2:~# time GPUSTACK_RUNTIME_LOG_LEVEL=debug GPUSTACK_RUNTIME_LOG_EXCEPTION=1 gpustack-runtime detect --format json
2026-08-27 15:14:40,121 - 1401 - gpustack_runtime.detector.amd - DEBUG - No AMD PCI devices found
2026-08-27 15:14:40,139 - 1401 - gpustack_runtime.detector.ascend - ERROR - Failed to initialize DCMI, loaded from libdcmi.so
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/gpustack_runtime/detector/ascend.py", line 77, in is_supported
    pydcmi.dcmi_init()
  File "/usr/local/lib/python3.11/dist-packages/gpustack_runtime/detector/pydcmi/__init__.py", line 835, in dcmi_init
    _dcmiCheckReturn(ret)
  File "/usr/local/lib/python3.11/dist-packages/gpustack_runtime/detector/pydcmi/__init__.py", line 280, in _dcmiCheckReturn
    raise DCMIError(ret)
gpustack_runtime.detector.pydcmi.DCMIError_NotSupport: Not Supported
2026-08-27 15:14:40,265 - 1401 - gpustack_runtime.detector.ascend - INFO - Initialized DCMI through the V2 API, the V1 API being unavailable
2026-08-27 15:14:42,433 - 1401 - gpustack_runtime.detector.cambricon - DEBUG - No Cambricon PCI devices found
2026-08-27 15:14:42,435 - 1401 - gpustack_runtime.detector.hygon - DEBUG - No Hygon PCI devices found
2026-08-27 15:14:42,436 - 1401 - gpustack_runtime.detector.iluvatar - DEBUG - No Iluvatar PCI devices found
2026-08-27 15:14:42,438 - 1401 - gpustack_runtime.detector.metax - DEBUG - No MetaX PCI devices found
2026-08-27 15:14:42,440 - 1401 - gpustack_runtime.detector.mthreads - DEBUG - No MThreads PCI devices found
2026-08-27 15:14:42,441 - 1401 - gpustack_runtime.detector.nvidia - DEBUG - No NVIDIA PCI devices found
[
  {
    "manufacturer": "ascend",
    "index": 0,
    "name": "Ascend950PR",
    "uuid": "0000:01:00.0",
    "driver_version": "25.7.rc1.6",
    "runtime_version": null,
    "runtime_version_original": null,
    "compute_capability": null,
    "cores": 28,
    "cores_utilization": 0,
    "memory": 131072,
    "memory_used": 121594,
    "memory_utilization": 92.77,
    "memory_status": "healthy",
    "temperature": 60,
    "power": null,
    "power_used": 267.8,
    "appendix": {
      "arch_family": "Ascend950PR",
      "bdf": "0000:01:00.0",
      "physical_id": 0,
      "numa": "0"
    }
  },
  {
    "manufacturer": "ascend",
    "index": 1,
    "name": "Ascend950PR",
    "uuid": "0000:11:00.0",
    "driver_version": "25.7.rc1.6",
    "runtime_version": null,
    "runtime_version_original": null,
    "compute_capability": null,
    "cores": 28,
    "cores_utilization": 0,
    "memory": 131072,
    "memory_used": 121594,
    "memory_utilization": 92.77,
    "memory_status": "healthy",
    "temperature": 51,
    "power": null,
    "power_used": 267.2,
    "appendix": {
      "arch_family": "Ascend950PR",
      "bdf": "0000:11:00.0",
      "physical_id": 1,
      "numa": "0"
    }
  },
  {
    "manufacturer": "ascend",
    "index": 2,
    "name": "Ascend950PR",
    "uuid": "0000:61:00.0",
    "driver_version": "25.7.rc1.6",
    "runtime_version": null,
    "runtime_version_original": null,
    "compute_capability": null,
    "cores": 28,
    "cores_utilization": 0,
    "memory": 131072,
    "memory_used": 121595,
    "memory_utilization": 92.77,
    "memory_status": "healthy",
    "temperature": 56,
    "power": null,
    "power_used": 227.1,
    "appendix": {
      "arch_family": "Ascend950PR",
      "bdf": "0000:61:00.0",
      "physical_id": 2,
      "numa": "0"
    }
  },
  {
    "manufacturer": "ascend",
    "index": 3,
    "name": "Ascend950PR",
    "uuid": "0000:71:00.0",
    "driver_version": "25.7.rc1.6",
    "runtime_version": null,
    "runtime_version_original": null,
    "compute_capability": null,
    "cores": 28,
    "cores_utilization": 0,
    "memory": 131072,
    "memory_used": 121595,
    "memory_utilization": 92.77,
    "memory_status": "healthy",
    "temperature": 58,
    "power": null,
    "power_used": 218.5,
    "appendix": {
      "arch_family": "Ascend950PR",
      "bdf": "0000:71:00.0",
      "physical_id": 3,
      "numa": "0"
    }
  },
  {
    "manufacturer": "ascend",
    "index": 4,
    "name": "Ascend950PR",
    "uuid": "0000:81:00.0",
    "driver_version": "25.7.rc1.6",
    "runtime_version": null,
    "runtime_version_original": null,
    "compute_capability": null,
    "cores": 28,
    "cores_utilization": 0,
    "memory": 131072,
    "memory_used": 5237,
    "memory_utilization": 4.0,
    "memory_status": "healthy",
    "temperature": 51,
    "power": null,
    "power_used": 199.0,
    "appendix": {
      "arch_family": "Ascend950PR",
      "bdf": "0000:81:00.0",
      "physical_id": 4,
      "numa": "1"
    }
  },
  {
    "manufacturer": "ascend",
    "index": 5,
    "name": "Ascend950PR",
    "uuid": "0000:91:00.0",
    "driver_version": "25.7.rc1.6",
    "runtime_version": null,
    "runtime_version_original": null,
    "compute_capability": null,
    "cores": 28,
    "cores_utilization": 0,
    "memory": 131072,
    "memory_used": 5236,
    "memory_utilization": 3.99,
    "memory_status": "healthy",
    "temperature": 49,
    "power": null,
    "power_used": 205.1,
    "appendix": {
      "arch_family": "Ascend950PR",
      "bdf": "0000:91:00.0",
      "physical_id": 5,
      "numa": "1"
    }
  },
  {
    "manufacturer": "ascend",
    "index": 6,
    "name": "Ascend950PR",
    "uuid": "0000:E1:00.0",
    "driver_version": "25.7.rc1.6",
    "runtime_version": null,
    "runtime_version_original": null,
    "compute_capability": null,
    "cores": 28,
    "cores_utilization": 0,
    "memory": 131072,
    "memory_used": 5236,
    "memory_utilization": 3.99,
    "memory_status": "healthy",
    "temperature": 55,
    "power": null,
    "power_used": 198.6,
    "appendix": {
      "arch_family": "Ascend950PR",
      "bdf": "0000:e1:00.0",
      "physical_id": 6,
      "numa": "1"
    }
  },
  {
    "manufacturer": "ascend",
    "index": 7,
    "name": "Ascend950PR",
    "uuid": "0000:F1:00.0",
    "driver_version": "25.7.rc1.6",
    "runtime_version": null,
    "runtime_version_original": null,
    "compute_capability": null,
    "cores": 28,
    "cores_utilization": 0,
    "memory": 131072,
    "memory_used": 5237,
    "memory_utilization": 4.0,
    "memory_status": "healthy",
    "temperature": 51,
    "power": null,
    "power_used": 205.5,
    "appendix": {
      "arch_family": "Ascend950PR",
      "bdf": "0000:f1:00.0",
      "physical_id": 7,
      "numa": "1"
    }
  }
]2026-08-27 15:14:42,443 - 1401 - gpustack_runtime.detector.thead - DEBUG - No T-Head PCI devices found


real	0m2.954s
user	0m0.343s
sys	0m0.201s

@gitlawr gitlawr left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gitlawr
gitlawr merged commit 59ea8c8 into gpustack:main Aug 27, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants