HDDS-15306. Expose Disk Balancer CLI in top-level datanode help and improve usability by Gargi-jais11 · Pull Request #10312 · apache/ozone

Gargi-jais11 · 2026-05-19T13:52:13Z

What changes were proposed in this pull request?

Make diskBalancer visible on top-level datanode help command which was hidden in this PR HDDS-13649
Improve diskBalancer help msg to explain how to terminate stdin input when using - to provide datanode addresses through standard input.
All the values in diskbalancer report should be rounded to 2 decimal places.
Only the IdealUsage: 0.14747954 | Threshold: 10.0% | ThresholdRange: (0.04747954, 0.24747954)
Utilization | VolumeDensity should be in percentage form.
In diskbalancer report for each dn storage report also add ozoneAvailable field and change TotalCapacity -> OzoneCapacity and UsedSpace -> OzoneUsed.
Add more clarity about delta moves in the diskbalancer status notes.
enable DiskBalancer by default.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15306

How was this patch tested?

Tested manually:

Before Fix:

// diskbalancer not appear in datanode cli option
Usage: ozone admin datanode [-hV] [--verbose] [COMMAND]
Datanode specific operations
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
      --verbose   More verbose output. Show the stack trace of the errors.
Commands:
  list          List info of datanodes
  decommission  Decommission a datanode
  maintenance   Put a datanode into Maintenance Mode
  recommission  Return a datanode to service
  status        Show status of datanodes
  usageinfo     List usage information of a datanode by IP address, hostname or
                  UUID.

                Legend (bytes):
                  Filesystem Capacity/Used/Available: raw filesystem stats for
                  the Datanode, aggregated across volumes.
                  Ozone Capacity/Used/Available: Ozone-usable stats after
                  reserved-space adjustment, aggregated across   volumes (see
                  hdds.datanode.dir.du.reserved / hdds.datanode.dir.du.reserved.
                  percent).
                  Reserved: hdds.datanode.dir.du.reserved: configured reserved
                  space for non-Ozone usage.
                  Committed: space pre-allocated for containers.
                  Free Space To Spare: minimum free space to keep before
                  closing containers.
 
 // not much info to use stdin -
 
 Get DiskBalancer status
      [<datanode address>...]
                  Datanode addresses: one or more, separated by spaces. To read
                    from stdin, specify '-' and supply one item per line.Port
                    is optional and defaults to 19864 (CLIENT_RPC port).
                    Examples: 'DN-1', 'DN-1:19864', '192.168.1.10'.
  -h, --help      Show this help message and exit. 
  
// Inconsistency in decimals and percentage for Disk Utilisation or Threshold
// no ozoneAvailable given
  
  dn-1 ~]# ozone admin datanode diskbalancer report --in-service-datanodes --verbose
Report result:
Datanode: dn1 (10.82.246.167:19864)
Aggregate VolumeDataDensity: 0.20786860859385214
IdealUsage: 0.14747954 | Threshold: 10.0% | ThresholdRange: (0.04747954, 0.24747954)
Volume Details:
StorageID                                     StoragePath                                TotalCapacity       UsedSpace   Container Pre-AllocatedSpace   EffectiveUsedSpace     Utilization   VolumeDensity
DS-71f33f87-9212-488c-9ae6-4db48c9a48f1       /hadoop-ozone/datanode/data/hdds               349.27 GB         4.71 GB                       26.82 GB             75.05 GB      0.21487346      0.06739392
DS-ac73c11c-993a-49f9-b97e-ea61145452e2       /hadoop-ozone/datanode/data2/hdds       195.78 GB         1.11 GB                        4.27 GB             16.36 GB      0.08357777      0.06390177
DS-5b27f109-0029-424a-9092-5ae7692b98db       hadoop-ozone/datanode/data3/hdds       195.78 GB         2.32 GB                       18.06 GB             30.86 GB      0.15760217      0.01012263
DS-3350ed2c-326c-4725-9fe2-755e85d17b02       hadoop-ozone/datanode/data4/hdds       195.78 GB         1.13 GB                        4.26 GB             15.86 GB      0.08102925      0.06645028

After Fix:

// diskbalancer visible in datanode cli option
Usage: ozone admin datanode [-hV] [--verbose] [COMMAND]
Datanode specific operations
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
      --verbose   More verbose output. Show the stack trace of the errors.
Commands:
  list          List info of datanodes
  decommission  Decommission a datanode
  maintenance   Put a datanode into Maintenance Mode
  recommission  Return a datanode to service
  status        Show status of datanodes
  usageinfo     List usage information of a datanode by IP address, hostname or
                  UUID.

                Legend (bytes):
                  Filesystem Capacity/Used/Available: raw filesystem stats for
                  the Datanode, aggregated across volumes.
                  Ozone Capacity/Used/Available: Ozone-usable stats after
                  reserved-space adjustment, aggregated across   volumes (see
                  hdds.datanode.dir.du.reserved / hdds.datanode.dir.du.reserved.
                  percent).
                  Reserved: hdds.datanode.dir.du.reserved: configured reserved
                  space for non-Ozone usage.
                  Committed: space pre-allocated for containers.
                  Free Space To Spare: minimum free space to keep before
                  closing containers.

  diskbalancer  DiskBalancer specific operations to ensure even disk
                  utilisation. It is disabled by default. To enable it, set
                  hdds.datanode.disk.balancer.enabled as true.
                  
  // improved stdin - info in help msg
  Usage: ozone admin datanode diskbalancer status [-hV] [--in-service-datanodes]
       [--json] [--verbose] [<datanode address>...]
Get DiskBalancer status
      [<datanode address>...]
                  Datanode addresses: one or more on the command line, OR read
                    from stdin with '-'.
                  Stdin usage:
                    ozone admin datanode diskbalancer <subcommand> -
                    Then type one datanode per line and end input:
                      - Linux/macOS: Ctrl-D
                      - Windows:     Ctrl-Z, then Enter
                  Examples:
                    # Piped (recommended for scripts)
                    echo -e "DN-1\nDN-2" | ozone admin datanode diskbalancer
                    status -
                    # From file having list of dns to balance
                    ozone admin datanode diskbalancer report - < datanode-lists.
                    txt
                  Port is optional and defaults to 19864 (CLIENT_RPC port).
                  Address examples: 'DN-1', 'DN-1:19864', '192.168.1.10'.
  -h, --help      Show this help message and exit.
      --in-service-datanodes
                  If set, the client will send DiskBalancer requests to all
                    available DataNodes in the HEALTHY and IN_SERVICE
                    operational state.
      --json      Format output as JSON
  -V, --version   Print version information and exit.
      --verbose   More verbose output. Show the stack trace of the errors.
      
  // round of value to 2 decimal places
  // *IdealUsage | Threshold | ThresholdRange | Utilization | VolumeDensity** should be in percentage form.
  // add **ozoneAvailable field** and change **TotalCapacity -> OzoneCapacity and UsedSpace -> OzoneUsed.**
  // explain delta move for source volume
 bash-5.1$ ozone admin datanode diskbalancer report - < datanode-list
Report result:
Datanode: ozone-datanode-3.ozone_default (172.18.0.8:19864)
Aggregate VolumeDataDensity: 0.18%
IdealUsage: 12.66% | Threshold: 0.01% | ThresholdRange: (12.65%, 12.67%)

Volume Details:

StorageID                                     StoragePath                                OzoneCapacity  OzoneAvailable       OzoneUsed   Container Pre-AllocatedSpace   EffectiveUsedSpace Utilization VolumeDensity
DS-78ca0f8e-6b4f-4639-aa91-ec2389f1c49f       /data/hdds1/hdds                              1006.75 GB       880.15 GB         5.42 GB                            0 B            126.60 GB 12.57%    0.09%
DS-45ac9e0c-b254-4585-b535-60467fde3be3       /data/hdds2/hdds                              1006.75 GB       880.15 GB         5.64 GB                        1.77 GB            128.37 GB 12.75%    0.09%
DS-ee5326f5-5933-4560-8b2c-75dcd8900b1b       /data/hdds3/hdds                              1006.75 GB       880.15 GB         5.53 GB                         907 MB            127.48 GB 12.66%    0.00%
Note:
  - Aggregate VolumeDataDensity: Sum of per-volume density (deviation from ideal); higher means more imbalance.
  - IdealUsage: Target utilization (0-100%) when volumes are evenly balanced.
  - ThresholdRange: Acceptable deviation (percent); volumes within IdealUsage +/- Threshold are considered balanced.
  - VolumeDensity: Deviation of a particular volume's utilization from IdealUsage.
  - Utilization: how much a particular volume is utilized effectiveUsedSpace / ozoneCapacity) in %.
  - OzoneCapacity: Ozone data volume capacity.
  - OzoneAvailable: Ozone data volume available space.
  - OzoneUsed: Ozone data volume used space.
  - ContainerPreAllocatedSpace: Space reserved for containers not yet written to disk.
  - EffectiveUsedSpace: This is the actual used space of volume which is visible to the diskBalancer : (ozoneCapacity minus ozoneAvailable) + containerPreAllocatedSpace + move delta.
  - move delta: source volume space to be reclaimed after move completion; this value is reflected only when diskBalancer is running else it is 0.

When DiskBalancer is disabled CLI is blocked and below output is seen:

bash-5.1$ ozone admin datanode diskbalancer report --in-service-datanodes
Disk Balancer is not enabled. Please enable the hdds.datanode.disk.balancer.enabled configuration key.

bash-5.1$ ozone admin datanode diskbalancer status --in-service-datanodes
Disk Balancer is not enabled. Please enable the hdds.datanode.disk.balancer.enabled configuration key.

bash-5.1$ ozone admin datanode diskbalancer start --in-service-datanodes
Disk Balancer is not enabled. Please enable the hdds.datanode.disk.balancer.enabled configuration key.

bash-5.1$ ozone admin datanode diskbalancer stop --in-service-datanodes
Disk Balancer is not enabled. Please enable the hdds.datanode.disk.balancer.enabled configuration key.

bash-5.1$ ozone admin datanode diskbalancer update -t 2 --in-service-datanodes
Disk Balancer is not enabled. Please enable the hdds.datanode.disk.balancer.enabled configuration key.

Copilot

Pull request overview

This PR improves DiskBalancer CLI discoverability and output usability by exposing the diskbalancer command in top-level datanode help, enhancing stdin-related help text, and changing the diskbalancer report output to show percentages/rounded values and additional per-volume fields.

Changes:

Unhide ozone admin datanode diskbalancer and improve CLI help text for reading datanode addresses from stdin (-).
Update diskbalancer report output to show selected ratios as percentages (2 decimals) and rename/add per-volume capacity/usage fields (including ozoneAvailable).
Update protobuf schema and tests to reflect the new per-volume report fields.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
hadoop-ozone/cli-admin/src/test/java/org/apache/hadoop/hdds/scm/cli/datanode/TestDiskBalancerSubCommands.java	Updates JSON-output assertions and report-proto generation to include the renamed/new per-volume fields.
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/DiskBalancerReportSubcommand.java	Formats report ratios as percent strings, renames/adds reported columns/JSON keys, and adds explanatory notes.
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/DiskBalancerCommands.java	Exposes the `diskbalancer` command in top-level datanode CLI help and updates description text.
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/DatanodeParameters.java	Improves help text explaining stdin usage termination and examples for `-`.
hadoop-hdds/interface-client/src/main/proto/hdds.proto	Renames/adds fields in `VolumeReportProto` for ozone capacity/available/used.
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/diskbalancer/TestDiskBalancerProtocolServer.java	Updates protocol server tests for the new `VolumeReportProto` fields.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java	Populates the updated `VolumeReportProto` fields when building volume reports.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Gargi-jais11 · 2026-05-20T10:58:19Z

@ChenSammi Please re-review, I have addressed all comments.

ChenSammi · 2026-05-26T06:04:14Z

@Gargi-jais11 , let's enable disk balancer by default in this patch too.

ChenSammi · 2026-06-02T03:12:10Z

@Gargi-jais11 , what's the CLI behavior if disk balancer is disabled?

Gargi-jais11 · 2026-06-02T03:51:10Z

@Gargi-jais11 , what's the CLI behavior if disk balancer is disabled?

When DiskBalancer is disabled CLI is blocked and below output is seen:

bash-5.1$ ozone admin datanode diskbalancer report --in-service-datanodes
Disk Balancer is not enabled. Please enable the hdds.datanode.disk.balancer.enabled configuration key.

bash-5.1$ ozone admin datanode diskbalancer status --in-service-datanodes
Disk Balancer is not enabled. Please enable the hdds.datanode.disk.balancer.enabled configuration key.

bash-5.1$ ozone admin datanode diskbalancer start --in-service-datanodes
Disk Balancer is not enabled. Please enable the hdds.datanode.disk.balancer.enabled configuration key.

bash-5.1$ ozone admin datanode diskbalancer stop --in-service-datanodes
Disk Balancer is not enabled. Please enable the hdds.datanode.disk.balancer.enabled configuration key.

bash-5.1$ ozone admin datanode diskbalancer update -t 2 --in-service-datanodes
Disk Balancer is not enabled. Please enable the hdds.datanode.disk.balancer.enabled configuration key.

unhide disbalancer option and improved usability

f07288c

Gargi-jais11 changed the title ~~HDDS-15306.~~ HDDS-15306. Expose Disk Balancer CLI in top-level datanode help and improve usability May 19, 2026

Gargi-jais11 marked this pull request as ready for review May 19, 2026 14:03

Gargi-jais11 requested review from ChenSammi and ssulav May 19, 2026 14:03

ChenSammi requested a review from Copilot May 20, 2026 03:28

Copilot started reviewing on behalf of ChenSammi May 20, 2026 03:29 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

address review comments

a196081

Gargi-jais11 added 2 commits May 21, 2026 09:56

Merge branch 'master' of github.com:apache/ozone into HDDS-15306

b4c99b0

fix test failure after master merge

492dbdf

ChenSammi reviewed May 26, 2026

View reviewed changes

Comment thread hadoop-hdds/interface-client/src/main/proto/hdds.proto Outdated

address review comment and enable diskbalancer by default

319e438

Gargi-jais11 requested a review from ChenSammi June 1, 2026 04:48

improve Ozonecontainer diskbalancer log

351fad6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-15306. Expose Disk Balancer CLI in top-level datanode help and improve usability#10312

HDDS-15306. Expose Disk Balancer CLI in top-level datanode help and improve usability#10312
Gargi-jais11 wants to merge 6 commits into
apache:masterfrom
Gargi-jais11:HDDS-15306

Gargi-jais11 commented May 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gargi-jais11 commented May 20, 2026

Uh oh!

Uh oh!

ChenSammi commented May 26, 2026

Uh oh!

ChenSammi commented Jun 2, 2026

Uh oh!

Gargi-jais11 commented Jun 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Gargi-jais11 commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gargi-jais11 commented May 20, 2026

Uh oh!

Uh oh!

ChenSammi commented May 26, 2026

Uh oh!

ChenSammi commented Jun 2, 2026

Uh oh!

Gargi-jais11 commented Jun 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Gargi-jais11 commented May 19, 2026 •

edited

Loading