Skip to content

[SYSTEMDS-3959] Report an error when a federated worker cannot bind its port - #2586

Open
gaturchenko wants to merge 1 commit into
apache:mainfrom
gaturchenko:fed-worker-invalid-port
Open

gaturchenko wants to merge 1 commit into
apache:mainfrom
gaturchenko:fed-worker-invalid-port

Conversation

@gaturchenko

@gaturchenko gaturchenko commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

The command systemds WORKER 80505 terminated silently with exit code 0. There are 2 causes:

  1. No range check for -w <port>, neither in bin/systemds nor when parsing the option
  2. FederatedWorker.run() captured every exception in a catch-all and logged only e.getMessage(), which is empty or ambigous for a bind failure. The same held for occupied and reserved ports, and for the -fedMonitoring backend

The following changes were implemented:

  • PortUtils (new): the shared range constants and validation, as well as explicit port and cause naming in case of errors;
  • DMLOptions: -w and -fedMonitoring reject non-integers and ports outside [1, 65535] at parse time, naming the incorrect option. A missing (optional) -w argument now falls back to the default port instead of throwing NumberFormatException.
  • FederatedWorker / FederatedMonitoringServer: a failed bind is logged through explainBindFailure and re-thrown as DMLRuntimeException. The monitoring backend now separates InterruptedException (a normal shutdown) from a real failure.
  • bin/systemds: adds a range check, a warning for reserved ports, and exit 1. Previously, an error was printed and the JVM was launched regardless.
  • FederatedWorkerPortTest (new): 14 tests covering the range predicates, option parsing, and the two worker paths (out-of-range and occupied port).

Resulting messages:

$ systemds WORKER 80505
error: Port 80505 for the federated worker is out of range, expected a port in [1, 65535]

$ systemds WORKER 80
warning: Port 80 for the federated worker is a reserved system port, binding it requires elevated privileges
ERROR federated.FederatedWorker: Federated worker stopped: no permission to bind port 80, ports below 1024 are reserved and require elevated privileges (Permission denied)

$ systemds WORKER 8001   # port taken
ERROR federated.FederatedWorker: Federated worker stopped: port 8001 is already in use (Address already in use)

@codecov

codecov Bot commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 64.10256% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.38%. Comparing base (9dccbc3) to head (1e8037d).
⚠️ Report is 17 commits behind head on main.

Files with missing lines Patch % Lines
...ederated/monitoring/FederatedMonitoringServer.java 12.50% 6 Missing and 1 partial ⚠️
...rc/main/java/org/apache/sysds/utils/PortUtils.java 66.66% 3 Missing and 4 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2586      +/-   ##
============================================
- Coverage     71.44%   71.38%   -0.07%     
- Complexity    50437    50550     +113     
============================================
  Files          1629     1633       +4     
  Lines        195513   196270     +757     
  Branches      38118    38208      +90     
============================================
+ Hits         139686   140101     +415     
- Misses        44867    45144     +277     
- Partials      10960    11025      +65     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gaturchenko

Copy link
Copy Markdown
Contributor Author

Waiting for the review by @ywcb00

@ywcb00 ywcb00 self-assigned this Aug 19, 2026
@gaturchenko
gaturchenko force-pushed the fed-worker-invalid-port branch from 0612897 to 1e8037d Compare August 28, 2026 08:21
Comment thread bin/systemds
Comment on lines +176 to +201
# verify that $1 is a port that a server can be bound to, otherwise abort with an error.
# $2 names the server the port is meant for.
function checkPort {
local port=$1
local target=$2
local re='^[0-9]+$'
if ! [[ $port =~ $re ]] ; then
echo "error: Port '$port' for the $target is not a number"
printUsage
exit 1
fi
# drop leading zeros, so that the length check below is not fooled by e.g. 0000080505
local num=$port
while [[ ${#num} -gt 1 && $num == 0* ]] ; do num=${num#0} ; done
# more than 5 digits is out of range by definition, and comparing it would silently
# overflow the 64 bit integers of the shell for very long inputs
if [ ${#num} -gt 5 ] || [ "$num" -lt 1 ] || [ "$num" -gt 65535 ] ; then
echo "error: Port $port for the $target is out of range, expected a port in [1, 65535]"
printUsage
exit 1
fi
if [ "$num" -lt 1024 ] ; then
echo "warning: Port $port for the $target is a reserved system port, binding it requires elevated privileges"
fi
}

@Baunsgaard Baunsgaard Sep 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only criticism. I am strongly against adding more logic to the systemds file. Can we not just have the error message and verification inside Java. This gives a single point of failure, rather than splitting the logic in two.

While there was some logic already verifying basic number compatibility, i think that should also just be cleaned up and moved into the java PortUtils you have defined.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

3 participants