YARN-11964: Clamp negative available resources in AllocateResponse to zero by ryukobayashi · Pull Request #8523 · apache/hadoop

ryukobayashi · 2026-06-01T08:32:46Z

Description of PR

YARN RM can transiently report negative available resources (e.g. cluster_capacity - allocated goes negative due to overload or node failures). DefaultAMSProcessor now applies Resources.componentwiseMax with Resources.none() before setting available resources in AllocateResponse, ensuring the AM always receives a non-negative value.

This is a re-implementation of YARN-11964 (reverted in 9c865dd). The previous fix applied the clamp inside Resource.castToIntSafely(), which was too broad and caused a regression in TestRLESparseResourceAllocation. This fix applies the clamp at the correct layer — in DefaultAMSProcessor where the AllocateResponse is built.

Contains content generated by Claude (Anthropic)

How was this patch tested?

Added TestDefaultAMSProcessor#testAvailableResourcesClampedToNonNegative which uses a custom scheduler that returns a negative resource limit and verifies the AllocateResponse always contains non-negative available resources.
Existing tests pass: TestApplicationMasterServiceCapacity, TestApplicationMasterServiceFair, TestApplicationMasterServiceInterceptor, TestRLESparseResourceAllocation.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

The PR includes the phrase "Contains content generated by Claude (Anthropic)"
My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html

hadoop-yetus · 2026-06-02T06:38:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	27m 21s		trunk passed
+1 💚	compile	0m 50s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	compile	0m 52s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	checkstyle	0m 51s		trunk passed
+1 💚	mvnsite	0m 55s		trunk passed
+1 💚	javadoc	0m 49s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 48s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 29s		trunk passed
+1 💚	shadedclient	16m 39s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 36s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 30s		the patch passed
+1 💚	mvnsite	0m 34s		the patch passed
+1 💚	javadoc	0m 29s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 30s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 20s		the patch passed
+1 💚	shadedclient	15m 52s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	91m 21s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		163m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/2/artifact/out/Dockerfile
GITHUB PR	#8523
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 61aa8d2782f7 5.15.0-179-generic #189-Ubuntu SMP Tue May 5 18:20:56 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a767604`
Default Java	Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/2/testReport/
Max. process+thread count	956 (vs. ulimit of 10000)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/2/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

pan3793 · 2026-06-03T03:19:19Z

@ryukobayashi, the Test hdfs-rbf failure and hang can be ignored, it is caused by another PR, and the author is working on the solution, but the Test mr is expected to pass. Since the GHA job runs on your forked repo, you have permission to rerun it if you think it's a flaky test which is irrelevant to your change.

ryukobayashi · 2026-06-03T04:59:34Z

@pan3793 Yes, I tried again as you suggested, but it failed. So, could you review this?

hadoop-yetus · 2026-06-09T08:51:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 49s		trunk passed
+1 💚	compile	0m 51s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	compile	0m 53s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	checkstyle	0m 47s		trunk passed
+1 💚	mvnsite	0m 55s		trunk passed
+1 💚	javadoc	0m 50s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 46s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 32s		trunk passed
+1 💚	shadedclient	16m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 30s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 28s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 16s		the patch passed
+1 💚	shadedclient	15m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	91m 16s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		162m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/3/artifact/out/Dockerfile
GITHUB PR	#8523
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 3d011f3ac43b 5.15.0-179-generic #189-Ubuntu SMP Tue May 5 18:20:56 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `03c7018`
Default Java	Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/3/testReport/
Max. process+thread count	956 (vs. ulimit of 10000)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/3/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

pan3793 · 2026-06-09T09:13:09Z

@ryukobayashi the mr failure is related to your change, try debugging locally.

slfan1989 · 2026-06-10T01:15:38Z

@ryukobayashi I’ll take a look at this PR today and share my feedback later.
cc: @pan3793

ryukobayashi · 2026-06-10T02:46:21Z

@pan3793 Thanks, I identified and fixed the parts where NPE was occurring.
@slfan1989 Thanks.

hadoop-yetus · 2026-06-10T05:28:14Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	26m 19s		trunk passed
+1 💚	compile	0m 49s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	compile	0m 50s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	checkstyle	0m 45s		trunk passed
+1 💚	mvnsite	0m 56s		trunk passed
+1 💚	javadoc	0m 50s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 45s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 34s		trunk passed
+1 💚	shadedclient	16m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 29s		the patch passed
+1 💚	compile	0m 34s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 34s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 28s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 18s		the patch passed
+1 💚	shadedclient	15m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	91m 7s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		162m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/4/artifact/out/Dockerfile
GITHUB PR	#8523
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ce988b2140bb 5.15.0-179-generic #189-Ubuntu SMP Tue May 5 18:20:56 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `905a1a7`
Default Java	Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/4/testReport/
Max. process+thread count	927 (vs. ulimit of 10000)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/4/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

ryukobayashi · 2026-07-01T04:56:21Z

@pan3793 @slfan1989 It seems the CI has passed, so could you please review it?

slfan1989

Thanks for the update. Applying the clamp while constructing AllocateResponse is the correct scope and avoids changing the semantics of the general-purpose Resource conversion methods. The null handling also preserves the previous behavior, and the latest CI results are clean. +1.

A few non-blocking suggestions:

The PR description links #8519 as the revert, but that PR is unrelated. Please link the actual revert commit (9c865dd01fb) instead.
Since the latest change fixes a null resourceLimit regression, it would be helpful to add a test that explicitly verifies the null path does not throw and preserves a null available-resource value.

… zero

ryukobayashi · 2026-07-27T05:49:13Z

@slfan1989 Thanks for your review. I've fixed both.

hadoop-yetus · 2026-07-27T08:38:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	7m 18s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	26m 12s		trunk passed
+1 💚	compile	0m 52s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	compile	0m 51s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	checkstyle	0m 49s		trunk passed
+1 💚	mvnsite	0m 54s		trunk passed
+1 💚	javadoc	0m 45s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 48s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 31s		trunk passed
+1 💚	shadedclient	16m 39s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 28s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 30s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 18s		the patch passed
+1 💚	shadedclient	15m 44s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	91m 26s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		169m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/5/artifact/out/Dockerfile
GITHUB PR	#8523
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 41dad9c22705 5.15.0-179-generic #189-Ubuntu SMP Tue May 5 18:20:56 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0647ffc`
Default Java	Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/5/testReport/
Max. process+thread count	977 (vs. ulimit of 10000)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/5/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2026-07-27T08:46:56Z

@ryukobayashi Thanks for the contribution! Merge into trunk. @pan3793 Thanks for the review!

github-actions Bot added YARN trunk labels Jun 1, 2026

ryukobayashi mentioned this pull request Jun 1, 2026

YARN-11964: Resource.castToIntSafely() should clamp negative values to 0 #8506

Merged

6 tasks

ryukobayashi force-pushed the YARN-11964 branch from a767604 to 03c7018 Compare June 9, 2026 06:08

pan3793 requested a review from slfan1989 June 9, 2026 06:15

slfan1989 approved these changes Jul 26, 2026

View reviewed changes

ryukobayashi added 4 commits July 27, 2026 14:35

YARN-11964: Clamp negative available resources in AllocateResponse to…

0dbe082

… zero

YARN-11964: Trigger CI

17857f6

YARN-11964: Handle null resourceLimit in DefaultAMSProcessor

593996c

YARN-11964: Add test for null resourceLimit

0647ffc

ryukobayashi force-pushed the YARN-11964 branch from 905a1a7 to 0647ffc Compare July 27, 2026 05:48

slfan1989 merged commit 673fc22 into apache:trunk Jul 27, 2026
5 of 6 checks passed

Uh oh!

Conversation

ryukobayashi commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

AI Tooling

Uh oh!

hadoop-yetus commented Jun 2, 2026

Uh oh!

pan3793 commented Jun 3, 2026

Uh oh!

ryukobayashi commented Jun 3, 2026

Uh oh!

hadoop-yetus commented Jun 9, 2026

Uh oh!

pan3793 commented Jun 9, 2026

Uh oh!

slfan1989 commented Jun 10, 2026

Uh oh!

ryukobayashi commented Jun 10, 2026

Uh oh!

hadoop-yetus commented Jun 10, 2026

Uh oh!

ryukobayashi commented Jul 1, 2026

Uh oh!

slfan1989 left a comment

Choose a reason for hiding this comment

Uh oh!

ryukobayashi commented Jul 27, 2026

Uh oh!

hadoop-yetus commented Jul 27, 2026

Uh oh!

Uh oh!

slfan1989 commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ryukobayashi commented Jun 1, 2026 •

edited

Loading