TEZ-4688 TEZ-4648: Tez upgrade to Hadoop 3.5.0 and jersey 2.x#474
TEZ-4688 TEZ-4648: Tez upgrade to Hadoop 3.5.0 and jersey 2.x#474maheshrajus wants to merge 11 commits into
Conversation
|
FYI, |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
@abstractdog @ayushtkn Can you please review and provide comments if any. Thanks ! |
|
@maheshrajus , can you please rebase this PR and resolve merge conflicts? |
|
@Aggarwal-Raghav Hi, i rebased and pushed the commit just now. Could you please take a look at this? Thanks ! |
|
The explicit dependency of |
This comment was marked as outdated.
This comment was marked as outdated.
@Aggarwal-Raghav Better this we will handle separately. thanks ! |
|
💔 -1 overall
This message was automatically generated. |
|
@abstractdog @ayushtkn @Aggarwal-Raghav |
|
@ayushtkn Could you please review and approve when you have a moment? Thank you ! |
|
🎊 +1 overall
This message was automatically generated. |
|
@abstractdog Look like the checkstyle task is catching incorrect one here. 'If' child has Indentation level 2. In this line i have not added any spaces except entityID string. Anything i am missing over here ? ./tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/SimpleHistoryParser.java:258: userName = subEntity.getString(Constants.ENTITY_ID);:15: 'if' child has incorrect indentation level 14, expected level should be 16. [Indentation] Code: |
we can ignore that checkstyle warning then, for me it's clearly a bug in checkstyle |
@abstractdog Thank you for confirming the Checkstyle-related. |
|
@abstractdog , please wait for some time before merging, I'm checking few things. Give me 1 hr. |
absolutely! I'm not about to merge this immediately |
|
I might be wrong, I was performing cluster testing of this change, and saw this error. In Hadoop jersery migration they have moved from Attaching complete log. |
@maheshrajus , did you face this in your env? |
@Aggarwal-Raghav Could you please try executing the same steps with my latest changes and confirm? I have fixed the Tez UI-related ATS fields according to the new behavior in Hadoop 3.5.0. With these changes, I am able to submit a DAG and view its status in the Tez UI. I have attached the screenshots above for reference. Thank you ! |
@Aggarwal-Raghav i got the below NoSuchMethodError(BufferRecycler.releaseToPool()) exception while testing tez-ui. Similar stack trace i found in attached logs.txt by you. With Latest code changes i cant find any exceptions and able to see the DAG status in tez UI. You also please verify once and confirm. Thank you ! BufferRecycler.releaseToPool() this method available only in jackson-core 2.18.x. We can find the fix changes in my latest pushed commit. Stack trace: |
|
The DAG has been submitted and the status is visible in the Tez UI with the latest fixes. @Aggarwal-Raghav Please review and confirm the changes so @abstractdog can merge them at his convenience. Thanks!
|
|
@maheshrajus , there are 2 things here.
The patch attached is how I would've handled it. Your jackson change is indeed required which was part of my future pr in hadoop-3.5.0, so I'm ok with that but exclusion from pig is required which is bringing jackson I got both the issue in my run and post that attaching the clean logs as well clean_log.txt. I have |
|
If the patches I've attached is something you are not ok with as you have not seen them in your environement that is totally understandable. I'm happy to raise a separate PR by myself if more discussion/questions is required there. |
Other Changes LGTM |
|
💔 -1 overall
This message was automatically generated. |
@Aggarwal-Raghav Did you run the steps above using the latest commit I pushed this afternoon? Are both of those issues still occurring with the latest changes? If they are, and if it's a Jersey 2.x issue, let's just include the fix in the current PR. |
yes, I'm still seeing |
|
💔 -1 overall
This message was automatically generated. |
@Aggarwal-Raghav Let me check and push the fixes. Thank you for the verification ! |
|
@Aggarwal-Raghav |
|
💔 -1 overall
This message was automatically generated. |
| <artifactId>snappy-java</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>org.apache.avro</groupId> |
There was a problem hiding this comment.
@maheshrajus , please revert this avro exclusion. UT's are working after that. As we have jackson-bom defined now, we don't need this. All jackson jar have 2.18.6 now
jackson-annotations-2.18.6.jar
jackson-core-2.18.6.jar
jackson-databind-2.18.6.jar
jackson-jaxrs-base-2.18.6.jar
jackson-jaxrs-json-provider-2.18.6.jar
jackson-module-jaxb-annotations-2.18.6.jar


What changes were proposed in this PR?
This PR upgrades Tez to support:
As part of the upgrade, several ATS Timeline API response format differences were observed between older Hadoop versions and Hadoop 3.5.0. This PR adds compatibility handling for those schema variations.
Main updates include:
Handled ATS compatibility differences including:
otherinfo↔otherInfoentity↔entityIdentitytype↔entityTypedomain/id↔domainIdAdditional fixes:
How was this patch tested?
jersey upgrade changes taken from PR : #429 by @abstractdog as they need for hadoop 3.5.0 upgrade.