Conversation
This was referenced Sep 28, 2026
sundapeng
force-pushed
the
oss-user-agent
branch
from
September 28, 2026 15:59
a8972f8 to
ef7d3d4
Compare
sundapeng
changed the title
sundapeng
changed the title
sundapeng
force-pushed
the
oss-user-agent
branch
from
September 28, 2026 16:35
ef7d3d4 to
7ea79df
Compare
OSS requests from OSSFileIO only carried the SDK's default User-Agent. They now carry Paimon's unified User-Agent format, module(transport;features) extended, passed to hadoop-aliyun as fs.oss.user.agent.prefix. The module defaults to Paimon/<version> and the transport is aliyun-sdk-java/<version>. Each part is read from fs.oss.user.agent.<part>, falling back to the catalog-wide user-agent.<part> keys that REST requests also use, so one setting covers both. dlf.access-tracking.extended-info is appended after the extended value. A user-set fs.oss.user.agent.prefix still wins. Both versions are written into OSSBuildVersions at build time by the templating-maven-plugin, so reading them needs no class loader lookup.
sundapeng
force-pushed
the
oss-user-agent
branch
from
September 29, 2026 01:08
7ea79df to
978cb19
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
OSS requests from
paimon-osscarry only the SDK's default User-Agent,aliyun-sdk-java/3.17.4(Linux/...;<java>), Hadoop/3.3.4, so Paimon traffic can't be told apart from other SDK traffic in OSS access logs. The DLF access tracking info (dlf.access-tracking.extended-info) never reaches the User-Agent either.This gives OSS requests Paimon's unified User-Agent format,
<module>(<transport>[;<feature>...])[ <extended>], which the REST client and the Python and Rust clients adopt as well:Paimon/<version>, transportaliyun-sdk-java/<version>. Both versions are written in at build time by thetemplating-maven-plugin, so no class loader lookup is involved.user-agent.module,user-agent.featuresanduser-agent.extended, which REST requests read too ([rest] Unify the REST User-Agent format #10308).fs.oss.user.agent.module/features/extendedoverride the matching part for OSS only.dlf.access-tracking.extended-info, which DLF sends with the data token, is appended after the extended part, so it never replaces a user-set value.OSSFileIOpasses the result to hadoop-aliyun asfs.oss.user.agent.prefix, and a user-set prefix still takes precedence.Examples (hadoop-aliyun appends
, Hadoop/<version>):Paimon/2.2-SNAPSHOT(aliyun-sdk-java/3.17.4), Hadoop/3.3.4user-agent.features=Flink,user-agent.extended=vvrPaimon/2.2-SNAPSHOT(aliyun-sdk-java/3.17.4;Flink) vvr, Hadoop/3.3.4Paimon/2.2-SNAPSHOT(aliyun-sdk-java/3.17.4;Flink) vvr tenantId/<uid> principalType/<type> userName/<name>, Hadoop/3.3.4fs.oss.user.agent.features=SparkPaimon/2.2-SNAPSHOT(aliyun-sdk-java/3.17.4;Spark) vvr tenantId/<uid> principalType/<type> userName/<name>, Hadoop/3.3.4fs.oss.user.agent.prefix=customcustom, Hadoop/3.3.4Related: #10308 (Java REST), #10321 (Jindo FileIO), #10307 (Python OSS), apache/paimon-rust#991 (Rust object storage).
Tests
OSSUserAgentTest: 8 new cases, 1 of them against a fake OSS endpoint; 32 tests inpaimon-oss-implpassed. Also checked on a live DLF catalog with access tracking enabled: the OSS access log shows the tracking info after the configured extended part.