Conversation
sundapeng
force-pushed
the
pypaimon-user-agent
branch
from
September 28, 2026 15:59
2d6a9d9 to
7fb4887
Compare
sundapeng
changed the title
sundapeng
force-pushed
the
pypaimon-user-agent
branch
from
September 28, 2026 16:35
7fb4887 to
01feed8
Compare
pypaimon hard-coded fs.oss.user.agent.features=pypaimon, which overwrote the user's features, and dropped dlf.access-tracking.extended-info. OSS requests now follow Paimon's unified User-Agent format, module(transport;features) extended. Each part comes from fs.oss.user.agent.<part>, or else from the catalog-wide user-agent.<part> shared with the REST client. pypaimon/<version> leads the features, a configured module is passed to the backend, and the access tracking info is appended after the extended value. The PyArrow S3 fallback sets AWS_SDK_UA_APP_ID when unset, so its requests carry app/pypaimon/<version>. The version is the one build_info embeds at build time.
sundapeng
force-pushed
the
pypaimon-user-agent
branch
from
September 29, 2026 01:08
01feed8 to
d1de2bc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
pypaimon hard-codes
fs.oss.user.agent.features=pypaimon, which overwrites any features the user sets, and drops the DLF access tracking info (dlf.access-tracking.extended-info). The PyArrow S3 fallback sends only the AWS SDK's default User-Agent, so its OSS requests can't be attributed to pypaimon at all.This gives OSS requests Paimon's unified User-Agent format,
<module>(<transport>[;<feature>...])[ <extended>], which the REST client and the Java and Rust clients adopt as well:user-agent.module,user-agent.featuresanduser-agent.extended, which REST requests read too ([python] Unify the REST User-Agent format #10309).fs.oss.user.agent.module/features/extendedoverride the matching part for OSS only.pypaimon/<version>leads the features.dlf.access-tracking.extended-info, which DLF sends with the data token, is appended after the extended part, so it never replaces a user-set value.fs.oss.impl=legacy) has no User-Agent option. It setsAWS_SDK_UA_APP_IDwhen unset, which addsapp/pypaimon/<version>; features and extended can't be carried on this path.build_infoembeds at build time, which snapshots already record as the writer version.Examples:
JindoSDK/6.15.504-nextarch(coro_http;pypaimon/2.2.dev)user-agent.features=Flink,user-agent.extended=vvr, on a DLF catalog with access tracking enabledJindoSDK/6.15.504-nextarch(coro_http;pypaimon/2.2.dev;Flink) vvr tenantId/<uid> principalType/<type> userName/<name>fs.oss.impl=legacyaws-sdk-cpp/1.11.800 ua/2.1 api/S3 os/Linux#... app/pypaimon/2.2.dev md/GCC#14.2.1 ...Related: #10309 (Python REST), #10306 (Java OSS), apache/paimon-rust#991 (Rust object storage). #10309 adds the same
build_info.version().Tests
oss_user_agent_test.pyandjindo_file_system_test.py: 16 new cases; 51 tests passed together withoss_legacy_mode_test.pyandpvfs_oss_filesystem_test.py. Also checked on a live DLF catalog with access tracking enabled: the OSS access log shows the tracking info after the configured extended part.