Skip to content

[jindo] Unify the Jindo User-Agent format - #10321

Open
sundapeng wants to merge 1 commit into
apache:masterfrom
sundapeng:jindo-user-agent
Open

sundapeng wants to merge 1 commit into
apache:masterfrom
sundapeng:jindo-user-agent

Conversation

@sundapeng

Copy link
Copy Markdown
Member

Purpose

JindoFileIO only appends the DLF access tracking info (dlf.access-tracking.extended-info) to fs.oss.user.agent.extended, so dls:// requests lose it. Paimon itself is identified only through the SDK's call-stack detection, as a bare Paimon without a version, and the catalog-wide user-agent.* options that the other Paimon clients read are ignored.

JindoFileIO now fills fs.oss.user.agent.* and fs.dls.user.agent.* with Paimon's unified User-Agent format, <module>(<transport>[;<feature>...])[ <extended>], the one paimon-oss (#10306), the REST client (#10308), pypaimon and paimon-rust adopt as well:

  • The SDK keeps its own module and transport, and Paimon/<version> leads the features. The version is written in at build time by the templating-maven-plugin, so no class loader lookup is involved.
  • user-agent.module, user-agent.features and user-agent.extended apply unless the fs.<scheme>.user.agent.* key for that part is set.
  • dlf.access-tracking.extended-info is appended after the extended part for both schemes, so it never replaces a user-set value.

Examples:

Options User-Agent
none JindoSDK/6.9.1-nextarch(coro_http;Paimon/2.2-SNAPSHOT)
user-agent.features=Flink, user-agent.extended=vvr JindoSDK/6.9.1-nextarch(coro_http;Paimon/2.2-SNAPSHOT;Flink) vvr
the above, on a DLF catalog with access tracking enabled JindoSDK/6.9.1-nextarch(coro_http;Paimon/2.2-SNAPSHOT;Flink) vvr tenantId/<uid> principalType/<type> userName/<name>
fs.oss.user.agent.features=morax/2.6.0 in addition JindoSDK/6.9.1-nextarch(coro_http;Paimon/2.2-SNAPSHOT;morax/2.6.0) vvr ...

Related: #10306 (Java OSS), #10308 (Java REST). The three PRs only share the templating-maven-plugin entry in the root pom.xml.

Tests

TestJindoUserAgent: 5 new cases; the existing TestJindoDlfAccessTracking still passes, 27 tests in paimon-jindo passed. Also checked on a live DLF catalog with access tracking enabled: the OSS access log shows Paimon/<version>, the configured features and extended part, and the tracking info.

JindoFileIO only appended dlf.access-tracking.extended-info to
fs.oss.user.agent.extended, so dls:// requests lost it, and Paimon was
identified only through Jindo's call-stack detection, without a version.

JindoFileIO now fills fs.oss.user.agent.* and fs.dls.user.agent.* with
Paimon's unified User-Agent: Paimon/<version> leads the features, the
catalog-wide user-agent.module/features/extended apply unless the
fs.<scheme>.user.agent.* key for that part is set, and the access
tracking info is appended after the extended part. The version is
written into JindoBuildVersions at build time by the
templating-maven-plugin, so reading it needs no class loader lookup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant