Skip to content

[core] Honor scan.manifest.parallelism during planning - #988

Merged
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/manifest-parallelism
Sep 28, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/manifest-parallelism

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #987

Native scan planning currently hard-codes 64 concurrent reads for manifest lists and manifest files. This ignores scan.manifest.parallelism, so a lower configured value does not protect object storage from request bursts when many planners start together.

Brief change log

  • Parse scan.manifest.parallelism in CoreOptions.
  • Default to the process CPU count when the option is absent, matching the Java option contract.
  • Apply the configured bound to incremental manifest-list reads and manifest-file reads.
  • Reject zero and invalid values before calling StreamExt::buffered.

For example, a table configured with scan.manifest.parallelism=8 now limits each native planner to eight concurrent manifest reads instead of the previous fixed 64.

Tests

  • cargo test -p paimon test_scan_manifest_parallelism --lib
  • cargo test -p paimon table::table_scan::tests --lib (108 passed)
  • cargo clippy -p paimon --lib --tests -- -D warnings
  • cargo fmt --all -- --check

API and Format

No public API or storage format change. This makes an existing table option effective in the native planner.

Documentation

No new option is introduced; behavior is aligned with the existing Java/Python option.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 73aab79 into apache:main Sep 28, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Native scan ignores scan.manifest.parallelism

2 participants