Repository navigation
Rename storage.s3.profiles to buckets, share S3StorageConfig for bucket entries - #6
Merged
papaharry merged 4 commits intoOct 6, 2026
Conversation
"Profiles" collides with AWS credential profiles; the map keys are bucket names, which also matches Hadoop's per-bucket S3A config. - Drop S3ProfileConfig: `buckets` is a map of S3StorageConfig, so serde, flavors, redaction, and Debug are shared with the primary backend. Nested `buckets` are rejected in StorageConfigs::validate. - apply_flavor recurses into bucket configs at load time. - The resolver builds one backend (config + lazy client) per bucket at construction, so resolve() has a single code path and needs no mutex.
- Reject `storage.s3.buckets` keys that are empty or contain `/`. Such keys can never match a parsed bucket name, so the bucket would silently fall back to the primary backend and its credentials. - Test bucket routing in the S3 factory: exact match, dotted names, bucket-only URIs, no prefix matching, and fallback to the primary. - Make the env-override test set QW_S3_ENDPOINT and QW_S3_FORCE_PATH_STYLE_ACCESS, and restore the test that unknown fields in a bucket entry are rejected. - Skip URI parsing when no buckets are configured, and drop the primary backend's duplicate copy of the bucket configs. - Docs: add the `buckets` property and the QW_S3_FORCE_PATH_STYLE_ACCESS env variable to the reference tables.
- Load a full node config with `storage.s3.buckets` to cover validation and flavor expansion on the real load path, including a quoted numeric bucket name and a rejected `s3://` key. - Check that the nested-buckets error comes from the nested check. - Docs: QW_S3_ENDPOINT and QW_S3_FORCE_PATH_STYLE_ACCESS apply to the primary backend only; every node must declare the same `buckets` entries; quote bucket names YAML would parse as non-strings.
dayaffe
force-pushed
the
david-yaffe/6523-s3-buckets
branch
from
October 6, 2026 17:53
bc86fd3 to
1faeb19
Compare
Remove tests already covered elsewhere: the full node-config load test (validation and flavor expansion are unit tested), the dotted-name and no-buckets factory tests (covered by the routing test), and the bucket unknown-field test (bucket entries share S3StorageConfig's deny_unknown_fields).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to quickwit-oss#6523. Merging this into your
mainupdates that PR.profiles→buckets(storage.s3.buckets.<bucket-name>): "profiles" is easy to confuse with AWS credential profiles, and the keys are bucket names, as in Hadoop's per-bucket S3A config.S3StorageConfiginstead of a separateS3ProfileConfig, so parsing, flavors, redaction, and unknown-field rejection are shared and can't drift. Nestedbucketsare rejected at validation.resolve()has a single code path and no mutex./likes3://my-bucket) fail at startup instead of silently falling back to the primary backend.QW_S3_FORCE_PATH_STYLE_ACCESS, that env overrides apply to the primary backend only, and that every node must declare the samebucketsentries.profiles:tobuckets:before deploying, otherwise Quickwit fails to start withunknown field profiles.After merging this, quickwit-oss#6523 will have a one-line conflict with upstream
mainindocs/configuration/storage-config.md(theQW_S3_MAX_CONCURRENCYrow); merging upstreammaininto yourmainagain resolves it.