Skip to content

feat(list): skip unaddressable keys (//, ., ..) from listings - #118

Draft
alukach wants to merge 1 commit into
worktree-fix-116-empty-path-segmentfrom
feat-116-skip-unaddressable-list-keys
Draft

feat(list): skip unaddressable keys (//, ., ..) from listings#118
alukach wants to merge 1 commit into
worktree-fix-116-empty-path-segmentfrom
feat-116-skip-unaddressable-list-keys

Conversation

@alukach

@alukach alukach commented Jul 21, 2026

Copy link
Copy Markdown
Member

What I'm changing

Stacks on #117 (base branch is worktree-fix-116-empty-path-segment, not main).

#117 stopped a single //-segment key from 503-ing an entire prefix, but as a side effect it surfaced those keys in the listing even though they can't be fetched: object_store::path::Path rejects a ///./.. key and validate_key returns 400 on any GET/HEAD/PUT/DELETE of one. That left a // object visible-but-unretrievable — a client sees it in the listing and then gets a 400 trying to fetch it.

This PR resolves that inconsistency the simple way: skip unaddressable keys from listings so LIST only advertises objects that are actually retrievable, logging the raw key at warn so operators can find and remediate the stray upload (as the issue suggested).

A diagnostic against the real bucket (obstore, the Python binding of the same Rust object_store crate) confirmed these keys fail client-side at path-parse for HEAD, GET, and LIST — they are unreachable through object_store by any route:

[ FAIL ] HEAD the // key: ValueError: ... Path "raw//..." contained empty path segment
[ FAIL ] GET the // key:  ValueError: ... contained empty path segment

Note: this is the intentionally-lazy treatment. It makes // objects invisible through the gateway rather than retrievable. The alternative — a custom byte-faithful presigner that makes // keys individually fetchable — was considered and deferred; it's more work (new SigV4 signing code) and only warranted if such objects are real data that must be reachable. For now they're treated as unaddressable.

How I did it

  • api/list.rshas_degenerate_segment (empty / . / .. path segment) filters collect_list_entries for both objects and common prefixes. Common prefixes strip exactly one trailing delimiter first, so raw// is caught while a genuine photos/ is not (trim_end_matches would wrongly clean raw// to raw).
  • Testsunaddressable_keys_and_prefixes_skipped_from_listing (unit) covers //, .., and a degenerate common prefix; the proxy end-to-end test now asserts the // key is skipped rather than surfaced.

Test plan

  • cargo test -p multistore --lib — 131 pass
  • cargo clippy -p multistore --all-targets — clean (no new warnings)
  • cargo check -p multistore-cf-workers --target wasm32-unknown-unknown — compiles
  • obstore diagnostic against the real ncar-cesm2-arise bucket confirms the // key is unreachable via object_store

🤖 Generated with Claude Code

Builds on the #116 LIST fix. That change stopped a single `//`-segment key
from 503-ing an entire prefix, but it surfaced such keys in the listing even
though they can't be fetched: `object_store::path::Path` rejects them and
`validate_key` 400s a GET/HEAD/PUT/DELETE of them. A `//` object was
therefore visible-but-unretrievable.

Skip these keys (and degenerate common prefixes) from listings so LIST only
advertises objects that are actually addressable, logging the raw key at
`warn` so operators can find and remediate the stray upload.

- api/list.rs: add `has_degenerate_segment` (empty / `.` / `..` segment) and
  filter it out of `collect_list_entries` for both objects and common
  prefixes. Common prefixes strip exactly one trailing delimiter first so
  `raw//` is caught while a normal `photos/` is not.
- Tests: `unaddressable_keys_and_prefixes_skipped_from_listing` (unit) and the
  proxy end-to-end test now asserts the `//` key is skipped, not surfaced.

Depends on #116 (this branch stacks on it — the lenient parser is what makes
per-key skipping possible; object_store fails the whole page).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Jul 21, 2026

Copy link
Copy Markdown

Claude finished @alukach's task in 3m 2s —— View job


✅ No blocking issues — safe to merge.

  • Gather context (PR diff, base branch)
  • Review api/list.rs and proxy.rs changes for correctness/security
  • Ponytail pass for over-engineering
  • Post final review

The //, ., .. filtering logic is correct and well-tested: has_degenerate_segment (crates/core/src/api/list.rs:305-308) correctly catches empty/./.. segments, and the common-prefix handling (crates/core/src/api/list.rs:395-407) correctly strips exactly one trailing / before checking — using trim_end_matches there would have wrongly cleaned raw// down to raw, so the strip_suffix choice (and the test covering it) is the right call. No security issues; this is server-side listing filtering only, no new injection surface.

One non-blocking correctness gap worth a follow-up:

  • has_degenerate_segment (crates/core/src/api/list.rs:305-308) duplicates half of the predicate already in validate_key (crates/core/src/api/request.rs:172-186), but drops its control-character check (key.bytes().any(|b| b < 0x20 || b == 0x7f)). A key containing a control byte would still 400 on GET/HEAD via validate_key, yet this PR's new filter won't catch it — the exact "visible-but-unretrievable" class the PR sets out to close, just for a different byte class. Delegating closes the gap and removes the duplication:
    fn is_unaddressable(key: &str) -> bool {
        crate::api::request::validate_key(key).is_err()
    }
    (safe to call on common prefixes too — validate_key("") returns Ok(()) for the bucket-level case, which never applies here.)

Simplify (ponytail)

  • crates/core/src/api/list.rs:305has_degenerate_segment reinvents validate_key's segment check instead of calling it; drop the duplicate predicate and call validate_key(key).is_err() instead (also closes the control-char gap above for free).

💰 Estimated review cost: $0.68 · 3m01s · 18 turns

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant