[fix](iceberg) Project the row filter before the delete-manifest prune in the cached scan plan - #67406
Open
raghav-reglobe wants to merge 1 commit into
Conversation
…e in the cached scan plan
cacheBackedFileScanTasks (the manifest-cache planning path shared by the
synchronous, streaming and COUNT(*) plans) prunes DELETE manifests with
ManifestEvaluator.forPartitionFilter(filterExpr, spec, caseSensitive)
passing the raw ROW filter. forPartitionFilter binds against the partition
struct, so this works by accident on identity-only specs (the partition
field keeps the source column name) and throws ValidationException
("Cannot find field 'ts' in struct: struct<... ts_month: int>") on any
spec with a transform — e.g. (identity(flag), month(ts)) — whenever the
filter references the transform's source column. The catch then aborts
the WHOLE cached plan into the SDK fallback, logging a WARN with a stack
trace per query:
[IcebergScanPlanProvider.planFileScanTask()] Iceberg plan with
manifest cache failed, falling back to SDK scan: Cannot find field
'ts' in struct: struct<1000: flag: optional boolean, 1001: ts_month:
optional int>
So on v2 tables with delete files and a time-transform partition spec,
every filtered query silently loses the manifest cache (planning latency
+ repeated catalog/storage manifest reads) and spams the warn log — we
measured ~1,100 such stacks per hour on one production FE where most
queries filter on the transform's source column.
The data-manifest side of the very same method (getMatchingManifest)
already projects the filter into partition space before building its
evaluator. Do the same for delete manifests: project with
Projections.inclusive(spec, caseSensitive) — predicates on non-partition
columns project to alwaysTrue(), so pruning semantics are unchanged. The
evaluator construction moves into a small package-private helper so the
behavior is unit-testable.
The new test builds a real InMemoryCatalog v2 table partitioned by
(identity(flag), month(ts)) with a position-delete file and pins: the raw
row filter still fails to bind (the projection stays load-bearing), the
projected evaluator keeps an overlapping month and prunes a far month,
the identity leg still prunes, and a residual-only filter keeps the
manifest.
Signed-off-by: Raghvendra Singh <raghav@cashify.in>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
raghav-reglobe
requested review from
924060929 and
CalvinKirs
as code owners
September 1, 2026 20:35
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: #64304 (where the plugin-world port of this path landed; the data-manifest side got the projection, the delete-manifest side did not)
Problem Summary:
cacheBackedFileScanTasks— the manifest-cache planning path shared by the synchronous, streaming, and COUNT(*) plans — prunes delete manifests with:passing the raw row filter.
forPartitionFilterbinds against the partition struct, so this works by accident on identity-only specs (the partition field keeps the source column name) and throws on any spec with a transform — e.g.(identity(flag), month(ts))stores the field asts_month: int— whenever the filter references the transform's source column. The catch inplanFileScanTaskthen aborts the whole cached plan into the SDK fallback, once per query:Net effect on v2 tables with delete files and a time-transform partition spec: every filtered query silently loses the manifest cache (planning latency + repeated catalog/storage manifest reads) and logs a WARN with a full stack trace. On one production FE where most queries filter on the transform's source column we measured ~1,100 such stacks per hour; the fallback also means the manifest cache is effectively dead for the hottest query shape on those tables.
The data-manifest side of the very same method (
getMatchingManifest) already projects the filter into partition space before building its evaluator. This PR does the same for delete manifests —Projections.inclusive(spec, caseSensitive).project(rowFilter)— via a small package-private helper so the behavior is unit-testable. An inclusive projection maps predicates on non-partition columns toalwaysTrue(), so pruning semantics are unchanged; the projected transform predicates now actually prune delete manifests instead of failing to bind.Release note
Fix Iceberg manifest-cache planning falling back to the SDK scan (with a per-query WARN + stack trace) for every filtered query on tables whose partition spec contains a transform (e.g.
month(ts)) and that carry delete files.Check List (For Author)
Test
IcebergScanPlanProviderDeleteManifestPruneTest(realInMemoryCatalogv2 table,(identity(flag), month(ts))spec, one position-delete file): pins that the raw row filter still fails to bind (the projection stays load-bearing), the projected evaluator keeps an overlapping month and prunes a far month, the identity leg still prunes, and a residual-only filter keeps the manifest.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merged this PR)