Prune unread nested parquet leaves when a projected column is cast to a narrower type (nested schema pruning)#23398
Draft
adriangb wants to merge 4 commits into
Draft
Prune unread nested parquet leaves when a projected column is cast to a narrower type (nested schema pruning)#23398adriangb wants to merge 4 commits into
adriangb wants to merge 4 commits into
Conversation
Registers the same wide list<struct> / struct parquet file with the
file's full schema and with a narrower declared schema, plus a
physically-narrow file as the floor. On main the narrow declared
schema reads exactly as many bytes as the full schema (the whole
column is fetched and clipped in memory by the adapter-inserted cast):
list_struct_narrow_schema: bytes_scanned=25.19 MB
list_struct_full_schema: bytes_scanned=25.19 MB
list_struct_physically_narrow: bytes_scanned=3.32 KB
select_events_narrow_schema 3.45 ms
select_events_full_schema 3.36 ms
select_events_physically_narrow 164 µs
Baseline for schema-driven nested projection pruning
(see apache/datafusion-comet#4859).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd
Move-only: `ParquetReadPlan`, `StructFieldAccess`, `build_projection_read_plan` and the leaf-index/schema-pruning helpers move from row_filter.rs (2100+ lines) into projection_read_plan.rs. `PushdownChecker`/`PushdownColumns` become pub(crate) so the new module can keep using them. No behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd
…ast to a narrower type When a table's logical schema declares a nested column narrower than the physical parquet type (e.g. logical events: list<struct<x,y>> over a file with list<struct<x,y,+18 more>>), the expr adapter rewrites the projection into CAST(events AS narrow_type) — and the projection mask derivation expanded that column to every physical leaf, fetching and decoding subfields the cast then threw away. This adds schema-driven nested projection pruning (the equivalent of Spark's ParquetReadSupport.clipParquetSchema): when a projected root column is consumed only through such a cast, the leaf ProjectionMask is computed by walking the physical and cast-target type trees, matching struct fields by name, recursing through matching List/LargeList wrappers. Clipping keys off the cast target because that is exactly the contract of the runtime nested cast (cast_column consumes source children only by target-name lookup), so it applies to any PhysicalExprAdapter that emits whole-column casts. Safety/fallback rules (the clip is total; worst case = full read): - maps, dictionaries, wrapper-kind mismatches: keep all leaves (cast_column has no name-based Map arm) - struct level with zero name overlap: keep first child (>=1 leaf per level preserves struct definition levels, so `s IS NULL` semantics survive) - leaf type promotion stays the cast's job: emitted types keep the physical leaf types - unions with get_field accesses on the same root are supported; any whole-column reference wins Gated by datafusion.execution.parquet.nested_projection_pruning (default true). Comet reported reading 1.35 TB where Spark read 30.9 GB for the same pruned ReadSchema. On the parquet_nested_schema_pruning benchmark the narrow-schema scan drops from 25.19 MB (full column) to the pruned leaves only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd
Same narrow-schema tables queried with
datafusion.execution.parquet.nested_projection_pruning=false, so the
before/after stays visible in one run:
select_events_narrow_schema 164 µs / 3.32 KB
select_events_narrow_schema_pruning_disabled 2.94 ms / 25.19 MB
select_events_physically_narrow (floor) 150 µs / 3.32 KB
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd
github-actions
Bot
added
documentation
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked on #23396 (move-only refactor) and #23397 (benchmark). Until those merge, this PR shows their commits too; the feature itself is the last two commits (
feat: prune unread nested parquet leaves...andbench: add pruning-disabled variants...), ~900 hand-written lines of which roughly half are tests. Supersedes #23392.Which issue does this PR close?
Rationale for this change
When a table's logical file schema declares a nested column narrower than the physical parquet type — e.g. the table declares
events: LIST<STRUCT<x, y>>over a file containingevents: LIST<STRUCT<x, y, +16 more fields>>— DataFusion reads every leaf of the column and drops the extra subfields in memory:DefaultPhysicalExprAdapterrewrites the projected column intoCAST(events AS narrow_type).get_fieldchains; the cast's inner column falls into theProjectionMask::rootspath, expanding to all physical leaves.nested_struct::cast_column) discards the unrequested subfields.Engines like Spark communicate nested projection pruning to the scan exactly this way — as a clipped read schema, not as
get_fieldexpressions (Spark'sSchemaPruningrule consumes the field-access expressions and bakes the result into the scan'srequiredSchema). Comet measured a production query reading 1.35 TB where plain Spark read 30.9 GB for the same prunedReadSchema. The same behavior is reproducible in pure DataFusion by declaring a narrower nested type inCREATE EXTERNAL TABLEand watchingbytes_scanned. Any embedder that hands DataFusion a pre-pruned schema (Comet, delta-rs, Iceberg integrations) benefits.This PR implements the equivalent of Spark's
ParquetReadSupport.clipParquetSchema: when a projected root column is consumed only through such a cast, the leafProjectionMaskis computed by walking the physical and cast-target type trees, matching struct fields by name, so only the leaves the cast retains are fetched and decoded. The adapter-inserted cast then degrades to a cheap reorder/rename/null-fill/promotion.Why key off the cast target instead of the logical-vs-physical schema diff? The cast target is exactly the contract of the runtime nested cast:
cast_columnconsumes source struct children only by looking up target field names, recursively through list wrappers, so physical subtrees the target never names are provably dead. It also works for customPhysicalExprAdapters (which are allowed to inject references to physical-only columns — schema-driven clipping could under-read there, expression-driven clipping degrades safely to a full read), and it requires no opener signature changes. A bareColumnwhose logical type is narrower than the physical file is not a working configuration today, so threading the logical schema would add no coverage.Benchmark results (see #23397; wide
list<struct>file, narrow declared schema):The
SUM(s['x'])shape on a schema-evolved struct — whereget_field(CAST(col))currently defeats the existing expression-level leaf clipping entirely (the bottom-up adapter rewrite meansPushdownCheckerno longer sees a plainColumnunderget_field) — improves ~80–90% as well, since it now clips to the cast target.What changes are included in this PR?
On top of the stacked #23396/#23397:
datafusion/datasource-parquet/src/nested_schema_pruning.rs:clip_for_castwalks physical vs cast-target types (name-matched struct children, case-sensitive to matchcast_column; recurses through matchingList/LargeListwrappers) and returns kept leaf offsets;prune_type_by_kept_offsetsderives the Arrow type the reader emits for a kept-offset union. The clip is total — maps (no name-based Map arm incast_column), dictionaries, wrapper-kind mismatches, and zero-name-overlap struct levels keep all their leaves, so the worst case is today's full read. Every struct level keeps ≥1 leaf, so struct definition levels (ands IS NULLsemantics) survive.PushdownCheckeroptionally collectsCastColumnAccess(aCastExprover a plainColumnwhererequires_nested_struct_cast(physical, cast_type)), enabled only for projection analysis — the row-filter path is mechanically untouched.build_projection_read_planunions cast-clipped leaves with existingget_fieldleaf accesses per root; any whole-column reference wins; roots with onlyget_fieldaccesses keep the bit-identical legacy path. A defensive leaf-count check falls back to a whole-root read if the arrow/parquet leaf mapping ever disagrees.datafusion.execution.parquet.nested_projection_pruning(defaulttrue), threaded source → morselizer → opener →DecoderProjection, plus proto field + docs +information_schemaentries. The flag exists purely as an escape hatch / bisection aid for a change in the IO path — there is no scenario where a correctly-working clip should be disabled — and could be deprecated after a release or two.Are these changes tested?
Yes:
ProjectionMask::leavesover a subset ofList<Struct>leaves emits exactly the predicted type and preserves null list rows / null struct elements.datafusion/core/tests/parquet/expr_adapter.rs, each asserting identical results with the flag on and off and thatbytes_scanneddrops by more than 2×: list-of-struct narrowing (subset + reorder +Int32→Int64leaf promotion + missing-subfield null-fill), top-level struct, struct-level nullability (s IS NULL),get_fieldon a narrowed struct, mixed whole-column + subfield access, filter pushdown enabled, and a scan mixing a physically-narrow and a wide file.datafusion/sqllogictest/test_files/parquet_nested_schema_pruning.sltcovering the end-to-end SQL path (CREATE EXTERNAL TABLEwith a narrower nested type) with the setting toggled both ways.Are there any user-facing changes?
A new configuration option,
datafusion.execution.parquet.nested_projection_pruning(defaulttrue), documented inconfigs.md. No API changes; behavior is semantically invisible (IO reduction only).Deliberately out of scope, noted for follow-ups:
Maparm innested_struct::cast_column).get_fieldpaths with cast targets (naively unsafe: null-filling a non-nullable logical sibling changes per-batch struct-compatibility validation outcomes).🤖 Generated with Claude Code
https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd