[GLUTEN][VL] Defer Delta deletion vector reads to executors

[malinjawi]

What changes are proposed in this pull request?

Native Delta scans currently load every on-disk deletion vector while the driver builds file splits. With 2,461 S3-backed DVs, this kept executors idle for about two minutes before the first Spark job.

This PR passes each DV's absolute path, offset, size, and Hadoop configuration with the Spark partition, then loads the payload during executor-side split serialization.

• On-disk DVs always load on executors; the serial driver-loading path is removed.
• Reads use Delta's checksum-validating APIs.
• Successful reads are memoized per task, and concurrent callers share one load.
• Failed reads are not cached, so task retries can read again.
• Metrics register after TaskContext becomes available.
• Inline DVs remain eager because their bytes are already in Delta metadata.
• The protobuf payload and native filtering are unchanged.

The production DV data-path change is confined to the JVM handoff. The PR also adds metrics, tests, documentation, and required CI baseline maintenance. spark.gluten.sql.columnar.filescan=false remains the broad fallback that disables native file scan. Direct Velox range reads are handled separately in #12867.

How was this patch tested?

• Scala formatting, generated configuration documentation, configuration suites, and git diff --check.
• Test-source compilation for Spark 3.3 / Delta 2.3, Spark 3.5 / Delta 3.3, and Spark 4.0 / Delta 4.0.
• Shared Delta 3.3/4.0 tests for serialization, executor materialization, concurrent single-read memoization, failed-read retry, and eager inline DVs without filesystem access.
• Version-specific native handoff tests assert executor-side metric propagation.
• The previous PR head passed Delta Spark UT. Both this PR and [GLUTEN][VL] Read Delta deletion vectors through Velox #12867 reported the same three now-passing baseline entries, which were removed under the gate's fail-on-fixed policy: one DV test and two unrelated type-widening tests.

Targeted DV reads

A Spark 3.5.4 validation build retained a temporary legacy switch solely to compare both paths from the same build. PR A used executor deferral with native range reading disabled; Legacy used driver loading with native range reading disabled. Native file scan and metadata row index were enabled in both arms. The final PR does not publish the legacy driver-loading switch.

Each arm ran three times in balanced order against the same snapshot and resources.

Query	Legacy median	PR A median	Speedup
Metadata count(*)	8.116 s	7.923 s	1.02x
Physical sum	123.592 s	12.332 s	10.02x
Physical grouped sum	114.912 s	6.753 s	17.02x
Combined physical scans	239.467 s	19.085 s	12.55x
Targeted three-query total	247.583 s	27.008 s	9.17x

The combined physical value is the median of each run's sum + grouped_sum time. SQL-start-to-first-job delay fell from 114.7 s to 0.64 s for sum, and from 111.2 s to 0.38 s for grouped sum.

The snapshot contained 2,461 DV-bearing files and 6,479,887,870 visible rows after applying DV cardinality 720,032,919. Every arm returned identical results.

Full Delta ingestion

A separate balanced three-round test processed all 24 TPC-DS SF2500 tables. Only store_sales carried DVs; native writing, output settings, inputs, and resources were fixed.

Scope	Legacy median	PR A median	Improvement
All 24 tables	820.56 s	693.24 s	15.52% lower
DV-bearing store_sales	372.35 s	254.08 s	31.76% lower
Other 23 tables	446.43 s	440.05 s	approximately unchanged

All runs completed 24 of 24 tables with matching schemas, partitioning, Delta metrics, row statistics, and corresponding output row/file counts.

The claims are limited to these Spark 3.5.4 workloads: 12.55x for the two targeted physical DV scans and 15.52% for full ingestion. They do not cover a full query suite, DV creation, native writing, or Spark 4 performance.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: IBM BOB

[github-actions]

Run Gluten Clickhouse CI on x86

[malinjawi]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[malinjawi]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI on x86

[malinjawi]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI on x86

[malinjawi]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI on x86

[malinjawi]

Run Gluten Clickhouse CI

[malinjawi]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI on x86

[malinjawi]

Run Gluten Clickhouse CI

[marin-ma]

Are there any circumstances that could benefit from driver loading? If not, the legacy implementation should be removed.

[malinjawi]

@marin-ma I checked the remaining cases and do not see a normal workload that benefits from loading on-disk DVs on the driver. The only theoretical differences are earlier failure for a missing/corrupt sidecar and avoiding a repeated DV request on a retried or speculative task; neither justifies preserving the serial planning path. The standard Delta fallback already reads DVs on executors.

I will remove the rollback config and the legacy on-disk driver branch, keep inline DVs eager because their bytes are already in Delta metadata, and retain spark.gluten.sql.columnar.filescan=false as the broader fallback. Please let me know if you see a compatibility case that requires a different direction.

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86