[GLUTEN-12377][VL][DO NOT MERGE] Run the Delta suite with Velox operator output validation

[felipepessoto]

What this does

Exposes Velox's debug.validate_output_from_operators as a Gluten config and turns it on for the Delta Spark UT.

Why

The DV bitmap row-index failures in #12377 are intermittent and report a meaningless value. The reason is that by the time anything notices, the damage is an out-of-bounds read: a Delta deletion-vector write scans only synthesized columns with a pushed-down filter, and on that path Velox emits a RowVector whose row-index child has no rows. That empty child is then wrapped in a dictionary, and reading it returns whatever heap memory follows. Whether that memory happens to land outside Delta's valid range decides whether the query aborts or silently accepts a wrong row index — which is why the failure moved from test to test and the reported value differed every time (9223372036854775807, -1, 0xe43315c000007f00, ...).

Velox can catch this at the source instead. The validation checks every operator's output for structural consistency — among other things, that a dictionary's indexes address rows its base vector actually has — and names the first operator to emit a malformed vector.

Companion PR

#12808 is this change plus one line pointing UPSTREAM_VELOX_PR_ID at the upstream fix, branched from the same commit on main. It is expected to be green. Red here and green there is the before/after evidence.

Expected result

This PR is expected to make the Delta suite fail, and that is the point. It should fail deterministically, naming the responsible operator:

Output validation failed for [operator: TableScan, plan node ID: 0]:
  Child vector has size 0 less than parent and parent has no nulls 10.

instead of intermittently with an arbitrary row index.

Once the upstream fix is picked up — via a Velox bump, or UPSTREAM_VELOX_PR_ID in ep/build-velox/src/get-velox.sh — the suite must go green again, which is what validates the fix.

• Velox issue: Row-index column is empty when a scan has a filter and no child readers facebookincubator/velox#18535
• Velox fix: fix(dwio): Correct row-index size when a scan has a filter and no child readers facebookincubator/velox#18536

Verification

This repo pins VELOX_BRANCH=dft-2026_08_17, in which the defect is confirmed still present: SelectiveStructColumnReaderBase::next() still passes outputRows() to setRowNumberField(), and useOutputRows() is still scanSpec_->hasFilter() || hasDeletion(). The fix from velox#18536 also applies cleanly to that revision, which is what #12808 relies on.

The evidence is this PR against #12808, which is identical to it except for one line that applies velox#18536 — same base commit, same pinned Velox, so the fix is the only variable.

From the first pair of runs:

Delta suite run	shards failed	Output validation failed	regressions
this PR (defect present)	7 of 8	present	—
#12808 (+ velox#18536)	2 of 8	0	0

Both of #12808's two failing shards reported Regressions (new failures): 0 and failed only because FAIL_ON_FIXED=true and two ImplicitStreamingMergeCastingSuite baseline entries now pass — unrelated to this change, caused by the cast-mode fix in #12051. Both PRs now carry the cherry-pick that removes those stale entries, and a fresh pair of runs is in flight.

So with the Velox fix applied, every failure attributable to this defect disappears across the whole Delta suite, and no unrelated invariant violations surface — which also answers whether enabling validation suite-wide is safe.

Separately, the same suite with deletion vectors but predicate pushdown off (...DVsPredPushOffSuite) shows no violation at all, consistent with the upstream analysis: the defect needs a filter on the scan.

How do we know this is the same defect as #12377, and not a different one?

The error text differs — validation reports Child vector has size 0 less than parent, while #12377 reports a bad row index — so it is worth stating why they are the same defect:

• The original failure's own diagnostic showed this exact vector. A capture of Delta bitmap row index cannot be negative: -5163232936757035008 carried encoding=DICTIONARY size=1 baseSize=0 valuesBytes=0 baseIndex=0: the aggregator was reading a dictionary indexing into an empty base. Same column, same TableScan[0] -> PartialAggregation[9] plan.
• The differing message is expected. Instrumenting the malformed read rather than the value showed ~176 out-of-bounds reads per suite pass, against roughly one crash per 25 passes. The old message only fires when the out-of-bounds read lands outside Delta's range; pointer-shaped values (~1.4e14) are positive and inside it, so they were accepted silently. Validation catches the cause every time and upstream of the symptom.
• Same trigger. Both occur only with predicate pushdown on, and neither occurs with it off.
• The fix removes both, and not by masking. flaky-error-patterns.txt quarantines the original error, so "0 regressions" alone would not settle this. In [DO NOT MERGE] Same as #12783, plus the upstream Velox fix (expected green) #12808's run the sampled shards report Quarantined flaky failures (ignored) | 0 together with zero occurrences of the original bitmap error — it did not happen, rather than happening and being hidden.

Scope

The config defaults to off, so nothing outside the Delta suite changes.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI (Claude Opus 5)

[felipepessoto]

CI status

The 3 delta-spark-test shard failures are pre-existing on main, not caused by this PR.
Same three regressions in the nightly main run
(31866072615):

• ImplicitMergeCastingSuite#MERGE overflow ... DECIMAL(3,1) -> DECIMAL(3,2)
• ImplicitStreamingMergeCastingSuite#Streaming MERGE overflow ... DECIMAL(3,1) -> DECIMAL(3,2)
• TypeWideningInsertSchemaEvolutionBasicSuite#INSERT - never automatic type widening DECIMAL(9,2) -> ...

All decimal casting / type widening; nothing to do with deletion vectors. This PR only touches
the DV bitmap aggregator's failure path plus a comment in the workflow file, so it cannot
affect them.

Did the diagnostic fire?

Not this run — no Delta bitmap aggregator input column in any shard log, i.e. #12377 did not
reproduce. Expected: it lands on roughly one test per full run and is intermittent. Re-running
to take another sample; the diagnostic only prints when the bounds check actually trips.

[felipepessoto]

Not intended to merge. This exists to produce a CI result, alongside #12808.

Result -- both branches share a base commit and pinned Velox, differing only by whether facebookincubator/velox#18536 is applied:

	Delta suite
this PR (validation on, defect present)	5+ of 8 shards failing, Output validation failed for [operator: TableScan, plan node ID: 0]
#12808 (same + velox#18536)	all 8 shards green

Plan from here: once velox#18536 merges it arrives through the regular weekly Velox bump, after which a separate PR removes the DV bitmap entries from flaky-error-patterns.txt and closes #12377. This PR and #12808 can be closed at that point.