Skip to content

[GLUTEN-11708][VL] Enable Bloom filter might_contain to subfield filter translation - #12793

Merged
zhztheplayer merged 2 commits into
apache:mainfrom
zhztheplayer:wip-bloom-pushdown
Aug 17, 2026
Merged

[GLUTEN-11708][VL] Enable Bloom filter might_contain to subfield filter translation#12793
zhztheplayer merged 2 commits into
apache:mainfrom
zhztheplayer:wip-bloom-pushdown

Conversation

@zhztheplayer

Copy link
Copy Markdown
Member

Fixes #11708

@zhztheplayer
zhztheplayer marked this pull request as ready for review August 17, 2026 13:25
Copilot AI lite review requested due to automatic review settings August 17, 2026 13:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Enables optional translation of Spark might_contain (Bloom filter) expressions into Velox subfield filters, controlled by a new static configuration flag.

Changes:

  • Added a new static config spark.gluten.sql.columnar.backend.velox.scan.bloomFilterPushdown.enabled (Scala + C++ + docs).
  • Extended SparkExprToSubfieldFilterParser to parse might_contain(bloomFilter, xxhash64_with_seed(seed, field)) into a Bloom-filter-backed Filter.
  • Wired the new config into Velox backend initialization when registering the parser.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
docs/velox-configuration.md Documents the new Velox scan Bloom filter pushdown config flag.
cpp/velox/operators/functions/SparkExprToSubfieldFilterParser.h Adds config-driven enablement for Bloom filter translation.
cpp/velox/operators/functions/SparkExprToSubfieldFilterParser.cc Implements constant-evaluation + might_contain parsing and a Bloom-filter-backed filter type.
cpp/velox/config/VeloxConfig.h Introduces C++ config key + default for enabling Bloom filter pushdown.
cpp/velox/compute/VeloxBackend.cc Passes the config flag into the parser registration.
backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala Adds the corresponding static Spark/Gluten config entry and getter.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cpp/velox/operators/functions/SparkExprToSubfieldFilterParser.cc
<< "cannot push down to subfield filter";
return std::nullopt;
}
auto seed = seedValue->as<SimpleVector<int64_t>>()->valueAt(0);
Comment thread cpp/velox/operators/functions/SparkExprToSubfieldFilterParser.cc
Comment thread cpp/velox/operators/functions/SparkExprToSubfieldFilterParser.cc
Comment thread cpp/velox/operators/functions/SparkExprToSubfieldFilterParser.cc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[VL] Translate bloom filter might_contain to a subfield filter

3 participants