Repository navigation
Emit masked gathers for predicated non-dense vector loads - #9684
Open
derek-gerstmann wants to merge 1 commit into
Open
derek-gerstmann wants to merge 1 commit into
derek-gerstmann wants to merge 1 commit into
Conversation
Predicated loads that aren't dense or stride -1 were scalarized through if_then_else, which emits a branch per lane. Vectorized loops with TailStrategy::GuardWithIf get predicated gathers in their tails, so a pipeline with many table lookups produced tens of thousands of tiny basic blocks, and LLVM's SLP vectorizer and LICM spent a very long time on them. One Photoshop generator took ~27 minutes to compile for AVX-512 (64-lane vectors), 826s of it in SLPVectorizerPass. Use llvm.masked.gather instead, for fixed-width vectors. The backend selects a native gather where it is legal and profitable, and otherwise scalarizes it late, after the mid-level optimizer has run. That generator now compiles in 83s for AVX-512 and 107s for AVX2 (previously 339s). Also guard the constant-zero index shortcut in CodeGen_LLVM::codegen_buffer_pointer with !index->getType()->isVectorTy() so vector zero indices still produce a vector of pointers for CreateMaskedGather. Co-authored-by: Bob Gardner <bgardner@adobe.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
abadams
reviewed
Nov 6, 2026
| add_streaming_load_metadata(gather); | ||
| } | ||
| value = gather; | ||
| } else { // It's not dense vector load, we need to scalarize it |
Member
There was a problem hiding this comment.
What actually reaches the else case now? Scalarizing non-fixed-width vectors sounds impossible.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Predicated loads that aren't dense or stride
-1were scalarized throughCall::if_then_else, which emits a conditional branch per lane. Vectorized loops withTailStrategy::GuardWithIfemit predicated gathers in their tails, so pipelines with many table lookups produced tens of thousands of tiny basic blocks (e.g. at 64 lanes on AVX-512), causing LLVM to spend hundreds of seconds inSLPVectorizerPassandLICM.Changes
src/CodeGen_LLVM.cpp:llvm.masked.gather(builder->CreateMaskedGather) inCodeGen_LLVM::codegen_predicated_loadfor fixed-width vector types (isa<FixedVectorType>(llvm_type_of(op->type))) so the backend either selects a native gather or scalarizes the intrinsic late after the mid-level optimizer has run.CodeGen_LLVM::codegen_buffer_pointerwith!index->getType()->isVectorTy()so constant-zero vector indices still return a vector of pointers rather than a scalar base pointer.test/correctness/predicated_gather.cpp,test/correctness/CMakeLists.txt:correctness_predicated_gathertesting predicated gathers across vector widths ({3, 4, 16, 64}), strides ({1, 3}), constant-zero vector indices (clamp(i, 0, 0)), and element types (float,uint8_t,int16_t,double), and verifying thatllvm.masked.gatheris emitted on x86.Breaking changes
None.
Checklist
Co-authored-by: Bob Gardner
Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com